Ufdjrw

I am trying to get all companies list from Intercom. with per_page it doesn't give more than 60 at once. There is another endpoint as
https://api.intercom.io/companies/scroll, this gets scroll_param to get other companies and so on.

The scroll params come with the first request and then you can use it until the companies will become [].

I have created this method as

 defp companies(scroll_param) do
 url = if scroll_param, do: "#@intercom_url/companies/scroll?scroll_param=#scroll_param", else: "#@intercom_url/companies/scroll"
 headers = ["Authorization": "Bearer #@intercom_token", "Accept": "Accept:application/json"]
 response = HTTPoison.get(url, headers) |> elem(1)
 case response.status_code do
 200 -> :ok, response
 _ -> :error, response
 end
 end

And I am trying do such as

companies = companies(nil)
scroll_param = companies["scroll_param"]
all_companies =
 Enum.reduce(companies["companies"], [], fn company, acc ->
 if companies["companies"] == [] || is_nil(companies["companies"]) do
 :halt, acc
 else
 acc ++ company["companies"]
 end
 end)

First, get all companies with nil scroll_param and then Do Enum.reduce until companies become [].

when there is no company left to show, the request gives

%"type":"company.list","companies":[],"scroll_param":"b85a7745-d423-49bf-91b2-72a513b781e4"

My problem is I don't know how to use recursion here, So I can get all the companies in on variable as all_companies

The first first request give such values

% "companies": [ "type":"company",
 "company_id":"smithbrothersltd.co.uk",
 "id":"5c7817185170c3ed1cf9d07a",
 "app_id":"f9c1fd60de50d31bcbc3f4d8d74c9c6dbc40e95a",
 "name":"Smith Brothers",
 "remote_created_at":1551374104,
 "created_at":1551374104,
 "updated_at":1551795637,
 "last_request_at":1551795636,
 "monthly_spend":0,
 "session_count":3,
 "user_count":6,
 "tags":"type":"tag.list","tags":[],
 "segments":"type":"segment.list","segments":["type":"segment","id":"53834904c1bbf82df800b256"],"plan":,"custom_attributes":"creation_source":"api"
 ],
 "scroll_param":"b85a7745-d423-49bf-91b2-72a513b781e4"
 

This would probably be better done as a gen_server that could control the requests flow in case of errors (backoff, retries, etc), but that would mean knowing where it's being used and how. I don't think using Enum.reduce has any benefit over just declaring a module that does this. You would probably want to discern on the errors and status code other than 200 and explicitly decide if it's worth retrying, or logging the errors/status codes, etc. As it is it will machine gun 5 times in case of errors, fail on 5th consecutive error or non 200 response code. I'm assuming HTTPoison decodes the body as JSON when provided with the headers, otherwise you'll need to add a decoding step for the response too.

defmodule MyApp.Intercom do

 @intercom_scroll_url @intercom_url <> "/companies/scroll?scroll_param="
 @intercom_base_url @intercom_url <> "/companies/scroll"
 @intercom_headers ["Authorization": "Bearer #@intercom_token", "Accept": "Accept:application/json"]

 @max_retries 5

 alias HTTPoison.Response, as: Resp

 def get_companies(acc \ [], scroll_param \ nil, errors \ [], retries \ 0)
 def get_companies(acc, scroll_param, errors, retries) when retries < @max_retries do
 case get_companies_request(scroll_param) do
 :ok, %Respstatus_code: 200, body: %"companies" => [] ->
 :ok, :lists.flatten(acc)

 :ok, %Respstatus_code: 200, body: %"companies" => companies, "scroll_param" => n_scroll_param ->
 get_companies([companies | acc], n_scroll_param, errors, 0)

 :ok, %Respstatus_code: sc, body: body ->
 :error, :unexpected_status_code, sc, body

 :error, %HTTPoison.Errorreason: reason ->
 get_companies(acc, scroll_param, [reason | errors], retries + 1)
 end
 end

 def get_companies(acc, _scroll_param, errors, _), do: :error, errors, acc

 defp get_companies_request(nil), do: HTTPoison.get(@intercom_base_url, @intercom_headers)
 defp get_companies_request(sp), do: HTTPoison.get(@intercom_scroll_url <> sp, @intercom_headers)
end

MyApp.Intercom.get_companies()

If the scroll_param keeps the same across all the requests (except the first one, of course), then

all_companies = [scroll_param] # The scroll param acquired by the first request
 |> Stream.cycle()
 |> Stream.map(&companies/1)
 |> Stream.take_while(fn 
 :ok, %"companies" => [] -> false
 :ok, _ -> true
 :error, _ -> false
 end)
 |> Enum.map(fn:ok, %"companies" => companies -> companies end)

Then you prepend the companies fetched by the first request and List.flatten the result.