# Pagination Automation ## Feature Introduction Pagination is often necessary to retrieve all results from a connector's API when data spans multiple pages. The advantages of automatic pagination over manual pagination are outlined [here](https://docs.locoia.com/automation/flow-builder/automatic-pagination#overview). ## Configuration on connector Here we have to fill the **Pagination Configuration** field in JSON format, the same one that's also used by [Remote Search Configuration](https://app.gitbook.com/s/xpN4SnIA1JI7uGLmi8QM/base-connector-setup/search-automation):
In most cases, the configuration will look something like this: ```json { "response_jsonpath": "$.{{ endpoint.split('/', 1)[0] }}", "query": "{\"page\": {{ page + 1 }}}" } ``` You can use Jinja in all fields in order to have conditional statements or do some simple calculations (e.g. as above `page + 1`). In the first request, the parameters are *not* used, only when pagination is needed it will be used. `response_jsonpath` is of course also used to point to the list of results of the first request ### `query` *Either `query`, `body`, or `header` needs to be specified* This follows mostly the same rules and logic as in the [Search configuration](https://docs.locoia.com/connectors/building-connectors/search-automation): The `query` is a string that contains a dictionary of query parameters that will be appended to the action call in order to do the pagination. In case the same parameter is given by the action itself, it will be overwritten by the parameter defined here. The `page` parameter starts with 1 and is increased by 1 for every subsequent call. {% hint style="info" %} Since most Connectors' paginate starts with 1, we do need to add `+ 1` to it in most cases, to start the first pagination request with 2 (as pointed out above, the first request is done independently of it, so the counting only starts after the second request). {% endhint %} {% hint style="info" %} The maximum page size is currently 100 in order to not accidentally run into endless loops or something similar. If we see that this needs to be adjusted, we can do so easily. {% endhint %} By using `previous_request.` we can reference to the previous requests's response, which is e.g. needed when dealing with next page tokens. {% hint style="info" %} If the `query`parameter is given a `GET` call is being done.\ If a `POST`call should be done instead, also specify `body` with `{}` as its value. {% endhint %} ### `response_jsonpath` (optional) This points to the list of results that should be used, which is relevant if the response is nested, e.g. in this example: ```json { "deals": [ { ... }, { ... } ], "meta": { ... } } ``` we expect the final output of the action to be a list of dictionaries, only containing entries that are inside the `deals` list. We have point to it using [JSON path](https://goessner.net/articles/JsonPath/) ([more examples below](#examples)). ### `endpoint` (optional) This can be used to adjust the default endpoint for all except the first page, which in some cases (e.g. [Dropbox](#dropbox-strange-cursor-based-with-special-pagination-endpoints)) needs to be extended to get the results of the next page with a next page token. You can also reference to `endpoint` in any of the parameters, which is often useful for e.g. the `response_jsonpath` ([see examples below](#examples)). ### `body` *Either `query`, `body`, or `header` needs to be specified* The `body` parameter is similar to the `query` parameter. It also contains a string which is a dictionary. However, instead of query parameters, it contains the request body that will be used in the calls. {% hint style="info" %} If the `body` parameter is given a `POST` call is automatically being done. {% endhint %} ### `replace_body` (optional) This defines whether the regular body that's being sent by the action should be replaced and only contains the pagination body as defined in the configuration. So far we only experienced this for one connector ([Dropbox](#dropbox-strange-cursor-based-with-special-pagination-endpoints)), so this can be left empty by default (`"replace_body": false` will be automatically added during saving of the configuration) It can be either `false` or `true`. ### `header` *Either `query`, `body`, or `header` needs to be specified* The `header` parameter is similar to the `body` parameter. It also contains a string which is a dictionary. However, instead of the request body, it contains the request headers that will be used in the calls. {% hint style="info" %} If the `header` parameter is given a `GET` call is automatically being done.\ If a `POST` call should be done instead, also specify body with `{}` as its value. {% endhint %} This needs to be used in combination with [multiple configurations](#multiple-configurations). ### Multiple configurations In order to add multiple pagination configurations for one Connector, the configuration needs to be a list of configurations and the parameter `when` needs to be used, except for the last statement, which can be a 'catch all' configuration. The `when` parameter will be checked from top to bottom and the one that matches first will be used. In case none match and a pagination configuration does not have a `when` parameter, that configuration will be used (i.e. similar to multiple `elif` and lastly an `else` statement). ## Configuration on action On the action just toggle the switch button next to **Supports automatic pagination** in order to show the **Retrieve all data** (`supports_automatic_pagination`) toggle on the action:
## Examples ### Freshsales - page / per\_page with nested response list ```json { "response_jsonpath": "$.{{ endpoint.split('/', 1)[0] }}", "query": "{\"page\": {{ page + 1 }}}" } ``` Here the interesting piece is the `response_jsonpath`:\ As mentioned above, the Freshsales response structure looks like this: ```json { "deals/contacts/sales_accounts": [ { ... }, { ... } ], "meta": { ... } } ``` where it's one of `deals/contacts/sales_accounts` depending on the endpoint (e.g. `deals/view/{id}`).\ With `{{ endpoint.split('/', 1)[0] }}` the endpoint is split into a list at every `/` and then the first element of that list (`[0]`) is accessed, which would be for the example `deals`.\ So the `response_jsonpath` is `$.deals`, which points directly to the list of records. ### Freshdesk - page / per\_page with raw list in response ```json { "query": "{\"page\": {{ page + 1 }}}" } ``` {% hint style="warning" %} Freshdesk returns the list of records without any nesting before it, thus we only need to specify the `query` parameter here. {% endhint %} ### Zoom - cursor-based token ```json { "response_jsonpath": "$.{{ endpoint.rsplit('/', 1)[1] }}", "query": "{\"next_page_token\": \"{{ previous_request.next_page_token }}\"}" } ``` The Zoom API works (in some parts, e.g. for [meeting participants](https://marketplace.zoom.us/docs/api-reference/zoom-api/meetings/pastmeetingparticipants)) with the next page token. The response from Zoom looks like this: ```json { ..., "next_page_token": "XYZ", "participants": [ { ... }, { ... } ] } ``` Thus, the query needs to reference to the previous request's response using `previous_request.` and then the key of the next page token, which in Zoom's case is on the top level and is called `next_page_token`. Here `rsplit` with a `maxsplit` of 1 (second parameter) is used to split the endpoint into two elements (starting from the right side of the string) ([difference between `split` and `rsplit`](https://www.tutorialsteacher.com/python/string-rsplit)). As the endpoint looks like `metrics/meetings/{meetingId}/participants`, the result of `{{ endpoint.rsplit('/', 1) }}` is `['metrics/meetings/{meetingId}', 'participants']` and the second element (`[1]`) is thus `participants`, which points to the list of records. ### Dropbox - strange cursor based with special pagination endpoints ```json { "replace_body": true, "response_jsonpath": "$.entries", "body": "{\"cursor\": \"{{ previous_request.cursor }}\"}", "endpoint": "{{ endpoint }}/continue" } ``` Dropbox, e.g. with the [list\_folders endpoint](https://www.dropbox.com/developers/documentation/http/documentation#files-list_folder-continue) is quite a special case and uses a different endpoint (and http method) for getting the next pages. In order to allow for this `endpoint` and `body` parameters have to be used. Furthermore, `replace_body` has to be set to `true`. The `response_jsonpath` is quite straightforward, as the list of records is nested in `entries` for every pagination-enabled endpoint, so `$.entries` is used. ### Braze - limit and offset ```json { "response_jsonpath": "$.emails", "query": "{\"offset\": {{ ((page + 1) * 100) + 1 }}, \"limit\": 100}" } ``` Braze, Xandr uses **limit and offset-based** pagination (e.g. for [this endpoint](https://www.braze.com/docs/api/endpoints/email/get_list_hard_bounces/#request-parameters)). Alternative keywords: **limit** / **start** e.g. [Pipedrive](https://developers.pipedrive.com/docs/api/v1/Activities#getActivities). This works very similarly to page number pagination: The offset parameter needs to be increased with each call, while the limit parameter needs to be a static value. Here we can set the `limit` query parameter to the default of `100` and then we have to multiple `page + 1` with the `limit` value and add 1 to the result, as we want to begin not with the 100th element (which we already got in the first call, but rather with the 101th element. {% hint style="info" %} This might work differently for other limit and offset-based paginations, as `offset` sometimes refers to the number of results that should be skipped. However, Braze defines `offset`as: “Optional beginning point in the list to retrieve from” {% endhint %} ### Plentific - limit and offset in header ```json { "header": "{\"X-Pagination-Offset\": {{ page * 200 }}, \"X-Pagination-Limit\": 200}", "response_jsonpath": "$" } ``` ### Spiri.bo - limit and skip Spiri.bo uses limit and skip based pagination, similar to SQL queries. \ This type of pagination requires the skip parameter to increase by the limit value with each call, while the limit parameter remains static.\ Here, the limit is set to 20, and the skip parameter is calculated by multiplying the page variable by the limit value. The first request starts with skip=0, and subsequent requests increase the skip value by 20 (i.e., skip = page \* 20). `/contracts?limit=20&skip=0` (fetches items 0-19) `/contracts?limit=20&skip=20` (fetches items 20-39) This approach ensures no items are lost during pagination, as each subsequent request fetches the next batch of records. ```json { "response_jsonpath": "$.data", "query": "{\"skip\": {{ page * 20 }}, \"limit\": 20}" } ``` ### Shopify - cursor-based token in response header [Shopify's pagination](https://shopify.dev/api/usage/pagination-rest) has a few challenges: * The pagination tokens are links in a single header value (called `Link`) (i.e. one field for both the previous and next link) * The `Link` header value is different for the first and last page, as in those cases it contains only one link * No filter query parameters are allowed to be set in pagination requests * The response jsonpath cannot be reliably derived from its endpoint These challenges can be solved as follows: * For the first two challenges, quite a bit of Jinja logic is required, as can be seen in the `endpoint` parameter * The query parameters can be removed from paginations requests, by setting them to `null` (as it has been done in the `query` parameter). This way, they will not be sent as query parameter at all * Lastly, JSON path filter expressions can be used in order to be highly flexible in terms of response bodies ```json { "endpoint": "{% set previous_link, _, next_link = previous_request_headers.Link.partition(', ') %}{% if 'next' in previous_link %}{% set next_link = previous_link %}{% endif %}{% if next_link != '' %}{{ next_link.split(';')[0].strip('<>') }}{% else %}{{ previous_link.split('?')[0] }}?page_info=END_PAGINATION{% endif %}", "query": "{\"created_at_min\": null, \"created_at_max\": null, \"updated_at_min\": null, \"updated_at_max\": null, \"status\": null}", "response_jsonpath": "$.*[?id]" } ``` ## Limitations Currently, we have a few known and probably further unknown limitations which might be temporary or permanent. ### Very low rate limits We automatically handle rate limits with the same logic that's used for [retries in regular flow runs](https://docs.locoia.com/automation/flow-debugger#how-flow-failure-is-handled-smart-retries). However, some APIs, such as Twitter, have a very low rate limit, where only a few requests every minute are allowed. In case many more pages than the rate limit per minute have to be retrieved, the request will most likely result in an error as the pagination request will automatically stop after having received too many errors