Locoia
  • Overview
  • Account and User Settings
    • User types
    • Adding Users
    • Teams
    • Access Permissions
    • 2 Factor Authentication 2FA
    • Versioning and Snapshots
    • Activity Log
  • Reset your Password
  • Invoices and Payments
  • Automation
    • Flow Builder
      • Flow Building Best Practices
      • Jinja Template Language
        • Jinja: (Custom) variables, wildcards and functions
        • Magic Code Samples
      • Connectors & APIs
        • Titles and References
        • Referencing data of objects, lists, arrays - how to pass data dynamically
        • Accessing Objects with JSONPath
        • Merging nested JSON objects
        • Parsing JSONs from String
        • Response Headers & Status Codes
        • Custom Data Fields
        • Wildcard API calls and actions
        • Response cleaning
      • Text Strings, Date & Time, Numbers and Currencies
        • Text and Strings
        • Dates & Time
        • Numbers (Thousand Separators, Currencies)
      • Email-formatting
      • Code Fields
      • Running single Flow steps
      • Flow run data retention, logging, and error notifications
      • Advanced View
      • Dynamic Title
      • Custom Error Handling
      • Error Handling Flows
      • Automatic Pagination
    • Flow Debugger
      • Automatic Retrying
      • Run Flows again
      • Troubleshooting Flows
    • Community Library
  • Connectors & Helpers
    • Connectors
      • Monday.com
      • ActiveCampaign
      • Aircall
      • Allthings
      • Amplitude
      • Animus
      • Assetti
      • Awork
      • AWS RDS Database - How to connect
      • bubble.io
      • Casavi
      • Chargebee
      • CleverReach
      • comgy
      • commercetools
      • Everreal
      • Exact Online
      • Facebook Marketing
      • Fahrländer Partner
      • FastBill
      • FILESTAGE.io
      • Freshdesk
      • Freshsales
      • Google Ads
      • Google Ads Lead Form
      • Google Analytics
      • Google Chat
      • Google Drive
      • Google Sheets
      • Gmail
      • HubSpot
      • Heyflow
      • iDWELL
      • ImmobilienScout24
      • Instagram Ads
      • Intercom
      • klaviyo
      • Kiwi Opening Doors
      • Klenty
      • Klipfolio
      • Kolibri CRM
      • konfipay
      • KUGU
      • Shopify
      • S3 AWS
      • SQS AWS
      • Lambda AWS
      • Learnster
      • lexoffice
      • LineMetrics
      • Linkedin
      • Locoia
      • Notion
      • MailGun
      • Makula
      • Microsoft Dynamics 365
      • Microsoft OneDrive
      • MixPanel
      • MongoDB
      • Odoo
      • OnFleet
      • OnOffice
      • Oracle NetSuite
      • Outbrain
      • Quickbooks
      • Trello
      • PandaDoc
      • Personio
      • Pinterest Ads
      • Pipedrive
      • Plentific
      • PriceHubble
      • relay
      • REALCUBE
      • Sage ERP
      • Salesforce
      • SAP
      • Scoro
      • Seafile
      • sevDesk
      • SharePoint
      • SharpSpring
      • Slack
      • Snapchat Marketing
      • Snowflake
      • Teamleader Focus
      • Teamwork.com
      • Tableau
      • TikTok
      • TinQwise
      • The Trade Desk
      • Twitter
      • Typeform
      • WordPress
      • Xero
      • Youtube
      • Zendesk
      • Zoho CRM
      • Zoom
    • Helpers
      • Scheduler
      • Webhook
      • Dict Helper
      • Spreadsheet Helper
      • REST Helper
      • Boolean Helper
      • Multi Case Helper
      • Looper
      • FTP Helper
      • CSV Helper
      • XLSX Helper
      • Mail Sender
      • Flow Trigger
      • File Storage Helper
      • Terminate Helper
      • Delay Helper
      • SQL Connector
      • PDF Helper
      • Zip Helper
      • Data Warehouse Helper
      • XML Helper
      • Form Helper
      • Arrow
      • Error Arrow
    • Authentication Types Available
      • Setting up authentication
      • OAuth1
      • OAuth2
      • Refreshable token
      • AWS Signature
      • Basic Auth and Other Simple Authentication Methods
      • How are API versioning and API updates handeled?
      • Custom OAuth2 clients (apps)
    • Building Connectors
      • Base Connector Setup
        • Connector Auth Validation
        • GraphQL APIs
        • Rendering with User Input
      • Building Connector Actions
        • Actions Examples
      • Search Automation
      • Pagination Automation
      • Uploading Files in Actions
      • Working with SOAP APIs
    • Super Actions
    • Webhook Trigger for Connectors
    • Data Mapping and Env Variables
  • Embed - White Label Portal
    • Embed Overview
      • 1. Embed Flow
        • 1.1 Creating Embed Flows
        • 1.2 Updating Embed Flows
        • 1.3 Embed Error Handling
        • 1.4 Setting up Callbacks for Integration activation/deactivation
        • 1.5 Setting up Remote search
        • 1.6 Setting up End User logs
      • 2. Configure Embed
        • 2.1 Embed Integration via SSO
        • 2.2 Proprietary connector setup
        • 2.3 Sharing level
        • 2.4 Consent screen
        • 2.5 Account Secrets
        • 2.7 Further settings
      • 3. Integrate Embed
        • 3.1 iframe vs native embed
        • 3.2 Customizing CSS
        • 3.3 Events emitted from iframe to parent window
      • 4. Embed for End User
        • 4.1 Embed Remote Search
        • 4.2 Embed End User Logs
      • 5. Test Embed Configuration
        • Testing example
      • 6. Embed Integrations and Connector Auths
    • Embed FAQs
  • Data and Dashboards
    • Dashboards & Insights
      • Introduction to Dashboards
      • Introduction to Insights
      • Introduction to Data Sources
      • Dashboard Filters
      • Insight Marketplace - Using Pre-Built Insights
      • Writing SQL Queries
      • Useful SQL Examples
      • Charts
        • Line Chart
        • Bar and Horizontal Bar Chart
        • Stat Card
        • Pie Chart
        • Gauge Chart
        • Donut Chart
        • Stacked Bar, Horizontal Stacked Bar, and Normalized Horizontal Stacked Bar
        • Multiple Line Chart
        • Pivot Table
        • Map Chart
  • Best Practice Guides
    • Integration Best Practices
    • Integration Check List
    • CSV Files in Excel
    • Multi-Tenant Flows
    • On-Premise Integrations
    • Database Connection Setup
    • Data and General Security
    • Using Tags
    • FAQ
  • API
    • Locoia API Authentication - Personal Access Token
    • Create Connector Authentication
  • Contact us
  • Status of Service
  • Data Privacy
  • Imprint
Powered by GitBook
On this page
  • Feature Introduction
  • Configuration on connector
  • query
  • response_jsonpath (optional)
  • endpoint (optional)
  • body
  • replace_body (optional)
  • header
  • Multiple configurations
  • Configuration on action
  • Examples
  • Freshsales - page / per_page with nested response list
  • Freshdesk - page / per_page with raw list in response
  • Zoom - cursor-based token
  • Dropbox - strange cursor based with special pagination endpoints
  • Braze - limit and offset
  • Plentific - limit and offset in header
  • Spiri.bo - limit and skip
  • Shopify - cursor-based token in response header
  • Limitations
  • Very low rate limits

Was this helpful?

  1. Connectors & Helpers
  2. Building Connectors

Pagination Automation

This will automatically do the entire pagination logic and will output a nice 'flat' list of records.

PreviousSearch AutomationNextUploading Files in Actions

Last updated 5 months ago

Was this helpful?

Feature Introduction

Pagination is often necessary to retrieve all results from a connector's API when data spans multiple pages. The advantages of automatic pagination over manual pagination are outlined .

Configuration on connector

Here we have to fill the Pagination Configuration field in JSON format, the same one that's also used by :

In most cases, the configuration will look something like this:

{
  "response_jsonpath": "$.{{ endpoint.split('/', 1)[0] }}",
  "query": "{\"page\": {{ page + 1 }}}"
}

You can use Jinja in all fields in order to have conditional statements or do some simple calculations (e.g. as above page + 1).

In the first request, the parameters are not used, only when pagination is needed it will be used. response_jsonpath is of course also used to point to the list of results of the first request

query

Either query, body, or header needs to be specified

The query is a string that contains a dictionary of query parameters that will be appended to the action call in order to do the pagination.

In case the same parameter is given by the action itself, it will be overwritten by the parameter defined here.

The page parameter starts with 1 and is increased by 1 for every subsequent call.

Since most Connectors' paginate starts with 1, we do need to add + 1 to it in most cases, to start the first pagination request with 2 (as pointed out above, the first request is done independently of it, so the counting only starts after the second request).

The maximum page size is currently 100 in order to not accidentally run into endless loops or something similar. If we see that this needs to be adjusted, we can do so easily.

By using previous_request. we can reference to the previous requests's response, which is e.g. needed when dealing with next page tokens.

If the queryparameter is given a GET call is being done. If a POSTcall should be done instead, also specify body with {} as its value.

response_jsonpath (optional)

This points to the list of results that should be used, which is relevant if the response is nested, e.g. in this example:

{
  "deals": [
    { ... },
    { ... }
  ],
  "meta": { ... }
}

we expect the final output of the action to be a list of dictionaries, only containing entries that are inside the deals list.

endpoint (optional)

body

Either query, body, or header needs to be specified

The body parameter is similar to the query parameter. It also contains a string which is a dictionary. However, instead of query parameters, it contains the request body that will be used in the calls.

If the body parameter is given a POST call is automatically being done.

replace_body (optional)

This defines whether the regular body that's being sent by the action should be replaced and only contains the pagination body as defined in the configuration.

It can be either false or true.

header

Either query, body, or header needs to be specified

The header parameter is similar to the body parameter. It also contains a string which is a dictionary. However, instead of the request body, it contains the request headers that will be used in the calls.

If the header parameter is given a GET call is automatically being done. If a POST call should be done instead, also specify body with {} as its value.

Multiple configurations

In order to add multiple pagination configurations for one Connector, the configuration needs to be a list of configurations and the parameter when needs to be used, except for the last statement, which can be a 'catch all' configuration.

The when parameter will be checked from top to bottom and the one that matches first will be used. In case none match and a pagination configuration does not have a when parameter, that configuration will be used (i.e. similar to multiple elif and lastly an else statement).

Configuration on action

On the action just toggle the switch button next to Supports automatic pagination in order to show the Retrieve all data (supports_automatic_pagination) toggle on the action:

Examples

Freshsales - page / per_page with nested response list

{
  "response_jsonpath": "$.{{ endpoint.split('/', 1)[0] }}",
  "query": "{\"page\": {{ page + 1 }}}"
}

Here the interesting piece is the response_jsonpath: As mentioned above, the Freshsales response structure looks like this:

{
  "deals/contacts/sales_accounts": [
     { ... },
     { ... }
  ],
  "meta":  { ... }
}

where it's one of deals/contacts/sales_accounts depending on the endpoint (e.g. deals/view/{id}). With {{ endpoint.split('/', 1)[0] }} the endpoint is split into a list at every / and then the first element of that list ([0]) is accessed, which would be for the example deals. So the response_jsonpath is $.deals, which points directly to the list of records.

Freshdesk - page / per_page with raw list in response

{
  "query": "{\"page\": {{ page + 1 }}}"
}

Freshdesk returns the list of records without any nesting before it, thus we only need to specify the query parameter here.

Zoom - cursor-based token

{
  "response_jsonpath": "$.{{ endpoint.rsplit('/', 1)[1] }}",
  "query": "{\"next_page_token\": \"{{ previous_request.next_page_token }}\"}"
}

The response from Zoom looks like this:

{
  ...,
  "next_page_token": "XYZ",
  "participants": [
     { ... },
     { ... }
  ]
}

Thus, the query needs to reference to the previous request's response using previous_request. and then the key of the next page token, which in Zoom's case is on the top level and is called next_page_token.

As the endpoint looks like metrics/meetings/{meetingId}/participants, the result of {{ endpoint.rsplit('/', 1) }} is ['metrics/meetings/{meetingId}', 'participants'] and the second element ([1]) is thus participants, which points to the list of records.

Dropbox - strange cursor based with special pagination endpoints

{
  "replace_body": true,
  "response_jsonpath": "$.entries",
  "body": "{\"cursor\": \"{{ previous_request.cursor }}\"}",
  "endpoint": "{{ endpoint }}/continue"
}

In order to allow for this endpoint and body parameters have to be used. Furthermore, replace_body has to be set to true.

The response_jsonpath is quite straightforward, as the list of records is nested in entries for every pagination-enabled endpoint, so $.entries is used.

Braze - limit and offset

{
  "response_jsonpath": "$.emails",
  "query": "{\"offset\": {{ ((page + 1) * 100) + 1 }}, \"limit\": 100}"
}

This works very similarly to page number pagination: The offset parameter needs to be increased with each call, while the limit parameter needs to be a static value.

Here we can set the limit query parameter to the default of 100 and then we have to multiple page + 1 with the limit value and add 1 to the result, as we want to begin not with the 100th element (which we already got in the first call, but rather with the 101th element.

This might work differently for other limit and offset-based paginations, as offset sometimes refers to the number of results that should be skipped. However, Braze defines offsetas: “Optional beginning point in the list to retrieve from”

Plentific - limit and offset in header

{
  "header": "{\"X-Pagination-Offset\": {{ page * 200 }}, \"X-Pagination-Limit\": 200}",
  "response_jsonpath": "$"
}

Spiri.bo - limit and skip

Spiri.bo uses limit and skip based pagination, similar to SQL queries. This type of pagination requires the skip parameter to increase by the limit value with each call, while the limit parameter remains static. Here, the limit is set to 20, and the skip parameter is calculated by multiplying the page variable by the limit value. The first request starts with skip=0, and subsequent requests increase the skip value by 20 (i.e., skip = page * 20).

/contracts?limit=20&skip=0 (fetches items 0-19)

/contracts?limit=20&skip=20 (fetches items 20-39)

This approach ensures no items are lost during pagination, as each subsequent request fetches the next batch of records.

{
  "response_jsonpath": "$.data",
  "query": "{\"skip\": {{ page * 20 }}, \"limit\": 20}"
}

Shopify - cursor-based token in response header

  • The pagination tokens are links in a single header value (called Link) (i.e. one field for both the previous and next link)

  • The Link header value is different for the first and last page, as in those cases it contains only one link

  • No filter query parameters are allowed to be set in pagination requests

  • The response jsonpath cannot be reliably derived from its endpoint

These challenges can be solved as follows:

  • For the first two challenges, quite a bit of Jinja logic is required, as can be seen in the endpoint parameter

  • The query parameters can be removed from paginations requests, by setting them to null (as it has been done in the query parameter). This way, they will not be sent as query parameter at all

  • Lastly, JSON path filter expressions can be used in order to be highly flexible in terms of response bodies

{
  "endpoint": "{% set previous_link, _, next_link = previous_request_headers.Link.partition(', ') %}{% if 'next' in previous_link %}{% set next_link = previous_link %}{% endif %}{% if next_link != '' %}{{ next_link.split(';')[0].strip('<>') }}{% else %}{{ previous_link.split('?')[0] }}?page_info=END_PAGINATION{% endif %}",
  "query": "{\"created_at_min\": null, \"created_at_max\": null, \"updated_at_min\": null, \"updated_at_max\": null, \"status\": null}",
  "response_jsonpath": "$.*[?id]"
}

Limitations

Currently, we have a few known and probably further unknown limitations which might be temporary or permanent.

Very low rate limits

However, some APIs, such as Twitter, have a very low rate limit, where only a few requests every minute are allowed. In case many more pages than the rate limit per minute have to be retrieved, the request will most likely result in an error as the pagination request will automatically stop after having received too many errors

This follows mostly the same rules and logic as in the :

We have point to it using ().

This can be used to adjust the default endpoint for all except the first page, which in some cases (e.g. ) needs to be extended to get the results of the next page with a next page token.

You can also reference to endpoint in any of the parameters, which is often useful for e.g. the response_jsonpath ().

So far we only experienced this for one connector (), so this can be left empty by default ("replace_body": false will be automatically added during saving of the configuration)

This needs to be used in combination with .

The Zoom API works (in some parts, e.g. for ) with the next page token.

Here rsplit with a maxsplit of 1 (second parameter) is used to split the endpoint into two elements (starting from the right side of the string) ().

Dropbox, e.g. with the is quite a special case and uses a different endpoint (and http method) for getting the next pages.

Braze, Xandr uses limit and offset-based pagination (e.g. for ).

Alternative keywords: limit / start e.g. .

has a few challenges:

We automatically handle rate limits with the same logic that's used for .

Search configuration
meeting participants
difference between split and rsplit
list_folders endpoint
this endpoint
Pipedrive
Shopify's pagination
retries in regular flow runs
JSON path
more examples below
Dropbox
see examples below
Dropbox
multiple configurations
here
Remote Search Configuration