Guide on how to handle the synchronization of large amounts of data from APIs.
nango.lastSyncDate
. You can use this timestamp to instruct the external API to send only the changes since that date. That way, you only receive and persist the modified records in the Nango cache.
For example, if you are syncing tens of thousands of contacts from a Salesforce account on an hourly basis, only a small portion of the contacts will be updated or created in any given hour. If you were doing a full refresh sync, you would need to fetch the entire contact list every hour, which is inefficient. With an incremental sync, you can fetch only the modified contacts from the past hour, as allowed by Salesforce.
Here is an example of a sync script that updates the list of Salesforce contact incrementally on each execution, leveraging nango.lastSyncDate
: