Detecting deletes in incremental syncs
Incremental syncs only fetch the delta (new/updated records) since the previous run, so Nango has no built-in way to know which records disappeared on the provider side. You must actively tell Nango which IDs have been removed by callingnango.batchDelete()
(full reference) inside the sync functions.
When can you use this?
You can usenango.batchDelete()
if the external API supports one of the following:
- A dedicated “recently deleted” endpoint (e.g.
GET /entities/deleted?since=...
) - The ability to filter or sort by a deletion timestamp
- The ability to filter or sort by last-modified timestamp and records include a flag like
is_deleted
,archived
, etc.
Example incremental sync with deletion detection
Detecting deletes in full refresh syncs
Full refresh syncs download all records when they run. Nango can therefore detect removals by computing the diff between two consecutive result sets. Enable this behaviour by calling thedeleteRecordsFromPreviousExecutions
function. (full reference).
deleteRecordsFromPreviousExecutions
does not work with incremental syncs because fetching the data incrementally prevents performing a diff and automatically detecting deletions.Example full refresh sync with deletion detection
How the algorithm works
- During the execution, Nango stores the list of record IDs.
- When calling
deleteRecordsFromPreviousExecutions
, Nango compares the new list with the old one. - Any records missing in the new list are marked as deleted (soft delete). They remain accessible from the Nango cache, but with
record._metadata.deleted === true
.
Be careful with exception handling when using
deleteRecordsFromPreviousExecutions
Nango only performs deletion detection (the “diff”) if a sync run completes successfully without any uncaught exceptions.If you’re using deleteRecordsFromPreviousExecutions
, exception handling becomes critical:- If your sync doesn’t fetch the full dataset, but still call
deleteRecordsFromPreviousExecutions
(e.g. you catch and swallow an exception), Nango will attempt the diff on an incomplete dataset. - This leads to false positives, where valid records are mistakenly considered deleted.
deleteRecordsFromPreviousExecutions
is not being called:- Let exceptions bubble up and interrupt the run.
- If you’re using
try/catch
, re-throw exceptions that indicate incomplete data.
How to use
deleteRecordsFromPreviousExecutions
safelyIf some records are incorrectly marked as deleted due to calling deleteRecordsFromPreviousExecutions
improperly, you can trigger a full resync (via the UI or API) to restore the correct data state.However, because deleteRecordsFromPreviousExecutions
relies on logic in your sync functions, mistakes there can lead to false deletions.We strongly recommend not performing irreversible destructive actions (like hard-deleting records in your system) based solely on deletions reported by Nango. A full resync should always be able to recover from issues.Troubleshooting deletion detection issues
Symptom | Likely cause |
---|---|
Records that still exist in the source API are shown as deleted in Nango | sync didn’t save all records (silent failures) before calling deleteRecordsFromPreviousExecutions |
You never see deleted records | Check if deletion detection is implemented for the sync. |
Questions, problems, feedback? Please reach out in the Slack community.