​

When to use syncs?

Sync enable to continuously sync data between your app and external APIs. This is useful when you need to:

• keep a list of records in sync between your app and an external system
• take action when records from an external system are created, updated and/or deleted

Syncs handles the critical infrastructure & logic necessary to continuously & reliably sync data:

• API requests & data transformation
• Unification of schemas across external APIs
• Logging & monitoring
• Rate-limit handling
• Automated retries
• Syncs API

​

Step 1: Setup the Nango CLI & nango-integrations folder

Install the Nango CLI globally:

npm install -g nango

Your Nango syncs live in your repo in a folder called nango-integrations. You can place this folder anywhere in your file tree, but we recommend you place it at the root level of your project.

In the folder where you want your integrations folder (e.g. root of your project), run:

nango init # Creates `./nango-integrations` with initial config

<YOUR-REPO>
    |
    nango-integrations
            |
            +- nango.yaml
            +- github-issues.ts
            +- asana-tasks.ts
            ...

Next, we need to authenticate the CLI. Add the following env vars (e.g. in an .env file in ./nango-integrations):

NANGO_SECRET_KEY_PROD='<prod-secret-key>'
NANGO_SECRET_KEY_DEV='<dev-secret-key>'

Get your prod and dev secret keys from the Project Settings tab (toggle between the prod and dev environment in the left nav bar).

​

Step 2: Create a sync

​

Configure your sync in nango.yaml

Open the nango.yaml file inside the nango-integrations folder and inspect its field:

integrations:
	asana-dev: # Integration name (must match an integration name in the Integrations tab of your Nango dashboard).
		asana-tasks: # Arbitrary (unique) sync name
			runs: every 30min # Sync frequency
			returns:
				- AsanaTask # Data model (defined below) as returned by your sync script

models: 
	AsanaTask: # Data model referenced above
		id: string # Required unique ID
        project_id: string
        name: string
        completed: boolean
        created_at: date
        modified_at: date

ExampleUser:
	id: string # Required, we recommend a string or number
    project_id: string
    names: string[] # An array of strings
    number_of_cats: number
    completed: boolean
    emails:
        personal_email: string
        business_email: string | null
        other_emails: string[] | null
    created_at: date # Date is a full timestamp with both date & time
    modified_at: date

Nango uses the models you define in nango.yaml to provide type safety when:

• you write sync scripts
• sync scripts run
• you fetch the synced data

Syncs and data models can be combined at will:

• A sync can return multiple data models: e.g. AsanaTasks and AsanaComments
• Multiple syncs can return the same data model: e.g. Asana and Linear can both return a Task

This lets you easily create your own unified API with standard data models tailored to your needs.

​

Write your sync

Modify the configuration of nango.yaml as you need and run (in ./nango-integrations):

nango generate

This will generate the scaffold for your sync script(s). Open any sync script (named [sync-name].ts) which contains the following template (for the Asana example above):

import { NangoSync, AsanaTask } from './models';

export default async function fetchData(nango: NangoSync): Promise<void> {
	// Integration code goes here.
}

Sync scripts mostly do 2 things. They:

• incrementally fetch data from external APIs (with HTTP requests)
• transform the external data into the models that you defined in nango.yaml

This logic is defined by you so that you can build arbitrarily custom and complex integrations.

To develop syncs locally and test them run the following within ./nango-integrations:

nango dev # Continuously watches integration files for changes.

Nango now watches your nango-integrations folder for changes and compiles the sync scripts & data models as needed. If there are any compilation errors (e.g. due to type issues) you can see them in the terminal where nango dev runs.

Fill in the fetchData method with your integration code (in the example here, we fetch tasks from Asana):

import { NangoSync, AsanaTask } from './models';

export default async function fetchData(nango: NangoSync): Promise<void> {

    // Get the user's workspaces & projects
    // For testing we just get the first project of the first workspace
    const workspaces = await paginate(nango, '/api/1.0/workspaces');
    const workspace = workspaces[0];

    const projects = await paginate(nango, '/api/1.0/projects', { workspace: workspace.gid });
    const project = projects[0];

    // Get all tasks for the project
    const filters = {
        project: project.gid,
        opt_fields: 'name,completed,created_at,modified_at'
    }
    const tasks = await paginate(nango, '/api/1.0/tasks', filters);
    let mappedTasks: AsanaTask[] = [];

    for (let task of tasks) {

        mappedTasks.push({
            id: task.gid,
            project_id: project.gid,
            name: task.name,
            completed: task.completed,
            created_at: task.created_at,
            modified_at: task.modified_at
        });

        if (mappedTasks.length > 49) {
            await nango.batchSave(mappedTasks, 'AsanaTask');
            mappedTasks = [];
        }
    }

    await nango.batchSave(mappedTasks, 'AsanaTask');
}

async function paginate(nango: NangoSync, endpoint: string, queryParams?: Record<string, string|string[]>) {
    const MAX_PAGE = 100;
    let results: any[] = [];
    let page = null;
    let callParams = queryParams || {};
    while (true) {

        if (page) {
            callParams['offset'] = `${page}`;
        }

        const resp = await nango.get({
            endpoint: endpoint,
            params: {
                limit: `${MAX_PAGE}`,
                ...callParams
            }
        });

        results = results.concat(resp.data.data);

        if (resp.data.next_page) {
            page = resp.data.next_page.offset;
        } else {
            break;
        }
    }

    return results;
}

Sync scripts return the created/update data to Nango with await nango.batchSave(data, 'ModelName');. You can split the data into as many batches as needed.

​

Dry run your sync

Before you deploy your sync to your cloud account you can test it locally to make sure it works as expected. You will probably use this a lot whilst developing your sync.

Use the dryrun function of the CLI:

nango dryrun asana-tasks <connection-id>

Because this is a dry run, the fetched data will not be stored in Nango. Instead dryrun will print the data as returned.

nango dryrun asana-tasks <connection-id> -l "2023-06-20T10:00:00.000Z"

​

Step 3: Deploy a sync

1. Deploy to the Dev environment

When your sync script is ready, you can deploy it to your Dev environment in Nango:

nango deploy dev

Inspect the Syncs & Actions tab to verify the deployment succeeded.

2. Deploy to the Prod environment

Once you are ready to deploy to production, run:

nango deploy prod

​

Step 4: Retrieve the synced data

Fetch the always-fresh data from the sync in your backend with the backend SDK or the REST API:

• Node SDK

• cURL

import { Nango }  from '@nangohq/node';
import type { AsanaTask } from '<path-to-nango-integrations>/models'

const nango = new Nango({ secretKey: '<your-dev-secret-key>' });

const result = await nango.getRecords<AsanaTask>({
    providerConfigKey: 'asana',
    connectionId: 'test-connection-id',
    model: 'AsanaTask'
});

Nango can also send you webhooks whenever new data is available from a sync.

That’s it!

​

Advanced Configuration

​

Handling API rate-limits

Nango has currently two approaches to handle rate limits, a generic/naive one and an API-specific one.

The generic & naive approach is based on retries & exponential-backoff. When you make network requests with the proxy in a sync with a high number of retries, exponential back-off will increase the delay between retries, augmenting the chances to go back under the rate-limit. But this “blind” approach is inefficient both in terms of optimising the time between requests and avoiding complex rate-limits.

The API-specific approach is based on reading the rate-limit headers returned by the external APIs. Nango observes these headers and pauses the sync job until the rate-limit is passed. This approach has the benefit of being more efficient both for minimizing sync durations and avoid failures due to rate-limiting.

This second approach requires to edit Nango’s providers.yaml file to indicate the rate-limit header name for a specific API (in the retry entry, under at or after fields):

Github example:

github:
    auth_mode: OAUTH2
    authorization_url: https://github.com/login/oauth/authorize
    token_url: https://github.com/login/oauth/access_token
    proxy:
        base_url: https://api.github.com
        retry:
            at: 'x-ratelimit-reset'
    docs: https://docs.github.com/en/rest

Discord example:

discord:
    auth_mode: OAUTH2
    authorization_url: https://discord.com/api/oauth2/authorize
    token_url: https://discord.com/api/oauth2/token
    authorization_params:
        response_type: code
    proxy:
        retry:
            after: 'Retry-After'

​

Handling record deletion

In Nango, syncs can detect record deletions. These deleted records are not removed but are flagged as “soft deleted” using metadata. This ensures that you can still view the records in Nango, but recognize them as deleted entities.

Each record has a _nango_metadata field that carries metadata, including deletion information:

_nango_metadata: {
    deleted_at: null,
    last_action: 'ADDED',
    first_seen_at: '2023-09-18T15:20:35.941305+00:00',
    last_modified_at: '2023-09-18T15:20:35.941305+00:00'
},

Deletion handling differs depending on the mode of the sync – whether it’s incremental (fetches only changes since the last sync run) or full refresh (fetches all data every time).

Incremental syncs

In incremental syncs, utilize the batchDelete method in your script to flag records as deleted:

await nango.batchDelete<User>(users, 'User');

Note: The deleted entities (in this case, users) need to provide just the unique id field.

Full refresh syncs

For syncs that operate in full refresh mode, add the track_deletes: true field to your nango.yaml configuration for the particular sync. This ensures that when a record previously returned by the sync script isn’t returned in the latest run, Nango will flag it as deleted.