Connecting data
Sync settings
Full refresh or incremental, cursor columns, schedules, column selection, and what happens when a source changes.
Each table in a connection syncs on its own. Open the connection, choose a table, and set its mode, schedule and columns.
Sync modes
| Mode | What happens on each run | Good for |
|---|---|---|
| Full refresh (replace) | The whole table is read again and replaces what was there. This is the default. | Small tables, and tables whose rows change without a reliable "last modified" column |
| Incremental (append new) | Only rows changed since the last run are transferred, using a cursor column. | Large tables that have a last-modified timestamp |
Some sources limit which modes a table allows. For example, Google Analytics 4's intraday table is full refresh only.
Cursor column
Incremental sync needs a cursor column: a column whose value goes up whenever a row changes. Datachef lists the cursors each source declares:
| Source | Cursor options |
|---|---|
| Salesforce | SystemModstamp, LastModifiedDate |
| Marketo | updatedAt (leads), activityDate (activities) |
| PostgreSQL, MySQL | Datachef suggests date, timestamp and integer columns |
For Salesforce, we recommend incremental on SystemModstamp, hourly.
How incremental tables handle changed records
An incremental table keeps every version of a record that changed, and queries only see the newest version of each one. So SELECT * FROM salesforce.opportunity returns one row per opportunity, as it stands now.
Switching a table to incremental, or changing its cursor column, reads the whole source table once on the next sync. That run replaces the table and the versions it kept.
Schedule
Choose how often each table syncs:
- Every 15 minutes
- Hourly
- Every 6 hours
- Daily (the default)
- Weekly
- Custom (cron), for example
0 */4 * * *for every four hours - Manual only, which syncs only when you start it yourself
The table's page shows when the next run is due. You can also sync a single table or a whole connection straight away.
Columns
By default a table syncs every column. To sync only some of them, select the ones you want. Unselected columns are left out of the query sent to the source, so they never leave it.
If you remove a column later:
- On a full refresh table, its values are gone after the next sync.
- On an incremental table, rows that were already synced keep their values.
Turning a table off
Turning a table's sync off stops new data from arriving. The rows already synced stay, and you can still query them.
When the source changes
- New columns and wider types in the source are added to the table automatically on the next sync.
- A source table that disappears is marked as retired. It's hidden and can't be scheduled, but the data already synced is kept.
Limits
- A single run reads at most 5,000,000 rows. A large incremental catch-up is spread across several runs.
- If a sync for a table is already queued or running, starting another one doesn't queue a duplicate.
Sync status
Each connection shows a status of pending, ok or error, with the details. Each run is listed with its outcome (queued, running, succeeded, failed or cancelled), the number of rows it read, and any error message.
Something missing or wrong on this page? Tell us