Skip to content

Syncs

A sync reads rows from a source, maps them, and writes them to a destination: source → mapping → destination. It is a job of the built-in type sync, so it has a schedule, Run now, a run history and the job alerts (failed run, no new data), and it belongs to a client like any job. Create one with New sync on the Jobs page.

What makes a sync different is its spec: configuration, never code, and never secrets (connections are named by role).

Today the source is a Google Sheet, read one of two ways:

  • As a service account (the default): through the Sheets API with a google_service_account connection. Share the sheet with the service account’s address as a Viewer.
  • By public link: the sheet’s public CSV export, with no sign-in. No connection is needed, but anyone with the link can read every column. Prefer a service account for a sheet with anything private in it.

You choose which tabs to read (all, some by name, or those whose name contains a word) and how to find each tab’s header row.

  • Columns are named by their header (contains some words, or equals). Only those columns are read; every other cell is dropped on the runner straight away.
  • A header containing pass, user, login, email or e-mail is never read, whatever the spec says, and a spec that asks for one is refused. You can add more words; you can’t take these away.
  • Fields are the destination’s fields, each an expression over the row’s columns, such as col.amount|int.
  • Keep rules say which rows are written, each with a reason shown for rows left out.
  • The key names each row. It must stay the same while the row means the same thing: never the row number, which changes when someone inserts a row. Writes update by key, so a re-run never adds a row twice.
Destination Writes
HTTP API (http_destination) Batches of up to 500 rows to your endpoint, with a bearer token, in the SREInsights sync protocol. Your API answers what it did with each row.
Postgres table (postgres_destination) One table with a sync_key text PRIMARY KEY column; inserts or updates each row by key, and skips unchanged rows.
Lake (s3_destination) A client’s rows staged as Parquet files under that client’s own prefix, for the lake to merge.

The exact protocol and table setup are in the sync spec reference.

A sync is saved paused. Before it can run:

  1. Select Preview. A runner does a dry run and shows each tab’s header and the columns it found, the rows it would leave out and why, and the first 50 rows it would write with what the destination would do with each.
  2. Once a preview of the current version has run, switch the sync on, or use Run now.

Changing the spec or the connections makes a new version, which must be previewed again. Until it is, switching it on or Run now answers 409 SYNC_NOT_PREVIEWED.

  • It never reads a never-read column, and it reports rows left out only by key and reason, never their values.
  • It runs inside the runner’s own process, not as a command, so on hosted runners the egress guard covers it.
  • It never deletes: rows deleted from the sheet stay where they were written.
  • A client’s sync reads through the client’s own source connection and writes to the client’s own destination, never shared ones.

Walk through one in Sync a Google Sheet.