Skip to content

One EC2 instance

This page shows how to run the whole of SREInsights on one box: Postgres, the control plane and TLS, in Docker Compose. Scripts in the repository’s deploy/ec2/ folder set it up, update it, and back it up.

You need access to the SREInsights repository, which is private: ask the SREInsights team.

deploy/ec2/docker-compose.yml (Compose project sreinsights) runs five services:

Service What it is
postgres postgres:16-alpine, with its data in the pgdata volume. Never published: only containers on the host reach it
migrate Applies migrations and seeds, then exits. Runs first on every start
app The API, the runner protocol and the web app, on port 8080 inside the host
worker Scheduler, alert processor and notification delivery
caddy caddy:2-alpine. Gets a certificate from Let’s Encrypt and proxies your domain to app. Publishes ports 80 and 443

Every setting comes from deploy/ec2/.env, which bootstrap.sh writes and git never sees.

  • An instance. Ubuntu 22.04 or 24.04, or Amazon Linux 2023. A t3.medium or t4g.medium (4 GB) can build the images itself. Anything smaller should pull published images instead (set SREINSIGHTS_IMAGE, below).
  • DNS. A record for your domain pointing at the instance’s public IP.
  • Ports 80 and 443 open in the instance’s security group. Caddy needs both for the certificate.

On the instance:

Terminal window
git clone <SREInsights repository URL> sreinsights && cd sreinsights
SREINSIGHTS_DOMAIN=sreinsights.example.com ACME_EMAIL=ops@example.com deploy/ec2/bootstrap.sh

bootstrap.sh:

  1. Installs Docker and the Compose plugin if they are missing (with apt-get or dnf), and adds you to the docker group. If docker ps then asks for permission, log out and in once, and run the script again.
  2. Writes deploy/ec2/.env with mode 600 and freshly generated passwords for Postgres and the three database roles, a session secret and a master key. It prints the master key once.
  3. Prepares deploy/ec2/initdb/, so Postgres creates the sreinsights database and its three roles (with deploy/postgres/setup.sql) on its first start.
  4. Builds the images on the box, or pulls them when SREINSIGHTS_IMAGE is set.
  5. Starts everything and waits up to 3 minutes for /readyz.

When it is ready it prints your URL and the next steps. Open the URL and sign up: the first sign-up creates your organization.

bootstrap.sh is safe to run again: it keeps an existing .env.

Set these in the environment when you run bootstrap.sh. It copies them into .env.

Variable Required What it does
SREINSIGHTS_DOMAIN Yes, on the first run Your domain. SREInsights answers at https://<domain>
ACME_EMAIL No The email Let’s Encrypt uses for certificate notices. Set it to your own
SREINSIGHTS_IMAGE No A published control plane image to pull instead of building on the box
SREINSIGHTS_RUNNER_IMAGE No The runner image the Runners page shows. Default ghcr.io/bintree-softwares/sreinsights-runner:latest
ADAM_API_KEY No Anthropic API key, to turn on ADAM
BACKUP_BUCKET No S3 bucket that backup.sh uploads dumps to

To change a setting later, edit deploy/ec2/.env and run docker compose up -d in deploy/ec2/. Settings you can add are in Environment variables.

Script What it does
deploy/ec2/update.sh [ref] Deploys a new version: checks out the git ref (default main), rebuilds or pulls the images, runs migrations, restarts
deploy/ec2/backup.sh Dumps the database to deploy/ec2/backups/, and to S3 when BACKUP_BUCKET is set
deploy/ec2/restore.sh <dump> Restores a dump made by backup.sh. Stops the app while it runs
deploy/ec2/hosted-runner.sh <org-slug> <token> [labels] Starts a runner container on the box for one organization
Terminal window
deploy/ec2/update.sh # the latest main
deploy/ec2/update.sh <ref> # a tag, branch or commit

It fetches and checks out the ref, then builds migrate, app and worker (or pulls them when .env has SREINSIGHTS_IMAGE). migrate runs and must succeed before app and worker restart. It removes unused images and waits up to a minute for /readyz. If the API doesn’t become ready, look at docker compose logs app migrate in deploy/ec2/.

Terminal window
deploy/ec2/backup.sh

It runs pg_dump inside the postgres container, in custom format, and writes deploy/ec2/backups/sreinsights-<UTC time>.sql.gz. With BACKUP_BUCKET in .env, it uploads the file to s3://<bucket>/sreinsights/, with the AWS CLI if it is installed or the amazon/aws-cli container if not. The instance role (or an AWS profile on the box) needs s3:PutObject on the bucket.

Local dumps older than BACKUP_KEEP_DAYS days (default 30) are deleted.

Run it nightly from cron:

Terminal window
(crontab -l 2>/dev/null; echo "15 3 * * * $(pwd)/deploy/ec2/backup.sh >> /var/log/sreinsights-backup.log 2>&1") | crontab -
Terminal window
deploy/ec2/restore.sh backups/sreinsights-20260916T031500Z.sql.gz

The path is relative to deploy/ec2/, or absolute. It asks you to type RESTORE, then:

  1. Stops app and worker.
  2. Replaces the sreinsights database with the dump (pg_restore --clean --if-exists --no-owner --no-privileges).
  3. Gives ownership back to sreinsights_owner. Grants come from the roles, not from the dump.
  4. Starts app and worker again.

A dump is only readable with the master key of the instance it came from. Restore onto a new instance with the old SREINSIGHTS_MASTER_KEY in its .env.

hosted-runner.sh starts an ordinary runner container on the instance, in one organization’s own runner group. Use it for an organization whose endpoints are reachable from the internet, so it needs no runner of its own.

Terminal window
deploy/ec2/hosted-runner.sh acme prt_… hosted,us
Argument What it is
<org-slug> Names the container sreinsights-runner-<org-slug>, the runner hosted-<org-slug>, and its state volume
<token> A registration token from that organization’s Runners page
[labels] Optional runner labels, comma-separated. Default hosted

The runner joins the Compose network and reaches the control plane at http://app:8080, with --stop-timeout 40 and the hardening flags. Its key lives in a named volume, so restarts need no new token. Running the script again replaces the container. The runner appears on the organization’s Runners page within a minute.

This is not a hosted runner group that every organization can pick. For those, see Hosted runner groups.

Run the operator commands in the app container, from deploy/ec2/:

Terminal window
docker compose exec app sreinsights clients list --org acme

Create the roles on the new server with deploy/postgres/setup.sql (any PostgreSQL 14 or newer), move the data, and change the three connection strings (DATABASE_OWNER_URL, DATABASE_URL, SYSTEM_DATABASE_URL) in deploy/ec2/docker-compose.yml.