Skip to content

Move a scheduled job into SREInsights

A script on a schedule somewhere (a crontab line, a launchd agent on a Mac, an EventBridge rule in front of a Lambda function) runs until the day it silently stops. Moved into SREInsights it gets a run history, Run now, the logs of each run, and an alert the first time it fails.

This guide moves one script. Repeat it for each.

Your script Move it in as
Does a task: refreshes a token, publishes a post, cleans up, calls an API A monitor with a script check. This guide.
Collects data day by day into storage, and should be able to backfill A job on your own job type. See the job type contract.
Only checks something (a page, a row count) A monitor with a built-in check type, no script needed.

A script check runs Node, Python or shell, for up to 10 minutes a run. The script passes when it exits 0, or when its assertion holds.

Runner When
A Lambda runner The script only needs the internet and AWS. Nothing runs between runs.
A Docker runner The script needs your private network, a local file system, or Docker.
Hosted runners Not for scripts: hosted runners don’t run them.

The runner image needs your script’s interpreter and libraries. The standard images have Node and a shell. For Python or extra packages, extend the image (see Lambda runner, step 1) and copy your code in, for example to /opt/jobs/.

SREInsights schedules are five-field cron expressions with a time zone, which you set on the monitor.

You have SREInsights cron
crontab */15 * * * * */15 * * * * (the same)
launchd StartInterval 900 */15 * * * *
launchd StartCalendarInterval Hour 6, Minute 30 30 6 * * *
launchd StartCalendarInterval Weekday 1, Hour 9, Minute 0 0 9 * * 1
EventBridge rate(15 minutes) */15 * * * *
EventBridge rate(1 hour) 0 * * * *
EventBridge cron(0 6 * * ? *) (UTC) 0 6 * * * with time zone UTC

The shortest interval is 30 seconds (60 seconds on an instance that runs as Lambda functions). A launchd RunAtLoad has no equivalent: use Run now once after you save.

By default a script fails when it exits with a code other than 0, and the failure message quotes the last lines of its output. That is usually all you need: keep the script’s own error handling, and make sure it exits non-zero when something goes wrong.

For more, the script can return JSON, and an assertion can judge it. Write the JSON to the file named in SREINSIGHTS_OUTPUT_FILE:

import json, os
result = {"refreshed": True, "expiresInDays": 59}
with open(os.environ["SREINSIGHTS_OUTPUT_FILE"], "w") as f:
json.dump(result, f)

The assertion then reads it as data, for example exitCode == 0 && data.expiresInDays > 7. A number in data (or data.value) is recorded as the run’s value, for threshold rules and charts.

Don’t put secrets in the script’s code or in the monitor’s env: both are saved with the monitor, in plain text, for anyone in the organization to read.

A script never sees connection secrets or the runner’s own secrets. It gets a short, fixed environment (PATH, HOME, locale, time zone, temp folder, CA and proxy settings), its own env, and whatever the runner’s operator lets it inherit with SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH. Use that list:

  • On a Lambda runner: pass the function’s role through, and read secrets from AWS Secrets Manager in the script:

    Terminal window
    SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH=AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN,AWS_REGION

    Give the function’s role secretsmanager:GetSecretValue on just the secrets your scripts need. Read the warning about what else that role can reach.

  • On a Docker runner: set the variable on the runner container and list its name:

    Terminal window
    -e INSTAGRAM_APP_SECRET=… -e SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH=INSTAGRAM_APP_SECRET

The list applies to runtime: local scripts only. Names starting SREINSIGHTS_SECRET_, the registration token and the runner’s identity are never passed.

  1. Monitors → New monitor → Check types → Script (under Code).

  2. Name it after what it does: Refresh Instagram token.

  3. Choose the language, keep the runtime local, and write the script. To run code you copied into the image, keep the script short:

    Terminal window
    set -e
    cd /opt/jobs
    python3.12 refresh_token.py

    Write with ADAM can draft or change the script from a description.

  4. Set the timeout: the default is 60 seconds, at most 10 minutes.

  5. Schedule (cron) and Time zone from step 3.

  6. Runner group: your Lambda group or Docker runner group.

  7. Select Create monitor, then Run now on its page. Expand the run to see the exit code, the output and the returned data.

The monitor already raises a high alert called Check failing when a run fails, and resolves it on the next success. Route it to a channel (Send alerts to your team). If one failure is too noisy for a job that runs often, add a rule such as “2 of the last 3 runs failed” (Alert rules).

Watch for these too:

  • skipped/no_runner: no runner took the run. It fires no alert. On a Lambda group, look at the function’s CloudWatch logs.
  • timeout: the script ran past its timeout.

Once a few runs have passed in SREInsights, stop the old one, so the work doesn’t run twice:

Terminal window
# cron: remove the line
crontab -e
# launchd: unload the agent
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.example.refresh-token.plist
# EventBridge: disable the rule
aws events disable-rule --name refresh-token-every-15-min

Keep the old definition somewhere for a while, in case you need to go back.

A Python script refreshes a long-lived API token every day at 03:00 and stores it in AWS Secrets Manager. It used to be a launchd agent on a laptop.

  1. Extend the Lambda runner image with Python and boto3, copy the script to /opt/jobs/refresh_token.py, and push it under a new tag.
  2. On the function, set SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH to the four AWS variables, and give its role read and write access to that one secret.
  3. Create a monitor Refresh API token: script, shell, python3.12 /opt/jobs/refresh_token.py, timeout 120 000 ms, schedule 0 3 * * *, your time zone, runner group jobs.
  4. Run now and check the output.
  5. Route Check failing alerts for it to your chat.
  6. launchctl bootout the old agent.

The laptop can now be off at 03:00.