Coach Precompute Scheduler Runbook

How the three Coach PLT-854 precompute jobs — a2subs, item-cf, and poe-eval — are scheduled, enabled, tested, and rolled back.

The one thing to know: these are NOT standalone FSD scheduled tasks anymore. They run on internal crons inside the always-on consumer-graph-worker (the PrecomputeScheduler component). There is no item-cf-runner.yml / coach-a2-substitutes-publisher.yml / coach-poe-eval.yml to deploy — you deploy consumer-graph-worker.yml and flip a flag.

Why (history)

They were originally written as separate FSD ECS scheduled tasks (when: schedule + cron(...)). But FSD has no scheduled-ECS-task construct — it renders that manifest into an invalid Terraform local-exec provisioner, so those tasks could never actually deploy. Rather than invent a scheduled-task mechanism, we run the jobs the way CGW already runs its other cron work (the notification scheduler, the DM CronLoop): as an in-process component in the long-running worker fleet.

The three jobs

Job	Cadence (UTC)	Warehouse	What it does	Valkey output
`a2subs`	every 6h — `0 /6 * *`	`PROD_XL`	Snowflake → Valkey A2 brand-swap candidates	`a2subs:*` (+ `a2subs:version` pointer)
`item-cf`	daily 07:00 — `0 7 * * *`	`PROD_X4`	Snowflake → Valkey item-CF neighbors	`similar_to:*` (+ `similar_to:version`)
`poe-eval`	(disabled)	`PROD_XL`	G5 POE-accuracy eval (report-only)	none (POSTs to the worker’s own `/v1/coach/handle`)

Per-job warehouse. item-CF runs on PROD_X4 (its stage-3 pair-cooccurrence self-join is heavy — on the shared PROD_XL it ground ~40 min and stalled; on PROD_X4 it finishes ~6.5 min). a2subs + poe-eval stay on the right-sized shared PROD_XL. Set via itemcf_warehouse (empty ⇒ the shared snowflake_warehouse).
Fleet-safe: every worker replica ticks each minute, but a per-(job, fire-minute) Valkey lock (precompute:lock:<job>:<yyyy-MM-ddTHH:mm>, SETNX + TTL, held to expiry, not released) guarantees exactly one replica runs each fire.
a2subs batches its Valkey write (500 keys/Exec) — a single unbounded write of ~185k keys i/o-times-out on the network; item-CF already batched.
Panic-isolated: a job panic is recovered — it can never crash the worker.
PROD-ONLY + dark by default. poe-eval is left cron-disabled (empty expr) until a coach-enabled cohort + CCS/category endpoints are configured.

Stage-validated in-worker 2026-07-23: item-CF → PROD_X4 349,896 items (~6.5 min), a2subs → PROD_XL 186,816 keys (batched), both in parallel with no contention (separate warehouses).

Config knobs

config/unified-worker-{env}.yaml → precompute_scheduler: (and PRECOMPUTE_* env overrides). Prod values ship in config/unified-worker-prod.yaml (dark). Valkey targets are reused from coach.itemcf_valkey_endpoint / coach.a2_valkey_endpoint so writer and reader always agree.

Key / env	Meaning
`enabled` / `PRECOMPUTE_SCHEDULER_ENABLED`	master gate (default `false`)
`a2_cron` / `itemcf_cron` / `poe_eval_cron`	per-job cron exprs (empty ⇒ that job off)
`snowflake_` / `PRECOMPUTE_SNOWFLAKE_`	account/user/role/warehouse/db/schema (shared default; a2subs + poe-eval)
`itemcf_warehouse` / `PRECOMPUTE_ITEMCF_WAREHOUSE`	item-CF-only warehouse override (prod: `PROD_X4`; empty ⇒ shared)
`snowflake_secret_arn` / `PRECOMPUTE_SNOWFLAKE_SECRET_ARN`	DPS key-pair secret ARN (prod; set from the FSD dep in `consumer-graph-worker.yml`)
`snowflake_password` / `PRECOMPUTE_SNOWFLAKE_PASSWORD`	password auth — stage testing only (SVC_PREPROCESSING); used when the secret ARN is empty
`itemcf_source_table` / `itemcf_sql_dir` / `a2_sql_file`	job inputs (SQL ships in the worker image)
`poe_endpoint` / `poe_user_ids` / `ccs_endpoint` / `category_endpoint`	poe-eval only

Auth precedence (internal/coach/sfauth): a non-empty secret ARN ⇒ key-pair; else password; neither ⇒ every job is skipped (a warning is logged, the worker idles — no crash).

Enable in PROD

The DPS secret + all prod config already ship (dark). To turn it on:

Set precompute_scheduler.enabled: true in config/unified-worker-prod.yaml (or PRECOMPUTE_SCHEDULER_ENABLED: "true" in consumer-graph-worker.yml, prod).
Deploy consumer-graph-worker:
- merging to main auto-deploys stage + prod (the Release Pipeline), or
- Actions → Deploy FSD → Run workflow with deployment-yaml=consumer-graph-worker.yml, environment=prod, a green-main image-tag.
Watch the deploy, then verify (below).

Enabling is a normal worker deploy — no separate scheduled-task deploy, no ephemeris registration, no prod-only guard. Coach serving stays gated per-user by IsCoachEnabled regardless.

Test in STAGE

Full Snowflake→Valkey E2E runs in stage against the DPS key-pair — the stage worker task role can read the prod-dataops secret (PRECOMPUTE_SNOWFLAKE_SECRET_ARN resolves to the prod ARN; validated 2026-07-23 — note a fresh task may briefly get AccessDenied right at startup before the IAM grant propagates, then succeed). So no SVC_PREPROCESSING password wiring is needed (though it’s supported as a fallback — see snowflake_password above).

Steps (add a temporary precompute_scheduler block to config/unified-worker-stage.yaml on a throwaway branch — do NOT merge it):

enabled: true; single-fire crons — a specific minute a few min after the expected boot, e.g. a2_cron: "38 23 * * *" + itemcf_cron: "38 23 * * *". Do NOT use a repeating */N cron shorter than a job’s runtime — a slow job piles up overlapping runs on the next tick.
itemcf_warehouse: "PROD_X4" (item-CF) + snowflake_warehouse: "PROD_XL" (a2subs). Different warehouses is what lets them run concurrently — both on PROD_XL queues/stalls (the pair- cooccurrence self-join contends). Set the rest of the snowflake_* connection (account/user/role/ db/schema) + itemcf_source_table + the SQL paths; leave snowflake_secret_arn: "" (the manifest env injects the DPS ARN). Valkey targets resolve to the stage experiment cache via coach.{itemcf,a2}_valkey_endpoint.
Deploy via Deploy to Stage (workflow_dispatch on your branch — builds the branch image). Confirm boot: precompute scheduler: started jobs=[...]. At the fire minute both jobs run in parallel; verify below. Restore stage to main (dark) + delete the throwaway branch after.

Scheduler-machinery-only check (no real load): same as above but leave the snowflake_* empty — the jobs fire, take the lock, and fail fast at Snowflake connect. Proves tick/lock/wiring without the ~6-min loads.

Note: the standalone binaries still exist (/usr/local/bin/{item_cf_loader,coach_a2_substitutes_loader,coach-poe-eval}) and remain the tool for ad-hoc manual stage seeding (the item-CF run recipe). The in-worker scheduler calls the same extracted cores (internal/coach/{itemcf,a2subs,poeeval}), so a passing standalone run already validates the load logic; the above validates the scheduler.

Verify a run landed

Logs (Loki): precompute scheduler: job complete job=<a2subs|item-cf>.
Valkey pointer advanced (the authoritative signal) — against the target cache:
- a2subs: GET a2subs:version → a fresh vYYYYMMDDThhmmssZ; a2subs:* keys present.
- item-cf: GET similar_to:version → fresh version; ~340K similar_to:* keys.
Exactly-once: only one replica logs job complete per fire; the others log another replica holds the lock — skipping.

Disable / roll back

Flip precompute_scheduler.enabled: false (or PRECOMPUTE_SCHEDULER_ENABLED=false) and redeploy the worker. Serving fails open — a2subs/item-cf just go stale, poe-eval is report-only. No FSD service to tear down.

Source of truth

Component: cmd/unified-worker/precompute_scheduler_component.go; cores: internal/coach/{itemcf,a2subs,poeeval,sfauth}.
Config: PrecomputeConfig in cmd/unified-worker/config.go; precompute_scheduler: in config/unified-worker-*.yaml; the DPS secret dep + PRECOMPUTE_SNOWFLAKE_SECRET_ARN in consumer-graph-worker.yml.
Precompute inventory / status tracker: doc/consumer-graph-worker/coach-valkey-precompute-inventory.md (external to this repo).