Persist-Retry DLQ Recovery Runbook

Persist-Retry DLQ Recovery Runbook

Scope: PLT-853 write-cutover Tier-A #3. What to do when the durable park-before-commit path (shipped in #337) sends a parked write to the terminal persist DLQ. Applies once prod is write_target=neptune (sole primary); the same path also backs the write_target=both async shadow today.

Topics (FSD consumer-graph-worker.yml)

Role	Topic	Notes
Retry	consumer-graph-persist-retry	drained by group consumer-graph-persist-retry-consumer
DLQ	consumer-graph-persist-dlq	produce-only / terminal — nothing consumes it

Retry cadence (defaults, cmd/unified-worker/config.go): base backoff 30s, max 30min, 10 cycles. This governs the two DLQ paths differently:

• Exhausted (persist_retry_exhausted) reaches the DLQ only after ~2.5h of sustained failure (Σ of the 10 backoffs) — that alert means a prolonged Neptune outage, never a transient blip.
• Oversized (persist_park_oversized_dlq) is routed to the DLQ immediately on the first park attempt (the >900 KB payload can never be produced), so it can fire at any time from a single pathological receipt.

Which alert fired

Alert (monitoring.yml)	Metric	Meaning
Persist Retry Exhausted (DLQ)	persist_retry_exhausted	a parked write failed all 10 cycles → DLQ. Full payload preserved in the DLQ record.
Persist Park Oversized (DLQ, data dropped)	persist_park_oversized_dlq	a park payload > 900 KB (pathological fan-out) → metadata-only DLQ record; inline data dropped.
Persist Retry Publish Failure	persist_retry_publish_failure	could not even produce a park to the retry/DLQ topic → MSK-produce outage (different fix, see §4).

Recovery — re-backfill the affected users (primary path)

The recovery mechanism is the existing, continuously-exercised /admin/backfill target=neptune endpoint (idempotent). It re-fetches a user’s full graph from source — Purchase History API for receipts AND BTS for shop-order factoids (internal/worker/orchestrator.go) — so it covers whichever write was parked (the durable-park path serves both the receipt and SHOP-factoid processors). We do not re-drive the Kafka DLQ topic by hand: a naive re-produce would carry Attempt=10 and be re-DLQ’d immediately, and the recovery re-fetches from source anyway. Live parked writes are recent, so a re-fetch reconstructs them exactly. (The oversized path’s own log line says “recover via backfill”.)

1. Extract the affected user_ids from Loki

Exhausted (persist_retry_handler.go):

{service_name="consumer-graph-worker", deployment_environment="prod"}
  |= "persist retry: exhausted" |= "sent to DLQ"

→ each line is ... user <USER_ID> op <OP> ....

Oversized (graph_processor.go):

{service_name="consumer-graph-worker", deployment_environment="prod"}
  |= "park payload oversized" |= "inline data dropped"

→ each line is ... user_id=<USER_ID> op=<OP>.

Collect the distinct user_ids over the alert window.

2. Re-backfill those users into Neptune

curl -X POST https://prod-consumer-graph-worker.us-east-1.prod-services.fetchrewards.com/admin/backfill \
  -H "Content-Type: application/json" \
  -d '{"user_ids": ["<id1>", "<id2>", ...], "target": "neptune"}'

FullLoad is idempotent (PURCHASED dedups by receipt id; SHOPS_AT is now receipt-deduped per #336; MERGE upserts) — safe to re-run. Use "target":"both" if run while prod is still write_target=both.

3. Verify

• The user’s writes land: /graph-audit prod neptune, or spot-check the affected receipt id in the user’s PURCHASED.receipt_ids_csv.
• Confirm the DLQ counters stop advancing (no NEW exhausted / oversized log lines). NOTE: these two alerts use the RAW cumulative counter (sum(counter) > 0), so they do not auto-clear when the events stop — they keep firing until the counter series resets (a full fleet redeploy). After recovering the affected users, ack/silence the alert; do not wait for include_resolution to clear it.

4. If the alert was Persist Retry Publish Failure (MSK-produce outage)

persist_retry_publish_failure increments in the shared producer (produceTo) whenever a park/republish can’t be produced to MSK — from three call sites. Whether anything is lost depends on which:

• Live sole-primary park (write_target=neptune) — log neptune primary write failed and park publish failed. The write also returns ErrNeedsRedelivery (offset not committed) → no loss; manifests as a consumer stall + Kafka lag (§9 operational finding), not DLQ growth. Fix MSK reachability; stuck receipts redeliver and land on recovery. No backfill.
• Retry-consumer republish / DLQ-publish — the handler returns an error, so the retry-topic message is not committed → redelivered on the retry topic → no loss. Same fix (MSK reachability); no backfill.
• write_target=both shadow drainer (pre-flip) — log persist shadow park publish failed (genuine loss, both); this ALSO trips the Persist Shadow Park Dropped alert. This shadow write is lost (Neo4j primary unaffected, but Neptune parity drifts). Recover: extract users from Loki |= "persist shadow park publish failed", then re-backfill via §2.

In all cases first fix MSK/broker reachability; only the shadow-drainer case needs a backfill.

Known limits

• >365-day RECEIPT data is NOT recoverable (the PH-API has a 365d cap, purchasehistory/client.go; BTS/shop-order re-fetch is not PH-API-bound). Live parked writes are recent, so this is N/A in practice; flagged only for completeness.
• First-ever write for a brand-new user, dropped: re-backfill still works here because we drive it by user_id from the Loki log line (not by a Neptune-keyed resweep), so there is no discovery hole for this runbook.

Testing this runbook

The recovery mechanism (/admin/backfill target=neptune) is exercised in prod continuously (see graph-backfill skill) — that half is proven. Forcing a natural exhaustion→DLQ to rehearse the full loop takes ~2.5h of sustained failure; to rehearse on stage, temporarily lower PersistRetryMaxCycles + PersistRetryBaseBackoff (config), point NEPTUNE_ENDPOINT at a connection-refused host (https://127.0.0.1:1, per the #337 stage validation), inject a receipt, watch it exhaust to consumer-graph-persist-dlq, then run steps 1–3 and confirm the write lands. Revert the config/endpoint after.