Backfill Runbook (Neptune)

Backfill Runbook (Neptune)

How to backfill the consumer graph’s Neptune store: enumerate the users already in Neptune, re-fetch each user’s purchase + shop-order history from the source-of-truth services, and FullLoad-overwrite it back into Neptune. This is the PLT-853 operation for correcting/completing Neptune data (source: neptune, target: neptune).

Backfilling Neo4j instead? The legacy Neo4j procedure (feature-flag gating, the bin/backfill CLI, category enrichment, the AI-Assistant cohort) lives in BACKFILL_RUNBOOK_NEO4J_LEGACY.md. Neo4j is still the prod primary today.

---

This runbook is skill-driven

The exact commands, live monitoring, and measured baselines live in three Claude Code skills. This doc is the orchestration layer + the Neptune-specific facts; run the skills for the blow-by-blow.

Skill	Role in a Neptune backfill
graph-backfill	Fire the backfill, run the writer-headroom failover-resize (4xl→16xl via reader-1), re-drive the DLQ, fail back + downsize
cgw-monitor	Live prod health during the multi-hour drain — DLQ depth, Neptune writer CPU, shadow-drop metrics, drain rate, fleet
graph-audit	Pre/post health + data-quality audit — run as /graph-audit prod neptune (Tier-1 openCypher, reader endpoint)

Invoke them by name in Claude Code (e.g. /graph-backfill prod, /cgw-monitor, /graph-audit prod neptune).

What a Neptune→Neptune backfill does

POST /admin/backfill {"source":"neptune","target":"neptune", ...}
  → enumerate ALL User node ids from Neptune (paginated cursor; the ~150K cohort
    already in the graph — no feature flag needed, the graph IS the cohort)
  → dedup → SQS main queue at rate_limit/s
  → workers pick up each user:
       → PH-API: ≤365-day receipt history  ─┐
       → BTS:    ≤365-day shop-order data   ├─ parallel, fail-fast
       → merge into one GraphData           ┘
       → FullLoad-overwrite the user's edges into Neptune (purchased_full_load.cypher)

• source: neptune enumerates the user IDs from Neptune itself, so the backfill targets exactly the users already present — no consumer_graph_ingestion flag step (that flag gates live Kafka ingestion, not admin backfill).
• target: neptune writes Neptune only; Neo4j is untouched and the worker’s global write_target stays neo4j. Neptune write failures are fatal per-message → they land in the DLQ (re-drivable). The request is fail-fast: target=neptune on a worker with no Neptune config is rejected with 400 before anything is enqueued.
• lookback_days is capped at 365 (server rejects >365 — the PH-API hard cap). Edges older than 365d cannot be recovered this way; closing full historical parity is a graph-to-graph copy, which is a decommission-time operation, not this backfill.
• FullLoad overwrite makes the write authoritative and idempotent — safe to re-run and safe alongside live dual-write (write_target=both) shadow traffic.

Prerequisites

• TempAdmin for prod-services (SQS + Neptune + ECS + CloudWatch). Request via https://aws-temp-access.fetchrewards.com/. The graph-audit/cgw-monitor skills handle SSO-expiry recovery.
• Worker config has neptune.endpoint set and its IAM role scoped to the cluster’s DbClusterResourceId (consumer-graph-worker.yml → neptune_cluster_resource_id) with connect + Read + Write + DeleteDataViaQuery.
• No Kafka-lag / feature-flag prep is required (that’s the Neo4j-legacy live-gate flow).

When to run

The job has two phases with very different sensitivities: a sender (~1h — enqueues SQS messages at rate_limit/s; 150K ÷ 40 ≈ 62 min) and a drain (~2h — workers process them). Wall time ~3h total.

Prefer off-peak / night, and start after the 03:00 UTC scheduler completes (~04:25 UTC).

• The scheduler runs weekly_forecast (~5 min, done ~03:05 UTC) then repurchase (~1h18m, done ~04:23 UTC). Start the backfill after ~04:25 UTC.
• Both scheduler jobs read Neptune in prod today (repurchase_read_target: neptune, forecast read_target: neptune). Running the backfill concurrently would (a) let the failover-resize’s reader/writer role-shuffle + ~14s blip disrupt those live reads, and (b) contend with repurchase’s ~1h18m of heavy Neptune reads. Sequencing after the scheduler avoids both.
• Off-peak = lower live receipt volume → more writer headroom (the measured ~3h / 16xl ~50–62% profile was an off-peak run).
• The ~04:25 UTC → next-03:00 UTC window is ~22h — ample for the ~3h run plus the Step-5 failback/downsize, which must finish before the next scheduler run so it reads a stable cluster.

Daytime is possible, but a redeploy can kill the run. Any CGW merge auto-deploys prod (rolling ECS task replacement):

• During the sender phase (~first hour): the deploy kills the in-memory, non-resumable sender → partial enqueue. If running in the day, freeze CGW merges for the sender window and don’t run the failover-resize mid-deploy. Recovery: re-fire (FullLoad is idempotent).
• During the drain (sent == total / status: completed): deploy-safe — a replaced worker’s in-flight SQS message redelivers via visibility timeout; no data loss.
• Daytime is also peak live-receipt traffic → less writer headroom.

Procedure

1. Pre-backfill audit — graph-audit

/graph-audit prod neptune

Record the baseline: per-label node/edge counts, PURCHASED data quality (epoch-ms + CSV columns, avg_interval_days, future timestamps), fan-out. Compared against the post-run audit in Step 6.

2. Writer headroom — graph-backfill failover-resize

A full ~150K run needs more Neptune writer headroom than the steady-state 4xl, especially concurrent with live write_target=both. Never resize the lone writer in-place (~15-min reboot = live write outage). Resize the standing reader and fail over — the graph-backfill skill has the exact commands. In short:

# bump reader-1 4xl→16xl (reader-only reboot; 4xl writer keeps serving), then failover
aws neptune modify-db-instance --db-instance-identifier consumer-graph-prod-reader-1 \
  --db-instance-class db.r6g.16xlarge --apply-immediately   # wait ~10-15m to available
aws neptune failover-db-cluster --db-cluster-identifier consumer-graph-prod \
  --target-db-instance-identifier consumer-graph-prod-reader-1
aws ecs update-service --cluster prod-fargate --service prod-consumer-graph-worker --desired-count 10

Writer-identity gotcha: after the failover the 16xl writer is the instance still named consumer-graph-prod-reader-1. Confirm the writer role before reading CPU.

2.5 Parameters to set before firing — and how to dial back

These are the levers that trade throughput against Neptune writer load. Set them before Step 3:

Parameter	Where	Baseline (large run)	Effect
Neptune writer class	failover-resize (Step 2)	db.r6g.16xlarge (steady-state 4xl)	Writer CPU headroom
ECS fleet	aws ecs update-service --cluster prod-fargate --service prod-consumer-graph-worker --desired-count 10	10	Write concurrency — the dominant knob for both drain rate and writer CPU/CME
Autoscaler min=max	aws application-autoscaling register-scalable-target --service-namespace ecs --resource-id service/prod-fargate/prod-consumer-graph-worker --scalable-dimension ecs:service:DesiredCount --min-capacity 10 --max-capacity 10	pin min=max=10 for the run	Stops a scale-IN from killing the non-resumable sender mid-run
rate_limit	backfill request body	40	SQS enqueue rate (the sender), msgs/s, 1–100
lookback_days	backfill request body	365	History window (hard-capped at 365)

Measured baselines (16xl writer, concurrent with live both):

• fleet-8 → ~5.8/s drain, writer ~25–40% CPU
• fleet-10 → ~7/s drain, writer ~50–62% CPU

If the writer CPU climbs (or the DLQ moves off 0):

• Signal — watch the DLQ, not the CME log volume. ConcurrentModificationExceptions (shared-Product MERGE collisions) scale with write concurrency (fleet), not writer CPU — they’re a lock conflict, expected, and self-heal on SQS redelivery while DLQ≈0. The real tipping point is writer CPU sustained >85% even on 16xl, or a DLQ climbing off 0 (messages exhausting SQS maxReceiveCount before contention clears).
• Levers: reduce the fleet (--desired-count, primary) and/or rate_limit. Both cut write concurrency → lower CPU + CME.
• Timing caveat — a scale-IN kills the non-resumable sender. Scaling up is always safe/additive; scaling in terminates the sender if it’s still enqueuing. So:
  • Sender still running (status: running, sent < total) and you must shed load: cancel (POST /admin/backfill/cancel), scale the fleet in, then re-run with a lower rate_limit/fleet. FullLoad is idempotent → re-running is safe.
  • Sender done (sent == total, status: completed): scaling the fleet in is safe — the drain just slows, no data loss (SQS redelivers).
• After the drain (Step 5): always fail back to the 4xl writer, downsize reader-1, and restore the autoscaler to min=3 max=10.

3. Fire the backfill — graph-backfill

curl -X POST https://prod-consumer-graph-worker.us-east-1.prod-services.fetchrewards.com/admin/backfill \
  -H "Content-Type: application/json" \
  -d '{"source":"neptune","target":"neptune","rate_limit":40,"lookback_days":365}'

Returns a backfill-<uuid> job ID. Only one backfill runs at a time.

4. Monitor the drain — cgw-monitor

/cgw-monitor

Watch, over the multi-hour drain:

• DLQ depth — the real tipping-point signal. ConcurrentModificationException log lines (shared-Product MERGE collisions) are expected and self-healing on SQS redelivery; only a growing DLQ means true failures — dial the fleet back if so.
• Neptune writer CPU — 16xl under fleet-10 healthy ~50–62%; sustained >85% = dial back.
• Live shadow drops (dual_shadow_write_dropped) — 0 / NoData = the backfill isn’t starving live dual-write.
• Drain rate = Δ(visible)/Δt; fleet-10 ≈ ~7/s. Fetch-I/O-bound by fleet size, so scale the fleet up (additive) if the writer has headroom.

5. DLQ re-drive + fail back — graph-backfill

Once the main queue drains to 0/0, re-drive the DLQ (CME failures succeed once contention clears — FullLoad is idempotent). Then always fail back to the original writer, downsize reader-1 to 4xl, restore the autoscaler to min=3 max=10, and set the fleet back to 3 (never leave the 16xl running) — commands in the skill’s Step 2c.

6. Post-backfill audit + compare — graph-audit

/graph-audit prod neptune

Compare against the Step 1 baseline (PURCHASED up, quality columns intact, future timestamps still 0). For true Neo4j↔Neptune parity use the migration-validator — raw counts differ by design (Neptune is Tier-1 and drops the #306 phantom non-product edges), so don’t read a raw count delta as a defect.

---

Use Case: 150K Neptune→Neptune backfill

The consumer graph holds ~150K users in Neptune. Re-fetch and FullLoad-overwrite their ≤365-day history to correct/complete Neptune from source-of-truth. This is the canonical run the skills were tuned on.

1. Pre-audit: /graph-audit prod neptune → record baseline.
2. Writer headroom: graph-backfill failover-resize reader-1 → 16xl, fail over, pin fleet to 10 (do this after any write_target=both flip is deployed so the ~14s failover blip lands on the async shadow, not the receipt critical path).
3. Fire: POST /admin/backfill {"source":"neptune","target":"neptune","rate_limit":40}.
4. Monitor: /cgw-monitor on a 3-min cadence through the drain.
5. Drain + DLQ: re-drive the DLQ after the main queue hits 0/0.
6. Fail back + downsize: restore the 4xl writer, downsize reader-1, fleet → 3.
7. Post-audit: /graph-audit prod neptune → compare.

Measured profile (prod, ~155K users, off-peak): ~3h wall time; 16xl writer ~50–62% CPU (fleet-10, concurrent with live both); DLQ ~0.0066%, all redrivable CME. No live shadow drops.

---

Request reference (POST /admin/backfill)

Field	Values	Default	Notes
source	"neptune" | "neo4j" | (or user_ids / s3_uri instead)	—	Exactly one required. neptune enumerates all Neptune User ids
target	"neptune" | "neo4j" | "both" | omitted	worker default (neo4j)	Fail-fast 400 if that backend isn’t configured on the worker
rate_limit	1–100	10	SQS enqueue rate (msgs/s). ~40 for a large run
lookback_days	1–365	365	Hard-capped at 365 (PH-API)

# specific users into Neptune (e.g. after a targeted fix)
curl -X POST <service-url>/admin/backfill \
  -d '{"user_ids":["u1","u2"],"target":"neptune"}'

# status / cancel
curl <service-url>/admin/backfill/status
curl -X POST <service-url>/admin/backfill/cancel

---

See also

• graph-backfill skill — run + failover-resize + DLQ re-drive
• cgw-monitor skill — live prod health during the drain
• graph-audit skill — pre/post Neptune audit (prod neptune)
• BACKFILL_RUNBOOK_NEO4J_LEGACY.md — the legacy Neo4j-target procedure
• Backfill Strategy — design details, dedup strategies
• consumer-graph-capacity-experiments/docs/neptune-migration-process.md — signed-curl fallback + IAM re-grant