ADR-0001: Drop the PD-1 → PD-2 SQS handoff; PD-1 posts the notification gateway directly

ADR-0001: Drop the PD-1 → PD-2 SQS handoff; PD-1 posts the notification gateway directly

• Status: Accepted (2026-06-16)
• Deciders: Ali Siddiqi (DM stack), with input from Steve Hollinger (Growth / notification-service)
• Affects: PD-1 (PLT-685), PD-2 (PLT-686), consumer-graph-worker.yml FSD infra
• Supersedes: PD-1 §4.5 / §4 stage 7 (SQS handoff), PD-2 §3–§4.6 (SQS consumer + dispatch-time re-validation)

Context

The original DM-stack design put an SQS queue ({env}-dm-delivery-queue) between PD-1 (the executor that fires DMs) and PD-2 (the delivery layer that calls notification-service). PD-1 published a message per fire; PD-2 consumed it, re-validated eligibility/flags at dispatch, translated send_time, and POSTed /v1/gateway/notify.

Steve questioned whether the queue is necessary, given the notification gateway already provides intake, dedup, quota, quiet-hours, and scheduling. We investigated the gateway’s real behavior (source: notification-service repo + docs) and the queue’s actual justifications. Several of the original rationales did not survive scrutiny.

What we verified about the notification gateway

• No intake queue in front of POST /v1/gateway/notify. Each request runs the 12 guardrails inline and writes to Valkey synchronously before returning 202. The HIGH/NORMAL SQS queues are downstream (dispatcher → sender), not an intake buffer.
• The gateway owns future scheduling. A request with scheduled_time is written to a Valkey sorted set and released by the dispatcher at the right instant — arbitrary future times, not bound by SQS’s 15-minute delay ceiling. Holding a deferred send is the gateway’s job, never SQS’s.
• The gateway owns dedup, quota, quiet hours. 24h idempotency on request_id; 100/user/day quota + score-based collision; quiet-hours reschedule. PD-2 was already specified NOT to re-implement these.
• Throughput is not the constraint. Our peak offered load is ~40–400 RPS (gated by the consumer worker pool), far under the gateway’s ~10k msg/s/instance and 3–10 ECS tasks. The gateway absorbs our bursts comfortably.
• The real shared-infra risk is Valkey, not notify volume. notification-service PR #148 (2026-04-02) drove prod Valkey to 100% engine CPU — root cause was device-registration cache-invalidation churn (3–5 DEL/registration), not gateway notify traffic. The fix toggled the device cache off for the worker/sender; the gateway hot path was untouched. So Valkey can be saturated, but our notify load is not what saturates it.

Corrections to earlier reasoning (recorded so we don’t relitigate)

Three arguments originally offered for keeping the queue were wrong and are withdrawn:

1. “The gateway can’t absorb bursts, so we need a queue.” — False. It absorbs our RPS.
2. “The gateway docs tell callers to keep an upstream queue and retry.” — No such text exists. The only “queue for retry” guidance in the docs is example client code for the 429 quota case, not a gateway directive.
3. “Removing the queue means rebuilding a deferral buffer for scheduled sends.” — False. The gateway’s Valkey scheduler is the deferral buffer; you pass scheduled_time.

The deciding question: does PD-2 need to re-validate at dispatch?

Dispatch-time re-validation is the only thing that needed a buffer after the dwell. Its value depends entirely on the fire→dispatch gap, which splits by send_time:

	immediate (Earnings Coach)	scheduled (Weekly Forecast)
fire→dispatch dwell	seconds	up to ~24h (per-user tz stagger)
can flag/eligibility go stale?	effectively no	yes

Decision input (Ali, 2026-06-16): the kill-switch and cohort flag are evaluated at fire time, not dispatch time. With that, PD-2’s three re-validation jobs all fall away:

1. cohort-flag re-check → gone (fire-time only),
2. kill-switch re-check → gone (fire-time only),
3. eligibility re-run → already a near-no-op (PD-2 §8.3 punts on re-enrichment, so it only re-checked a possibly-stale payload) → not worth a dispatch-time stage on its own.

With no work that must happen after the dwell, the deferred hold is delegated to the gateway’s scheduled_time, and there is nothing left for an SQS buffer to do.

Decision

Remove {env}-dm-delivery-queue and its DLQ. PD-1’s pipeline posts /v1/gateway/notify directly (via the existing github.com/fetch-rewards/notification-service/pkg/client SDK already used by the live handlers), passing scheduled_time for scheduled/window types so the gateway owns the hold. The gateway continues to own dedup, quota, quiet hours, and scheduling.

PD-2 is no longer an SQS consumer. Its remaining responsibilities (the NotifyRequest field mapping, send_time translation, the per-type weekly cap, the producer="pd2" audit write) fold into a thin gateway-client + audit step invoked inline at the end of PD-1’s pipeline. The PD-1/PD-2 split becomes a code-organization boundary, not a process / queue boundary.

Consequences

Positive

• One fewer piece of infra (queue + DLQ) and one fewer network hop.
• No producer/consumer dual-process; simpler ops + audit (one writer per fire).
• Scheduling/dedup/quota live in exactly one place (the gateway) — no risk of two sources of truth.

Negative / tradeoffs (accepted)

• Retry durability. SQS gave automatic visibility-timeout retries on transient gateway 5xx. Direct POST means retry is in-process (bounded retry-with-backoff). If a pod dies mid-retry, that fire is not auto-redelivered. Mitigation: the synchronous ScheduleEvent audit row already records failed with the stage, so a lost fire is visible and re-fireable; and the next cron tick is the natural retry for scheduled types. Acceptable because DMs are not transactional/critical sends.
• Burst concurrency moves into PD-1. The fan-out (≤50k fires/tick) must be throttled by PD-1’s worker pool + a bounded-concurrency gateway client, rather than by SQS consumer count. This is a client-side concurrency limit, not new infra.
• Per-type weekly cap now runs inline (Valkey Lua check, as PD-2 §4.5 specified) rather than in a separate consumer — same logic, different host.

Implementation

See the implementation plan in the PR that lands this ADR. Summary:

• internal/dm/executor: replace the Handoff interface (SQS publish) with a Dispatcher interface (gateway POST); rename stage 7 from “SQS handoff” to “dispatch.”
• New internal/dm/dispatch/: gateway-client adapter over the notification-service SDK, doing the §4.1 field mapping + send_time translation + per-type cap + inline retry, then the producer="pd2" audit UpdateItem.
• Delete the Handoff SQS adapter work (never built) and the {env}-dm-delivery-queue + DLQ FSD dependency from consumer-graph-worker.yml.
• Spec edits to PD-1 + PD-2 to match (this PR).