ADR-0002: Generalize pipeline stage 2 from a fixed CCS enricher to a ref-resolved Enrich stage

ADR-0002: Generalize pipeline stage 2 from a fixed CCS enricher to a ref-resolved Enrich stage

• Status: Proposed (2026-06-17) — open questions resolved 2026-06-17 (see below); ready to accept
• Deciders: Ali Siddiqi (DM stack)
• Affects: PD-1 (PLT-685), PD-3 (PLT-751 — registration schema), Earnings / Post-Scan Coach (the motivating type)
• Relates to: ADR-0001 (the dispatch rearchitecture), ../dm-execution-e2e.md §3.3

Context

PD-1’s pipeline (internal/dm/executor/pipeline.go) is seven stages. Four of them — candidate (1), eligibility (3), ranking (4), copy (5) — are generic adapters that resolve a per-type ref out of the PD-3 registration through the symbol table (internal/dm/registry/callable.go): the adapter is shared, the behavior is the type’s registered Go function. Adding a DM type means registering its functions and pointing its YAML at them; no pipeline change.

Stage 2 (enrichment) is the odd one out. It is welded to a single concrete service:

// cmd/unified-worker/dm_executor_component.go (composition root)
Enricher: ccs.NewEnricher(ctxClient, ccs.Options{}),

internal/dm/executor/deps.go

type Enricher interface {
    Enrich(ctx context.Context, candidates []any) (enriched []any, err error)
}

Every DM type, regardless of what external context its copy actually needs, gets the Consumer-Context-Service product/offer enrichment — and only that. There is no per-type hook on this stage the way there is on 1/3/4/5.

Motivating case: Earnings / Post-Scan Coach

The Coach needs to call a Points Optimization Engine (POE) endpoint to derive the advice its copy renders — “given this user’s just-scanned receipt, what’s the points-optimal next action?” That is, semantically, enrichment: fetch external context for the candidate set so the copy stage has something to render. But there is no clean place to put it:

• It is not candidate selection (stage 1) — the candidate already exists (the receipt / the user); we’re decorating it.
• It is not eligibility (stage 3) — it doesn’t filter; it adds data.
• It is not copy (stage 5) — copy renders a payload from already-assembled data; making it do a network call to the POE conflates “fetch context” with “format context.”

It is enrichment. The only reason it doesn’t fit is that stage 2 is hardcoded to CCS rather than being ref-resolved like its sibling stages.

Workaround considered (and why it’s not enough)

The Coach could call the POE inside its candidate extractor / candidate-query callable (stage 1) — that callable is the type’s own Go code and can do anything. This ships the Coach with zero platform change. But it is a workaround, not the right seam:

• it conflates “select candidates” with “fetch external context,” the exact muddle the staged pipeline exists to prevent;
• it sidesteps stage 2’s batching contract (CCS enrichment is batched; an ad-hoc POE call in stage 1 has no shared batching/concurrency posture);
• it makes “this type needs an external engine to inform its content” an invisible implementation detail of one callable instead of a declared, reviewable capability.

If a second type later needs the same shape, the workaround calcifies into a pattern we didn’t design.

Decision

Generalize stage 2 from a fixed CCS enricher to a ref-resolved Enrich stage, mirroring the candidate/eligibility/ranking pattern exactly. CCS stops being privileged: it becomes one registered enricher among potentially several. A DM type declares which enricher(s) it wants in its YAML; the pipeline resolves the ref(s) through the symbol table at startup (fail-fast under WithCallableResolution()).

This is deliberately the same mechanism stages 1/3/4/5 already use — not a new concept. Stage 2 simply joins them.

PD-3 schema (additive, registry types.go)

A new optional enrich block on DMTypeRegistration. Optional + ordered list so the common case stays a one-liner, a type can opt out of CCS entirely, and a type can chain enrichers when it genuinely needs more than one source.

The Earnings Coach at launch needs only the POE, not CCS — so it lists a single enricher:

enrich:
  - ref: "earningscoach.enrich:points_optimization"   # POE-backed; the Coach's only enricher
    parameters: { timeout_ms: 800 }

A future type that needs both an external engine and CCS product data declares the chain; enrichers run in declared order and each sees the prior’s output (§ sequencing, decided):

enrich:
  - ref: "somefuture.enrich:external_signal"
  - ref: "ccs.enrich:products"

// registry/types.go — reuse the existing Callable shape (ref + params), like ranking
type DMTypeRegistration struct {
    // ...
    Enrich []Callable `yaml:"enrich,omitempty" json:"enrich,omitempty"`
    // ...
}

Backward-compatible default (decided: implicit): a registration with no enrich block gets the default CCS enricher, so every shipped type (Weekly Forecast) keeps its exact current behavior with no YAML edit. A type that wants no CCS — like the Earnings Coach, which uses only the POE — lists its own enricher explicitly (enrich: [earningscoach.enrich:points_optimization]), and the presence of an enrich block replaces the default entirely (it is not appended to). So: empty block → CCS; non-empty block → exactly what’s listed, in order.

Symbol table (registry callable.go)

Add an EnricherFunc category, alongside the existing four:

// signature mirrors the Enricher interface — batch in, batch out
type EnricherFunc func(ctx context.Context, candidates []any, params map[string]any) ([]any, error)

func RegisterEnricher(ref string, fn EnricherFunc) { /* same dup/grammar guards */ }
func LookupEnricher(ref string) (EnricherFunc, bool)

Executor (new EnrichAdapter, parallels runner.CandidateAdapter)

A generic adapter that resolves the registration’s enrich refs and runs them in order, threading the candidate set through each. It implements the existing executor.Enricher interface, so pipeline.go does not change — only what gets injected does:

// internal/dm/runner (or internal/dm/enrich)
type EnrichAdapter struct{ /* nothing per-instance; resolves per fire from dm.Enrich */ }

func (a *EnrichAdapter) Enrich(ctx, candidates) ([]any, error) {
    // resolve dm.Enrich refs via registry.LookupEnricher; run in declared order;
    // fall back to the default CCS enricher when dm.Enrich is empty.
}

Note the Enrich interface today takes only (ctx, candidates) and has no dm argument. Two clean options:

1. add dm registry.DMTypeRegistration to the Enrich signature (matches every other stage adapter, which all receive dm), or
2. have the pipeline pass the resolved enricher chain in per fire. Recommendation: (1) — make Enrich(ctx, dm, candidates) consistent with Candidates(ctx, dm, fe) / Filter(ctx, dm, enriched) / Rank(ctx, dm, kept).

CCS becomes a registered enricher

internal/dm/ccs keeps its batched implementation; it just also exposes a registration:

// registered at worker startup (needs the live ctxclient.Client), same pattern as the
// Weekly Forecast candidate query bind in PR-F:
enrich.RegisterCCS(deps.ContextServiceClient)   // RegisterEnricher("ccs.enrich:products", ...)

Consequences

Positive

• “This type needs an external engine to inform its content” becomes a declared, reviewable capability in YAML, not a buried side effect of a candidate callable.
• The Coach’s POE call lands in the semantically correct stage with the right batching seam.
• Stage 2 becomes consistent with 1/3/4/5 — one fewer special case in the pipeline.
• Enricher chaining (N ordered sources) is expressible without bespoke code, for the future types that need it; a single-enricher type (the Coach) stays a one-liner.

Negative / costs

• PD-3 schema change (additive) → regenerate dm/_schema.json; the Python CI lint + LSP pick it up.
• One signature change (Enrich gains dm) and a new symbol-table category — small, but touches the executor + the composition root + the CCS package.
• An enricher that does I/O per fire (POE) needs the same care CCS already takes: batching, a per-call timeout, and degrade-open semantics (a single-candidate enrichment error drops that candidate; a batch error fails the fire) — the existing Enricher contract in deps.go already specifies this, so the contract is reused, not invented.

Neutral

• No behavior change for Weekly Forecast (implicit CCS default).
• Independent of ADR-0001; touches stage 2, not the dispatch rearchitecture.

Resolved decisions (2026-06-17)

1. CCS default — DECIDED: implicit. No enrich block → the default CCS enricher runs, so Weekly Forecast needs no YAML edit. A non-empty enrich block replaces the default with exactly what’s listed (it is not appended to), so a type can opt out of CCS entirely — as the Earnings Coach does, running only the POE.
2. Enricher failure policy — DECIDED: every enricher is required (fail-closed); no opt-out knob. A batch error from any enricher fails the fire (failed, reason=“enrich”); a single-candidate error drops that candidate — exactly the existing Enricher contract in deps.go, applied uniformly to each enricher in the chain. There is no per-ref required flag: enrichers are always required. (The single-enricher case — e.g. the Coach at launch — needs no policy at all; this rule only governs chains, and it governs them the same way the single-enricher contract already works.) If a genuine soft-signal/optional enricher use case ever appears, adding an opt-out is a future additive change — not built speculatively now.
3. Sequencing — DECIDED: ordered, each sees the prior’s output. Enrichers run in the order declared in the enrich list; each receives the candidate set as enriched by the previous one. This is what makes multi-enricher composition (e.g. an external-signal enricher feeding CCS) well-defined, and it is the contract the EnrichAdapter implements.

Implementation sketch (when accepted)

1. registry/types.go: add Enrich []Callable; regenerate _schema.json.
2. registry/callable.go: add EnricherFunc + RegisterEnricher + LookupEnricher; add enrich to resolveCallables so refs fail-fast at load.
3. executor/deps.go: change Enricher.Enrich to (ctx, dm, candidates).
4. New EnrichAdapter resolving dm.Enrich in order (empty → default CCS); every enricher required — any batch error fails the fire.
5. internal/dm/ccs: expose RegisterCCS(client) registering ccs.enrich:products.
6. Composition root: register CCS + the Coach’s POE enricher at startup; inject EnrichAdapter instead of the bare CCS enricher.
7. Earnings Coach YAML: enrich: [earningscoach.enrich:points_optimization] (POE only, no CCS).
8. Tests: enrich-chain ordering (each sees prior output), empty-block default-to-CCS, and any-enricher batch-error → fire fails.