Opik Integration Guide

Opik Integration Guide

Opik integration provides prompt management, versioning, and evaluation capabilities for the Consumer Agent.

Quick Start

1. Setup Credentials

Local Development (.env):

OPIK_API_KEY=your-api-key

Production Deployment:

• OPIK_API_KEY stored in AWS Secrets Manager as consumer-agent/opik-api-key
• Automatically injected via consumer-agent.yml (FSD deployment)

Get credentials from Comet.com > Opik section.

2. Enable Opik

Set opik.enabled: true in settings.yaml. Environment-specific projects (consumer-agent-dev/stage/prod) and fixed workspace (“consumer-agent”) are configured there.

Architecture

Two Evaluation Modes:

Aspect	CLI (Offline)	Online (Production)
Execution	Developer machine	Opik cloud
Trigger	Manual command	Automatic (every trace)
LLM-as-Judge	Local GEval → OpenAI	Opik → OpenAI
Cost	Developer pays	Opik pricing
Dataset	Curated test cases	Real user traffic
Use For	Testing prompts, A/B tests	Production monitoring, alerts

Components:

• PromptEvaluator (src/consumer_agent/evaluation/evaluator.py) - CLI evaluation orchestration
• OpikClient (src/consumer_agent/evaluation/client.py) - Opik SDK wrapper for dataset/prompt CRUD
• Metrics (src/consumer_agent/metrics/) - 7 metrics (4 LLM-as-judge + 3 rule-based)
• Automation Rules (scripts/register_llm_judge_rules.py) - Online evaluation setup

Chat CLI Tracking

The chat CLI interface (consumer-agent chat) includes Opik tracking for local testing and development interactions.

Configuration

Chat CLI uses the same environment-specific project configuration as the rest of the application. See Quick Start → Enable Opik for the full settings.yaml configuration. Local development (default environment) uses the consumer-agent-dev project.

What is Tracked

Each conversation turn is logged as a trace with:

Input: User message Output: Assistant response Metadata:

• locale: User locale (en, es-419)
• user_id: User ID if provided
• latitude/longitude: Location if provided
• enabled_components: Active component types
• reasoning_length: Length of reasoning text
• response_length: Length of response text

Graceful Degradation

If Opik Cloud is unavailable or credentials are missing, the chat CLI continues working without tracking. No errors are shown to the user.

View Chat Traces

1. Go to Opik Dashboard
2. Select workspace: consumer-agent
3. Select project: consumer-agent-dev (for local development)
4. View all local chat interactions with metadata
5. Filter by locale, user_id, or components, or tags: [“chat-cli”, “local-testing”]

Implementation

The tracking implementation is in src/consumer_agent/cli/chat.py:273-402:

# Initialize Opik client if enabled
opik_client = None
trace = None
if settings.opik.enabled and settings.opik.api_key:
    opik_client = opik.Opik()

# Wrap agent streaming in trace
if opik_client:
    trace = opik_client.trace(
        name="chat_interaction",
        input={"message": user_message},
        project_name=settings.opik.chat_project_name,
        metadata={...}
    )

Use Cases

• Local Testing: Track development conversations for debugging
• Integration Testing: Verify prompt changes work as expected
• Component Testing: Test specific component behaviors
• Locale Testing: Verify translations and locale handling
• Tool Testing: Monitor MCP tool calls and responses

Comparison with Production Tracking

Aspect	Chat CLI	FastAPI Service
Project	consumer-agent-chat-cli	consumer-agent
Purpose	Local testing	Production monitoring
Tracking	Enabled (OpikTracer)	Enabled (OpikTracer)
Evaluation	Manual review	Automated rules (LLM-as-judge)
Users	Developers	Real users
thread_id	UUID per session	episode_id (multi-turn grouping)

Production Deployment

FSD Configuration: OPIK_API_KEY secret configured in consumer-agent.yml (AWS Secrets Manager path: consumer-agent/opik-api-key).

Environment-Specific Projects:

Environment	Project Name
default (local)	consumer-agent-dev
stage	consumer-agent-stage
prod	consumer-agent-prod

Workspace: Fixed as “consumer-agent” in settings.yaml across all environments.

Custom Metrics

Consumer-agent implements 7 custom metrics in src/consumer_agent/metrics/:

Metric	Type	Scale	Purpose	When to Use
ResponseQualityMetric	LLM-as-judge	1-10	Overall response quality	All evaluations
GreetingQualityMetric	LLM-as-judge	0-1	Landing page greeting validation	Landing page only
CapabilityAlignmentMetric	LLM-as-judge	0-1	Suggestions align with bot capabilities	Suggestion validation
ConversationFlowMetric	LLM-as-judge	0-1	Multi-turn conversation coherence	Multi-turn conversations
SuggestionValidationMetric	Rule-based	0-1	Exactly 5 suggestions, 50-75 chars each	Landing page
DiversityMetric	Rule-based	0-1	No duplicates or similar suggestions	Landing page
ToolUsageMetric	Rule-based	0 or 1	Binary check if tools were called	Items requiring tools

LLM-as-judge metrics use GEval with gpt-5-mini (4 API calls per evaluation). Rule-based metrics use deterministic logic (no API cost).

Usage: consumer-agent opik eval run --dataset test-v1 --metrics response_quality,greeting_quality

For detailed implementations, see src/consumer_agent/metrics/ directory.

Prompt Management

Environment-Driven Source Selection

Prompts are loaded based on environment configuration in settings.yaml:

Environment	Source	Behavior
default (local)	file	Load from prompts/ directory
stage	opik	Load from Opik (strict mode - fail if unavailable)
prod	opik	Load from Opik (strict mode - fail if unavailable)

Prompt Definitions

All prompts are defined in agent_config.yaml under prompts.definitions:

prompts:
  file:
    directory: "prompts"
    components_directory: "prompts/components"

  definitions:
    conversational:
      file: conversational.txt
      opik: conversational@latest
    prompt-suggestions:
      file: prompt-suggestions.txt
      opik: prompt-suggestions@latest

Using PromptManager

from consumer_agent.prompts import create_prompt_manager

pm = create_prompt_manager()

# Load prompt (source determined by environment)
prompt = pm.load_prompt("conversational")

# Build complete system prompt for an agent
system_prompt = pm.build_system_prompt(
    agent_id="conversational",
    enabled_components=["offer-list"],
    locale="en",
    latitude=41.8781,
    longitude=-87.6298,
)

Versioning with Opik

Each prompt definition includes an opik reference with explicit version:

• conversational@latest - resolves via get_prompt() in local (default) env
• conversational@abc123 - pin to specific commit (local dev only)

In stage/prod the @version portion is ignored; the loader resolves the version by env tag instead (see “Environment-Based Promotion” below).

Environment-Based Promotion (PLT-311)

Prompts in stage and prod are selected via version-level tags, not the @version reference. The same Opik workspace (consumer-agent) is shared across environments; tags decide which version each env loads.

ENV_FOR_DYNACONF	How OpikSource resolves a prompt
default	get_prompt(name) — @latest or pinned commit (file source preferred)
stage	search_prompts('name = "<NAME>" and tags contains "stage"')
prod	search_prompts('name = "<NAME>" and tags contains "prod"')

Dual FAIL-HARD invariants (zero ambiguity, no silent stale prompts):

• Zero matches — if tags contains "<env>" returns no version for a prompt name referenced by agent_config.yaml, the loader raises PromptNotFoundError. Pods crash on first request rather than serving an unpredictable fallback.
• Multiple matches — if two versions of the same prompt are tagged with the same env, the loader raises. The invariant is exactly one version per (prompt, env). Move the tag from the old version to the new one rather than tagging both.

Cache behavior — OpikSource._cache is first-call-wins, no TTL. A tag move in the Opik UI does not propagate to running pods; the next pod restart (deploy) is required. This matches the project’s feature-flag caching pattern.

Pre-Tag CLI

Before merging PLT-311 (or any later prompt change), every config-referenced prompt’s intended version must be tagged for the target env, or the FAIL-HARD code crashes pods on the next deploy.

# Dry-run first to verify the plan
ENV_FOR_DYNACONF=stage consumer-agent opik prompt pre-tag --env stage --dry-run

# Tag every config-referenced prompt's @latest with the env label
ENV_FOR_DYNACONF=stage consumer-agent opik prompt pre-tag --env stage
ENV_FOR_DYNACONF=stage consumer-agent opik prompt pre-tag --env prod

# Tag only one prompt (e.g. after a steady-state edit)
ENV_FOR_DYNACONF=stage consumer-agent opik prompt pre-tag --env stage --prompt conversational

The CLI:

1. Walks agent_config.prompts.definitions to enumerate Opik names (main prompts + components).
2. Looks up each prompt’s most-recent version id via OpikClient.get_latest_prompt_version_id(name).
3. Applies the env tag additively (merge_tags=True) so an existing prod tag is preserved when adding stage and vice versa.
4. Verifies via count_prompts_tagged(env) that exactly N prompts are tagged; mismatch is reported as a warning before exit.

Opik credentials are read from settings.opik — pass ENV_FOR_DYNACONF to select the right project/workspace context.

Workflow A: Shipping PLT-311 (one-time)

1. Publish any prompt missing from Opik (e.g. locale-instructions): consumer-agent opik prompt create -n locale-instructions -f prompts/components/locale-instructions.yml.
2. Pre-tag every config-referenced prompt with both prod and stage (CLI commands above).
3. Verify: consumer-agent opik prompt list plus client.search_prompts('tags contains "prod"') returns the expected count (currently 11).
4. Merge the PR.
5. Deploy stage: gh workflow run stage-deploy-fsd.yaml --ref main. ECS rolling-replaces the task; the new task loads each prompt via the env-tag filter and caches first-call-wins.
6. Verify on stage: drive a few stream requests, check Loki for "Loaded ... from Opik (env-tag: stage)" lines.
7. Deploy prod: gh workflow run prod-deploy-fsd.yaml --ref main.

Workflow B: Steady-State Prompt Change

For every future prompt edit (after PLT-311 is live):

1. Edit prompts/<name>.txt (or the component YAML), open PR.
2. Merge to main. Opik still has the OLD version only; running pods continue to serve it.
3. Publish the new version to Opik: consumer-agent opik prompt create -n <name> -f prompts/<name>.txt. Creates commit B with no tags; commit A still has [stage, prod].
4. Move the stage tag from A to B via the Opik UI (or a future consumer-agent opik prompt promote wrapper — not in scope for PLT-311). Running stage pods still serve A from cache; no outage.
5. Trigger a stage deploy: gh workflow run stage-deploy-fsd.yaml --ref main. New tasks pick up B; old tasks drain.
6. Verify on stage (eval, replay queries, etc.).
7. Repeat steps 4-6 for the prod tag.

Update Prompts (Legacy / Local-Only)

When working in default env (file source), no Opik interaction is required:

1. Edit prompt file in prompts/ directory
2. Test locally (uses file source via PromptManager)
3. When ready to ship, follow Workflow B above

Evaluation

Evaluation Context

Configure evaluation section in agent_config.yaml with real test user IDs to avoid 404 errors from MCP tools. See file for full config structure.

Create Evaluation Dataset

from consumer_agent.evaluation import PromptEvaluator

evaluator = PromptEvaluator()

# Define test cases
items = [
    {
        "input": "What coffee offers are available?",
        "expected_output": "List of coffee offers with points and details"
    },
    {
        "input": "Show me nearby snack offers",
        "expected_output": "Snack offers with location information"
    },
    {
        "input": "How do I redeem points?",
        "expected_output": "Point redemption instructions"
    }
]

# Create dataset in Opik
dataset = evaluator.create_dataset("prompt-eval-v1", items)
print(f"Dataset created: {dataset['id']}")

Run Evaluation

from consumer_agent.agent import Agent
from consumer_agent.factory import create_chat_model
from opik.evaluation.metrics import Hallucination, AnswerRelevance

# Setup agent
model = create_chat_model()
agent = Agent(model=model)

# Define metrics to track
metrics = [
    Hallucination(),      # Detects made-up information
    AnswerRelevance(),   # Measures answer quality
]

# Run evaluation on dataset
results = evaluator.evaluate_prompt(
    agent=agent,
    dataset_name="prompt-eval-v1",
    scoring_metrics=metrics
)

# Metrics automatically saved to Opik dashboard

View Results

All evaluation metrics are automatically saved to Opik:

1. Go to Opik Dashboard
2. Navigate to your project: consumer-agent
3. View evaluation results with all metrics
4. Compare prompt versions side-by-side

Available Metrics

Built-in metrics from Opik:

• Hallucination: Detects fabricated information
• AnswerRelevance: Measures answer quality and relevance
• Moderation: Checks for toxic/inappropriate content
• ContextPrecision: Measures retrieval accuracy (RAG systems)
• ContextRecall: Measures retrieval completeness (RAG systems)

CLI Commands

Manage datasets and prompts using command-line interface.

Dataset Commands

Create Dataset

Create dataset from JSONL file (one JSON object per line):

# Create dataset from local file
consumer-agent opik dataset create \
  --name sample-test-v1 \
  --file data/raw/sample.jsonl \
  --description "Test dataset description"

# File format (JSONL):
{"input": "...", "expected_output": "...", "metadata": {...}}
{"input": "...", "expected_output": "...", "metadata": {...}}

List Datasets

# List all datasets in Opik
consumer-agent opik dataset list

Export Dataset

# Export dataset to JSONL file
consumer-agent opik dataset export \
  --name sample-test-v1 \
  --output data/external/exported.jsonl

Prompt Commands

Create/Update Prompt

# Create or update prompt from file
consumer-agent opik prompt create \
  --name conversational-v3 \
  --file prompts/conversational-v3.txt

Get Prompt

# Get latest version
consumer-agent opik prompt get --name conversational-v3

# Get specific version
consumer-agent opik prompt get \
  --name conversational-v3 \
  --commit abc123def456

View Prompt History

# View version history
consumer-agent opik prompt history --name conversational-v3

Evaluation Commands

Run Evaluation

Run evaluation on a dataset with auto-selected or custom metrics:

# Run with auto-selected metrics (based on dataset metadata)
consumer-agent opik eval run --dataset test-eval-v1

# Run with specific prompt version
consumer-agent opik eval run \
  --dataset test-eval-v1 \
  --prompt conversational-v4

# Run with custom metrics
consumer-agent opik eval run \
  --dataset test-eval-v1 \
  --metrics suggestion_validation,diversity,tool_usage

# Run with custom model
consumer-agent opik eval run \
  --dataset test-eval-v1 \
  --model gpt-4

# Run with custom experiment name
consumer-agent opik eval run \
  --dataset test-eval-v1 \
  --experiment "prompt-v4-test"

Available Metrics: See Custom Metrics section above for detailed metric descriptions.

Auto-Selection: When no metrics specified, automatically selects based on dataset metadata:

• type: "landing" triggers suggestion_validation + diversity
• requires_tools: true triggers tool_usage

Online Evaluation

Automated evaluations run on production traffic in real-time using LLM-as-judge rules. Setup: python scripts/register_llm_judge_rules.py. View results in Opik Dashboard > Experiments.

Additional Resources

• Opik Documentation
• Evaluation Metrics Guide
• Online Evaluation Rules