Location-Based Tool Selection Fix

Date: 2025-12-09 Status: Complete Author: Prakash Chaudhary Jira: PLT-312

Executive Summary

When users ask for location-based offers without having shared their location, the agent blocked on location 90.6% of the time - asking for ZIP code or city before attempting to help. This fix adds explicit fallback logic to use search_offers when location is unavailable.

Key Findings (Final Evaluation - Manual Review)

Before fix: Agent blocked on location 90.6% of the time, only provided offers 6.2% of time
After fix: Agent uses search_offers fallback 78.1% of the time, never blocks on location
Improvement: 100% elimination of “blocks on location” behavior (29/32 -> 0/32)
Success rate: 9.4% -> 100% acceptable behavior (manual classification)

Recommendation

Deploy the prompt fix to production. The 4-line change dramatically improves user experience by providing offers instead of asking for location.

Problem Statement

When a user asks for “nearby offers” but hasn’t shared their location (lat=0, lon=0), the agent blocked on location 90.6% of the time - asking for ZIP code or city before attempting to help.

Expected behavior: Silently fall back to search_offers for nationwide results instead of asking for location.

Solution

Add explicit instructions to the conversational prompt for location-based tool selection:

## Location & Privacy

* If user coordinates are available, use them silently for `search_nearby_offers`. If not, use `search_offers` instead.
* **Never** ask the user for location, ZIP code, or coordinates.
* Never display GPS coordinates or raw addresses. Prefer "nearby" or store names.
* Provide address/distance **only if asked**.

Changes Made

Updated the conversational prompt:

Diff:

* Never display GPS coordinates or raw addresses by default. Prefer "nearby" or store names.
* Provide an address/distance **only if asked**.
* Do not reveal internal locations, office addresses, or employee-specific info.
* If user coordinates are available, use them silently for `search_nearby_offers`. If not, use `search_offers` instead.
* **Never** ask the user for location, ZIP code, or coordinates.
* Never display GPS coordinates or raw addresses. Prefer "nearby" or store names.
* Provide address/distance **only if asked**.

Test Results

Full Evaluation (32 Samples - Final Run with Manual Review)

Test Setup:

32 diverse queries covering “nearby”, “local”, “near me”, and location-implicit requests
Location: Not provided (lat=0, lon=0)
Agent: Full conversational agent with all MCP tools (7 tools)
Concurrency: 8 parallel requests
Classification: Manual review of each response (not keyword-based)

Automated Metrics (keyword detection):

Metric	Original Prompt	Fixed Prompt	Change
Asked for location (keyword)	25/32 (78.1%)	1/32 (3.1%)	-75%
Used search_offers	2/32 (6.2%)	24/32 (75.0%)	+68.8%
Used search_nearby_offers	0/32 (0.0%)	0/32 (0.0%)	No change
Used neither tool	30/32 (93.8%)	8/32 (25.0%)	-68.8%

Manual Classification Results:

Behavior Category	Original Prompt	Fixed Prompt	Definition
Blocks on location	29/32 (90.6%)	0/32 (0.0%)	Asks for location BEFORE attempting to help
Uses search_offers	2/32 (6.2%)	25/32 (78.1%)	Calls search_offers as fallback
Clarifies intent	0/32 (0.0%)	6/32 (18.8%)	Asks about item/category (acceptable)
Provides info	1/32 (3.1%)	1/32 (3.1%)	Gives helpful info without tool

Key Observations:

100% elimination of blocking behavior: Manual review confirms no “blocks on location” cases with fixed prompt
Keyword detection overestimates failures: Automated detection flags error handling messages as “asking for location” when agent actually tried first
Intent clarification is acceptable: Agent asking “grocery or retail?” is NOT the same as asking for location

Example: Before vs After

Original Prompt (Blocks on location):

Query: "find me coffee offers near me"
Tools: []
Response: "Sure - I can look up nearby coffee offers. Share your ZIP code (or city)
or say 'use my device location' so I can search nearby."
Result: User gets no offers, must provide location to continue

Fixed Prompt (Falls back to search_offers):

Query: "find me coffee offers near me"
Tools: [search_offers, llm_feedback]
Response: "I found these coffee offers you can earn points on:
- Starbucks Frappuccino: 50 points
- Dunkin' K-Cups: 75 points
- Peet's Coffee bags: 100 points
These are available nationwide."
Result: User gets helpful offers immediately, no location required

Future Considerations

ZIP Code Fallback: If user has ZIP code but not precise location, could use ZIP-based search as intermediate fallback before nationwide search. LiDAR team previously tested this but results were mixed - worth revisiting for assistant use case where graceful degradation may be more acceptable than in main UI.
Location Permission Request: Agent could send a “request allow location” event so mobile can prompt user for location permission, with a soft nudge like “We found these offers based on your area. For more accurate nearby results, you can share your location.” This requires mobile-side implementation. See Figma designs.

Methodology

Test Environment

Agent Configuration: Full conversational agent (agent_config.yaml)
Tools Enabled: All Rover MCP tools + web_search
Components Enabled: general-instructions, offer-list, prompt-suggestion
Model: gpt-5-mini with low reasoning effort
User ID: 67800c1c62de6fde188967ec (test user)

Test Execution

32-Sample Evaluation:

32 diverse queries covering “nearby”, “local”, “near me”, and location-implicit requests
Each query run once with original prompt, once with fixed prompt
Manual classification of each response into behavior categories

Sample Queries:

- find me coffee offers near me
- show me deals near me
- nearby grocery offers
- offers near my location
- local grocery offers
- stores around me with deals
- offers in my area
- help me find nearby grocery deals
... (32 total queries)

Limitations

Single test user ID, but representative of all users without location
LLM behavior is non-deterministic; results may vary slightly between runs

Conclusion

The location-based tool selection fix resolves the critical issue of the agent blocking on location requests. The 4-line prompt change:

Eliminates 100% of “blocks on location” behavior (90.6% -> 0%)
Increases acceptable behavior from 9.4% to 100%
78.1% of queries now use search_offers fallback (vs 6.2% before)
Agent never asks for location before attempting to help
Aligns with privacy policy (“never ask”)

Key Insight from Manual Review: Keyword-based detection overestimates failures by flagging error handling messages. Manual classification reveals the fix is even more effective than automated metrics suggest.