OpenAI Apps SDK Integration Guide

OpenAI Apps SDK Integration Guide

Overview

This document explains how to integrate rover-mcp with the OpenAI Apps SDK (ChatGPT developer mode) to render rich UI widgets alongside MCP tool responses. The integration is based on analysis of the openai-apps-sdk-examples Pizzaz server.

Architecture

MCP + Apps SDK Flow

ChatGPT Client
    ↓ 1. User query: "Find coffee offers near me"
    ↓
MCP Server (rover-mcp)
    ↓ 2. Execute search_nearby_offers tool
    ↓ 3. Return response with _meta.openai/outputTemplate
    ↓
ChatGPT Client
    ↓ 4. Read _meta.openai/outputTemplate: "ui://widget/offer-map.html"
    ↓ 5. Call ReadResource("ui://widget/offer-map.html")
    ↓
MCP Server
    ↓ 6. Return HTML/JS/CSS bundle
    ↓
ChatGPT Client
    ↓ 7. Render widget in iframe with window.openai global
    ↓ 8. Widget reads toolOutput from window.openai
    ↓ 9. Widget renders offers on map

Key Components

1. MCP Protocol Handlers (already implemented in rover-mcp)

   • ListTools - Advertise available tools
   • CallTool - Execute tool and return results

2. Resource Handlers (need to add)

   • ListResources - Advertise available widget templates
   • ReadResource - Serve widget HTML/JS/CSS bundles
   • ListResourceTemplates - List widget template URIs

3. Widget Bundles (need to create)

   • Self-contained HTML files with inline CSS/JS
   • React components built with Vite
   • Communicate via window.openai global object

Tool Response Structure

Standard Response (Current)

{
  "content": [
    {
      "type": "text",
      "text": "Found 5 coffee offers near you!"
    }
  ]
}

Apps SDK Enhanced Response (Target)

{
  "content": [
    {
      "type": "text",
      "text": "Found 5 coffee offers near you!"
    }
  ],
  "structuredContent": {
    "offers": [
      {
        "id": "offer-123",
        "description": "Get 500 points on Starbucks coffee",
        "points": 500,
        "category": "Coffee"
      }
    ],
    "query": "coffee",
    "userLocation": {
      "latitude": 37.7749,
      "longitude": -122.4194
    }
  },
  "_meta": {
    "openai/outputTemplate": "ui://widget/offer-list.html",
    "openai/toolInvocation/invoking": "Searching for offers...",
    "openai/toolInvocation/invoked": "Found offers near you!",
    "openai/widgetAccessible": true,
    "openai/resultCanProduceWidget": true
  }
}

Metadata Fields

Field	Purpose	Example
openai/outputTemplate	URI of widget to render	"ui://widget/offer-map.html"
openai/toolInvocation/invoking	Loading message shown while tool executes	"Searching for offers..."
openai/toolInvocation/invoked	Success message shown after tool completes	"Found offers!"
openai/widgetAccessible	Widget is accessibility-compliant	true
openai/resultCanProduceWidget	Response includes widget support	true

Resource Management

Resource Structure

From pizzaz_server_node/src/server.ts:143-149:

const resources: Resource[] = widgets.map((widget) => ({
  uri: widget.templateUri,              // "ui://widget/pizza-map.html"
  name: widget.title,                   // "Show Pizza Map"
  description: `${widget.title} widget markup`,
  mimeType: "text/html+skybridge",      // Special MIME type for widgets
  _meta: widgetMeta(widget)
}));

For rover-mcp

We should define resources for each widget type:

[
  {
    "uri": "ui://widget/offer-list.html",
    "name": "Offer List",
    "description": "Display offers in a card list",
    "mimeType": "text/html+skybridge",
    "_meta": {
      "openai/outputTemplate": "ui://widget/offer-list.html",
      "openai/widgetAccessible": true
    }
  },
  {
    "uri": "ui://widget/offer-map.html",
    "name": "Offer Map",
    "description": "Display offers on an interactive map",
    "mimeType": "text/html+skybridge",
    "_meta": {
      "openai/outputTemplate": "ui://widget/offer-map.html",
      "openai/widgetAccessible": true
    }
  }
]

ReadResource Response

When ChatGPT requests ui://widget/offer-list.html, the server returns:

{
  "contents": [
    {
      "uri": "ui://widget/offer-list.html",
      "mimeType": "text/html+skybridge",
      "text": "<div id=\"offer-root\"></div>\n<link rel=\"stylesheet\" href=\"https://cdn.example.com/offer-list-2d2b.css\">\n<script type=\"module\" src=\"https://cdn.example.com/offer-list-2d2b.js\"></script>",
      "_meta": {
        "openai/outputTemplate": "ui://widget/offer-list.html",
        "openai/widgetAccessible": true
      }
    }
  ]
}

Widget Development

window.openai Global Object

Widgets access ChatGPT context via window.openai:

interface WindowOpenAI {
  // Visual context
  theme: "light" | "dark";
  displayMode: "pip" | "inline" | "fullscreen";
  maxHeight: number;
  locale: string;

  // Data from MCP server
  toolInput: Record<string, unknown>;      // Original tool arguments
  toolOutput: Record<string, unknown>;     // structuredContent from response
  toolResponseMetadata: Record<string, unknown>;  // _meta from response

  // Persistent widget state (survives tool calls)
  widgetState: Record<string, unknown> | null;
  setWidgetState: (state: Record<string, unknown>) => Promise<void>;

  // Actions
  callTool: (name: string, args: Record<string, unknown>) => Promise<{ result: string }>;
  sendFollowUpMessage: (args: { prompt: string }) => Promise<void>;
  openExternal: (payload: { href: string }) => void;
  requestDisplayMode: (args: { mode: "pip" | "inline" | "fullscreen" }) => Promise<{ mode: string }>;
}

React Hooks (from openai-apps-sdk-examples)

// Access window.openai properties reactively
import { useOpenAiGlobal } from "./use-openai-global";
const theme = useOpenAiGlobal("theme");
const displayMode = useOpenAiGlobal("displayMode");

// Get structured data from MCP server
import { useWidgetProps } from "./use-widget-props";
const { offers, query } = useWidgetProps<{
  offers: Offer[];
  query: string;
}>();

// Persistent widget state
import { useWidgetState } from "./use-widget-state";
const [selectedOffer, setSelectedOffer] = useWidgetState<{ offerId: string }>();

Example Widget Component

src/widgets/offer-list/index.tsx

import React from "react";
import { createRoot } from "react-dom/client";
import { useOpenAiGlobal } from "../use-openai-global";
import { useWidgetProps } from "../use-widget-props";

interface OfferData {
  offers: Array<{
    id: string;
    description: string;
    points: number;
    category: string;
  }>;
  query: string;
}

function OfferListWidget() {
  const theme = useOpenAiGlobal("theme");
  const displayMode = useOpenAiGlobal("displayMode");
  const data = useWidgetProps<OfferData>();

  if (!data?.offers) {
    return <div>No offers found</div>;
  }

  return (
    <div className={`offer-list ${theme} ${displayMode}`}>
      <h2>Found {data.offers.length} offers for "{data.query}"</h2>
      {data.offers.map((offer) => (
        <div key={offer.id} className="offer-card">
          <div className="offer-points">{offer.points} pts</div>
          <div className="offer-description">{offer.description}</div>
          <div className="offer-category">{offer.category}</div>
        </div>
      ))}
    </div>
  );
}

createRoot(document.getElementById("offer-root")).render(<OfferListWidget />);

Implementation Plan for rover-mcp

Phase 1: Add Resource Handlers

1. Implement MCP Resource Protocol in internal/server/server.go

   • Add ListResources handler
   • Add ReadResource handler
   • Add ListResourceTemplates handler

2. Define Widget Registry

   type Widget struct {
       ID           string
       Title        string
       TemplateURI  string
       Invoking     string
       Invoked      string
       HTML         string
   }
   
   var widgets = []Widget{
       {
           ID:          "offer-list",
           Title:       "Offer List",
           TemplateURI: "ui://widget/offer-list.html",
           Invoking:    "Searching for offers...",
           Invoked:     "Found offers!",
           HTML:        loadWidget("offer-list.html"),
       },
       {
           ID:          "offer-map",
           Title:       "Offer Map",
           TemplateURI: "ui://widget/offer-map.html",
           Invoking:    "Mapping nearby offers...",
           Invoked:     "Mapped offers near you!",
           HTML:        loadWidget("offer-map.html"),
       },
   }

3. Update Tool Responses in internal/tools/*.go

   func (t *SearchOffersHandler) Execute(ctx context.Context, input map[string]interface{}) (map[string]interface{}, error) {
       // ... existing search logic ...
   
       return map[string]interface{}{
           "content": []map[string]interface{}{
               {
                   "type": "text",
                   "text": fmt.Sprintf("Found %d offers matching '%s'", len(offers), query),
               },
           },
           "structuredContent": map[string]interface{}{
               "offers": offers,
               "query":  query,
               "userID": userID,
           },
           "_meta": map[string]interface{}{
               "openai/outputTemplate":            "ui://widget/offer-list.html",
               "openai/toolInvocation/invoking":   "Searching for offers...",
               "openai/toolInvocation/invoked":    "Found offers!",
               "openai/widgetAccessible":          true,
               "openai/resultCanProduceWidget":    true,
           },
       }, nil
   }

Phase 2: Create Widget Bundles

1. Setup Widget Build System

   mkdir widgets
   cd widgets
   pnpm init
   pnpm add -D vite @vitejs/plugin-react
   pnpm add react react-dom

2. Create Widget Components

   • widgets/src/offer-list/ - Card-based list view
   • widgets/src/offer-map/ - Map view with markers (like Pizzaz)
   • widgets/src/use-openai-global.ts - Hook utilities
   • widgets/src/use-widget-props.ts - Data access hook

3. Build and Bundle

   pnpm run build  # Outputs to widgets/dist/

4. Embed or Host Widgets

   • Option A: Embed in Go binary with go:embed
   • Option B: Host on CDN (S3 + CloudFront)
   • Option C: Serve via MCP ReadResource endpoint

Phase 3: Testing

1. Local Testing with MCP Inspector

   # Install MCP Inspector
   npx @modelcontextprotocol/inspector
   
   # Test resource listing
   # Test resource reading
   # Test tool execution with _meta

2. ChatGPT Developer Mode

   • Enable developer mode in ChatGPT settings
   • Add rover-mcp as connector
   • Use ngrok for local testing: ngrok http 8080
   • Add connector URL: https://<id>.ngrok-free.app/mcp

Widget Design Patterns

1. Responsive Layouts

Handle three display modes from server.ts:183-190:

const displayMode = useOpenAiGlobal("displayMode");

<div style={{
  height: displayMode === "fullscreen" ? maxHeight - 40 : 480,
  borderRadius: displayMode === "fullscreen" ? 0 : "1.5rem"
}}>

2. Theme Support

const theme = useOpenAiGlobal("theme");

<div className={`widget ${theme === "dark" ? "dark" : ""}`}>
  <style>{`
    .widget { background: white; color: black; }
    .widget.dark { background: #1a1a1a; color: white; }
  `}</style>
</div>

3. Interactive Actions

// Call another MCP tool from widget
const handleRefresh = async () => {
  const result = await window.openai.callTool("search_offers", {
    query: searchTerm,
    user_id: userId
  });
};

// Send follow-up prompt
const handleAskQuestion = () => {
  window.openai.sendFollowUpMessage({
    prompt: "Show me vegetarian offers"
  });
};

// Open external link
const handleViewDetails = () => {
  window.openai.openExternal({
    href: "https://fetchrewards.com/offers/123"
  });
};

4. Persistent State

// State survives across tool calls
const [mapCenter, setMapCenter] = useWidgetState({
  lat: 37.7749,
  lng: -122.4194,
  zoom: 12
});

// When user pans map, persist location
const handleMapMove = (center: LatLng, zoom: number) => {
  setMapCenter({ lat: center.lat, lng: center.lng, zoom });
};

Reference Implementation

The Pizzaz Node server (/Users/s.hollinger/experiments/openai-apps-sdk-examples/pizzaz_server_node/) demonstrates:

SSE Transport (server.ts:240-331)

// GET /mcp - Establish SSE connection
async function handleSseRequest(res: ServerResponse) {
  res.setHeader("Access-Control-Allow-Origin", "*");
  const server = createPizzazServer();
  const transport = new SSEServerTransport(postPath, res);
  await server.connect(transport);
}

// POST /mcp/messages?sessionId=... - Client sends messages
async function handlePostMessage(req, res, url) {
  const sessionId = url.searchParams.get("sessionId");
  const session = sessions.get(sessionId);
  await session.transport.handlePostMessage(req, res);
}

Widget Registry (server.ts:43-117)

const widgets: PizzazWidget[] = [
  {
    id: "pizza-map",
    title: "Show Pizza Map",
    templateUri: "ui://widget/pizza-map.html",
    invoking: "Hand-tossing a map",
    invoked: "Served a fresh map",
    html: `
      <div id="pizzaz-root"></div>
      <link rel="stylesheet" href="https://persistent.oaistatic.com/ecosystem-built-assets/pizzaz-0038.css">
      <script type="module" src="https://persistent.oaistatic.com/ecosystem-built-assets/pizzaz-0038.js"></script>
    `,
    responseText: "Rendered a pizza map!"
  },
  // ... more widgets
];

Tool Handler (server.ts:204-225)

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const widget = widgetsById.get(request.params.name);
  const args = toolInputParser.parse(request.params.arguments ?? {});

  return {
    content: [{ type: "text", text: widget.responseText }],
    structuredContent: { pizzaTopping: args.pizzaTopping },
    _meta: widgetMeta(widget)
  };
});

Resource Handler (server.ts:177-194)

server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
  const widget = widgetsByUri.get(request.params.uri);

  return {
    contents: [{
      uri: widget.templateUri,
      mimeType: "text/html+skybridge",
      text: widget.html,
      _meta: widgetMeta(widget)
    }]
  };
});

Security Considerations

1. Widget Isolation

   • Widgets run in sandboxed iframe
   • Can only access window.openai API
   • Cannot access parent page DOM or cookies

2. Content Security Policy

   • Host widgets on CDN with HTTPS
   • Use SRI (Subresource Integrity) for script tags
   • Validate all user inputs in widgets

3. Authentication

   • Widget cannot access MCP auth tokens
   • All tool calls go through ChatGPT client
   • User context maintained by ChatGPT

Next Steps

1. Create POC Widget

   • Build simple offer list widget
   • Test with MCP Inspector
   • Validate data flow

2. Implement Resource Handlers

   • Add to Go server
   • Support widget serving
   • Test with ChatGPT developer mode

3. Production Widgets

   • Offer card list (inline/pip mode)
   • Offer map (fullscreen mode)
   • Retailer directory
   • Offer details panel

Resources

• MCP Specification
• OpenAI Apps SDK Examples
• Pizzaz Server Reference
• Widget Types Reference