AI Integration

The AI system uses LangChain for multi-provider support with a modular agent architecture.

Multi-LLM Support

The AI system routes every chat and embedding request through one server-only LlmClient. Set AI_LLM_TRANSPORT=direct to call OpenAI, Anthropic, and Google with their native credentials, or openrouter to use one OpenRouter key and the canonical gateway model mappings in config/ai.ts.

File Structure

config/
└── ai.ts                 # Models, providers, pricing configuration

lib/ai/
├── index.ts              # Main exports
├── llm-client.ts         # Sole external AI boundary (direct + OpenRouter)
├── circuit-breaker.ts    # Upstash/in-memory closed-open-half-open route state
├── providers/
│   └── index.ts          # Model/provider discovery only; no external calls
├── agents/
│   ├── index.ts          # Agent registry
│   ├── types.ts          # Agent interfaces
│   ├── base-agent.ts     # Base class for all agents
│   ├── chat/index.ts     # Chat agent
│   ├── code-assistant/   # Code assistant agent
│   ├── translator/       # Translator agent
│   └── writer/           # Writer agent
├── cache/
│   └── index.ts          # Prompt caching utilities — applied to the Anthropic system prompt in base-agent.ts (default `5m`, optional `1h` TTL via `AnthropicCacheTTL`)
└── langgraph.ts          # Message building utilities

app/api/ai/
└── stream/route.ts       # SSE streaming endpoint

Supported Providers

Model Configuration

Models are defined in config/ai.ts with current direct/OpenRouter pricing, capabilities, sampling compatibility, and a canonical openRouterModelId. The curated text catalogue was last reviewed on 2026-08-18 against official vendor documentation and OpenRouter’s live model endpoint. Retired entries and modality-specific image-generation endpoints are not selectable chat models. Update this single catalogue when adding a route; do not create a second gateway-only model list.

Unified LLM boundary and ZDR

lib/ai/llm-client.ts is the only module allowed to construct provider SDKs or call chat/embedding endpoints. In OpenRouter mode every request carries provider.zdr=true, data_collection="deny", require_parameters=true, and attribution headers. The fixture used by pnpm run qa:local:private rejects a request if any of those fields is absent. This routing control is not a substitute for reviewing OpenRouter and the selected upstream provider in your DPA.

Direct mode cannot manufacture account-level ZDR by code. Direct OpenAI requests set store=false and the LangChain ZDR flag, but the operator must obtain and verify the provider's organization/project retention controls. Anthropic and Google retention likewise follows the configured API account and contract.

Circuits, retries, and model fallback

The centralized client uses the existing Upstash connection for shared production circuit state. Five transient failures in 30 seconds open one transport/model route for 30 seconds; after cooldown, one atomically leased half-open probe decides whether to close or reopen it. Store errors are redacted, rate-limited, and fail open so Redis cannot become a global AI availability dependency. Development and deterministic tests use the same state machine in process memory.

Direct vendors receive one cancellation-aware jittered retry, while OpenRouter receives no application retry because the gateway already routes among eligible providers for the same model. Before any content is emitted, an open circuit or transient failure may move once to the configured cross-provider fallback. Agent allow-lists still apply. Any partial output makes the failure terminal, and embeddings never cross models because mixing vector spaces corrupts similarity search. ai_requests.metadata.routing stores only bounded model/route/circuit outcome summaries; prompts, responses, identities, secrets, and raw provider bodies are excluded.

Built-in Agents

Five agents ship out of the box — Chat (default), Code Assistant, Translator, Writer, and Knowledge Base (RAG) — each defined in lib/ai/agents/ and registered in lib/ai/agents/index.ts. All extend BaseAgent; specialized agents may declare an allowedModels list that also constrains fallback. Credits are charged uniformly across agents — 1 credit per LLM token (input + output), deducted once post-stream against provider-reported usage from the model that actually completed the answer.

For the full architecture, registry pattern, custom-agent recipe, and advanced override hooks (like the RAG agent's prepareRAGContext() pre-fetch), see Built-in Agents.

Chat Interface

A full-featured AI conversation page lives at /private-dashboard/chat — Server Component shell with a Client Component (components/private/chat-interface.tsx) for streaming, agent switching, and session history. Data reads come from core/chat/queries.ts; writes go through core/chat/mutations.ts. Sessions are persisted in chat_sessions, messages in chat_messages, both RLS-scoped by membership.

For the file structure, hook API (useChat), session management rules, SSE handling, and pagination details, see Chat Interface.

RAG Document Chat

The Knowledge Base agent lets users upload TXT/MD/PDF files at /private-dashboard/documents, chunks and embeds them through the same selected LlmClient transport, and answers questions by retrieving relevant chunks from a pgvector HNSW index. Similarity search uses a membership-enforced match_document_chunks_text SECURITY DEFINER RPC; the match cutoff is configurable via embeddingConfig.ragMatchThreshold (default 0.1 in config/ai.ts). Credits are deducted 1:1 with the embedding tokens consumed.

For the embedding-model table, credit math, RPC details, API routes, and the Knowledge Base page walkthrough, see RAG & Documents.