Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.formal.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

LLM resources are only supported on macOS.
LLM resources work differently from other Formal resource types. Rather than routing traffic through a running in your infrastructure (a reverse proxy), LLM traffic is proxied directly by the Formal Desktop App acting as a local forward proxy. When you run formal connect for an LLM resource, the Desktop App starts a local HTTPS listener and forwards requests directly to the upstream LLM provider (e.g. api.openai.com, api.anthropic.com). No Connector is involved in the data path. 
your app  →  localhost (Desktop App forward proxy)  →  LLM provider API

​
Creating LLM Resources

You must create an LLM resource in Formal for each LLM API you want to intercept. The resource tells the Desktop App which upstream hostname and port to forward traffic to. 
resource "formal_resource" "anthropic" {
  name       = "anthropic"
  technology = "llm"
  hostname   = "api.anthropic.com"
  port       = 443
}

resource "formal_resource" "chatgpt" {
  name       = "chatgpt"
  technology = "llm"
  hostname   = "chatgpt.com"
  port       = 443
}

​
Connecting

formal connect anthropic

​
Observability

​
Log source

Because LLM traffic does not pass through a Connector, logs appear with source: desktop rather than source: connector. You can filter for these logs in the Logs page with: 
source:desktop

​
What is captured

The Desktop App captures the full content of each LLM request and response, including:
  • Model and provider — the model name extracted from the request body
  • Messages — the full prompt and completion text
  • Tool calls — any tool/function calls the model makes, including their names, arguments, and results
  • Token usage — prompt tokens, completion tokens, and total tokens
  • Streaming responses — individual chunks are reassembled into the complete response before logging

​
Session replay

Each formal connect invocation for an LLM resource creates a session. Full session replay is available in the Sessions page. The session replay shows every request sent and response received during the connection, including all tool call exchanges.

​
Policies

You can enforce policies on LLM sessions at the session, request, and response stages. Use session rules to block a connection before it reaches the upstream provider, request rules to inspect or rewrite request headers, and response rules to inspect tool calls, the model, the provider, and token usage. Request-stage rewrite policies can add, replace, or remove headers before the Desktop App forwards the request to the LLM provider. See LLM policy inputs for the full list of available fields and examples.