Skip to main content

​
Overview

LLM resources are only supported on macOS.
LLM resources work differently from other Formal resource types. Rather than routing traffic through a running in your infrastructure (a reverse proxy), LLM traffic is proxied directly by the Formal Endpoint acting as a local forward proxy. When you run formal connect for an LLM resource, the Formal Endpoint starts a local HTTPS listener and forwards requests directly to the upstream LLM provider (e.g. chatgpt.com, api.anthropic.com). No Connector is involved in the data path.

​
Creating LLM Resources

You must create an LLM resource in Formal for each LLM API you want to intercept. The resource tells the Formal Endpoint which upstream hostname and port to forward traffic to.

​
Connecting

​
Observability

​
Log source

Because LLM traffic does not pass through a Connector, logs appear with source: desktop rather than source: connector. You can filter for these logs in the Logs page with:
To encrypt or strip Formal Endpoint logs, create a log configuration with Source set to Desktop. See Logs Configuration.

​
What is captured

The Formal Endpoint captures the full content of each LLM request and response, including:
  • Model and provider — the model name extracted from the request body
  • Messages — the full prompt and completion text
  • Tool calls — any tool/function calls the model makes, including their names, arguments, and results
  • Token usage — prompt tokens, completion tokens, and total tokens
  • Streaming responses — individual chunks are reassembled into the complete response before logging

​
Session replay

Each formal connect invocation for an LLM resource creates a session. Full session replay is available in the Sessions page. The session replay shows every request sent and response received during the connection, including all tool call exchanges.

​
Policies

You can enforce policies on LLM sessions at the session, request, and response stages. Use session rules to block a connection before it reaches the upstream provider, request rules to inspect or rewrite request headers and bodies, and response rules to inspect tool calls, the model, the provider, and token usage. Request-stage rewrite policies can:
  • Add, replace, or remove headers before the Formal Endpoint forwards the request
  • Replace the body (for example with regex.replace on input.http.body) to redact sensitive prompt text such as SSNs without blocking the session
See LLM policy inputs for the full list of available fields and examples, including redacting SSNs from request messages.