Skip to main content

​
Overview

LLM resources let Formal inspect and control traffic to LLM provider APIs, such as api.anthropic.com or chatgpt.com. Formal can proxy this traffic in two places:
  • Formal Endpoint: Transparent Mode intercepts LLM traffic on macOS and Linux. The Endpoint evaluates policies and forwards requests to the provider.
  • Connector: A in your infrastructure evaluates policies and forwards requests to the provider. Traffic reaches it from the Formal Endpoint or directly from your applications.

​
Creating LLM Resources

You must create an LLM resource in Formal for each LLM API you want to intercept. The resource tells Formal which upstream hostname and port to forward traffic to.

​
Connecting

​
Through the Formal Endpoint

LLM resources do not support formal connect. Instead, enable Transparent Mode and write a network rule that matches LLM traffic. This rule matches known agents on macOS and Linux, and known LLM provider hostnames on Linux:
To have a Connector evaluate this traffic instead of the Endpoint, set forward_to_connector in the rule’s outputs. The LLM resource must be reachable through a Connector listener.

​
Through a Connector

Applications in your infrastructure can send LLM traffic to a Connector directly. Link the LLM resource to a Connector listener, then replace the provider hostname with the Connector hostname. When the listener serves several resources, prefix the hostname with the resource name, as described in Smart Routing:
  • Before: https://api.anthropic.com/v1/messages
  • After: https://anthropic.<connector-hostname>/v1/messages
As with HTTP resources, set the X-Formal-User-Username and X-Formal-User-Password headers to associate requests with a Formal identity.

​
Observability

​
Log source

Logs appear with source: desktop when the Formal Endpoint evaluates the traffic, and with source: connector when a Connector does. Filter for them in the Logs page with source:desktop or source:connector. To encrypt or strip these logs, create a log configuration with Source set to Endpoint or Connector. See Logs Configuration.

​
What is captured

Formal captures the full content of each LLM request and response, including:
  • Model and provider — the model name extracted from the request body
  • Messages — the full prompt and completion text
  • Tool calls — any tool/function calls the model makes, including their names, arguments, and results
  • Token usage — prompt tokens, completion tokens, and total tokens
  • Streaming responses — individual chunks are reassembled into the complete response before logging

​
Session replay

Full session replay is available in the Sessions page. The session replay shows every request sent and response received, including all tool call exchanges.

​
Policies

You can enforce policies on LLM sessions at the session, request, and response stages. Use session rules to block a connection before it reaches the upstream provider, request rules to inspect or rewrite request headers and bodies, and response rules to inspect tool calls, the model, the provider, and token usage. Request-stage rewrite policies can:
  • Add, replace, or remove headers before Formal forwards the request
  • Replace the body (for example with regex.replace on input.http.body) to redact sensitive prompt text such as SSNs without blocking the session
See LLM policy inputs for the full list of available fields and examples, including redacting SSNs from request messages. To allow only your Anthropic organization, see Restrict Anthropic Sign-Ins to One Org.