> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM

> How Formal proxies LLM API traffic through the Desktop App forward proxy

export const G = ({term, anchor, children}) => {
  const href = anchor ? `/docs/glossary/index#${anchor}` : `/docs/glossary/index`;
  return <a href={href} className="glossary-link" style={{
    textDecoration: "underline",
    textDecorationLine: "underline",
    textDecorationColor: "#6b7280",
    textDecorationThickness: "1px",
    textUnderlineOffset: "2px",
    color: "inherit",
    transition: "text-decoration-color 0.2s ease",
    borderBottom: "none"
  }} onMouseEnter={e => e.target.style.textDecorationColor = "#fff"} onMouseLeave={e => e.target.style.textDecorationColor = "#6b7280"}>
  {children || term}
</a>;
};

## Overview

<Note>
  LLM resources are only supported on **macOS**.
</Note>

LLM resources work differently from other Formal resource types. Rather than routing traffic through a <G anchor="connector">Connector</G> running in your infrastructure (a reverse proxy), LLM traffic is proxied directly by the **Formal Desktop App** acting as a local forward proxy.

When you run `formal connect` for an LLM resource, the Desktop App starts a local HTTPS listener and forwards requests directly to the upstream LLM provider (e.g. `api.openai.com`, `api.anthropic.com`). No Connector is involved in the data path.

```
your app  →  localhost (Desktop App forward proxy)  →  LLM provider API
```

## Creating LLM Resources

You must create an LLM resource in Formal for each LLM API you want to intercept. The resource tells the Desktop App which upstream hostname and port to forward traffic to.

```hcl theme={null}
resource "formal_resource" "anthropic" {
  name       = "anthropic"
  technology = "llm"
  hostname   = "api.anthropic.com"
  port       = 443
}

resource "formal_resource" "chatgpt" {
  name       = "chatgpt"
  technology = "llm"
  hostname   = "chatgpt.com"
  port       = 443
}
```

## Connecting

```bash theme={null}
formal connect anthropic
```

## Observability

### Log source

Because LLM traffic does not pass through a Connector, logs appear with `source: desktop` rather than `source: connector`. You can filter for these logs in the [Logs page](https://app.formal.ai/logs) with:

```
source:desktop
```

### What is captured

The Desktop App captures the full content of each LLM request and response, including:

* **Model and provider** — the model name extracted from the request body
* **Messages** — the full prompt and completion text
* **Tool calls** — any tool/function calls the model makes, including their names, arguments, and results
* **Token usage** — prompt tokens, completion tokens, and total tokens
* **Streaming responses** — individual chunks are reassembled into the complete response before logging

### Session replay

Each `formal connect` invocation for an LLM resource creates a session. Full session replay is available in the [Sessions page](https://app.formal.ai/sessions). The session replay shows every request sent and response received during the connection, including all tool call exchanges.

## Policies

You can enforce policies on LLM sessions at the `session`, `request`, and `response` stages. Use `session` rules to block a connection before it reaches the upstream provider, `request` rules to inspect or rewrite request headers, and `response` rules to inspect tool calls, the model, the provider, and token usage.

Request-stage rewrite policies can add, replace, or remove headers before the Desktop App forwards the request to the LLM provider. See [LLM policy inputs](/docs/guides/policies/evaluation#llm-resources) for the full list of available fields and examples.