> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Formal Kubernetes Egress Agent

> Apply Formal policies to traffic from Kubernetes application containers

## Overview

The Formal Kubernetes Egress Agent enables applying Formal to arbitrary
traffic coming from Kubernetes application containers. The Kubernetes
Egress Agent operates like the [Formal Endpoint](/docs/guides/client-apps/desktop-app)
but runs as a native sidecar outside of the application container instead
of being installed on the host. You can then configure
[policies](/docs/guides/policies/introduction) against application
container traffic like you would for the Formal Endpoint.

```mermaid theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
flowchart LR
    subgraph pod["Pod"]
        app["Application container"]
        sidecar["Formal Kubernetes Agent<br/>native sidecar"]
    end

    internet["Public internet"]

    app --> sidecar --> internet
```

## Cluster requirements

Your cluster must meet these requirements before you run the sidecar.

| Requirement                             | Why it is required                                                                                                                                                                     |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Kubernetes 1.29+**                    | Native sidecars use `restartPolicy: Always` on an init container.                                                                                                                      |
| **Sidecar `NET_ADMIN`**                 | The sidecar adds the `NET_ADMIN` Linux capability so it can intercept Pod egress. The cluster does not need a `NET_ADMIN` setting.                                                     |
| **Privileged Pod Security Standard**    | The sidecar runs as root and adds `NET_ADMIN`. Restricted and Baseline [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) deny those Pods. |
| **Outbound HTTPS to the Control Plane** | The sidecar must reach `https://api.joinformal.com`.                                                                                                                                   |
| **Image pull access**                   | Nodes must pull the Formal Kubernetes Egress image from ECR or GAR.                                                                                                                    |

**Verify:**

```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
kubectl version
```

The server version must be 1.29 or later.

## Installation

Install the agent as a native sidecar on your application Pod. The
recommended path is the
[Helm chart](https://github.com/formalco/helm-charts/tree/main/charts/kubernetes-egress).
It injects the sidecar onto labeled Pods via a
[MutatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/mutating-admission-policy/).

If you already control the workload spec, you can
[add the sidecar to your spec](#add-the-sidecar-to-your-spec) instead.

### Add the Formal API Key Secret

The sidecar authenticates as a Formal machine user. Create an API key for that
user and store it as a Kubernetes Secret in each namespace that runs labeled
Pods.

The chart does not create this Secret.

<Steps>
  <Step title="Create a machine user">
    Go to [Users](https://app.formal.ai/users) and click **Create User**.
    Choose **Machine**. Give the user a name that matches the workload.
  </Step>

  <Step title="Create an API key">
    Go to [API Keys](https://app.formal.ai/api-keys) and click **Create API
    Key**. Assign the key to that machine user.
  </Step>

  <Step title="Create the Secret">
    Create `formal-kubernetes-egress` in every namespace that will run
    labeled Pods. The sidecar reads `FORMAL_API_KEY` from the
    `formal-api-key` key.

    ```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
    kubectl create secret generic formal-kubernetes-egress \
      --namespace agents \
      --from-literal=formal-api-key=<YOUR_API_KEY>
    ```
  </Step>

  <Step title="Verify">
    ```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
    kubectl get secret formal-kubernetes-egress --namespace agents
    ```

    The Secret must exist before you create labeled Pods in that namespace.
  </Step>
</Steps>

## Deploy with Helm

The chart is published as `kubernetes-egress` in the
[Formal Helm Charts](https://github.com/formalco/helm-charts) repository.

The chart needs Kubernetes 1.36+ and cluster RBAC to manage
[MutatingAdmissionPolicy](https://kubernetes.io/docs/reference/kubernetes-api/admissionregistration/mutating-admission-policy-v1/)
and
[MutatingAdmissionPolicyBinding](https://kubernetes.io/docs/reference/kubernetes-api/admissionregistration/mutating-admission-policy-binding-v1/).

### Apply the inject label

The policy binding mutates Pods that have this label set to `true`.

```text theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
formal.ai/kubernetes-egress-inject: "true"
```

Put it on the Pod template (`spec.template.metadata.labels`) of your
Deployment, StatefulSet, DaemonSet, Job, or CronJob:

```yaml theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
spec:
  template:
    metadata:
      labels:
        formal.ai/kubernetes-egress-inject: "true"
```

The policy mutates Pod CREATE only. Labeling an existing Pod does not inject
the sidecar.

**Verify:**

```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
kubectl get pods --namespace agents --show-labels
kubectl get pods -l formal.ai/kubernetes-egress-inject=true --all-namespaces
```

Expected output includes `formal.ai/kubernetes-egress-inject=true` on the
workload Pods.

To stop injecting, remove the label from the Pod template and roll the
workload. Existing Pods keep their sidecar until you replace them.

<Steps>
  <Step title="Add the Formal Helm repository">
    ```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
    helm repo add formal https://formalco.github.io/helm-charts
    helm repo update
    ```
  </Step>

  <Step title="Install the chart">
    ```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
    helm upgrade --install formal-kubernetes-egress formal/kubernetes-egress \
      --namespace formal \
      --create-namespace
    ```

    The chart installs cluster-scoped admission objects only.
  </Step>

  <Step title="Label Pods to inject">
    Apply the [inject label](#apply-the-inject-label) on the
    workload Pod template. New Pods created with that label are injected.
  </Step>

  <Step title="Verify">
    ```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
    kubectl get mutatingadmissionpolicy formal-kubernetes-egress
    kubectl get mutatingadmissionpolicybinding formal-kubernetes-egress
    kubectl get pods -l formal.ai/kubernetes-egress-inject=true --namespace agents \
      -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.initContainers[*].name}{"\n"}{end}'
    ```

    New labeled Pods include a `formal-kubernetes-egress` init container.
  </Step>
</Steps>

Use `helm upgrade --install` for every apply.

### Image pull credentials

The default image is:

`654654333078.dkr.ecr.us-east-1.amazonaws.com/formalco-prod-kubernetes-egress:latest`

On AKS, pull from GAR with
[`azure-gar-cred`](https://github.com/formalco/helm-charts/tree/main/charts/azure-gar-cred).
Set `pullWithCredentials: true` and the GAR repository:

```bash theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
helm upgrade --install formal-kubernetes-egress formal/kubernetes-egress \
  --namespace formal \
  --set pullWithCredentials=true \
  --set sidecar.image.repository=us-docker.pkg.dev/formal-public-assets/formalco-prod-kubernetes-egress/formalco-prod-kubernetes-egress
```

Use `ecr-cred` when you pull Formal ECR images outside AWS.
Do not install both credential charts in the same namespace.

## Add the sidecar to your spec

If you already control the workload chart or manifests, add this native
sidecar to the Pod template instead of using Formal's Helm chart.

```yaml theme={"languages":{"custom":["/languages/cel.json","/languages/rego.json"]}}
spec:
  initContainers:
    - name: formal-kubernetes-egress
      image: 654654333078.dkr.ecr.us-east-1.amazonaws.com/formalco-prod-kubernetes-egress:latest
      imagePullPolicy: Always
      restartPolicy: Always
      env:
        - name: FORMAL_API_KEY
          valueFrom:
            secretKeyRef:
              name: formal-kubernetes-egress
              key: formal-api-key
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
          add: ["NET_ADMIN"]
      volumeMounts:
        - name: formal-ca
          mountPath: /var/lib/formal/ca
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi
  volumes:
    - name: formal-ca
      emptyDir: {}
```

On each application container, mount `formal-ca` at `/formal-ca` and set
the [TLS trust](#tls-trust) environment variables.

## TLS trust

The sidecar intercepts TLS and presents Formal's local CA. Injection mounts
`/formal-ca/localca-cert.pem` on every application container and sets these
environment variables so common HTTP clients trust that CA.

| Variable              | Clients                                |
| --------------------- | -------------------------------------- |
| `SSL_CERT_FILE`       | OpenSSL, curl, Python `ssl`, many CLIs |
| `NIX_SSL_CERT_FILE`   | Nix                                    |
| `NODE_EXTRA_CA_CERTS` | Node.js                                |
| `REQUESTS_CA_BUNDLE`  | Python `requests`, `urllib3`           |
| `PIP_CERT`            | pip                                    |
| `AWS_CA_BUNDLE`       | AWS SDK, boto3                         |
| `GIT_SSL_CAINFO`      | git                                    |
| `CARGO_HTTP_CAINFO`   | cargo                                  |

Runtimes that ignore these variables need their own trust configuration
pointed at `/formal-ca/localca-cert.pem`. That includes Java truststores, Go
binaries that embed a CA pool, and some Python clients that pin
[certifi](https://pypi.org/project/certifi/) (for example httpx).

## Troubleshooting

<AccordionGroup>
  <Accordion title="Sidecar is missing from new Pods">
    **Possible causes:**

    * The Pod spec does not include the sidecar
    * Pod is missing `formal.ai/kubernetes-egress-inject: "true"`
    * The label is present with a different value
    * The label is on the Deployment or Namespace, not the Pod
    * MutatingAdmissionPolicy is not installed

    **Solution:**

    1. Confirm the Pod spec includes `formal-kubernetes-egress`, or confirm the Pod label
    2. For controllers, confirm `spec.template.metadata.labels`
    3. Confirm the admission objects exist if you use the injector
    4. Create a new Pod. Labeling a running Pod does not inject
  </Accordion>

  <Accordion title="Pods fail to start after injection">
    **Possible causes:**

    * Secret `formal-kubernetes-egress` is missing
    * Image pull failed
    * Pod Security Standard drops `NET_ADMIN`

    **Solution:**

    1. Create the Secret in that namespace
    2. Confirm image pull credentials
    3. Use Privileged Pod Security, or exempt the namespace
  </Accordion>

  <Accordion title="Helm install fails on Kubernetes version">
    **Cause:** The chart requires Kubernetes 1.36 or later.\
    **Fix:** Add the sidecar to your spec, or upgrade the control plane
    and reinstall the chart.
  </Accordion>

  <Accordion title="Application TLS handshake fails after injection">
    **Possible causes:**

    * The client ignores the injected CA environment variables
    * The client uses a bundled CA store (Java, certifi, a static Go binary)

    **Solution:**

    1. Point the client at `/formal-ca/localca-cert.pem`
    2. Confirm the container has that file: `kubectl exec` and `ls /formal-ca`
    3. Restart the Pod so it picks up the current admission mutation
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Formal Endpoint" icon="desktop" href="/docs/guides/client-apps/desktop-app">
    See how the Endpoint authenticates and routes traffic
  </Card>

  <Card title="Write Policies" icon="shield-check" href="/docs/guides/policies/introduction">
    Control intercepted egress with Rego policies
  </Card>

  <Card title="View Logs" icon="file-lines" href="/docs/guides/observability/logs">
    Search Endpoint logs for intercepted requests
  </Card>

  <Card title="Machine users" icon="users" href="/docs/guides/core-concepts/identities">
    Manage the identity the sidecar authenticates as
  </Card>
</CardGroup>