Formal Kubernetes Egress Agent - Formal Documentation

Overview

The Formal Kubernetes Egress Agent enables applying Formal to arbitrary traffic coming from Kubernetes application containers. The Kubernetes Egress Agent operates like the Formal Endpoint but runs as a native sidecar outside of the application container instead of being installed on the host. You can then configure policies against application container traffic like you would for the Formal Endpoint.

Cluster requirements

Your cluster must meet these requirements before you run the sidecar.

Requirement	Why it is required
Kubernetes 1.29+	Native sidecars use `restartPolicy: Always` on an init container.
Sidecar `NET_ADMIN`	The sidecar adds the `NET_ADMIN` Linux capability so it can intercept Pod egress. The cluster does not need a `NET_ADMIN` setting.
Privileged Pod Security Standard	The sidecar runs as root and adds `NET_ADMIN`. Restricted and Baseline Pod Security Standards deny those Pods.
Outbound HTTPS to the Control Plane	The sidecar must reach `https://api.joinformal.com`.
Image pull access	Nodes must pull the Formal Kubernetes Egress image from ECR or GAR.

Verify:

kubectl version

The server version must be 1.29 or later.

Installation

Install the agent as a native sidecar on your application Pod. The recommended path is the Helm chart. It injects the sidecar onto labeled Pods via a MutatingAdmissionPolicy. If you already control the workload spec, you can add the sidecar to your spec instead.

Add the Formal API Key Secret

The sidecar authenticates as a Formal machine user. Create an API key for that user and store it as a Kubernetes Secret in each namespace that runs labeled Pods. The chart does not create this Secret.

Create a machine user

Go to Users and click Create User. Choose Machine. Give the user a name that matches the workload.

Create an API key

Go to API Keys and click Create API Key. Assign the key to that machine user.

Create the Secret

Create formal-kubernetes-egress in every namespace that will run labeled Pods. The sidecar reads FORMAL_API_KEY from the formal-api-key key.

kubectl create secret generic formal-kubernetes-egress \
  --namespace agents \
  --from-literal=formal-api-key=<YOUR_API_KEY>

Verify

kubectl get secret formal-kubernetes-egress --namespace agents

The Secret must exist before you create labeled Pods in that namespace.

Deploy with Helm

The chart is published as kubernetes-egress in the Formal Helm Charts repository. The chart needs Kubernetes 1.36+ and cluster RBAC to manage MutatingAdmissionPolicy and MutatingAdmissionPolicyBinding.

Apply the inject label

The policy binding mutates Pods that have this label set to true.

formal.ai/kubernetes-egress-inject: "true"

Put it on the Pod template (spec.template.metadata.labels) of your Deployment, StatefulSet, DaemonSet, Job, or CronJob:

spec:
  template:
    metadata:
      labels:
        formal.ai/kubernetes-egress-inject: "true"

The policy mutates Pod CREATE only. Labeling an existing Pod does not inject the sidecar. Verify:

kubectl get pods --namespace agents --show-labels
kubectl get pods -l formal.ai/kubernetes-egress-inject=true --all-namespaces

Expected output includes formal.ai/kubernetes-egress-inject=true on the workload Pods. To stop injecting, remove the label from the Pod template and roll the workload. Existing Pods keep their sidecar until you replace them.

Add the Formal Helm repository

helm repo add formal https://formalco.github.io/helm-charts
helm repo update

Install the chart

helm upgrade --install formal-kubernetes-egress formal/kubernetes-egress \
  --namespace formal \
  --create-namespace

The chart installs cluster-scoped admission objects only.

Label Pods to inject

Apply the inject label on the workload Pod template. New Pods created with that label are injected.

Verify

kubectl get mutatingadmissionpolicy formal-kubernetes-egress
kubectl get mutatingadmissionpolicybinding formal-kubernetes-egress
kubectl get pods -l formal.ai/kubernetes-egress-inject=true --namespace agents \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.initContainers[*].name}{"\n"}{end}'

New labeled Pods include a formal-kubernetes-egress init container.

Use helm upgrade --install for every apply.

Image pull credentials

The default image is: 654654333078.dkr.ecr.us-east-1.amazonaws.com/formalco-prod-kubernetes-egress:latest On AKS, pull from GAR with azure-gar-cred. Set pullWithCredentials: true and the GAR repository:

helm upgrade --install formal-kubernetes-egress formal/kubernetes-egress \
  --namespace formal \
  --set pullWithCredentials=true \
  --set sidecar.image.repository=us-docker.pkg.dev/formal-public-assets/formalco-prod-kubernetes-egress/formalco-prod-kubernetes-egress

Use ecr-cred when you pull Formal ECR images outside AWS. Do not install both credential charts in the same namespace.

Add the sidecar to your spec

If you already control the workload chart or manifests, add this native sidecar to the Pod template instead of using Formal’s Helm chart.

spec:
  initContainers:
    - name: formal-kubernetes-egress
      image: 654654333078.dkr.ecr.us-east-1.amazonaws.com/formalco-prod-kubernetes-egress:latest
      imagePullPolicy: Always
      restartPolicy: Always
      env:
        - name: FORMAL_API_KEY
          valueFrom:
            secretKeyRef:
              name: formal-kubernetes-egress
              key: formal-api-key
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
          add: ["NET_ADMIN"]
      volumeMounts:
        - name: formal-ca
          mountPath: /var/lib/formal/ca
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi
  volumes:
    - name: formal-ca
      emptyDir: {}

On each application container, mount formal-ca at /formal-ca and set the TLS trust environment variables.

TLS trust

The sidecar intercepts TLS and presents Formal’s local CA. Injection mounts /formal-ca/localca-cert.pem on every application container and sets these environment variables so common HTTP clients trust that CA.

Variable	Clients
`SSL_CERT_FILE`	OpenSSL, curl, Python `ssl`, many CLIs
`NIX_SSL_CERT_FILE`	Nix
`NODE_EXTRA_CA_CERTS`	Node.js
`REQUESTS_CA_BUNDLE`	Python `requests`, `urllib3`
`PIP_CERT`	pip
`AWS_CA_BUNDLE`	AWS SDK, boto3
`GIT_SSL_CAINFO`	git
`CARGO_HTTP_CAINFO`	cargo

Runtimes that ignore these variables need their own trust configuration pointed at /formal-ca/localca-cert.pem. That includes Java truststores, Go binaries that embed a CA pool, and some Python clients that pin certifi (for example httpx).

Troubleshooting

Sidecar is missing from new Pods

Possible causes:

The Pod spec does not include the sidecar
Pod is missing formal.ai/kubernetes-egress-inject: "true"
The label is present with a different value
The label is on the Deployment or Namespace, not the Pod
MutatingAdmissionPolicy is not installed

Solution:

Confirm the Pod spec includes formal-kubernetes-egress, or confirm the Pod label
For controllers, confirm spec.template.metadata.labels
Confirm the admission objects exist if you use the injector
Create a new Pod. Labeling a running Pod does not inject

Pods fail to start after injection

Possible causes:

Secret formal-kubernetes-egress is missing
Image pull failed
Pod Security Standard drops NET_ADMIN

Solution:

Create the Secret in that namespace
Confirm image pull credentials
Use Privileged Pod Security, or exempt the namespace

Helm install fails on Kubernetes version

Cause: The chart requires Kubernetes 1.36 or later.
Fix: Add the sidecar to your spec, or upgrade the control plane and reinstall the chart.

Application TLS handshake fails after injection

Possible causes:

The client ignores the injected CA environment variables
The client uses a bundled CA store (Java, certifi, a static Go binary)

Solution:

Point the client at /formal-ca/localca-cert.pem
Confirm the container has that file: kubectl exec and ls /formal-ca
Restart the Pod so it picks up the current admission mutation

Next Steps

Formal Endpoint

See how the Endpoint authenticates and routes traffic

Write Policies

Control intercepted egress with Rego policies

View Logs

Search Endpoint logs for intercepted requests

Machine users

Manage the identity the sidecar authenticates as

​Overview

​Cluster requirements

​Installation

​Add the Formal API Key Secret

​Deploy with Helm

​Apply the inject label

​Image pull credentials

​Add the sidecar to your spec

​TLS trust

​Troubleshooting

​Next Steps

Formal Endpoint

Write Policies

View Logs

Machine users

Overview

Cluster requirements

Installation

Add the Formal API Key Secret

Deploy with Helm

Apply the inject label

Image pull credentials

Add the sidecar to your spec

TLS trust

Troubleshooting

Next Steps