Skip to main content
Your Formal Connector deployment will require different resource requirements depending on the specifics of your setup. Factors like the number of active connections you expect to pass through the Connector at once, the policies you configure, the payload sizes of requests and responses, and more can affect the health of individual Connector instances. There are two ways we help our customers get observability into their Connectors: Logs and Metrics.

Logs

The Connector emits logs to stdout/stderr that can be captured in your log collector of choice. For example, if deploying your Connector on ECS Fargate using Terraform, use logConfiguration to route these logs to be sent to your observability tool of choice:

Metrics

The Connector emits metrics over OTLP/gRPC to an OpenTelemetry Collector such as the Datadog Agent to 127.0.0.1 port 4317 by default. This endpoint can be customized via connector configuration settings. The Connector currently emits the following metrics:
  • formal.connector.instance.heartbeat: A heartbeat gauge metric once every 30 seconds to verify Connector instance uptime.
  • formal.connector.memory.usage_percent: A gauge of how much of the memory reserved by the Go runtime is in use. It is heap plus stack in use (Alloc + StackInuse) divided by total memory obtained from the OS (Sys). It does not use the container memory limit, the host memory, or GOMEMLIMIT, so values of 70-90% are normal even when the container uses little of its limit. To alert or autoscale on memory, use container or host memory usage against its limit instead.
  • formal.connector.connections_opened: A counter that is incremented every time the instance accepts a new TCP connection from a client.
  • formal.connector.connections_closed: A counter that is incremented every time the instance closes a TCP connection from a client.
  • formal.connector.active_connections: A gauge metric of the current number of open TCP client connections that the instance has.
  • formal.connector.resource_health_check: A gauge metric of the latest health check result for each resource. The status="success" or status="failure" series matching the latest result is 1, and the other is 0.

Deploying an OpenTelemetry Collector

By default the Connector sends metrics to an OpenTelemetry Collector at 127.0.0.1 port 4317, so a Collector on the same host works out of the box. For this setup, run the Datadog Agent as a container on that host (so it is reachable over 127.0.0.1) and set the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317 environment variable. You can also configure the Connector to send metrics to a different hostname and port using the Connector configuration settings, either from the Control Plane or with Terraform. In Terraform, use the otel_endpoint_hostname and otel_endpoint_port fields in your formal_connector_configuration resource to specify a custom OpenTelemetry endpoint. If nothing is listening on the configured endpoint, the Connector will refrain from sending metrics entirely. Note: these metrics are not sent over TLS, so make sure the OpenTelemetry endpoint is in your trusted network.

Datadog Dashboard

To visualize connector health in a single place, we provide a template dashboard for you to import into your Datadog instance.
  • Enter a dashboard name.
  • Choose the New Dashboard button.
  • In the new dashboard, choose the Configure button.
  • Choose the Import dashboard JSON button.
  • Upload the dashboard JSON file downloaded in the previous steps.

Grafana Dashboard

If you use Grafana with Prometheus, we provide a template dashboard for you to import into your Grafana instance. This dashboard expects metrics to be collected via an OpenTelemetry Collector configured with a Prometheus exporter, which Prometheus then scrapes.
  • Download the template JSON.
  • Sign in to your Grafana instance.
  • Navigate to Dashboards in the side menu.
  • Choose the New button, then Import.
  • Upload the dashboard JSON file or paste its contents.
  • Select your Prometheus data source when prompted.
  • Choose Import.