Log Integration

Overview

Formal can forward all activity logs to your SIEM, data lake, or observability platform. This enables centralized log management, long-term retention, compliance reporting, and integration with your existing security tools.

Supported Platforms

AWS S3
Google Cloud Storage (GCS)
Azure Blob Storage
Datadog
Splunk
Panther

By default, Formal forwards all logs to your configured destination, including Insights findings.

Setup

AWS S3

The AWS S3 log integration requires an AWS Cloud Integration with S3 access enabled.

Set Up AWS Integration

First, configure an AWS Cloud Integration with:

AllowS3Access: true
S3BucketARN: Your S3 bucket ARN

Navigate to SIEM

Go to SIEM

Create Integration

Click Create Integration

Select AWS S3

Choose AWS S3 as your provider

Configure

S3 Bucket Name: Your bucket name
Cloud Integration: Select your AWS Cloud Integration
Compression: the codec log objects are written with (see Compression)

Terraform

# First, create S3 bucket
resource "aws_s3_bucket" "formal_logs" {
  bucket = "formal-connector-logs"
}

# Create AWS Cloud Integration
resource "formal_integration_cloud" "aws" {
  name         = "aws-integration"
  cloud_region = "us-east-1"

  aws {
    template_version    = "1.2.0"
    allow_s3_access     = true
    s3_bucket_arn       = "${aws_s3_bucket.formal_logs.arn}/*"
  }
}

# Deploy CloudFormation stack (see AWS Integration guide)
resource "aws_cloudformation_stack" "formal" {
  # ... (see AWS Integration docs)
}

# Create log integration
resource "formal_integration_log" "s3" {
  name = "s3-log-drain"

  aws_s3 {
    s3_bucket_name       = aws_s3_bucket.formal_logs.bucket
    cloud_integration_id = formal_integration_cloud.aws.id
    compression          = "gzip"
  }
}

Google Cloud Storage

The GCS log integration requires a GCP Cloud Account with Cloud Storage access enabled.

Set Up GCP Integration

First, configure a GCP Cloud Account with:

allow_gcs_access: true
gcs_buckets: the buckets Formal may write to (empty allows every bucket in the project)

Navigate to SIEM

Go to SIEM

Create Integration

Click Create Integration

Select Google Cloud Storage

Choose Google Cloud Storage as your provider

Configure

GCS Bucket Name: your bucket name
Cloud Account: select your GCP Cloud Account
Compression: the codec log objects are written with (see Compression)

Terraform

# Create GCP Cloud Account with Cloud Storage access (see GCP Integration guide)
resource "formal_integration_cloud" "gcp" {
  name = "gcp-integration"

  gcp {
    project_id                              = "my-gcp-project"
    enable_compute_instances_autodiscovery  = true
    enable_gke_clusters_autodiscovery       = true
    enable_cloudsql_instances_autodiscovery = true
    allow_gcs_access                        = true
  }
}

# Create log integration
resource "formal_integration_log" "gcs" {
  name = "gcs-log-drain"

  gcs {
    cloud_integration_id = formal_integration_cloud.gcp.id
    gcs_bucket_name      = "my-formal-logs"
    compression          = "gzip"
  }
}

Creating the GCP Cloud Account also requires the terraform-formal-gcp module and an activation resource. See the GCP Integration guide for the full setup.

Azure Blob Storage

The Blob Storage log integration requires an Azure Cloud Account with Blob Storage access enabled.

Set Up Azure Integration

First, configure an Azure Cloud Account with:

allow_blob_access: true
blob_storage_accounts: the storage accounts Formal may write to (empty allows every account in scope)

Navigate to SIEM

Go to SIEM

Create Integration

Click Create Integration

Select Azure Blob Storage

Choose Azure Blob Storage as your provider

Configure

Storage Account: the account to write to. Accounts named on the Cloud Account are offered as a list.
Container: the container within that account
Cloud Account: select your Azure Cloud Account
Compression: the codec log objects are written with (see Compression)

The Terraform provider does not yet support Blob Storage log integrations. Create them from the Control Plane or the API.

Datadog

Get Datadog Credentials

From your Datadog account, retrieve:

Application Key
API Key
Site (e.g., datadoghq.com, datadoghq.eu)

Navigate to SIEM

Go to SIEM

Create Integration

Click Create Integration

Select Datadog

Choose Datadog as your provider

Enter Credentials

Application Key: Your Datadog Application Key
API Key: Your Datadog API key
Site: Your Datadog site

Terraform

resource "formal_integration_log" "datadog" {
  name = "datadog-log-drain"

  datadog {
    account_id = var.datadog_application_key
    api_key    = var.datadog_api_key
    site       = "datadoghq.com"  # or datadoghq.eu, etc.
  }
}

Datadog Dashboard

Import this template JSON for a sample of some of the analyses you can do with the Datadog log integration.

Splunk

Create Splunk HEC Token

In Splunk, create a new HTTP Event Collector (HEC) token

Navigate to SIEM

Go to SIEM

Create Integration

Click Create Integration

Select Splunk

Choose Splunk as your provider

Enter Configuration

Access Token: Your HEC token
Host: Your Splunk instance hostname
Port: HEC port (usually 8088)

Terraform

resource "formal_integration_log" "splunk" {
  name = "splunk-log-drain"

  splunk {
    access_token = var.splunk_hec_token
    host         = "splunk.example.com"
    port         = 8088
  }
}

Panther

Custom.Formal.Logs schema

Compression

Log objects written to AWS S3, Google Cloud Storage, and Azure Blob Storage can be sent as compressed files.

Codec	Object name	Notes
`none`	`.json`	Default. Plain JSON.
`gzip`	`.json.gz`	Widest tool support, and the format most log pipelines expect.
`zstd`	`.json.zst`	Smaller objects and faster decompression, with narrower support.

Compression applies per log object, and Formal currently writes one object per log line. Compression reduces stored bytes but not the number of objects.

Formal records the codec in the object name and leaves the Content-Encoding metadata unset.

Object Storage Layout

For AWS S3, Google Cloud Storage, and Azure Blob Storage, Formal writes objects under:

formal/logs/{source}/{year}/{month}/{day}/{file}

{source} is the Connector’s name for Connector logs. For other logs, it is desktop, control-plane, workflow, or insights. Datadog and Splunk receive the same value as the service name. See Control Plane Logs for Control Plane-specific details.

Use Cases

Compliance and Auditing

Forward logs to long-term storage for compliance requirements:

# Archive to S3 for 7 years (SOC 2, HIPAA, etc.)
resource "formal_integration_log" "compliance_archive" {
  name = "compliance-s3-archive"

  aws_s3 {
    s3_bucket_name       = aws_s3_bucket.compliance_logs.bucket
    cloud_integration_id = formal_integration_cloud.aws.id
  }
}

# Configure S3 lifecycle policy
resource "aws_s3_bucket_lifecycle_configuration" "compliance" {
  bucket = aws_s3_bucket.compliance_logs.id

  rule {
    id     = "archive-old-logs"
    status = "Enabled"

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 2555  # 7 years
    }
  }
}

Real-Time Security Monitoring

Send logs to your SIEM for real-time threat detection:

# Send to Datadog for real-time monitoring
resource "formal_integration_log" "security_monitoring" {
  name = "datadog-security"

  datadog {
    account_id = var.datadog_application_key
    api_key    = var.datadog_api_key
    site       = "datadoghq.com"
  }
}

Create alerts in Datadog for:

Failed authentication attempts
Policy violations
Unusual query patterns
Off-hours access

Data Lake Integration

Forward logs to your data lake for analytics:

# Send to S3 data lake
resource "formal_integration_log" "data_lake" {
  name = "data-lake-integration"

  aws_s3 {
    s3_bucket_name       = "my-data-lake-formal-logs"
    cloud_integration_id = formal_integration_cloud.aws.id
  }
}

Then use Athena, Redshift Spectrum, or Databricks to analyze:

User access patterns
Query performance
Policy effectiveness
Resource utilization

Multi-Destination Forwarding

Send logs to multiple destinations:

# Real-time monitoring
resource "formal_integration_log" "splunk_realtime" {
  name = "splunk-realtime"
  splunk {
    access_token = var.splunk_hec_token
    host         = "splunk.example.com"
    port         = 8088
  }
}

# Long-term archive
resource "formal_integration_log" "s3_archive" {
  name = "s3-archive"
  aws_s3 {
    s3_bucket_name       = "formal-logs-archive"
    cloud_integration_id = formal_integration_cloud.aws.id
  }
}

# Security analytics
resource "formal_integration_log" "datadog_security" {
  name = "datadog-security"
  datadog {
    account_id = var.datadog_application_key
    api_key    = var.datadog_api_key
    site       = "datadoghq.com"
  }
}

Getting Started

Example Use Cases

AI Governance

Configuring Formal

Integrations

Configuration

Overview

Supported Platforms

Setup

AWS S3

Terraform

Google Cloud Storage

Terraform

Azure Blob Storage

Datadog

Terraform

Datadog Dashboard

Splunk

Terraform

Panther

Compression

Object Storage Layout

Use Cases

Compliance and Auditing

Real-Time Security Monitoring

Data Lake Integration

Multi-Destination Forwarding

Next Steps

Cloud Accounts

View Logs

​Overview

​Supported Platforms

​Setup

​AWS S3

​Terraform

​Google Cloud Storage

​Terraform

​Azure Blob Storage

​Datadog

​Terraform

​Datadog Dashboard

​Splunk

​Terraform

​Panther

​Compression

​Object Storage Layout

​Use Cases

​Compliance and Auditing

​Real-Time Security Monitoring

​Data Lake Integration

​Multi-Destination Forwarding

​Next Steps

Cloud Accounts

View Logs

Overview

Supported Platforms

Setup

AWS S3

Terraform

Google Cloud Storage

Terraform

Azure Blob Storage

Datadog

Terraform

Datadog Dashboard

Splunk

Terraform

Panther

Compression

Object Storage Layout

Use Cases

Compliance and Auditing

Real-Time Security Monitoring

Data Lake Integration

Multi-Destination Forwarding

Next Steps