Reducing Datadog Custom Metric Ingestion Costs

The Datadog EKS agent can add a lot of high cardinality tags, even when the lowest cardinality settings are enabled. This often leads to high custom metric ingestion costs.

To mitigate this, you can:

1. Run vector.dev with StatsD enabled (recommended approach)
2. Use Stripe Veneur with Datadog exporter

1. Vector.dev with StatsD

Using vector.dev allows you to collect and forward StatsD metrics to Datadog without the extra high-cardinality tags that the default EKS agent adds.

Installation via Helm

Use the following values.yaml configuration, ensuring podHostNetwork is enabled:

role: "Agent"
service:
  enabled: false

podHostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

containerPorts:
  - name: statsd
    containerPort: 9125
    protocol: UDP

env:
  - name: CLUSTER_ENV
    value: dev
  - name: DD_API_KEY
    valueFrom:
      secretKeyRef:
        name: datadog-secret
        key: api-key

customConfig:
  data_dir: /vector-data-dir
  sources:
    statsd:
      type: statsd
      address: "0.0.0.0:9125"
      mode: udp
  transforms:
    add_env_identifier_tags:
      type: remap
      inputs: ["statsd"]
      source: |
        .tags.env = get_env_var!("CLUSTER_ENV")
    aggregate_statsd:
      type: aggregate
      inputs: ["add_env_identifier_tags"]
      interval_ms: 10000   
      mode: Auto

  sinks:
    datadog_metrics:
      type: datadog_metrics
      inputs: ["aggregate_statsd"]
      default_api_key: "${DD_API_KEY}"
      site: "us5.datadoghq.com"
    console:
      type: console
      inputs: ["aggregate_statsd"]
      encoding:
        codec: json

Application Configuration

In your application’s Kubernetes manifest, configure it to send StatsD metrics directly to the Vector agent running on the node:

env:
  - name: DOGSTATSD_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DOGSTATSD_PORT
    value: "9125"

2. Stripe Veneur with Datadog Exporter

As an alternative, you can use Stripe Veneur to collect StatsD metrics and forward them to Datadog.
This approach is similar to vector.dev but requires more setup.

Steps

1. Deploy Veneur as a DaemonSet with hostNetwork enabled.
2. Configure your application to send StatsD metrics to Veneur’s UDP port.
3. In Veneur’s configuration, add a Datadog sink with your API key and site.
4. Apply Kubernetes environment variables to point your application to Veneur.

The key principle is the same — collect StatsD metrics close to the application, strip unnecessary tags, and forward to Datadog.

Summary

• Preferred: Vector.dev with StatsD → Datadog
• Alternative: Stripe Veneur → Datadog
• Both methods reduce ingestion costs by avoiding unnecessary high-cardinality tags from the Datadog EKS agent.