OpenTelemetry Delta Temporality and Datadog

When you ship OpenTelemetry metrics to Datadog, the choice between cumulative and delta temporality changes how your dashboards behave. This post covers the practical differences, what Datadog actually stores, and how to query each instrument type correctly.

Temporality

OpenTelemetry supports two temporalities for counters and histograms:

• Cumulative — each export sends the running total since process start. The value only goes up (for monotonic counters).
• Delta — each export sends the change since the last export. The value resets after each push.

Configure delta temporality with a TemporalitySelector on the OTLP exporter:

func deltaSelector(kind sdkmetric.InstrumentKind) metricdata.Temporality {
    switch kind {
    case sdkmetric.InstrumentKindCounter,
        sdkmetric.InstrumentKindHistogram,
        sdkmetric.InstrumentKindObservableCounter:
        return metricdata.DeltaTemporality
    default:
        return metricdata.CumulativeTemporality
    }
}

metricExp, _ := otlpmetrichttp.New(ctx,
    otlpmetrichttp.WithTemporalitySelector(deltaSelector),
)

Gauges always use cumulative temporality — they represent point-in-time state, not accumulations.

What Datadog Actually Stores

In my experience, Datadog appears to store counter data as deltas internally regardless of the temporality you send. When I switched from cumulative to delta, the queried values looked the same — the metric was always represented as a delta, even before the change. This suggests Datadog diffs consecutive cumulative points on ingest to derive the delta.

The advantage of sending delta is avoiding counter-reset artifacts. When a container restarts, a cumulative counter drops from a large value to zero. Datadog must detect and discard the negative diff, which can produce spurious spikes or dips. With delta, each data point is self-contained and container restarts are invisible.

Instrument Types at Export Time

Counter

Each export sends the delta (count since last export), not a running total. In Datadog, use the bare query (no modifier) to see the average count per export interval — this is the primary throughput view. Use .as_count() to see the total count over the rollup bucket — this is useful for spotting bursts where more events landed in a specific bucket, even when the average rate looks smooth.

Be careful with .as_rate() — it computes a per-second rate by dividing the count by the export interval. If your PeriodicReader interval is long (e.g., 60 seconds), you only get one data point per minute, and the per-second rate is a coarse estimate. With a shorter export interval (e.g., 10 seconds), .as_rate() becomes more meaningful. Match your expectations to your export frequency.

Histogram

Every .Record() call increments a bucket count and adds to an exact sum/count. Export sends the bucket counts, sum, count, min, and max. Datadog computes percentiles (p50/p95/p99) per interval from the buckets, and exact averages from sum/count. Individual values are not preserved — you see distributions, not data points.

Do not use .as_rate() or .as_count() on histograms — those are counter-only modifiers. Use the aggregation prefix (avg:, p50:, p95:, max:) to control the view.

Gauge

Only the last .Record() value survives to export. Use for point-in-time state (e.g., buffer depth, queue length), not per-event measurements.

Querying Counters in Datadog

With delta temporality, there is no cumulative reset artifact when a container restarts. However, there is a subtlety with aggregation during container rotation: when a container is replaced, the old container may emit a final low-value data point while the new container ramps up. The default avg aggregation averages these together, which halves the apparent rate.

For timeseries graphs, use max aggregation to avoid this:

max:my.service.requests{$env}

This takes the highest value across containers at each time bucket, which reflects the active container’s actual throughput. Datadog automatically buckets the data points based on the selected timeframe (e.g., 5-minute buckets for a 4-hour view, 20-second buckets for a 1-hour view) and averages values within each bucket. Regardless of which bucket size is in effect, the metric always shows the average count per minute — the bucket size only affects granularity (smoother vs. more detailed), not the unit.

For KPI widgets (query_value), use the last aggregator to display the most recent data point:

max:my.service.requests{$env}  (aggregator: last)

Querying Histograms in Datadog

Use the bare query with an aggregation prefix — no modifiers needed:

avg:my.service.latency_ms{*}     — typical latency
p95:my.service.latency_ms{*}     — worst-case latency (for alerting)
avg:my.service.batch_size{*}     — average batch size
max:my.service.batch_size{*}     — largest batch in the bucket

.as_rate() on a histogram divides the sum by time. For a latency metric, this produces “milliseconds per second” — meaningless. For a batch size metric, it produces “rows per second” — technically valid but misleading; use a counter for throughput instead.

.as_count() on a histogram gives the number of observations (how many events were recorded), not the recorded values.

Summary

Instrument	Temporality	Datadog query	What you see
Counter	Delta	max:metric{*}	Average count per export interval
Counter	Delta	max:metric{*}.as_count()	Total count per rollup bucket
Counter	Delta	max:metric{*}.as_rate()	Per-second rate (only useful with short export intervals)
Histogram	Delta	avg:metric{*}	Average value per bucket
Histogram	Delta	p95:metric{*}	95th percentile per bucket
Histogram	Delta	.as_rate() / .as_count()	Do not use
Gauge	Cumulative	avg:metric{*}	Point-in-time value