opentelemetry source histogram discrepancy

[tony-resendiz]

Question

hello, we recently ran into an issue when testing histogram metrics through the opentelemetry source component.

we have this client code (based on these docs):

import time

from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.metrics import Observation
from opentelemetry.sdk.metrics import (
    Counter,
    Histogram,
    # _Gauge,
    MeterProvider,
    ObservableCounter,
    ObservableGauge,
    ObservableUpDownCounter,
    UpDownCounter,
)
from opentelemetry.sdk.metrics.export import (
    AggregationTemporality,
    PeriodicExportingMetricReader,
)

load_dotenv()

exporter = OTLPMetricExporter(
    endpoint="http://vector_collector_default:4318/v1/metrics",
    preferred_temporality={
        Counter: AggregationTemporality.DELTA,
        UpDownCounter: AggregationTemporality.CUMULATIVE,
        # Gauge was missing in their recommendation docs - can add this based on ObservableGauge
        # Python SDK maybe prefers ObservableGauge (_Gauge is protected) so we'll try that instead
        # _Gauge: AggregationTemporality.CUMULATIVE,
        Histogram: AggregationTemporality.DELTA,
        ObservableCounter: AggregationTemporality.DELTA,
        ObservableUpDownCounter: AggregationTemporality.CUMULATIVE,
        ObservableGauge: AggregationTemporality.CUMULATIVE,
    },
)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=5_000)
provider = MeterProvider(metric_readers=[reader])

meter = provider.get_meter("my-meter")
counter = meter.create_counter("test.recommended.vector.counter")
updown = meter.create_up_down_counter("test.recommended.vector.updown")
observableGauge = meter.create_observable_gauge(
    name="test.recommended.vector.gauge",
    callbacks=[lambda options: [Observation(1, {})]],
)
histogram = meter.create_histogram("test.recommended.vector.histogram")

while True:
    # observable gauge uses callback, so no need to set
    counter.add(1)
    updown.add(1)
    histogram.record(1)

    time.sleep(2)

and vector_collector_default service has this config:

[log_schema]
level_key = "status"
level = "info"

[api]
enabled = true
address = "0.0.0.0:8787"
playground = false

[sources.opentelemetry]
type = "opentelemetry"
grpc.address = "0.0.0.0:4317"
http.address = "0.0.0.0:4318"

[sinks.datadog_metrics_sink]
type = "datadog_metrics"
inputs = ["opentelemetry.metrics"]
default_api_key = "${DD_API_KEY}"
endpoint = "https://metrics.agent.datadoghq.com"

the result is an average of ~1.5 for the histogram in datadog:

tapping the inputs of the datadog metrics sink the values seemed normal:

{
  "name": "test.recommended.vector.histogram",
  "tags": {
    "resource.service.name": "unknown_service",
    "resource.telemetry.sdk.language": "python",
    "resource.telemetry.sdk.name": "opentelemetry",
    "resource.telemetry.sdk.version": "1.38.0",
    "scope.name": "my-meter"
  },
  "timestamp": "2025-11-19T23:01:41.664304624Z",
  "kind": "incremental",
  "aggregated_histogram": {
    "buckets": [
      {
        "upper_limit": 0.0,
        "count": 0
      },
      {
        "upper_limit": 5.0,
        "count": 2
      },
      {
        "upper_limit": 10.0,
        "count": 0
      },
      {
        "upper_limit": 25.0,
        "count": 0
      },
      {
        "upper_limit": 50.0,
        "count": 0
      },
      {
        "upper_limit": 75.0,
        "count": 0
      },
      {
        "upper_limit": 100.0,
        "count": 0
      },
      {
        "upper_limit": 250.0,
        "count": 0
      },
      {
        "upper_limit": 500.0,
        "count": 0
      },
      {
        "upper_limit": 750.0,
        "count": 0
      },
      {
        "upper_limit": 1000.0,
        "count": 0
      },
      {
        "upper_limit": 2500.0,
        "count": 0
      },
      {
        "upper_limit": 5000.0,
        "count": 0
      },
      {
        "upper_limit": 7500.0,
        "count": 0
      },
      {
        "upper_limit": 10000.0,
        "count": 0
      },
      {
        "upper_limit": "inf",
        "count": 0
      }
    ],
    "count": 2,
    "sum": 2.0
  }
}

i found this discussion about using protobuf and vrl to remap the event so i tweaked the client code slightly and targeted a different vector using an otel sink (targeting datadog otlp metric intake) with this config:

[log_schema]
level_key = "status"
level = "info"

[api]
enabled = true
address = "0.0.0.0:8787"
playground = false

[sources.opentelemetry]
type = "opentelemetry"
grpc.address = "0.0.0.0:4317"
http.address = "0.0.0.0:4318"

[transforms.convert_metric_to_log]
type   = "metric_to_log"
inputs = ["opentelemetry.metrics"]

[transforms.route_metrics]
type = "exclusive_route"
inputs = ["convert_metric_to_log"]

    [[transforms.route_metrics.routes]]
    name = "histogram"
    condition = 'exists(.aggregated_histogram)'

    [[transforms.route_metrics.routes]]
    name = "gauge"
    condition = 'exists(.gauge)'

    [[transforms.route_metrics.routes]]
    name = "counter"
    condition = 'exists(.counter)'

[transforms.remap_histogram]
type = "remap"
inputs = ["route_metrics.histogram"]
source = '''
# pull out tags
attrs = []
for_each(object!(.tags)) -> |k, v| {
  attrs = push(attrs, {
    "key": k,
    "value": { "stringValue": to_string!(v) }
  })
}

# get bucket counts and explicit bounds
bucket_counts = []
explicit_bounds = []
for_each(array!(.aggregated_histogram.buckets)) -> |_index, bucket| {
  if (bucket.upper_limit != "inf") {
    bucket_counts = push(bucket_counts, bucket.count)
    explicit_bounds = push(explicit_bounds, bucket.upper_limit)
  }
}

# get count and sum
count = .aggregated_histogram.count
sum = .aggregated_histogram.sum

# timestamp
unix_timestamp = to_unix_timestamp!(.timestamp, unit: "nanoseconds")

# fill in the otel histogram format
. = {
  "resourceMetrics": [
    {
      "resource": {
        "attributes": attrs
      },
      "scopeMetrics": [
        {
          "metrics": [
            {
              "histogram": {
                # TODO: verify CUMULATIVE vs DELTA
                # this needs to be set to DELTA, if histograms are sent as CUMULATIVE it will be rejected by Datadog
                "aggregationTemporality": "AGGREGATION_TEMPORALITY_DELTA",
                "dataPoints": [
                  {
                    "bucketCounts": bucket_counts,
                    "explicitBounds": explicit_bounds,
                    # TODO: min and max aren't included in the vector metric output
                    # see if there is an impact when these aren't included
                    # "min": 1.0,
                    # "max": 1.0,
                    "count": count,
                    "sum": sum,
                    # these timestamps are required or metric doesn't appear
                    "startTimeUnixNano": unix_timestamp,
                    "timeUnixNano": unix_timestamp
                  }
                ]
              },
              "name": .name
            }
          ],
          "scope": {
            "name": "my-meter"
          }
        }
      ]
    }
  ],
  "timestamp": .timestamp
}
'''

[transforms.remap_counter]
type = "remap"
inputs = ["route_metrics.counter"]
source = '''
# pull out tags
attrs = []
for_each(object!(.tags)) -> |k, v| {
  attrs = push(attrs, {
    "key": k,
    "value": { "stringValue": to_string!(v) }
  })
}

# timestamp
unix_timestamp = to_unix_timestamp!(.timestamp, unit: "nanoseconds")

. = {
  "resourceMetrics": [
    {
      "resource": {
        "attributes": attrs
      },
      "scopeMetrics": [
        {
          "metrics": [
            {
              "name": .name,
              "sum": {
                "aggregationTemporality": "AGGREGATION_TEMPORALITY_DELTA",
                "dataPoints": [
                  {
                    "asInt": to_int!(.counter.value),
                    "startTimeUnixNano": unix_timestamp,
                    "timeUnixNano": unix_timestamp
                  }
                ],
                "isMonotonic": true
              }
            },
          ],
          "scope": {
            "name": "my-meter"
          }
        }
      ]
    }
  ],
  "timestamp": .timestamp
}
'''

[transforms.remap_gauge]
type = "remap"
inputs = ["route_metrics.gauge"]
source = '''
# pull out tags
attrs = []
for_each(object!(.tags)) -> |k, v| {
  attrs = push(attrs, {
    "key": k,
    "value": { "stringValue": to_string!(v) }
  })
}

# timestamp
unix_timestamp = to_unix_timestamp!(.timestamp, unit: "nanoseconds")

. = {
  "resourceMetrics": [
    {
      "resource": {
        "attributes": attrs
      },
      "scopeMetrics": [
        {
          "metrics": [
            {
              "gauge": {
                "dataPoints": [
                  {
                    "asInt": to_int!(.gauge.value),
                    "timeUnixNano": unix_timestamp
                  }
                ]
              },
              "name": .name
            }
          ],
          "scope": {
            "name": "my-meter"
          }
        }
      ]
    }
  ],
  "timestamp": .timestamp
}
'''

[sinks.otlp_metrics_intake_sink]
type   = "opentelemetry"
inputs = ["remap_histogram", "remap_counter", "remap_gauge"]
protocol.type = "http"
protocol.uri  = "https://otlp.datadoghq.com/v1/metrics"
protocol.method = "post"
protocol.encoding.codec = "otlp"
protocol.headers."dd-api-key" = "${DD_API_KEY}"
protocol.headers."content-type" = "application/x-protobuf"
protocol.headers."dd-otel-metric-config" = '{"resource_attributes_as_tags": true, "instrumentation_scope_metadata_as_tags": true, "histograms": {"mode": "distributions", "send_aggregation_metrics": true}}'

this results in the desired avg of 1.0

thinking it could be issue with the dd apis, i opened a support ticket with them but they redirected me here. any ideas if this is an issue with the dd metrics sink? or something else entirely? thanks for your time.

Vector Config

No response

Vector Logs

No response

[pront]

Thanks for the thorough writeup and the workaround, that's very useful for pinning this down. I was OOO when this was originally reported.

Closing since this is covered by #24365