OpenTelemetry Cardinality Limits for SREs

Metrics fail quietly when their dimensions get out of control. A request counter can look healthy in aggregate while a route or error attribute undercounts because the SDK folds excess combinations into overflow. The recent OpenTelemetry cardinality limits guide explains where that reliability risk starts.

What Changed

OpenTelemetry metrics SDKs enforce an aggregation cardinality limit for each metric stream. The guide calls out the default limit of 2000 attribute combinations and clarifies what happens after it is reached.

New combinations are not thrown away. Their values are added to an overflow data point marked with otel.metric.overflow=true. Totals stay correct, but the original measurement attributes disappear.

For operators, this distinction matters. A dashboard that shows total HTTP requests may stay accurate. A dashboard grouped by http.route, success, or tenant_id can undercount after overflow starts.

Why SREs Should Care

Cardinality overflow creates a nasty class of observability failure: the telemetry pipeline still works, but some queries become less trustworthy.

The risky cases are familiar:

• Raw URLs instead of route templates
• User IDs, tenant IDs, request IDs, or session IDs on metrics
• Unbounded exception strings or database query text
• Feature flags, regions, and status labels multiplied together without a budget

The behavior is broader than many teams expect. When a measurement overflows, every measurement attribute on that data point is removed. Even success=false can disappear from grouped queries if it rides along with an overflowing combination.

That can affect SLO burn-rate alerts, error-rate dashboards, capacity forecasts, and incident triage. It is especially relevant for multi-tenant platforms where teams are tempted to put tenant identity on every metric.

A Production Checklist

Start with the metrics that drive paging alerts. For each one, write down the dimensions that must remain queryable during an incident.

Then audit instrumentation and Collector pipelines:

# Look for high-cardinality metric attributes in app code and config
rg "tenant_id|user_id|request_id|session_id|url|exception|query" .

# Watch for overflow in your metrics backend
otel.metric.overflow=true

Use OpenTelemetry Views to drop or rename unsafe metric attributes before they hit aggregation. Keep high-cardinality context in traces or logs where it belongs, then correlate from metrics to traces during investigation.

If you raise a cardinality limit, treat it as a capacity change. A per-process SDK limit does not automatically bound backend series growth across a fleet.

Operational Tips

Create separate metric tiers. Paging metrics should use stable dimensions such as route templates, methods, status classes, workload names, and bounded error categories. Exploratory metrics can carry richer labels, but keep them away from critical alerts until their cardinality is understood.

Add an overflow dashboard next to your telemetry ingestion dashboard. If otel.metric.overflow=true appears for a service, make that visible during review. It is usually a modeling issue, not a backend issue.

For AI-assisted SRE workflows, pass overflow state into the incident context. An agent should know when a metric breakdown may be incomplete before it recommends action.

Conclusion

OpenTelemetry cardinality limits are a safety feature, but they are not a promise that every metric query stays valid. SRE teams should monitor overflow, budget dimensions deliberately, and keep high-cardinality context out of paging metrics.

Akmatori helps SRE teams connect alerts, metrics, logs, traces, and runbooks into governed incident workflows. Powered by Gcore, Akmatori gives operators the control they need when telemetry quality decides the next action.