Turn on an AI coding agent’s telemetry and it throws a wall of text at you. It’s expensive to store and what’s worse, you can’t make sense of it.
It’s all OpenTelemetry native, though. So you can shape it in the Collector before sending it to a telemetry backend. Which leaves one question. What do you actually need?
The requirement isn’t to collect less, it’s to keep what answers questions
I gave Claude Code a small task. Fix two failing tests, add a few boundary tests. It did. With every content flag on, that one run produced 181 telemetry records and 289 KB of JSON.
What I needed was answers to these six questions:
| Question | What answers it |
|---|---|
| What is this costing, per model? | claude_code.cost.usage |
| Tokens per session, and how much is cache? | claude_code.token.usage by type |
| Which tools fail, and how often? | tool_result events, success |
| Are people rejecting the agent’s edits? | tool_decision events |
| Where is the latency? | llm_request spans, ttft_ms |
| What broke? | api_error events, error_class |
Every answer is a counter or a short attribute. I don’t need prompts, file bodies or diffs. If no dashboard, alert, or query reads a field, it isn’t observability. It’s storage you’re paying to index.
Where the volume actually lives
An agent runs a loop: Prompt, model call, tool call, result, decide, repeat. One interaction is a claude_code.interaction span with an llm_request child per model turn and a tool child per tool call. With OTEL_LOG_TOOL_CONTENT on, each tool span gets a tool.output event holding whatever the tool returned.
That’s a directory listing. When the tool is Read, the event is the file. A 40 KB file becomes a 40 KB span event.
Logs have the same shape. tool_result and tool_decision events carry tool_parameters, the full arguments to the tool. For a Write that’s the whole file. Same content as the span event, stored twice.
Metrics carry identity instead. Every data point has session.id, user.id, user.email and two account ids. Each new session created a fresh set of time series that never goes away. Twenty five sessions gave me 300 series for eight metrics.
The default instrumentation is written for debugging one agent run, not for running a fleet. Tool content lands in traces and logs both. Metrics are keyed on attributes that belong on the resource, or nowhere. None of that is wrong for a single developer watching a single session. It’s wrong at 200 sessions a day. The agent won’t change its defaults for you, so the pipeline has to.
What we’re building
One OTLP receiver that’ll ingest Claude Code’s internal telemetry. Every signal is fanned into two pipelines, a raw one and an optimized one, each with its own exporter, all six landing in the same Grafana LGTM container.
The raw pipeline is the control group. It carries a service.name and a batch processor. Nothing else. The optimized pipeline carries the processors we’ll add in this tutorial. The same receiver feeds both, so both paths see byte identical input.
That’s what makes the comparison honest. Both data sets land in Grafana. One dashboard reads both and draws them side by side, red for raw, green for optimized. Cardinality, bytes per minute, record counts. The gap between the lines is the storage you stop paying for.
In production you’d point the raw exporters at object storage, or drop them altogether. Here they point at Grafana because I want to show you the difference.
# docker-compose.yaml
services:
lgtm:
image: grafana/otel-lgtm:0.33.1
ports:
- "3000:3000"
collector:
image: otel/opentelemetry-collector-contrib:0.161.0
depends_on: [lgtm]
volumes:
- ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "4317:4317"
- "4318:4318"
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp/lgtm_raw_logs:
endpoint: lgtm:4317
tls:
insecure: true
sending_queue:
enabled: true
otlp/lgtm_opt_logs:
endpoint: lgtm:4317
tls:
insecure: true
sending_queue:
enabled: true
# four more exactly like this:
# otlp/lgtm_raw_metrics, otlp/lgtm_raw_traces,
# otlp/lgtm_opt_metrics, otlp/lgtm_opt_traces
service:
pipelines:
logs/raw:
receivers: [otlp]
processors: [batch]
exporters: [otlp/lgtm_raw_logs]
logs/optimized:
receivers: [otlp]
processors: [batch] # the reduction goes here
exporters: [otlp/lgtm_opt_logs]
# four more exactly like this:
# metrics/raw, metrics/optimized, traces/raw, traces/optimized
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_ASSISTANT_RESPONSES=1
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_LOG_TOOL_CONTENT=1Step 2. Logs: Drop, delete, trim, hash
Filter. api_request carries a model name and a request id. Everything else on it lives in other signals. Drop the whole event. Roughly a quarter of log records.
processors:
filter/drop_api_request:
error_mode: ignore
logs:
log_record:
- 'attributes["event.name"] == "api_request"'Delete. Six attributes off every record. tool_parameters is the big one. prompt_length and response_length stay, so you still see size trends without the text.
transform/logs_delete_fields:
error_mode: ignore
log_statements:
- context: log
statements:
- delete_key(log.attributes, "prompt")
- delete_key(log.attributes, "response")
- delete_key(log.attributes, "tool_parameters")
- delete_key(log.attributes, "workspace.host_paths")
- delete_key(log.attributes, "user.email")
- delete_key(log.attributes, "terminal.type")
Hash and trim. Hash session.id and user.id, cap error at 256 characters.
transform/logs_hash_and_trim:
error_mode: ignore
log_statements:
- context: log
statements:
- set(log.attributes["session.id"], Substring(SHA256(log.attributes["session.id"]), 0, 16)) where log.attributes["session.id"] != nil
- set(log.attributes["user.id"], Substring(SHA256(log.attributes["user.id"]), 0, 16)) where log.attributes["user.id"] != nil
- set(log.attributes["error"], Substring(log.attributes["error"], 0, 256)) where log.attributes["error"] != nil and Len(log.attributes["error"]) > 256
Truncate the digest. A full SHA256 is 64 characters replacing a 36 character UUID, twice per record. The first time I ran this, log bytes went up. Sixteen hex characters is plenty.
logs/optimized:
receivers: [otlp]
processors: [filter/drop_api_request, transform/logs_delete_fields, transform/logs_hash_and_trim, batch]
exporters: [otlp/lgtm_opt_logs]Log bytes roughly halve.
Step 3. Metrics: Delete the dimension
The problem here is cardinality, not bytes. Delete six attributes from every data point and 300 series become 12 regardless if the content flags are on or off.
transform/metrics_delete_fields:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
- delete_key(datapoint.attributes, "session.id")
- delete_key(datapoint.attributes, "user.account_uuid")
- delete_key(datapoint.attributes, "user.account_id")
- delete_key(datapoint.attributes, "user.id")
- delete_key(datapoint.attributes, "user.email")
- delete_key(datapoint.attributes, "terminal.type")Sum the sessions, then convert. Claude Code exports delta temporality and Prometheus wants cumulative, so delta_to_cumulative has to sit at the end of this pipeline. With session.id gone, every running session writes into the same twelve series, and the converter only accepts points in timestamp order. Sum the sessions per window first.
# 1. Collect two seconds of exports. The size is out of reach on purpose,
# so the timeout is always what flushes.
batch/metrics_window:
send_batch_size: 1000000
send_batch_max_size: 0
timeout: 2s
# 2. Put every metric of the same name under one resource.
groupbyattrs/compact: {}
# 3. Give every point in the window the same timestamp, then sum the points
# whose attributes are now identical.
transform/metrics_merge_sessions:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
- set(datapoint.time, TruncateTime(Now(), Duration("1s")))
- set(datapoint.start_time, datapoint.time)
- context: metric
statements:
- aggregate_on_attributes("sum")
# 4. Only now convert. On the raw pipeline too, or the raw side never
# reaches Prometheus and there is nothing to compare against.
delta_to_cumulative:
max_stale: 5m
metrics/optimized:
receivers: [otlp]
processors: [transform/metrics_delete_fields, batch/metrics_window, groupbyattrs/compact, transform/metrics_merge_sessions, delta_to_cumulative, batch]
exporters: [otlp/lgtm_opt_metrics]
Batch and wait two seconds, give everything one timestamp, add up the points, then convert. Optimized totals match raw to within half a percent.
Note: This only works if one Collector sees every session, so behind a load balancer it belongs on a single gateway.
Metric bytes fall by more than any other signal. Six attributes on every point was most of the payload. And, removing the session.id keeps the cardinality in check.
Step 4. Traces: Trim, don’t sample
Traces are the biggest stream and the one people want to sample first. I’ll use one filter and two transforms instead.
Filter. Drop tool.execution when the tool succeeded. The parent span already has tool_name, duration_ms and result_tokens. The child is only valuable when it fails. This alone removes a fifth of the trace bytes.
filter/drop_successful_tool_execution:
error_mode: ignore
traces:
span:
- 'name == "claude_code.tool.execution" and attributes["success"] == true and attributes["error_class"] == nil'
Span context. Delete what repeats, hash what identifies.
transform/traces_span:
error_mode: ignore
trace_statements:
- context: span
statements:
- delete_key(span.attributes, "user_prompt") where span.name == "claude_code.interaction"
- delete_key(span.attributes, "user.email")
- delete_key(span.attributes, "terminal.type")
- delete_key(span.attributes, "user.account_uuid")
- delete_key(span.attributes, "user.account_id")
- delete_key(span.attributes, "organization.id")
- delete_key(span.attributes, "span.type")
- delete_key(span.attributes, "tool_use_id")
- delete_key(span.attributes, "request_id")
- delete_key(span.attributes, "model")
- delete_key(span.attributes, "stop_reason")
- delete_key(span.attributes, "full_command") where span.attributes["full_command"] != nil and IsMatch(span.attributes["full_command"], "(sk-ant-|gh[pousr]_|AKIA[0-9A-Z]{16}|(?i)(password|secret|token)\\s*=)")
- set(span.attributes["user.id"], Substring(SHA256(span.attributes["user.id"]), 0, 16)) where span.attributes["user.id"] != nil
- set(span.attributes["session.id"], Substring(SHA256(span.attributes["session.id"]), 0, 16)) where span.attributes["session.id"] != nil
Twelve deletes. tool_use_id, request_id, model and stop_reason each have a twin in gen_ai.tool.call.id, gen_ai.response.id, gen_ai.request.model and gen_ai.response.finish_reasons. Keep the gen_ai.* semconv names. A dashboard that has never heard of Claude Code still understands those. The full_command statement is the one secret guard, on the one field that can carry a secret.
Span event context. Cap content, output and diff at 512 characters, move them onto the span, and add content_truncated_by_pipeline so nobody mistakes a cut body for a short one.
transform/traces_spanevent:
error_mode: ignore
trace_statements:
- context: spanevent
statements:
- set(span.attributes["content_truncated_by_pipeline"], true) where spanevent.attributes["content"] != nil and Len(spanevent.attributes["content"]) > 512
- set(span.attributes["content"], Substring(spanevent.attributes["content"], 0, 512)) where spanevent.attributes["content"] != nil and Len(spanevent.attributes["content"]) > 512
- delete_key(spanevent.attributes, "content")
# the same three statements for "output" and for "diff"Every span survives. The ls -la stays readable. A 40 KB file read does not, and that’s the point.
traces/optimized:
receivers: [otlp]
processors: [filter/drop_successful_tool_execution, transform/traces_span, transform/traces_spanevent, batch]
exporters: [otlp/lgtm_opt_traces]
Trace bytes roughly halve without sampling.
Sampling would cut further. I left it out because it trades away the one thing traces are for, pulling up the specific run someone is asking about. Agent interactions also run for minutes, and tail_sampling decides after decision_wait, so a tool that fails at minute three arrives after the verdict. If truncation isn’t enough, this is the next step. This block below keeps errors, slow traces, and 10 percent of the rest.
tail_sampling:
decision_wait: 120s
num_traces: 50000
decision_cache:
sampled_cache_size: 100000
non_sampled_cache_size: 100000
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: slow
type: latency
latency:
threshold_ms: 60000
- name: baseline
type: probabilistic
probabilistic:
sampling_percentage: 10
Step 5. Prove nothing you needed was lost
The objection is always the same.
“We lose data.”
Let’s check. The Collector counts its own bytes. Turn on detailed self telemetry and send it to the same Grafana.
service:
telemetry:
metrics:
level: detailed
readers:
- periodic:
interval: 10000
exporter:
otlp:
protocol: grpc
endpoint: http://lgtm:4317
insecure: true
Name the two paths and tag the metric points so the queries can tell them apart. Four processor definitions, then every pipeline picks up the ones it needs.
resource/name_raw:
attributes:
- {key: service.name, value: claude-code-raw, action: upsert}
resource/name_optimized:
attributes:
- {key: service.name, value: claude-code-optimized, action: upsert}
transform/tag_raw:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
- set(datapoint.attributes["pipeline"], "raw")
transform/tag_optimized:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
- set(datapoint.attributes["pipeline"], "optimized")
service:
pipelines:
logs/raw:
receivers: [otlp]
processors: [resource/name_raw, batch]
exporters: [otlp/lgtm_raw_logs]
logs/optimized:
receivers: [otlp]
processors: [filter/drop_api_request, transform/logs_delete_fields, transform/logs_hash_and_trim, resource/name_optimized, batch]
exporters: [otlp/lgtm_opt_logs]
metrics/raw:
receivers: [otlp]
processors: [transform/tag_raw, resource/name_raw, delta_to_cumulative, batch]
exporters: [otlp/lgtm_raw_metrics]
metrics/optimized:
receivers: [otlp]
processors: [transform/metrics_delete_fields, transform/tag_optimized, resource/name_optimized, batch/metrics_window, groupbyattrs/compact, transform/metrics_merge_sessions, delta_to_cumulative, batch]
exporters: [otlp/lgtm_opt_metrics]
traces/raw:
receivers: [otlp]
processors: [resource/name_raw, batch]
exporters: [otlp/lgtm_raw_traces]
traces/optimized:
receivers: [otlp]
processors: [filter/drop_successful_tool_execution, transform/traces_span, transform/traces_spanevent, resource/name_optimized, batch]
exporters: [otlp/lgtm_opt_traces]
The dashboard is three query shapes.
# cardinality, from the data itself
count by (pipeline) ({__name__=~”claude_code_.*”, pipeline=~”raw|optimized”})
# bytes per minute, per signal, before and after
sum(increase(otelcol_exporter_enqueue_size_bytes_sum{exporter=”otlp/lgtm_raw_logs”}[1m]))
sum(increase(otelcol_exporter_enqueue_size_bytes_sum{exporter=”otlp/lgtm_opt_logs”}[1m]))
# record counts, per exporter
sum by (exporter) (otelcol_exporter_sent_log_records_total)
otelcol_exporter_enqueue_size_bytes only exists at level: detailed with sending_queue enabled, and it’s labelled by exporter, not signal.
Top row, cardinality. 300 raw, 12 optimized. In a real fleet the red line climbs with every session. Green is flat because there’s no per session dimension left to grow along. Middle row, bytes per minute per signal. The gap is what you stop paying for. Bottom row, record counts. Logs lose api_request, traces lose successful tool.execution, metrics arrive pre-summed. None of it is sampling.
The second dashboard reads only the optimized stream and asks the six questions.
Tokens by type, cost by model, log volume, lines added and removed. Twelve series, every panel populated, and all my questions answered.
What’s next?
If you’ve written off agent telemetry as “all prompts, nothing to dashboard,” the constraint is narrower than it looks. You don’t need the transcript in your backend. You need six answers. Use the layout above to configure processing and start cutting storage costs right away.