From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes

Posted on September 30, 2026 by Deepak Biswas, Senior Engineering Manager, Central Monitoring and Disaster Recovery, Atlassian

CNCF projects highlighted in this post

Every SaaS company has the same uncomfortable question after a major incident: who noticed first, the monitoring or the customers? For a long time our honest answer was “it depends”. This post describes how a small team rebuilt the detection platform behind our automated incident creation (“AutoHOT”, where HOT is our internal name for a high-severity incident) on Apache Kafka, Apache Flink on Kubernetes and OpenTelemetry, what we measured month by month for eighteen months, and what we would do differently.

It is not a success story with a bow on it. In-scope recall went from about 60% to a peak of 86%, then fell back to 64% in a bad month. Precision is still below where we want it. We think the numbers are more useful than the slogans.

The use case

We run more than ten cloud products for millions of tenants. Each product emits client-side operational telemetry for every user action: an event when a task starts, and one when it succeeds, fails or is abandoned, tagged with tenant, user, the named “experience” (view issue, edit page, view board) and the HTTP status. Across all products, there are billions of events a day, and it is the closest thing we have to ground truth about whether customers can actually do their work.

The job of central monitoring is to turn that stream into three answers, fast:

1. Is something wrong? An error rate or a volume drop that a human would call an incident.
2. How big is it? Distinct users and tenants impacted, and where: which tenant, database shard, region, or globally.
3. Who should be woken up, at what severity, with what evidence?

The first-generation system did this with a Node.js aggregator fed from the event bus through a cloud queue, with an in-memory cache for de-duplication and task pairing, running on roughly 90 virtual machines. It worked, and it taught us the domain. It also had three problems we could not tune away:

• Latency. Event-to-metric took more than 40 seconds on a good day. Detectors then needed several minutes of windowed data on top of that.
• Noisy neighbours. The queue was shared infrastructure. Two incidents in one year were caused by another tenant’s lag on the same queue, which is a bad property for the system that is supposed to detect incidents.
• Cost that scaled with onboarding. Adding products and experiences took the run cost from roughly $120K to $230K a year, and the cache tier hit 100% CPU during a routine change.

We also had a correctness problem that only became visible once we had better data. Client-side telemetry only exists when a page loads. A hard-down database shard produces silence, not errors. Any design that only looks for failures will read silence as health.

Design goals

We wrote down five goals before choosing technology, and they are worth stating because they explain most of the trade-offs below.

Architecture

[Figure 1 ] End-to-end architecture: upstream producers and transport, the Apache Flink job, the storage and signal plane, the AutoHOT decision engine, and downstream consumers The full system, upstream to downstream. Green boxes are open-source projects, amber are managed or commercial services kept behind stable interfaces, blue are services our team owns, purple are consumers.

Reading left to right:

Upstream. Products emit operational events through an analytics gateway into the company event bus, which is built on Apache Kafka. Rather than consume the whole firehose, we register a server-side subscription filter, kept as code: an allow-list of products and experiences, with experiment and synthetic traffic dropped at the bus. The matching fraction lands on a dedicated Kafka topic with a seven-day retention that doubles as our replay window. Two side inputs feed the pipeline: a tenant-context service (tenant to shard, region, edition, licence count) and a configuration repository.

Stream processing. One Apache Flink 1.20 job, deployed through the Flink Kubernetes Operator, does the work. Details in the next section.

Storage and signal plane. Metrics go to a Prometheus-compatible time-series database via OpenTelemetry, and to a commercial metrics service where our detectors currently run. Per-minute impact aggregates go to a multi-region key-value store and to Apache Parquet files in object storage, from where a lakehouse builds the contractual SLA tables. A small Go GraphQL service, the Impact API, serves “who is impacted and how many” questions from those stores.

Decision plane. The AutoHOT engine (Go, active-active in two regions) turns detector alerts into anomalies, quantifies impact every minute through the Impact API, applies a scoped severity matrix, suppresses blips, creates the incident ticket, pages the right teams and keeps re-evaluating until the impact clears.

Downstream. Responders, the tenant health dashboard, enterprise SLA reporting, status and comms tooling, product-team SLO dashboards, and post-incident analytics that feed thresholds back into configuration.

What we built

1. Let the bus do the filtering

The first decision was to stop being a “direct consumer of everything”. Operational events are about 55% of all traffic on the bus. Our subscription filter is a ~770-line YAML file generated from the same product configuration that drives everything else, so the pipeline only sees the products and experiences it is responsible for. This alone removed a whole class of scaling work and made the topic small enough that seven days of retention is cheap.

2. One Flink job, deliberately

We run a single job rather than one per product. The graph is:

kafka-source
  -> filter-and-parse            (drop events > 15 min old, 401s, no-user events)
  -> apply-transformations       (config-driven regex/rule transforms)
  -> keyBy(product, subproduct, tenant)
  -> event-enricher              (async, tenant-context sidecar, cache, bulkhead + circuit breaker)
  -> fan-out:
       metrics       -> OpenTelemetry (OTLP) exporter -> Prometheus-compatible TSDB
                     -> StatsD (legacy metric names preserved)
       logs          -> log analytics
       user-dedup    -> 30 s keyed TTL state -> distinct-user counters
       aggregate     -> 60 s tumbling windows keyed (product, subproduct, experience, tenant)
                        carrying HyperLogLog sketches
                     -> key-value store (idempotent PutItem on (PK, SK))
                     -> Parquet FileSink (committed on checkpoint)
       side streams  -> 4xx impact, error-message impact, taskStart<->terminal pairing (timers)
       detectors     -> per-minute tallies -> thresholds -> alert queue   (shipped dark)

Some of the choices that matter:

• Async enrichment with a bulkhead and a circuit breaker. The tenant-context lookup is the one external call in the hot path. It runs through a sidecar with a 500k-entry in-memory cache, a baked read-only tenant map as the first-level lookup, a semaphore bulkhead that fails fast at 50 in-flight calls, and a circuit breaker that opens on a 50% failure or slow-call rate. Before this, a slow dependency back-pressured the whole job. After it, we have not had an enrichment-induced stall.

• HyperLogLog instead of user IDs. “Distinct impacted users” has to be mergeable across minutes, tenants, shards and regions. Summing per-row counts double-counts. Each 60-second window carries an HLL sketch; the Impact API unions sketches at read time for any grouping the caller asks for. The cost is format coupling between the Java producer and the Go consumer (we shipped one hotfix when the sketch precision parameter drifted) and about 1.5% error, which nobody has noticed.

• Idempotent sinks, at-least-once metrics. The key-value store key is (product, day, hour, job) plus (window, tenant, subproduct, experience), so replaying the topic rewrites identical rows. Parquet files roll on checkpoint, so they are exactly-once. Metrics are fire-and-forget: a restart produces a two to three minute dip followed by a replay surge, and we accept that because detectors read the pre-aggregated store, not the raw counters.

• Preserve the old metric names. The StatsD path emits under the legacy metric names. Product teams’ SLO dashboards and detectors did not have to change on cutover day, which is the only reason cutover day was boring.

• RocksDB, 30-second checkpoints, savepoint upgrades. State lives in RocksDB, checkpoints go to object storage every 30 seconds, and upgrades restore from savepoints keyed by stable operator UIDs. We learned the hard way that a topology change deployed with “undeploy and redeploy” discards state silently; we now treat operator UIDs as an API.

• Autoscaler tuning is a reliability feature. Early on every rescale was a full job restart and we saw around eight restarts a day. Setting the operator autoscaler to a 0.7 target utilisation, a 0.3 boundary, vertex parallelism between 18 and 25 and a three-hour scale-down interval took that to zero. Our production deploy pipeline now fails if the job restarts more than three times in ten minutes.

• Configuration as the control plane. A single products.yaml with a CUE schema generates the Kafka subscription filter, the transforms, the detector thresholds and the sink routing. Since mid-2026 the job hot-loads a runtime bundle of that config from object storage, so onboarding an experience no longer needs a redeploy.

Operator design: knowing when to merge and when to split

One of the more nuanced lessons from building this job was that operator granularity isn’t a one-time architectural decision — it’s something you revisit as the job matures under real load. We learned this in both directions.

Merge when you’re doing redundant work. Early on, the pipeline had a filter followed by a map: the filter would parse the JSON to decide if a message was worth processing, and then the map would parse it again to deserialize it into a typed Payload. Combining these into a single flatMap — parse once, emit only if valid — eliminated the redundant deserialization entirely. The same principle led us to use Flink side outputs for fan-out to log analytics, StatsD, and OpenTelemetry rather than separate operators: one pass through the graph, three destinations.

Split when operators have different scaling needs. The opposite lesson came later, under production load. We had our JSON parsing and transformation logic running in a single operator, but the enrichment step — doing async lookups to resolve tenant metadata — was pegging at 100% CPU and becoming a bottleneck for the entire pipeline. The fix was to explicitly separate “Phase 1: parse and filter (I/O bound)” from “Phase 2: apply transformations (CPU bound)” so each could be assigned its own parallelism independent of the other. What looks like one logical step on a whiteboard can become two distinct operators in Flink the moment they have different resource profiles under real traffic.

The heuristic we landed on: merge operators when they touch the same data in the same pass, split them when they need to scale independently. Flink’s uid() annotations on every operator enforce this discipline — once you name and checkpoint an operator, you’re committed to its boundary, which forces you to think carefully before drawing it.

3. The decision engine

Detectors fire on the metrics. The alert travels through the paging service, a pub/sub topic and one queue per engine region. Both regions consume every alert; correctness comes from a lock row in a multi-region table keyed by a business de-duplication ID, never the queue message ID, with a five-minute takeover if the owning region stalls. The incident-manager API adds a second, authoritative lock, which is the last line of defence against replication lag.

Every minute, the engine asks the Impact API for the window from fifteen minutes before the anomaly to now, raw and noise-filtered, and applies a scoped severity matrix (for example, ~75 distinct users in one tenant, ~200 on one shard, ~500 in one region, ~1,000 globally, or ~20,000 on a critical experience). Tenants with two or fewer impacted users are treated as noise. A blip check watches the per-minute unique-user series for fifteen minutes before a lower-severity ticket is cut — this blip detection alone prevented paging people ~300 times for false positives in FY26. When a ticket is created, the engine pages the top five dependent services by tier, opens the chat channel, triggers automated comms, provides details on which tenants need to be notified of potential impact, initiates an AI-assisted first investigation, and continually monitors impact for severity upgrades and re-evaluates until the impact clears. Additionally, it detects sudden drops in telemetry volume to spot incidents with missing telemetry (accounting for expected dips on holidays and weekends).

Two design details paid for themselves. Retries are scaled to message value: control messages get five, impact-analysis messages get twenty and are never dropped by the queue’s receive count, because losing impact data is worse than duplicate work. And the data source is pinned per anomaly, so a feature flag flipped mid-incident cannot mix two sources inside one incident record.

4. Watching the watcher with OpenTelemetry

A detection platform that fails silently is worse than none, because people stop watching the dashboards. Three things keep us honest:

• OpenTelemetry end to end. The Flink job exports metrics over OTLP to a Prometheus-compatible TSDB (Grafana Mimir), and the engine emits OpenTelemetry traces and OTLP metrics. That gives us one vendor-neutral view across a Java stream job and a Go control plane, and it is the path we intend to move detectors onto as we retire the commercial metrics hop. The one constraint we hit was series cardinality: we cap the tenant-health namespace at two million series and pre-aggregate by tenant-minute rather than emitting per-user series.

• A synthetic deep-check every 15 minutes injects a fake alert and drives it through anomaly, impact analysis, ticket creation and closure. If any hop breaks, we know within a quarter of an hour, not at the next real incident.

• Self-monitoring detectors on the platform: divergence between the new pipeline and the legacy path during the migration, zero writes to the impact store, enrichment error burn rate, and a per-message good/bad SLO on the engine. The engine’s SLO is counted per queue-message outcome, not per HTTP call, because “the API returned 200” says nothing about whether a ticket was created.

Results

Platform

The impact dashboard’s own bill fell from more than $20,000 to about $1,000 a month once it read the pre-aggregated store instead of scanning logs.

Detection outcomes, month by month

We review every major incident monthly and classify it: in scope for automated detection (it hit an instrumented experience) or not, and if in scope, detected or missed.

Figure 2: Month-by-month in-scope recall, core-product recall and scope ratio from March 2025 to August 2026, and the share of auto-created incidents later rejected by severity cohort in mid-2026  Left: in-scope recall (blue), recall on the three core products (green) and the share of all major incidents that fall on instrumented experiences (dashed). Right: the share of auto-created incidents later rejected, by severity cohort and month, against the full-year rates.

Three things stand out.

In-scope recall improved, unevenly. From 60% in March 2025 to 86% in June 2026 and 80% in July, then 64% in August. On the three core products it reached 100% in three separate months. The improvement tracks specific engineering: blip suppression and Sev3 early-warning tickets in late 2025, the Flink cutover and volume-drop detection in May 2026, 4xx detection for the largest product in April.

Overall coverage did not improve, and that is the important number. Over the nine months from November 2025 to July 2026 there were 263 major incidents. 117 (44.5%) were on instrumented experiences; we detected 80 of those (68.4% in scope), which is 30.4% of all major incidents. The scope ratio fell from about 60% to between 21% and 33% as the incident mix diversified into new products and non-reliability causes. Recall is a property of the detector; coverage is a property of what you chose to instrument. We had been reporting the first and celebrating it as the second.

Precision is a family of numbers, not one. On covered scope for the core products it sits around 90%. Across every auto-created ticket including low-severity early warnings it was 79% in February 2026 and 68% in March. In mid-2026 the rejection share ran between 19% and 41% for low-severity early warnings and between 21% and 40% for Sev1 and Sev2 tickets, against full-year rates of roughly 21% and 15%. Of everything rejected between January and August 2026, 48% was process and bookkeeping (stale, duplicate, no reason recorded), 26% was real signal below the incident bar, 16% was the monitoring being wrong, and 11% had a cause outside the service. The June rise in the chart came from a burst of false volume-drop tickets caused by ingestion lag in the metrics backend being read as zero; the fix was a 120-second minimum delay on volume-drop detectors, a deliberate two-minute trade of detection time for precision.

For the full fiscal year, about 85% of the Sev2 incidents we raised automatically were confirmed real, lower-severity early warnings ran at 70 to 79% accuracy, and blip suppression cut rejected blip tickets by about 80%. Where we have time-to-detect data for Sev1 and Sev2 incidents in mid-2026, the median from impact start to detection was one to eight minutes. In one Sev1 the automated ticket fired nine minutes before the human one; in another, a slow-growing login failure went undetected for ten days.

What did not work

• Silence still looks like health. The two worst misses of the period were a database shard that went hard-down (no page load, no events) and a stall in our own upstream event pipeline that left the detector starved. Both were invisible to a design that looks only for failures. The fixes are volume-drop detectors (which caught a Sev1 in May 2026 and also produced the June false positives), telemetry-freshness checks on the pipeline itself, and adding non-client signals such as edge 5xx rates and synthetic probes.
• Quantification lagged detection. In one Sev1 the ticket was raised early but the impact count showed about two thousand users when the real number was over eighty thousand, because the counting store only holds tenant-minutes with failures. The incident was severity-two during response and severity-one in the review. Detection and quantification are different problems with different data needs.
• Read-time aggregation collapsed under concurrency. The Impact API unions minute-grain sketches on every request with no roll-up and no shared cache. About a hundred concurrent dashboard users during an incident took it down. A daily roll-up written by Flink and a short-TTL cache are the fix; the lesson is that “we can answer any grouping at read time” is a promise you pay for on your worst day.
• The detection input is single-region. The engine is active-active, but the event gateway, the bus subscription and the Flink job live in one region. During a regional cloud outage our compute was healthy and our detection was blind. Making the pipeline multi-region is the main reliability work in front of us.
• Rejection hygiene is a data-quality problem. Almost half of rejected tickets had no recorded reason. We now require one, and we split “auto-recovered” and “maintenance window” from “false positive”, because tuning on a mixed bucket tunes the wrong thing.

What we would tell another team

1. Filter at the bus, not in the job. Subscription filters as code kept the topic small, the job simple and the cost linear in the events you care about.
2. Design the sink keys before the topology. Idempotent (partition, sort) keys are what make Kafka replay a recovery tool instead of a double-counting hazard.
3. Sketches, not IDs. Distinct counts that must merge across dimensions belong in HyperLogLog. Accept the format coupling and test it.
4. Treat operator UIDs and autoscaler settings as production configuration. Most of our early Flink pain was self-inflicted by redeploys and rescale storms.
5. Report recall and coverage separately. Recall is your detector’s quality; coverage is your instrumentation decision. Conflating them hides the real gap.
6. Instrument the detector with the same rigour as the product. OpenTelemetry across languages, a synthetic end-to-end check and per-outcome SLOs are what turn “the dashboards look fine” into evidence.
7. Be honest about silence. Client-side telemetry is a superb signal for degradation and a useless one for hard-down. Pair it with something that emits when nothing else does.

What is next

We are moving detection onto the OpenTelemetry-fed time-series store and enabling the Flink-native detector branch, which removes two hops from the path between event and alert. We are making the pipeline active-active across regions with in-job de-duplication and the job name in the storage key. We are adding a daily roll-up and cache in front of the Impact API, building telemetry-coverage SLOs per product so that the scope ratio becomes a managed number rather than a surprise, and we are targeting recall above 90% and precision above 90% on instrumented scope with a Sev3 detection P90 under 90 minutes. We will publish the same monthly numbers when we get there, including the months that go the wrong way.