Last month, I watched three senior engineers burn four hours debugging a “mysterious” Kubernetes issue that turned out to be a kubectl version upgrade. The same week, another team spent an entire night hunting phantom load balancer bugs when a certificate rotation had broken mobile clients with certificate pinning.
These aren’t stories about incompetent engineers. These are stories about brilliant people falling into the same cognitive trap that catches everyone: diving deep instead of looking broad.
The Question Nobody Asks
Here’s what happened with the kubectl disaster. The symptoms looked serious: cron jobs failing, secrets not updating across namespaces, image pulls crashing left and right. Classic distributed systems chaos, right?
The team immediately went into full forensic mode. Pod logs. RBAC permissions. Service account configurations. Network policies. Container registry connectivity. They even started questioning whether someone had compromised their infrastructure.
Four hours later, an exhausted engineer finally asked the obvious question: “Wait, did we change anything recently?”
One git log later, they found their smoking gun: a kubectl version bump from earlier that day. The new version handled secret types differently. Five minutes to roll back, problem solved.
The brutal truth? They spent 240 minutes debugging something that took 5 minutes to fix, all because they forgot to ask a 10-second question.
The Change Principle (It’s Not Rocket Science)
Production systems don’t just randomly decide to break on Tuesday afternoon. Something always changes first. Always.
The change usually falls into one of four buckets:
- Code: deployments, feature flags, that “quick fix” someone pushed
- Infrastructure: server upgrades, network changes, scaling operations
- Configuration: environment variables, that toggle, you flipped “just to test”
- Traffic: load spikes, geographic shifts, bot attacks
Once you accept this reality, debugging becomes less about technical wizardry and more about detective work. Good detectives don’t start by analyzing the murder weapon—they start with “who was here when it happened?”
The Revert Rebellion
Here’s where I lose most engineers: your first instinct shouldn’t be to understand the problem. It should be to undo whatever caused it.
This goes against everything we’re taught. Engineers love understanding root causes. We want to learn, to fix things properly, to write post-mortems that impress people with our technical depth.
But production users don’t care about your learning journey. They care about working software.
The certificate pinning incident I mentioned? Operations had rotated SSL certificates overnight. The mobile app couldn’t connect because it was checking for the old certificate fingerprint. Web browsers worked fine because they don’t do certificate pinning.
The “proper” debugging approach: dive into TLS handshake logs, analyze certificate validation failures, and understand the cryptographic details of pinning implementations.
The first-principle approach: “What changed? Oh, certificates. Let’s roll back.”
One took six hours. The other took six minutes.
The Exotic Bug Fallacy
Every engineer has a favorite theory about why their system is broken. Usually, it involves something impressive-sounding:
“It might be a race condition in our distributed cache.” “Could be a memory leak in the JVM garbage collector.” “I bet it’s that new Kubernetes version causing networking issues.”
Here’s the uncomfortable truth: 95% of production issues are caused by boring, obvious changes that you made recently. The remaining 5% are caused by boring, obvious changes that someone else made recently.
Your code is almost certainly the problem. Your recent deployment is almost certainly the problem. That configuration change you made this morning is almost certainly the problem.
Exotic platform bugs make great conference talks, but they’re terrible debugging hypotheses.
The Debug Stack
When something breaks, work through this hierarchy:
- What did I deploy recently?
- What infrastructure changed?
- What configuration changed?
- What external dependencies changed?
- Is there a platform bug?
Most issues resolve at level 1 or 2. If you’re investigating level 5 before exhausting levels 1-4, you’re debugging in hard mode for no reason.
The Meta Problem
After years of watching engineers debug in circles, I realized the real problem isn’t technical complexity. It changes visibility.
When production breaks, teams shouldn’t have to play “did anyone deploy anything?” detective. They shouldn’t have to dig through multiple repositories, chat histories, and change management systems to build a timeline.
This insight sparked our current product, which has unified change management as a core principle that captures everything: code, infrastructure, and configuration in one timeline. When alerts fire, the ‘what changed?’ question has an immediate answer.
Next time your production system decides to have a bad day, resist every instinct to dive deep. Take a breath. Ask the right question: “What changed?”
Then fix it first, and understand it later.
Your users will thank you. Your sleep schedule will thank you. And your ego might take a small hit, but that’s probably good for you anyway.