What a live call actually is
A live incident call is not a debugging session that happens to have an audience. It is a trust exercise that happens to involve debugging. The Customer joined the bridge because something they depend on is broken, they cannot fix it themselves, and they need to know two things at every moment: is anyone competent working this, and is that work moving. Everything in this module serves those two signals.
The technical problem and the call are separate workloads, and they compete for the same brain. The engineers who are good at this are not smarter than everyone else. They have a structure that lets them think about the problem while the structure carries the call. That structure is what this module teaches: a first five minutes script, a theory board you run out loud, a narration discipline, a rule for saying “I do not know yet,” a decision test for going async, a closing ritual, and a write up format.
One framing rule before anything else: on a live call you are the technical owner. Not the most senior person present, not the account owner, not the person who filed the ticket. The technical owner is the person who holds the current theory of the failure and decides what gets tested next. If that is you, act like it from the first minute, because a call with no technical owner produces forty minutes of people reading logs at each other.
The first five minutes
The first five minutes decide the shape of the next hour. Resist the pull to start debugging immediately, because debugging without a frame means debugging the wrong thing confidently. You need three facts on the table before you touch a terminal, and you get them by asking, out loud, in this order.
Establish impact. What is actually broken, for whom, right now? Not the Customer’s diagnosis (“the VPN is down”) but the observable symptom (“users in the warehouse cannot reach the inventory app on node-a since about 9:40”). Push past the summary to specifics: is it all users or some, all destinations or one, total failure or degraded? Impact scopes the problem and sets the urgency honestly. A total outage and a single slow path get different pacing, and pretending otherwise wastes either trust or time.
Establish timeline. When did this start, as precisely as anyone can say? When was it last known working? Was the onset sharp or gradual? A sharp onset at 9:41 points at an event. A gradual degradation over a week points at drift, growth, or an expiring credential. The timeline is the spine every piece of evidence gets pinned to for the rest of the call, so get it early and say it back: “So this was working at 9:30, broken by 9:45, and nothing has recovered on its own since. Everyone agree?”
Establish what changed. In the window before onset, what changed anywhere near this system? Deploys, config pushes, ACL edits, key rotations, OS updates, network changes, vendor maintenance, DNS changes. Ask the question even though the first answer is almost always “nothing changed.” That answer is almost never true; it means “nothing I personally changed that I currently connect to this symptom.” Ask again in specifics: any changes to the tailnet policy file? Any nodes re-authenticated? Any infrastructure work last night? The change that caused the incident is usually in the room, unmentioned, because its owner has not connected it to the symptom yet.
Then say the frame back in one breath: “Here is what I have. Warehouse users lost access to the inventory app at roughly 9:41, sharp onset, other apps unaffected, and there was an ACL change deployed at 9:35. That change is my starting theory. I am going to verify connectivity from an affected node now.” Thirty seconds of speech, and the call now has a shared model, a leading theory, and visible motion.
Running the theory board out loud
Once you have a frame, the investigation is a loop: hold one leading theory, test it, keep it or kill it, repeat. Everyone debugs this way internally. The craft on a live call is doing it out loud, as a visible board with three columns you keep repeating.
Current leading theory. One sentence, stated as a claim about a mechanism, not a vibe. “The 9:35 ACL change removed the rule that allowed warehouse nodes to reach node-a on port 443” is a theory. “Something with the ACLs” is not, because it cannot be tested or killed.
Evidence for. Why this theory leads. Onset lines up with the change, the failure scope matches the rule’s scope, the symptom is a clean block rather than a timeout.
Evidence that would kill it. This is the column that separates investigation from confirmation bias, and it is the one engineers skip. Before you run a test, say what result would falsify the theory: “If a warehouse node can reach node-a on 443 right now, this theory is dead.” Then run the test. Stating the kill condition first does three things: it forces the theory to be concrete, it commits you to honoring an inconvenient result, and it shows the Customer you are hunting the truth rather than defending a guess.
The loop sounds like this in practice: “Leading theory: the ACL change. Evidence for: timing and scope. Kill test: I will run a connectivity check from an affected node to node-a; if traffic passes, the theory dies. Running it now.” Then the result: “Traffic is blocked at the destination. Theory survives, and I am promoting it. Next I want the diff of that ACL change.” When a theory dies, say that just as plainly: “That kills the ACL theory. Traffic passes fine from node-b, so the block is not policy. New leading theory: the app on node-a itself. Here is why.”
Killed theories are progress and you should frame them that way. “We have ruled out policy and basic connectivity, which means this is on the host” is a sentence that moves the call forward. A silent engineer who has ruled out three things sounds identical to a silent engineer who has learned nothing.
Narrating without hand waving
Narration is how the Customer sees motion. The mistake is thinking narration means talking constantly. It means marking transitions. You speak at four moments: when you start something (“I am checking whether the affected node can reach node-a directly”), when you get a result (“it cannot, and the failure mode is a block, not a timeout”), when the theory changes, and when you need something from someone (“I need whoever owns the policy file to pull up the 9:35 diff”).
Between those moments, silence is fine if you have bounded it: “I am going to be quiet for about three minutes while I read this log.” Bounded silence reads as focus. Unbounded silence reads as absence.
Concreteness is what separates narration from hand waving. In a Tailscale context the difference sounds like this. Hand waving: “I am checking the network.” Narration: “I am running tailscale status on the affected node to see the state of its connections to its peers, then tailscale ping node-a to test the path over the tailnet specifically, because that isolates the tailnet layer from the application. If the ping works and the app still fails, this is not a connectivity problem.” Same work, but the second version teaches the Customer your model of the system, and a Customer who understands your model stops asking “any update?” because they can see where you are in it.
Two narration rules that pay for themselves. First, never speculate upward: do not float causes to the Customer that you have not promoted to leading theory, because every speculative cause you mention becomes a fact in someone’s notes and you will spend the postmortem unwinding it. Second, never assign blame mid-call, even to a vendor, even to a config, even when you are sure. “The 9:35 change is my leading theory” is a claim about your investigation. “Your team broke it at 9:35” is a claim about people, and it changes the room instantly and never in a useful direction.
Saying “I do not know yet”
At some point on every real incident you will be asked a question you cannot answer: what caused this, when will it be fixed, is data affected. The pressure to produce an answer is enormous, and yielding to it is the most expensive mistake in this entire module, because a confident wrong answer costs you the thing the call runs on.
The discipline is a fixed sentence shape: “I do not know yet. Here is how we find out.” The second half is mandatory. “I do not know” alone is an ending; with the second half it is a plan. “I do not know whether data was affected yet. Here is how we find out: once we confirm the block was at the policy layer, we will know requests were rejected outright rather than partially processed, and I will be able to answer definitively. I expect to know within the hour.”
Notice what that does. It replaces a guess with a method, it names the evidence that will settle the question, and it attaches a time. Customers do not actually need you to know everything. They need to trust what you say, and the fastest way to build that trust is to visibly refuse to guess. The engineer who says “I do not know yet, here is how we find out” three times and is then precisely right the fourth time owns the room. The engineer who guessed three times does not get a fourth.
The same shape works for the estimate question. “When will it be fixed” before you have a confirmed cause deserves: “I cannot give you a fix time yet because I do not have a confirmed cause, and a fix time without a cause would be a guess. What I can commit to: we will have the cause confirmed or ruled out within thirty minutes, and I will give you a real estimate at that point.” That is an answer. It is not the answer they wanted, but it is one they can plan around, which is what an answer is for.
When to stop live debugging and go async
Live calls have a failure mode where they outlive their usefulness: eight people on a bridge watching one person read logs. The bridge itself has a cost. It burns the Customer’s staff, it pressures you toward fast shallow tests over slow decisive ones, and it makes deep work worse. The technical owner decides when the call stops paying rent, and there is a test for it.
Every ten minutes or so, ask yourself two questions. First: is there a live hypothesis I can test in the next few minutes that needs someone on this call, their access, their confirmation, their eyes? If yes, keep going, the bridge is earning its cost. Second, if no: is anyone on this call producing information I cannot get async? If both answers are no, the call is done, and prolonging it is theater.
Certain moments almost always mark the transition. You have confirmed cause and the fix is a change that needs review or a maintenance window. The next step is a long running task: collecting diagnostics with tailscale bugreport and working with upstream support, a log trawl, a reproduction attempt in a lab. You are waiting on a third party. Or the incident has been mitigated and only root cause work remains. In all of these, the honest sentence is: “We are at a point where keeping everyone live is not making this faster. Here is what happens next and who owns it, and I would like to take this async.”
The credibility of that sentence depends entirely on what follows it, which is the closing ritual.
Closing the call
Never let an incident call dissolve. Calls that end with “okay, we will keep looking and circle back” have not ended, they have evaporated, and everything discussed evaporates with them. The close is a ritual with three parts, and you perform it out loud before anyone drops.
Owned next steps. Every open thread gets exactly one name and one action. Not a team, a name. “I am pulling the policy diff and testing the revert in the lab. Dana is confirming with the app team that node-a’s service logs cover the window. Priya is identifying the change approver.” A step without an owner is a step that will not happen; every incident veteran has watched “someone should check the logs” go unchecked for two days.
A time commitment. When does the Customer hear from you next, regardless of progress? “You will have a written update from me by 2:00pm even if the update is that we are still testing.” The commitment is to communication, not to resolution, which means you can always keep it, and keeping it is the point. An update that says “no change, next update at 4:00” maintains trust. Silence until you have good news destroys it, because from the Customer’s side silence and abandonment are indistinguishable.
The state of the record. Say where the ongoing record lives: the ticket, the shared channel, the doc. One place, named on the call, so nobody is reconstructing the incident from four chat fragments tomorrow.
Then, before you drop, restate the whole board once in under a minute: impact, timeline, leading theory, what has been ruled out, next steps with owners, next update time. This sixty second summary is the highest leverage minute of the entire call. It is what everyone actually remembers, and it becomes the skeleton of the write up.
The write up after
The write up is not paperwork after the incident; it is the last act of the incident. Until it exists, the incident lives in fragments across a bridge recording nobody will replay and the memories of people who each heard a different call. Write it the same day while the ordering of events is still cheap to reconstruct, and keep it short enough that it gets read.
The shape mirrors the call, which is why running the call well makes the write up nearly free:
Impact. What broke, for whom, from when to when. Numbers where you have them.
Timeline. Timestamped, factual, no interpretation mixed in. First bad moment, detection, key findings, mitigation, resolution. The interpretation goes in the next section, and keeping them separate matters, because the timeline is what everyone must agree on even when they disagree about the cause.
Cause. The mechanism, stated the same way you stated theories on the call: as a claim about the system. Include the theories you killed and what killed them. This is not padding; it is the part a future engineer facing similar symptoms will actually reuse, and it shows the Customer the investigation was a hunt, not a lucky guess.
What we are doing about it. Each item with an owner and a date, in the same discipline as the call close. Distinguish “done,” “committed with a date,” and “under consideration” honestly. A write up that promises six improvements and delivers one teaches the Customer to discount everything you write.
What we still do not know. If open questions remain, say so, with the method for closing them. The write up is the last place to apply the “I do not know yet, here is how we find out” discipline, and applying it in writing, where it can be checked later, is what makes the next incident call with this Customer start from trust instead of from zero.
Send it to the Customer proactively. A write up they had to ask for is worth half of one that arrived unprompted, because the unprompted one says the thing every part of this module is designed to say: someone owns this, and the work is still moving even when you cannot see it.
Cross references
- Module 11 covers the diagnostic tools referenced in the narration examples in depth: status, ping, netcheck, bugreport, and how to read what they return.
- Module 03 explains the path and relay mechanics behind connectivity theories you will state on calls about unreachable peers.
- Module 05 covers policy evaluation, the layer implicated in this module’s running ACL example.
- Module 02 explains the control plane, the layer to reason about when nodes lose state that connectivity tests alone cannot explain.
- Module 10 covers the operational practices (logging, alerting, change discipline) that determine how much evidence exists before your call starts.