Tailscale Field Guide

Drill · policy · difficulty ●●● · 6 min

The service name that died in the grants migration

"Since the ACL migration, https://billing.velvet-lizard.ts.net times out for the finance team, but the old server URL still works."

Which failure areas the drills cover connectivity 3 drills identity 3 drills policy 2 drills dns 2 drills routing 3 drills platform 2 drills bar length is the count, not a difficulty or importance score

The ticket

Monday, first thing, urgency high because invoicing is blocked. Over the weekend the platform team shipped two changes at once: they migrated the tailnet policy file from legacy acls to grants, and they moved the billing app behind a Tailscale Service named svc:billing so it can survive host migrations. Tailscale Services require one or more devices running v1.86.0 or later (kb-services). The fleet is on 1.88.x, so versions are not the story. The announcement told everyone to use the new URL. Finance did.

“The new billing link times out for my whole team. The old link to the server still works, so the app is clearly fine. Whatever you changed this weekend broke the new thing you told us to use.”

Evidence provided

The first responder gathered a failing request, a working control, and the post-migration policy excerpt.

finance-laptop$ curl -sS --max-time 10 https://billing.velvet-lizard.ts.net/
curl: (28) Connection timed out after 10001 milliseconds

finance-laptop$ curl -sS -o /dev/null -w '%{http_code}\n' https://cloud-1.velvet-lizard.ts.net:8443/
200
"grants": [
  {"src": ["group:finance"],  "dst": ["tag:billing-host"], "ip": ["tcp:8443"]},
  {"src": ["group:platform"], "dst": ["svc:billing"],      "ip": ["443"]}
]

The service host is cloud-1 (tagged tag:billing-host), advertising svc:billing with a TCP endpoint mapping the service’s port 443 to the app on 8443. A platform engineer notes, unhelpfully for the outage but helpfully for the diagnosis, that the new URL “works fine for me.”

Hypothesis tree

The Customer’s frame is “the new thing is broken.” The interesting fact is the split: the same user, same laptop, same app, reachable by node name and port, unreachable by service name. Anything that would break the app, the host, or the network in general would break both paths.

Hypothesis tree: service name times out, direct node access worksbilling service name times outdirect node access worksMagicDNS failureplatform user resolvesand connects fineruled outService host problemconsole: approved andadvertising endpointsruled outApp process downdirect :8443 returns 200app is healthyruled outNo grant for svc dstfinance has tag dst onlyno dst svc:billingconfirmed

Investigation

  1. Reproduce on an affected machine. curl --max-time 10 https://billing.velvet-lizard.ts.net/ from a finance laptop times out. Consistent, not intermittent. That shape (timeout, not refused) already smells like filtered traffic rather than a dead listener.

  2. Run the working control from the same machine. curl https://cloud-1.velvet-lizard.ts.net:8443/ returns 200. This rules out the app being down, the laptop being offline, and any general connectivity or client problem on the finance side. Same user, same node, same physical server.

  3. Reproduce from a platform machine. The service URL returns 200 for a group:platform member. This rules out the service definition being broken, the host advertisement being stuck, and any service-wide outage. Whatever is wrong varies by who is asking. In a policy system, “works for group X, fails for group Y” is a policy symptom (Module 05).

  4. Check the service in the admin console. svc:billing exists, its host cloud-1 is approved and advertising the TCP endpoint. Service host advertisements require approval by an Admin, Network admin, or Owner unless auto-approval is configured (kb-services), so a pending approval was a live hypothesis. It is not the problem here.

  5. Read the grants section, searching for who can reach the service. Grants use the fields src, dst, ip, app, via, and srcPosture (kb-acl-syntax). Search the file for svc:billing: exactly one hit, and its src is group:platform. The finance grant, produced by mechanically translating the old ACL line, has dst: ["tag:billing-host"] with ip: ["tcp:8443"]. Nothing grants finance the service destination. Grants deny by default just as ACLs do, so the absence is the answer.

  6. Confirm the semantics before declaring victory. Per the Services docs, users reach a service through its TailVIP or its MagicDNS name, and access is controlled by grant policies that reference the service with the svc: prefix in dst, with the ip field naming the service ports (kb-services, kb-grants). A grant whose dst is the host’s tag authorizes connections to that node’s own addresses and ports. The service is a different destination identity. The two working data points (finance direct, platform via service) and one failing data point (finance via service) are all predicted exactly by the policy file as written. Confirmed.

Root cause

Two weekend workstreams collided in one policy file. Workstream one translated legacy ACLs to grants. That translation was faithful: the old rule {"action": "accept", "src": ["group:finance"], "dst": ["tag:billing-host:8443"]} became {"src": ["group:finance"], "dst": ["tag:billing-host"], "ip": ["tcp:8443"]}. Legacy ACLs express network-layer rules with action, src, dst, and optional proto; grants restructure this into src, dst, and ip, and add capabilities ACLs never had, including application-layer app grants and via route scoping (kb-acl-syntax, kb-grants). A mechanical translation can only preserve what the old rules said, and the old rules predate the service, so no translator output could ever contain svc:billing.

Workstream two created the service and granted it to group:platform for testing. Nobody owned the join between the two streams: granting the new destination to the population that was told to use it. Grants and legacy ACLs can coexist in the same policy file (kb-grants), so nothing forced a full review at cutover, and the save went through because nothing in the file was invalid. It was merely incomplete, and default deny turned incomplete into an outage (Module 05, Module 02 for how the resulting filter reaches each node).

Fix and prevention

Immediate fix. One grant:

{"src": ["group:finance"], "dst": ["svc:billing"], "ip": ["443"]}

Save, then verify from the finance laptop: the service URL returns 200. Total change: one line, scoped to one service and one port.

Durable prevention. This is precisely what the tests section exists for. A test names a source and then lists destinations under accept and under deny, and the allowed destination forms include svc:my-service for the Tailscale Virtual IP addresses of a Service. If an assertion fails, Tailscale rejects the updated tailnet policy file with an error (kb-acl-syntax). The save that caused this outage would have been rejected at the console if this had existed on Friday:

"tests": [
  {"src": "fin-lead@example.com", "proto": "tcp", "accept": ["svc:billing:443"]},
  {"src": "fin-lead@example.com", "deny": ["tag:billing-host:22"]}
]

Adopt the rule behind the fix: every access path you announce to users gets a test in the same change that announces it. Then the policy file cannot drift away from its promises, no matter how many migrations run over it.

The handoff package

Prepared in case this had been a Services defect; it was not.

The trap

The weak investigation anchors on “the new URL is broken” and goes spelunking in DNS: flushing caches, restarting clients, toggling MagicDNS, because the visible difference between the working and failing requests is the hostname. That path can consume a morning and produce a false fix the first time a cache flush coincides with someone testing from the wrong (platform) laptop. The other weak move is the panic revert of the whole migration on Monday morning, which is worse than doing nothing: it throws away a working policy structure, still does not grant finance the service (the legacy file never had it), and now two changes are tangled instead of one. The strong move noticed in step 3 that access varied by group, which converts the whole ticket into a sixty-second read of the grants section. When the same request works for one principal and fails for another, stop debugging infrastructure and start reading policy.

Sources

  1. Grants checked 2026-08-10
  2. Syntax reference for the tailnet policy file checked 2026-08-10
  3. Tailscale Services checked 2026-08-10

All drills