Tailscale Field Guide

Drill · dns · difficulty ●●● · 6 min

Internal names suddenly resolve to public IPs

"Every laptop resolves gitlab.corp.example.com to a public address since this morning. It worked yesterday."

Which failure areas the drills cover connectivity 3 drills identity 3 drills policy 2 drills dns 2 drills routing 3 drills platform 2 drills bar length is the count, not a difficulty or importance score

The ticket

A platform team runs a tailnet of about 200 devices with MagicDNS enabled. Internal services live under corp.example.com, served by an internal resolver at 10.0.5.53 that sits behind a subnet router (node-b, advertising an approved 10.0.0.0/16). At 10:05 UTC the Customer opens a high urgency ticket: engineers cannot reach GitLab or the artifact registry, and browsers are throwing certificate warnings because they are landing on the wrong server entirely.

“Since about 09:30 UTC every laptop resolves gitlab.corp.example.com to a public address and hits a certificate warning. Our DNS server has not changed. This all worked yesterday.”

Evidence provided

The first responder collected a query from an affected macOS client, node-a:

$ dig gitlab.corp.example.com A

; <<>> DiG 9.18.24 <<>> gitlab.corp.example.com A
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23817
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; ANSWER SECTION:
gitlab.corp.example.com. 300	IN	A	203.0.113.20

;; Query time: 24 msec
;; SERVER: 100.100.100.100#53(100.100.100.100)
;; WHEN: Mon Aug 10 10:02:41 UTC 2026

The expected answer is 10.0.20.14. Two facts in this one capture are worth more than everything else in the ticket: the answer came from 100.100.100.100, and it is a clean NOERROR carrying a public address.

Hypothesis tree

Classify the wrongness before touching anything. “Right name, wrong answer” and “no answer” are different diseases: a dead resolver produces SERVFAIL, REFUSED, or a timeout; it can never produce a confident public A record. That single distinction prunes half the tree before you run a command.

Hypothesis tree: right name, wrong answer Internal names return public IPs NOERROR, answered by 100.100.100.100 Internal resolver down or unreachable Would be SERVFAIL or timeout, never a clean public A record Client bypassing Tailscale DNS SERVER field would show a LAN or public resolver, not quad-100 Split DNS route missing from tailnet DNS config tailscale dns status shows no route; queries fall through to globals Public zone tampered or hijacked Record is legitimate and predates the incident Discriminators: who answered (SERVER), what kind of wrong (status), what config was pushed

Investigation

  1. Confirm the query path. The dig output already shows SERVER: 100.100.100.100#53. The OS handed the query to Tailscale’s local resolver, exactly as designed. This rules out the bypass branch: resolv.conf drift, another VPN fighting for DNS, a hardcoded resolver in the app.

  2. Classify the wrong. status: NOERROR with a public A record. A dead or unreachable internal resolver cannot manufacture that answer; it fails loudly. This demotes “resolver down” from primary suspect to impossible-as-sole-cause. Someone answered this query honestly, from the wrong view of the zone.

  3. Ask the internal resolver directly. From the same client, through the still-approved 10.0.0.0/16 subnet route:

    $ dig @10.0.5.53 gitlab.corp.example.com A +short
10.0.20.14

    Correct answer, 30 ms. The resolver is healthy, the subnet route works, ACLs permit the traffic. “Nothing changed on the DNS server” is now verified fact, not Customer assertion.

  4. Read the DNS config the control plane pushed. On the client (the dns command is available in Tailscale v1.74.0 and later (kb-cli); output abridged):

    $ tailscale dns status

=== 'Use Tailscale DNS' status ===

Tailscale DNS: enabled.

=== MagicDNS configuration ===

MagicDNS: enabled tailnet-wide (suffix = velo-cirrus.ts.net)

Resolvers (in preference order):
  - 1.1.1.1

Split DNS Routes:
  (no routes configured: split DNS disabled)

    Yesterday this listed a route sending corp.example.com to 10.0.5.53. Today the table is empty. The client is faithfully executing a configuration that no longer contains the rule. That rules out client caching, per-device weirdness, and platform quirks: every device got the same push.

  5. Check the admin console DNS page. Under Nameservers, the global resolvers are present, but the restricted nameserver row (10.0.5.53, restricted to corp.example.com) is gone. Change history and the admin who made it line up: at 09:26 UTC someone swapped global nameservers and deleted the restricted row in the same edit session. Onset “about 09:30” matches.

  6. Explain the public answer. dig @9.9.9.9 gitlab.corp.example.com from outside the tailnet returns the same 203.0.113.20: a years-old public wildcard for *.corp.example.com pointing at the company’s web gateway. The public record is legitimate, ruling out the hijack branch.

Root cause

Split DNS in Tailscale is a suffix-to-resolver routing table, stored in the admin console DNS page and pushed to every client by the control plane (Module 02). A restricted nameserver entry said: queries matching corp.example.com go only to 10.0.5.53. During a routine nameserver edit, an admin deleted that entry. Per the DNS KB, a restricted nameserver only applies to queries matching a specific search domain, while a global nameserver handles queries for any domain, so with the restricted row gone the internal names stopped matching anything special and became ordinary queries for the global resolver. The quad-100 resolver on each client (Module 06) kept doing its job perfectly; the table it was given simply had one less row.

The reason this presented as wrong answers rather than failures is split-horizon DNS: the same names exist in the public zone with different records. If the public wildcard had not existed, every lookup would have returned NXDOMAIN, the ticket would have said “names stopped resolving,” and the missing-route diagnosis would have been nearly instant. This is the signature to memorize (Module 11): no answer points at a server or path; the right name with a wrong answer points at query routing, and on a tailnet, query routing is the split DNS table.

Fix and prevention

Immediate. In the admin console DNS page, add the nameserver back: Add nameserver, Custom, 10.0.5.53, then restrict it to the search domain corp.example.com so it becomes a restricted nameserver again. The control plane pushes the change without any client restart. Verify on an affected client: dig gitlab.corp.example.com now returns 10.0.20.14 from 100.100.100.100, and tailscale dns status lists the route again. Total client-side action required: none, which is also your proof of the mechanism.

Durable.

  1. Treat the DNS page as production configuration. Split DNS lives in the admin console, not in the policy file, so it does not ride through your ACL review flow. Give it an equivalent: a documented change process and a second person on any nameserver edit.
  2. Add a canary: a scheduled job on a tailnet node runs dig canary.corp.example.com and alerts if the answer falls outside 10.0.0.0/16. This converts the silent failure mode into a paged one, and it would have caught this at 09:27 instead of 10:05.
  3. If the public wildcard is not load-bearing, remove it. A failure mode of NXDOMAIN is a gift: loud, obvious, and impossible to mistake for an application bug.

The handoff package

Summary: All tailnet clients resolve corp.example.com names to public IPs; split DNS restricted nameserver entry absent from tailnet DNS config after an 09:26 UTC admin console edit. Repro: On any client with MagicDNS enabled, dig gitlab.corp.example.com returns 203.0.113.20 (public wildcard) from 100.100.100.100; expected 10.0.20.14. Log evidence: 10:02:41 UTC, node-a: NOERROR public answer via quad-100 (dig capture attached). 09:26 UTC: DNS page nameserver edit removing restricted entry 10.0.5.53 for corp.example.com. 10:41 UTC, node-a: dig @10.0.5.53 returns correct internal record. Version matrix: Clients v1.84.0 (macOS, Linux); subnet router node-b v1.84.0 (Linux); MagicDNS enabled tailnet-wide. Impact scope: ~200 devices, every name under corp.example.com, 09:26 to 11:10 UTC. Ruled out: internal resolver health, subnet route to 10.0.0.0/16, ACLs, client DNS bypass, public zone tampering, client caching. Proposed owning area: none in product; admin configuration change. If escalated at all: control plane DNS configuration UX (deleting a restricted nameserver warns no differently than deleting a global one).

The trap

The weak version of this investigation hears “DNS is broken” and starts restarting things: the internal resolver, tailscaled, the laptops. Cache flushes everywhere. Each restart takes long enough that someone believes it worked, then the next lookup disproves it, and two hours vanish. The evidence that shortcuts all of it was in the very first dig: the SERVER field says who answered, and the status field says what kind of wrong you have. A clean NOERROR with a public address through quad-100 can only mean the query was routed somewhere that answers from the public view, and on a tailnet exactly one table decides that routing.

Sources

  1. DNS in Tailscale checked 2026-08-10
  2. MagicDNS checked 2026-08-10
  3. Tailscale CLI checked 2026-08-10
  4. What is 100.100.100.100? checked 2026-08-10

All drills