Skip to main content
November 18, 2026

When ubuntu.com Vanished From DNS: Your CI's Hidden Deps

When ubuntu.com Vanished From DNS: Your CI's Hidden Deps

On 11 November 2026, CI pipelines failed in a step most teams never think about: apt-get install. From about 13:48 UTC, GitHub-hosted Ubuntu runners could not resolve archive.ubuntu.com, security.ubuntu.com, or azure.archive.ubuntu.com. Code, tests, and runners were all fine. One vendor's name resolution broke, and with it every job that installed a system package.

This post walks through what the public record shows, why retries and fallbacks did not help, and a checklist that makes upstream DNS and package mirrors a dependency you can see and survive.

What the Public Record Shows

The clearest timeline is in actions/runner-images#15296. The reporter saw npx playwright install --with-deps chromium exit with code 100 on three tries at 13:48, 14:02, and 14:24 UTC. The same job on the same image had worked at 13:10 UTC. A second issue, #15297, reported the same failure on ubuntu-22.04 from about 14:00 UTC.

The logs looked like this:

Temporary failure resolving 'azure.archive.ubuntu.com'
Temporary failure resolving 'archive.ubuntu.com'
Temporary failure resolving 'security.ubuntu.com'
E: Unable to fetch some archives

Commenters then narrowed it down quickly:

  • Pinning from ubuntu-26.04 (ubuntu-latest) to ubuntu-24.04 did not help. With #15297 covering 22.04, the failure hit 22.04, 24.04, and 26.04 alike.
  • A plain getent hosts archive.ubuntu.com returned nothing on six attempts between about 16:05 and 16:10 UTC. GitHub, crates.io, and packages.microsoft.com resolved and downloaded normally in the same job. So the failure was name resolution, not apt and not the network.
  • Jobs in the same workflow that did not install apt packages passed.

At 16:23 UTC a GitHub staff member wrote that they were aware of a widespread outage reported by Canonical, and that both Azure and Canonical package mirrors were impacted. At 18:14 UTC they reported the Azure mirror as healthy. When the issue was closed the next day, the closing comment said that the Canonical outage "also broke name resolution for the Ubuntu archive hosts".

Canonical's status feed shows the recovery. Its DNS component returned to Operational at 17:19 UTC on 11 November. cloud-images.ubuntu.com followed at 16:54, login.ubuntu.com at 17:37, ppa.launchpad.net at 17:46, keyserver.ubuntu.com at 18:40, and launchpad.net at 12:15 UTC the next day. We could not find a published root cause from Canonical.

Why Retries Made It Worse

Three details from the public reports are worth noting, because they will happen again with the next vendor.

apt is slow to give up. In archestra-ai/archestra#10584, apt spent about five minutes timing out on each entry in the runner's /etc/apt/apt-mirrors.txt before it exited 100. Another team reported about 27 minutes of apt retries per run. A red build is cheap. A build that holds a runner for half an hour and then turns red costs you queue time for every other job.

apt-get update only warns. One commenter noted that apt update treated the resolution errors as warnings, so apt install ran on stale indexes and failed later. The error you see first is not always the real one.

The fallback was in the same failure domain. In mihazs/lazysh#1268, the install script fell back from the Azure mirror to archive.ubuntu.com. That is also a Canonical name, so the fallback failed in the same way. A release was blocked for hours after four attempts. The issue also points out that a rerun uses the workflow files of the original commit, so a fix you merge during the outage only helps the next release, not the one that is stuck.

Your Hidden Dependency Map

Most pipelines have more external hosts than the team thinks. Use this table to decide what to fix first.

Dependency Typical CI step What happens when its DNS fails Cheap fix Durable fix
Distro archive (archive.ubuntu.com) apt-get install Long timeouts, exit 100 Skip installs already on the image Bake packages into your own image
Distro security mirror apt-get upgrade in Dockerfiles Same as above Pin base image digests Rebuild base images on a schedule, not per build
Browser deps (Playwright --with-deps) E2E setup Pulls apt under the hood --only-shell plus an ldd check Prebuilt test image
PPAs and keyservers add-apt-repository Key import fails before install Ship the key in the repo Mirror the PPA in your own proxy
Cloud images VM or Packer builds Image download fails Cache the image in object storage Promote images through your own registry

The workaround in the Playwright row comes from the #15296 thread. A commenter ran npx playwright install --only-shell chromium, checked the binary with ldd, and called install-deps only if a library showed not found. On ubuntu-latest only fonts were missing.

The Check-and-Cache Checklist

1. List the hosts

Run one full pipeline with egress logging, or read the sources directly:

# Every apt source the runner will use
grep -rhE '^(deb |URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null
# GitHub-hosted runners also read a mirror list
cat /etc/apt/apt-mirrors.txt 2>/dev/null

Add language registries, base images, and any curl | sh installers. Anything on this list can fail your build.

2. Check that fallbacks are really different

A different hostname is not a different failure domain. Compare who answers for each name:

dig +short NS ubuntu.com
dig +short NS kernel.org

If your primary and fallback mirrors share a parent domain or nameservers, they will fail together. In the lazysh issue, mirrors.edge.kernel.org and ftp.osuosl.org were listed as mirrors that still resolved during the outage. Canonical publishes the official archive mirror list. If you build for arm64, check that your fallback carries ubuntu-ports, since not every mirror does.

3. Fall back only when the primary fails

The archestra fix is a good pattern. Before each apt call, a script checks whether azure.archive.ubuntu.com resolves. Only if it does not, the script rewrites the mirror list to https://mirrors.edge.kernel.org/ubuntu/. When DNS is healthy, nothing changes. apt still verifies the indexes against Ubuntu's archive key, so a third-party mirror does not weaken package trust. Never add --allow-unauthenticated or trusted=yes to make a fallback work.

if ! getent hosts azure.archive.ubuntu.com >/dev/null; then
  echo "Ubuntu mirror DNS down, switching to fallback mirror"
  # rewrite /etc/apt/apt-mirrors.txt to the fallback mirror here
fi

4. Do not install what you already have

The lazysh issue proposes skipping apt-get install for packages that dpkg -s reports as installed. Hosted runner images ship many common packages. A job that touches the network only when it must has fewer ways to fail.

5. Put a cache with serve-stale in front of self-hosted runners

Ephemeral runners start with an empty DNS cache, so caching on the runner itself does not help. A long-lived resolver for your runner fleet can help. RFC 8767 describes serving stale answers when the authoritative servers cannot be reached. Unbound implements this with serve-expired, and its serve-expired-ttl defaults to one day. This also explains why resolvers can disagree during an authoritative outage: each one has its own cache contents, TTLs, and stale policy.

DNS is only half of it. A package proxy (apt-cacher-ng, Nexus, Artifactory, or your own registry for prebuilt images) means a vendor outage costs you new packages, not every build.

6. Bound the blast radius

Set timeout-minutes on install steps so a DNS failure fails fast instead of holding runners for half an hour. Bound apt's own retries with -o Acquire::Retries=2. Fail on apt-get update warnings in critical jobs, so you see the real error first.

7. Watch the vendor, not just your build

Canonical's RSS status feed is plain XML and easy to poll. One trap from this incident: a commenter noticed that the archive.ubuntu.com component still showed Operational during the outage. The feed still lists that component's last change as 7 October. A green component is not proof. Test resolution from your own network.

Teach Your Triage the Pattern

The signature of this failure is clear once you know it. Many unrelated jobs fail at the same step, with Temporary failure resolving, across every image version, while other hosts in the same job resolve fine. An on-call engineer who has never seen it may spend an hour rerunning builds.

That pattern fits well in an Akmatori skill: a SKILL.md file with instructions the agent reads when an incident matches its description. A skill for CI failures can tell the agent to group failing jobs by step and error string, check whether the failing names share a vendor, compare with a name that still resolves, and check the vendor's status feed before anyone touches the pipeline. For the wider view on CI as a production dependency, see When GitHub Actions Becomes an Outage, and for your own zones, DNS resilience with anycast and failover.

Conclusion

Nothing in these pipelines was broken. They depended on a vendor's DNS that nobody had listed, with a fallback in the same failure domain and timeouts long enough to fill the queue. List the hosts, make sure fallbacks are really independent, cache what you can, and fail fast on what you cannot. Then the next vendor outage turns into a warning in a log instead of a blocked release.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.

Automate incident response and prevent on-call burnout with AI-driven agents!