Nginx RCE Season: Move Cert Delivery Off the Web Server

Every new nginx CVE brings the same reaction: anyone who cared about certificate handling left nginx long ago. Two replacements come up. Caddy runs ACME itself, or one frontend renews every certificate into Consul and the fleet reads it from there.
Both patterns work. Neither is a patch, and neither takes the private key out of the process that just had a remote code execution bug. This post covers what they change and what to check before you copy them. Patch first: the HTTP/2 triage checklist covers finding every nginx that speaks h2.
What CVE-2026-123007 Actually Allows
The CVE record from F5 describes a flow control window bug in ngx_http_v2_module that can cause a heap buffer overflow, "typically followed by an NGINX worker segmentation fault". It adds that "attackers can execute code on systems with Address Space Layout Randomization (ASLR) disabled or when the attacker can bypass ASLR." F5 scores it CVSS 3.1 8.1 (High, attack complexity High) and CVSS 4.0 9.2 (Critical). nginx.org rates it "major" and fixes it in 1.31.9 and 1.30.7.
So, per the vendor, a crash needs only HTTP/2, and code execution needs an ASLR bypass. Wiz Research, one of the reporters, closes that gap: "Typical memory safety mitigations such as ASLR do not protect against these issues, as attackers can use either vulnerabilities for memory read access prior to code execution." Wiz says it confirmed remote code execution with an ASLR bypass on official 1.31.7 and 1.31.8 builds, over TLS on 64-bit systems, and that ordinary static files can reach the vulnerable path. CISA's SSVC entry in the record still says exploitation "none".
The practical reading: the DoS is easy, and the RCE needs more work, but per Wiz the bug supplies the memory read an ASLR bypass needs. Plan as if the worker can be taken over.
What a Worker Takeover Reaches
A TLS terminator must hold the private key of every certificate it serves, because it signs handshakes. No delivery pattern changes that. What the pattern changes is everything around the key:
- The ACME account key. Whoever holds it can order and revoke certificates for your names.
- The DNS API token. DNS-01 challenges need write access to your zone. A token scoped to a whole zone is worth more to an attacker than any one certificate.
- Write access to the certificate store. If every edge can write to the shared store, one compromised edge can replace certificates for the whole fleet.
- The reload path. If renewal means "write a file and signal the server", a silent reload failure becomes a silent expiry.
So ask "which process holds which of these four things", not "which server is safer".
Pattern 1: Caddy Runs ACME Itself
Caddy's Automatic HTTPS "provisions TLS certificates for all your sites and keeps them renewed", with "no downtime, extra configuration, or separate tooling". By default, certificate management runs in the background of the running server, so there is no separate reload step to forget. If a renewal fails, Caddy retries, switches challenge types, falls back from Let's Encrypt to ZeroSSL, and then backs off for up to 30 days. Its ACME library, CertMagic, is written in Go and creates a new key for each certificate by default.
That removes the reload failure mode and brings memory safety. It also puts the ACME account key and any DNS token inside the internet-facing process. A standalone Caddy does not move cert delivery off the web server; it makes the web server the ACME client.
nginx now has a first-party option too. The ngx_http_acme_module implements ACMEv2 and serves the result through $acme_certificate. Same property: the ACME client lives inside the process that had the bug.
Pattern 2: One Renewer Writes to Consul, the Fleet Reads
The second pattern splits roles: one host renews, the store holds the result, edges read. It comes in two shapes.
Caddy cluster on Consul storage. The caddy-tlsconsul plugin stores CertMagic data in Consul KV. Per Caddy's storage docs, Caddy instances that use the same storage "coordinate certificate management as a cluster". The plugin's documented ACL policy gives each instance write on the key prefix and on sessions. Every edge is a writer: a cluster, not a single renewer.
Renewer plus consul-template for nginx. A dedicated host runs the ACME client and writes PEMs into KV. On each edge, consul-template renders the cert and key to disk and runs a reload command. Edges hold only a read token. This shape actually moves the ACME key and DNS token off the edge.
Check These Before You Put Private Keys in Consul
The data directory is outside Consul's threat model. The Consul security model says every server persists "all KV, service registrations, ACL tokens" there, and "any read or write to this directory allows an attacker to access and tamper with that data." Treat server disks and backups as key material.
Application-side encryption must use your key. caddy-tlsconsul encrypts values with AES-GCM, but its DefaultAESKey is a constant in the public source, consultls-1234567890-caddytls-32, and the README examples use the same string. If you never set aes_key or CADDY_CLUSTERING_CONSUL_AESKEY, the encryption protects nothing.
ACLs default to allow. In the ACL configuration reference, default_policy defaults to allow; the security model says to set deny. Then give the renewer write on its prefix and each edge read on only the prefixes it serves, so one edge cannot read every site's key.
TLS to the agent and between agents. Without TLS, keys cross the network in plaintext on every read. The plugin's example sets tls_enabled "false" and tls_insecure "true"; change both. The security model also notes that the only way to enforce that no client talks to a server unencrypted is to also enable verify_incoming, which requires client certificates.
File permissions on the edge. consul-template's configuration docs say that if no file exists at the destination, "the permissions are 0644". Set perms = 0600 and user/group on the key template, or the first render makes the key world-readable.
Reloads that fail quietly. nginx re-reads its configuration on HUP. If applying it fails, it "rolls back changes and continues to work with old configuration". A broken render or a mismatched cert and key leaves the old certificate serving, with no outage, until it expires. Use nginx -t && nginx -s reload as the command, make its failure loud, and probe expiry from outside.
To skip reloads, ssl_certificate accepts variables since 1.15.9, but then "a certificate will be loaded for each SSL handshake". ssl_certificate_cache limits that cost.
Decision Table
| Setup | ACME key and DNS token live on | Who can overwrite certs | Renewal needs reload | Main trap |
|---|---|---|---|---|
| ACME client on each nginx host | Every edge | Each edge, for itself | Yes | Silent reload rollback |
nginx ngx_http_acme_module |
Every edge, in nginx | Each edge, for itself | No (variable certs) | ACME client in the exposed process |
| Standalone Caddy | Every edge, in Caddy | Each edge, for itself | No | Same, but memory-safe Go |
| Caddy cluster on Consul | Every edge | Every edge, for all names | No | Default AES key, write tokens everywhere |
| One renewer, Consul, consul-template, nginx | Renewer only | Renewer only | Yes | 0644 keys, allow-by-default ACLs |
If an edge compromise must not hand over the DNS token or swap certificates fleet-wide, only the last row gets there, and it keeps the reload problem.
Checklist Before Copying the Pattern
- Patch nginx to 1.30.7 or 1.31.9, or your distro's fixed build. Migration is not mitigation.
- Write down which process holds the ACME account key and the DNS token today.
- Set Consul
default_policytodeny; scope renewer tokens towriteand edge tokens toreadper prefix. - Enable TLS to Consul agents with certificate verification on.
- Set your own AES key if you use caddy-tlsconsul. Grep your configs for the default string.
- Set
perms = 0600and ownership on every key template. - Make reload failures alert, and monitor certificate expiry from outside the host.
- Treat Consul server disks and their backups as private key storage.
Where an Agent Helps
The inventory is mechanical; the migration call is not. In Akmatori, an SSH connector is read-only by default. Add nginx -T and openssl x509 -noout -enddate -in * as allowed command patterns, and the agent can list which hosts terminate TLS with nginx, which certificates they load, and when each expires. Allowed commands can never contain $, a backtick or a redirect, so keep each check a plain command. A scheduled agent job from the cron jobs API can run that inventory on a cron schedule and post the result to a channel. Choosing Caddy or a central renewer stays with your team.
Conclusion
Moving certificate delivery off nginx does not move the private key out of the worker that serves it. It can move the ACME account, the DNS token, and the power to overwrite certificates fleet-wide. A single renewer writing to Consul does that, if Consul runs with deny-by-default ACLs, TLS, your own encryption key, and 0600 files on the edge. Caddy removes the reload trap but keeps the ACME client on the edge. Patch first, then pick the row that matches your threat.
If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, inventory TLS endpoints, and flag certificates before they expire. Powered by Gcore for global infrastructure reliability.
