We moved our images to a CDN and the origin kept serving them

Our documentation is heavy with screenshots. Every portal feature, captured in light and dark, at two widths, across ten locales, which adds up to something north of a hundred megabytes of images. All of it was sitting in the app's own static bundle, served from the origin, which is the wrong place for a hundred megabytes of pictures that never change. So we moved it to Cloudflare R2 and put a CDN in front, one per environment: cdn.authagonal.io for production and a matching CDN host for our dev site.

The app picks the right CDN host at runtime by looking at the hostname it is being served from. If the page is on authagonal.io, images come from cdn.authagonal.io. Simple, no build-time configuration, one image reference that resolves correctly on either environment. We shipped it, watched the browser pull every image cleanly from the CDN, and then looked at the origin's traffic. The origin was still serving the images. Every one, on first load, exactly as before.

The page a crawler sees is not the page a browser builds

To explain why, I have to explain what our marketing pages actually are. They are a single-page app, but a single-page app is a blank shell until JavaScript runs, and a blank shell is bad for search engines and bad for the time it takes the first meaningful thing to appear. So at build time we prerender: we start the site, open each page in a headless browser, let it render fully, and save the resulting HTML. That prerendered HTML is what we serve first. It already has the content in it, the crawler sees a real page, the user sees text and images immediately, and then the JavaScript takes over.

Here is the problem hiding in that description. The prerender runs the site on 127.0.0.1, a loopback address on the build machine. So when the app asks "what hostname am I on, so I can pick a CDN," the answer during prerender is "a local IP address," which matches neither authagonal.io nor the dev host. The runtime host-derivation, the clever part, quietly falls through to its only remaining option: the origin-relative path. And that origin path is what gets frozen into the prerendered HTML.

So every visitor received HTML with origin image URLs baked in, and the browser did the obvious thing: it fetched them, from the origin, on first paint. A moment later the JavaScript re-rendered the page, re-derived the hostname, this time getting the real one, and swapped the images to the CDN. Two fetches for every image, the first one hitting exactly the server we were trying to spare, and completely invisible unless you were watching the origin's logs, because the page looked and worked perfectly.

The fix that made things worse for eleven minutes

The first fix was the tempting one. If the problem is that a wrong URL gets frozen into the prerendered HTML, then emit no URL during prerender at all. Leave the image source blank in the snapshot, and let the JavaScript fill in the correct CDN URL once it runs and knows the real hostname. No wrong URL, no origin fetch.

It shipped, and it was wrong, and we replaced it about eleven minutes later. Blanking the image source in the prerendered HTML defeats the entire reason we prerender. The crawler that reads the static page now sees no image, and the browser has nothing to show until JavaScript has run, which is the slow path we build the prerendered HTML specifically to avoid. We had removed the double fetch by removing the image from the one version of the page that exists before JavaScript. That trades a traffic problem for a search-and-first-paint problem, which is a worse trade.

The lesson from those eleven minutes: the prerendered HTML has to carry a real, working image reference. The bug was never that a URL was present. It was that the wrong URL was present, and blanking it just swaps one defect for another. What we actually needed was for the correct URL to be knowable at prerender time, and it is not, because at prerender time nobody knows which environment will serve the page.

Deciding at the last possible moment

The correct URL depends on the request host, and the request host is not known until there is a request. Prerender happens once, at build, and produces one file that is served on both environments. So the environment-specific part of the URL cannot be decided at build. It has to be decided per request, and the only thing in the stack that sees every request and its host is the web server at the edge.

So the prerender no longer writes a URL or a blank. It writes a placeholder, a literal sentinel string, https://cdn.authagonal.io, in place of the CDN host. The prerendered HTML that leaves the build says https://cdn.authagonal.io/screenshots/whatever.webp, which is neither an origin URL nor an empty source. It is a URL with a hole in it.

nginx fills the hole on the way out. It maps the request's host to the right CDN base, the non-production host to its own dev CDN, authagonal.io to https://cdn.authagonal.io, anything else to empty so the URL collapses back to origin-relative as a safe local fallback. Then a single directive rewrites the sentinel to that value as the HTML streams to the client:

map $host $asset_cdn_base {
    default              "";
    "~*example\.dev$"    "https://cdn.example.dev";
    "~*authagonal\.io$"  "https://cdn.authagonal.io";
}

sub_filter 'https://cdn.authagonal.io' $asset_cdn_base;

The static HTML now leaves the origin already carrying real CDN URLs, correct for whichever host asked. The browser fetches straight from the CDN on first paint. There is no wrong first fetch to undo, because there is no wrong URL, and there was never a blank one either. The crawler sees a complete page with real image links. One file serves both environments, and neither one needs a separate build.

There is a small elegance to which parts of the response get rewritten. The rewrite is scoped to HTML only, which is nginx's default. That matters because the same sentinel string appears in the JavaScript bundle too, as the fallback branch of the host-derivation code. On the client that branch is dead code, since the browser has a real hostname and never emits the sentinel, so we specifically do not want nginx touching the bundle. Leaving the rewrite at its HTML-only default gets that for free. We learned, incidentally, that writing the default out explicitly is worse than leaving it off, because a redundant declaration makes nginx warn about a duplicate.

Why not just rewrite the file once on startup

The obvious objection is that this is a lot of per-request work for a value that only has two possibilities. Why not rewrite the HTML once when the container boots, bake in the right host, and serve a static file with no filter?

Because the container cannot write to itself. The static site runs in nginx as a non-root user with a read-only root filesystem and every Linux capability dropped, with a single small writable scratch directory for nginx's own temp files. That is a deliberate hardening posture: a web server that serves untrusted traffic should not be able to modify the files it serves, so that an attacker who finds a foothold finds nothing to rewrite. A boot-time rewrite of the HTML would need exactly the write access we took away on purpose. The read-only filesystem is not an obstacle to work around, it is a property we want, and it rules out the boot-time rewrite cleanly. The per-request rewrite touches no files at all, which is why it is compatible with a server that owns nothing it can change.

We also did not want to bake the CDN host in at build and produce two different image bundles, one per environment, because that reintroduces the environment-specific build we had just gotten rid of. The point of host-derivation was one artifact for both environments. The sentinel keeps that promise: one build, one file, the environment resolved at the edge at the last possible moment.

Prerendering freezes environment values, so resolve them at the edge

Prerendering freezes a moment. Anything your app decides by looking at its environment gets decided, at prerender time, against the build's environment, which is a loopback address and nobody's real host. So a value that is correct at runtime becomes wrong the instant it is snapshotted, and it fails silently, because the prerendered page still renders, just pointing at the wrong place.

The general fix is to not resolve environment-specific values during prerender at all. Emit a placeholder, carry it through the static artifact intact, and resolve it at the layer that actually sees the environment, which for a per-request value is the edge. It is late binding, applied to a string in an HTML file: leave the hole open through every stage that does not know the answer, and fill it at the one stage that does.

If you would rather your identity provider's own documentation and assets were already served correctly off the edge, that is one of the many small things Authagonal has already argued out so you can spend the argument on your own product instead.