The password rotation that quietly broke my deploys

September 26, 2026

Here’s an awkward admission for a post on this particular site: for six days, the site you’re reading didn’t update. The nightly refresh of the live infrastructure page ran every night, failed every night, and told nobody. The page just kept serving the last good build, which looked completely fine — it was only wrong.

The cause was me doing the right thing, slightly incompletely.

The right thing

A password for an admin account on my self-hosted Git server had ended up somewhere it shouldn’t have been. I won’t go into where — the short version is “a place that got copied into other places,” which is how these things usually go.

So I rotated it. I did it properly, too: generated a long random replacement, then actually checked that the old password now got a 401 and the new one got a 200. I found the one automation I knew was using it, moved it to a scoped deploy key, and crossed the item off.

That verification step is the part I was proud of. It’s also the part that gave me false confidence. “The old credential no longer works” is a statement about the credential. It says nothing about who was still holding it.

What was still holding it

Two things, it turned out — and neither of them was on my list.

• The container stack for my media apps. Portainer polls the Git repo every five minutes and redeploys when something changes. Its stored Git credentials were… that password. From the moment of rotation it logged authentication required every five minutes, for four days. Every container kept running happily on its last config, so nothing looked broken — any change I’d pushed in those four days would simply never have arrived.
• This website’s build. Every push to the site’s repo fires a webhook; a small script on the web server pulls the repo, builds the Astro site, and swaps in the new files. The pull used — you can guess — the same password. For six days every build started, failed at the very first git fetch, and stopped. I found it when I swapped in a new photo of the rack and it simply never showed up.

The build log was perfectly honest about all of this. Six days of:

fatal: Authentication failed

The problem is that nobody reads a log that isn’t shouting.

Why nothing noticed

Both failures have the same shape, and it’s worth naming because it’s everywhere:

The pipeline fails, the output stays. A failed deploy doesn’t take the old deploy down. The site kept serving; the containers kept running. Every “is it up?” check I have — and I have dozens — kept reporting green, because it was up. It was just frozen.

Uptime monitoring answers “is it reachable?” It doesn’t answer “is it current?” Those are different questions, and I’d only been asking the first one.

The fix

Two changes, one for each half of the lesson.

1. Every consumer gets its own credential. Both pipelines now use their own read-only, repository-scoped access token. The site’s build can read the site’s repo and nothing else; I tested that a write attempt with it gets refused. If one of those tokens ever leaks, the blast radius is “someone can read a repo”, and I can revoke it without touching anything else. The admin password is now used by exactly zero automations — which is what it should have been all along.

2. Freshness gets its own alarm. The site’s build script now ends by sending a heartbeat to Uptime Kuma. That monitor doesn’t check whether the site responds — it expects to hear from the build, and if 36 hours go by without a successful one, I get a notification. Since the site rebuilds every night, a single missed night is enough to trip it.

This is a “dead man’s switch” rather than a health check, and I like it a lot. It doesn’t care why the build stopped succeeding — expired credential, full disk, a dependency that won’t install — only that it did.

What I’d tell past me

• A rotation isn’t done when the old credential stops working. It’s done when every consumer has been moved. Before rotating, list who uses the thing. Search configs, not just your memory. If you can’t list the consumers, that’s the finding.
• A password is not an automation credential. Humans get passwords. Machines get tokens — scoped to one job, one per consumer, revocable on their own.
• Anything that runs on a schedule needs a “did it run?” alarm, not just an “is it up?” one. Backups, deploys, syncs, certificate renewals: if success is silent and failure is also silent, you’re not monitoring it.
• Find the source, not the copies. When I went hunting for where the password had spread, most of the hits were copies of copies — exports and backups of the one real place it lived. Scrub the copies and leave the source, and it’ll just leak again.

The slightly funny part

The whole reason I noticed the site was stuck is that I was replacing the homepage photo — and the old photo turned out to have also been missing for a while, silently hidden by a bit of defensive code that removes the image if it fails to load. Two quiet failures, stacked on top of each other, on the homepage of a site about running things properly.

It’s fixed now. And if it breaks again, I’ll hear about it within a day and a half, which is a lot better than whenever I next happen to look.

← all posts