Our lockout after five wrong passwords never locked anyone out

Account lockout is table stakes. Five wrong passwords and the account is locked for ten minutes, which turns online password guessing from a real threat into a non-starter: an attacker gets five tries per account per ten minutes, which is nothing. We had it. The threshold was five, the window was ten minutes, the code that checked it was correct, and it worked perfectly every time we tested it, because every time we tested it we typed the wrong password five times in a row, one after another, like a person.

An attacker is not a person typing one after another. An attacker sends five hundred at once. And when we did that, the counter that was supposed to stop at five barely moved, and the account never locked. You could throw an unlimited number of guesses at a single account and the one control designed to stop exactly that sat there doing nothing, not because the logic was wrong but because it assumed it would only ever be asked one question at a time.

A counter that loses count

The failed-login code did the obvious thing. On a wrong password it loaded the user, added one to their failed-attempt count, checked whether the new value had reached five, and saved the user back. Read, modify, write. Written out as a person would think about it, it is correct.

Written out as two requests arriving at the same moment, it is not. Both read the count, and both read the same value, say three. Both add one and get four. Both write four back. Two failed attempts happened, and the count went from three to four. One of the increments simply vanished, overwritten by the other, because both were working from the same stale starting point and the second write landed on top of the first as if the first had never happened.

Now fire five hundred wrong passwords at once. They pile onto each other in exactly this way, each one reading a low count and writing a slightly-less-low count, increments evaporating by the hundred. The counter drifts up a little and stops nowhere near five. The lockout threshold is never crossed, not because the check is wrong, it is checked faithfully every single time, but because the number it checks is a lie assembled by a crowd of requests all overwriting each other. Brute-force protection defeated not by a logic flaw but by concurrency, which is the kind of flaw that passes every serial test you will ever write and fails the moment it meets a real attacker, who is the only party with a motive to send requests in parallel.

Where this bug lived matters. It was not buried in some storage layer. It was right there in the login handler, in plain sight, a ++ and a save, the most natural code in the world. Nothing about reading it suggests danger. You have to be thinking about two of it running at once for the bug to appear, and reading code one line at a time is precisely the state of mind in which you are not.

Making a counter atomic on a store that cannot count

The fix has to make the whole read-modify-write indivisible: no other attempt may slip in between our read and our write. The interesting part is that we are on a key-value store with no increment operation. You cannot tell it "add one to this field," which is the primitive a database with atomic counters would hand you. You can only read a row and write a row.

What the store does give you is a conditional write. Every row carries a version tag that changes on every write, and you can say: write this row back, but only if its version is still the one I read. If someone else wrote in the meantime, the version moved, and your write is rejected rather than silently clobbering theirs.

That turns the lost-increment race into a caught collision. Read the row and its version. Add one. Try to write back conditioned on that version. If it succeeds, your increment definitely landed on top of the value you read, with nothing lost. If it is rejected, it means another attempt beat you to it, so you re-read the now-higher count and try again. Loop on rejection, a bounded handful of times, and every increment is forced to serialize behind the last one instead of overwriting it. The count becomes true again, and the five-hundred-at-once attack now walks the counter straight to five and trips the lock, which is the entire point.

We carried the same shape to our other storage backend, which also has no increment but also offers a conditional write, this time gated on the counter value itself rather than a version tag: write the new count only if the stored count is still what I read. Same guarantee, expressed in that store's vocabulary. The lesson generalises past any one database: when you lack an atomic increment, a conditional write plus a retry loop reconstructs one, because the condition is what refuses the stale overwrite that was losing your count.

One deliberate asymmetry is worth noting. We made the failure path atomic and left the success path, which resets the counter to zero on a correct login, as a plain last-writer-wins write. Concurrency does not threaten a reset: several successful logins racing all want to write zero, and it does not matter which wins. Only the increment is adversarial, because only the increment is the thing an attacker is trying to hold down. You spend the cost of conditional writes and retries where an adversary is pushing, and not where they are not.

Assume someone is counting

The real value of this bug was the frame of mind it forced, because the same day we fixed it we went looking for everything else that quietly assumed one-request-at-a-time or assumed nobody was measuring, and found a small cluster of siblings.

Our login responses were telling attackers which accounts existed. A missing account, a disabled one, a locked one, and a wrong password each produced a distinguishable answer, and the no-such-user path even returned faster because it skipped the password hash entirely, so an attacker could learn who has an account by watching either the status or the clock. We collapsed the answers: an unknown user and a wrong password now return the identical response, and an unknown user is still verified against a fixed dummy hash so the timing is the same whether or not the account exists. The account exists or not, and you cannot tell from the outside.

Registration had the same leak in a harder form, because you cannot simply always say success when someone tries to register an email that is taken, since that would seem to hand them the account. So a registration for an already-registered address now returns the very same neutral success a fresh signup returns, spends the same password-hash time, creates nothing, and instead sends the real owner an email saying somebody tried to register their address. The attacker learns nothing. The actual owner is quietly told.

And the password-reset form was an email cannon: no per-address limit, so you could make us fire reset emails at any address on repeat, an attack aimed at a third party and at our own sending reputation. It now rate-limits per target email and, over the cap, skips the send while returning the same success response as always, so throttling itself does not become a new way to probe which addresses exist.

Every one of those is the lockout bug's lesson wearing different clothes. Design as though an attacker is sending requests in parallel, timing your responses, and counting. Because the one thing you can be sure of is that the person attacking your login is not doing it one polite request at a time.

Why a security counter has to be tested concurrently

A concurrency bug in a security control is worse than a concurrency bug anywhere else, for a specific reason: the party with the strongest motive to trigger it is the attacker, and the conditions that trigger it, many simultaneous requests, are exactly the conditions an attack creates and a test almost never does. So a lockout, a rate limit, a quota, any control built on a counter, cannot be validated by incrementing it in a loop and watching it trip. It has to be hit concurrently, because concurrency is the attacker's native environment and the serial version of the test proves only that your control works against an adversary who has agreed to be polite.

And when you do harden the counter, the tool on a store without atomic increments is the conditional write: read the version, write only if it has not moved, retry if it has. It rebuilds the atomic increment the database did not give you, out of the one guarantee it did.

If you would rather run on an identity provider that already assumes the attacker is counting in parallel, Authagonal built these controls to be hit all at once, because that is the only way they were ever going to be hit for real.