Nginx rate limiting for French users behind a shared ISP NAT

Two weeks ago our small API (single EU region, Postgres 14, Nginx 1.22 in front, ~40 req/s at peak) started returning 429s to users who insisted they had only refreshed once. The logs told a different story: one /24 was driving 60% of the offending requests. It turned out to be a major French mobile carrier’s CGNAT pool, so thousands of subscribers were sharing a handful of public addresses. Our limit_req_zone $binary_remote_addr was treating all of them as one client.

The fix was not to loosen the limit but to key it on something we actually controlled. We already issue a signed session cookie at login, so we moved the key to that when present, falling back to the address only for anonymous traffic:

map $cookie_session $rate_key {
    default  $binary_remote_addr;
    "~^[A-Za-z0-9._-]{20,}$"  $cookie_session;
}
limit_req_zone $rate_key zone=api:10m rate=20r/s;
limit_req zone=api burst=40 nodelay;

We also added limit_req_status 429 and a Retry-After: 2 header so the mobile app could back off instead of hammering. The map keeps the regular expression narrow: a malformed or spoofed cookie falls through to the address key rather than collapsing every anonymous user into one bucket.

Why key on the session instead of the address: the attacker or the noisy retry loop is usually tied to a client we can identify, and carriers rotate addresses faster than we can maintain a blocklist. The trade-off is memory and fairness. A 10m zone holds roughly 160k keys at this key length, which is fine now but will not scale past a few million monthly sessions without sharding or moving to a Redis-backed limiter. There is also a privacy cost: we are hashing session identifiers in a rate-limit table, so retention needs to be minutes, not days.

Where this breaks: users who share a cookie (kiosks, family tablets) get one budget; clients that never log in still share the carrier NAT problem; and the map regex will silently fail to match if your cookie name differs from what you think. I hit that last one for an hour because the upstream app trimmed the cookie’s quotes differently than my test request. Also, Nginx reloads are not atomic for zone state: after a reload the counters reset, so a burst right after deploy can slip through.

Now the question I am still sitting with: for anonymous traffic behind a shared French mobile NAT, is it better to key on address plus User-Agent, or to push the limit down to a token issued by a lightweight challenge endpoint? I have not measured the false-positive rate of the former, and I would rather hear from someone who runs a public endpoint in France than guess.