How we hunt request smuggling without breaking anything

Request smuggling has a reputation for being a dark art, and the demos do not help. Someone pastes a strange-looking request into a tool, a server does something it should not, applause. What the demo skips is the ninety percent of the work that makes the result trustworthy, and that part is unglamorous and it is the entire job. Here is how we actually do it. But first, what the bug even is.

Almost no website is a single computer. Your request hits a front-of-house server, a proxy or a load balancer or a CDN edge, which reads it, decides where it goes, and forwards it to the actual application behind it. Both of them have to agree on one boring thing: where your request ends and the next person's begins. They read the same stream of bytes off the same shared connection and chop it into separate requests. Smuggling is what happens when you can craft a request the two of them chop differently. The front-of-house thinks your request ended here; the back-end thinks it ended there. The leftover bytes, the part the front-end thought was still yours but the back-end did not, get glued onto the front of whoever's request comes next.

That is the whole trick, and it is nastier than it sounds, because the next request on that connection might be a stranger's. An attacker who controls those leftover bytes can prepend a hidden request to your traffic. They can redirect your session to a page they wrote, harvest the response that was meant for you, slip a poisoned answer into a shared cache so everyone gets served the attacker's content, or walk straight past a login check the front-end already waved through. One malformed request, no malware, no stolen password, just two servers that disagreed about punctuation. That is why it is worth hunting, and why it has to be hunted carefully.

Start with the edge, not the exploit. A smuggling bug is a disagreement between a front-end and a back-end, so the first thing worth knowing is whether a given service even has a front-end and a back-end that could disagree. That is recon, and ours is entirely passive. We read the published response headers, look at which protocols the server offers, and notice when a Via header reveals two hops where one stack hands off to another. A single hop cannot disagree with itself. A hand-off is where the seam lives. None of this sends anything unusual; it is the same traffic any browser generates.

Fingerprint, then form a hypothesis, not a claim. If the edge advertises a stack with known framing history, or the hand-off looks like an old proxy in front of a modern one (or the other way round), that is a lead. It tells us where to look, not what we will find. We have trained ourselves hard on this distinction, because the failure mode of every scanner is to print a CVE match and call it a vulnerability. A version string is a hypothesis. The probe is what tests it.

Prove it in a lab you own, against a back-end that can't lie. This is the centre of the whole method. We stand up the real proxies (nginx, HAProxy, Envoy, and the HTTP/3 terminators) in a local rig, and behind each one we put a back-end of our own whose only job is to log the literal bytes the proxy forwarded to it. Then we point Keith's probes at the front and read the truth off the back. There is no inference involved. Either the proxy forwarded a request whose framing disagrees with what we sent, or it did not.

Most of the time, it did not. We have swept current nginx, HAProxy across several major versions, and Envoy with our full probe set, and they did the right thing. They reconciled the framing, rejected the mismatch, and stripped the headers that do not belong. Those are clean negatives, and we record them as carefully as we would record a hit, because a method that only ever finds things is a method that is fooling itself. Our earlier post on the HTTP/3 FIN desync was exactly this: a clean negative against nginx, proven over the wire rather than asserted.

The HTTP/3 frontier is wide open, and that is the discipline test. HTTP/3 smuggling is almost unexplored, not because the surface is small but because almost every tool speaks HTTP/1 and /2 only, and the few that speak /3 do it through a conformant library that will not let you send the bug. Keith was built to send it. That makes /3 the most productive place to look right now, and because it is new and the false-positive rate could be brutal, it is also where "send the exact malformed image, read the exact downstream truth" matters most. A timing oracle that says the back-end hung is a hint. The back-end's own byte log is the proof.

Patch gaps reward permutation. When a vendor fixes a framing bug quietly, the fix is sometimes keyed to the exact byte pattern that was reported, or only to one protocol's intake path. So we keep a permutation matrix that varies one axis at a time: the method, the precise way a Transfer-Encoding header is obfuscated, how a chunk size is written, and which protocol carries the request to the back-end (raw HTTP/1, or downgraded from /2 or /3). A fix that matches one exact string can miss the same bug wearing a tab instead of a space, or arriving over a downgrade path the patch never touched. The matrix is how you find out methodically, instead of by guessing.

And the line we don't cross. Two rules sit above all of this. We only ever point live probes at infrastructure we are explicitly authorised to test, through sanctioned disclosure programs, with their rate limits and identifying headers honoured. And impact, when we demonstrate it, is always demonstrated to ourselves: our own request prefixing our own follow-up request on a single connection. We never poison a cache key a real user might hit, and never hijack a stranger's request to prove a point. If a finding cannot be shown without harming a third party, it does not get shown. It gets written up and disclosed.

That last point is worth dwelling on, because it is where the build-in-public log goes quiet on purpose. When a real terminator does forward the malformed image, when the lab back-end logs a request that disagrees with what we sent, that is the good day. It is also the day the work stops being a blog post and becomes a coordinated-disclosure email to the people who maintain the code. The bytes are the contribution. The fix landing upstream is the point.

Simon Morley researches infrastructure security and is the founder of NullRabbit. About / contact.