Aaron Hakala

QUIC and HTTP/3

If one packet gets lost, TCP holds back everything that came after it, because it has no idea those bytes belong to different responses. One lost packet stalls the whole connection.

This week I fell into HTTP/3 and QUIC, the transport protocol. I haven't worked much with networking and I found some of the concepts from QUIC very interesting. Let's get into it.

TCP is over forty years old. Protocol ossification: middleboxes that are sensitive to the wire format limit the evolvability of the protocol. Wikipedia article on protocol ossification. "QUIC is the first IETF transport protocol to have been designed with deliberate anti-ossification properties." It's everywhere and changing it is close to impossible because many routers, firewalls and middleboxes misbehave when they see something unfamiliar. QUIC sidesteps this by running on top of UDP, usually in user space, with nearly all of its headers encrypted. The network has nothing to latch onto, so the protocol stays free to evolve. HTTP/3 runs on the IETF version of QUIC, RFC 9000.

Here is a short write-up of what I have understood QUIC does, and the problem that made it click for me.

Two layers

Quick background first. Networking is built in layers, and two of them matter here. The transport layer (TCP, or QUIC) gets bytes from one machine to another. It cuts them into packets, keeps track of where each piece belongs, sends again anything that gets lost, and puts everything back in the right order. The application layer (HTTP) knows what the bytes actually mean: this is a request, this is an image, these bytes are the stylesheet. Each layer only reads its own labels. I'll demonstrate what this looks like in a diagram later in this write-up.

Most of the time, that separation is great. But this is where HTTP/2 runs into trouble. HTTP/2 sends lots of responses over one TCP connection, and if one packet gets lost, TCP holds back everything that came after it, because it has no idea those bytes belong to different responses. One lost packet stalls the whole connection.

One packet each

HTTP/2 runs on TCP, and TCP offers exactly one stream per connection. HTTP/2 multiplexes requests by slicing them into frames and interleaving those frames into that single byte sequence. TCP numbers every byte and hands bytes to the application strictly in order. It has no idea that one range is image and the next is CSS. HTTP/2 frames do carry stream identifiers, but they live inside the TCP bytes, where TCP never looks. RFC 9113, §4.1.

responses HTTP/2 frames TCP segments image, stream 1 css, stream 3 A1 A2 A3 A4 B1 B2 B3 A1 B1 A2 B2 A3 B3 A4 seq 0 seq 300 seq 600 seq 900 responses image, stream 1 css, stream 3 A1 A2 A3 A4 B1 B2 B3 HTTP/2 frames A1 B1 A2 B2 A3 B3 A4 seq 0 seq 300 seq 600 seq 900 TCP segments

HTTP/2 cuts each response into frames and takes turns sending them on one connection. TCP then cuts that single line of bytes into segments, without knowing where one response ends and the next begins. The segment at seq 300 is the one shown in the next figure.

HTTP/3 keeps the many streams but hands them to the transport. On QUIC, a packet carries frames, and each stream frame says which stream it belongs to and at what offset. The receiver reassembles every stream on its own. The STREAM frame carries a stream ID, an offset and the data. RFC 9000, §19.8.

HTTP/2 over TCP one TCP segment TCP header seq 300 HTTP/2 frame stream 1 image data HTTP/2 frame stream 3 css data bytes 300 to 599 HTTP/2 names the stream inside the payload TCP sees one numbered run of bytes, nothing more HTTP/3 over QUIC one QUIC packet QUIC header packet 7 STREAM frame id 0 · offset 1200 image data STREAM frame id 4 · offset 800 css data the stream ID and offset are part of QUIC itself QUIC still sees two streams, each with its own byte offsets HTTP/2 over TCP one TCP segment TCP headerseq 300 HTTP/2 frame stream 1 image data HTTP/2 frame stream 3 css data bytes 300 to 599 HTTP/2 names the stream inside the payload TCP sees one numbered run of bytes, nothing more HTTP/3 over QUIC one QUIC packet QUIC headerpacket 7 STREAM frame id 0 · offset 1200 image data STREAM frame id 4 · offset 800 css data the stream ID and offset are part of QUIC itself QUIC still sees two streams, each with its own byte offsets
View as

One packet from each connection. Switch to the transport's view to see what TCP and QUIC can actually read. The stream numbers differ because HTTP/2 client streams are odd and QUIC client streams are multiples of four.

When a packet is lost

This is where the placement of streams matters. Below, both connections send the same packets and lose the same one. Solid squares have been handed to the application. Faded squares have arrived but are being held back.

Seven packets, one image (A) and one stylesheet (B), packet 3 lost. Click a packet in flight, or a button above, to drop or restore it.

Packet 3 carries the second piece of the image. On the TCP connection, the CSS that arrives after it sits in the receive buffer. The bytes are already in the machine's memory, but TCP will not release them until the gap is filled. On the QUIC connection, the CSS is delivered as it arrives, and only the image waits. The simulation resends lost data 110 ms after the original send, roughly when both TCP and QUIC notice that three later packets got through. RFC 5681, §3.2; RFC 9002, §6.1.1.

Watch the retransmission too. TCP resends the same byte range. QUIC puts the lost frames into a brand new packet with a new number, because QUIC packet numbers never repeat. The packet is gone; only the stream data is worth recovering. New numbers solve two problems. When TCP resends, it cannot tell whether an acknowledgement is for the original or the resend, which muddles its round-trip time estimates and loss detection (RFC 9002, §4.2). And the packet number is part of QUIC's encryption input, so it must never be reused (RFC 9001, §5.3).

Two caveats

Streams still share the connection's congestion control. A loss tells the sender to slow down, and it slows down for every stream. The simulation ignores this.

Separate streams do not mean separate packets. A QUIC sender is free to pack frames from several streams into one packet. The spec advises against it unless the packet would otherwise go out mostly empty. If that packet is lost, every stream in it has a gap, and they all wait, just as they would on TCP. "Implementations are advised to include as few streams as necessary in outgoing packets without losing transmission efficiency to underfilled packets." RFC 9000, §13.

Why services mostly stay on TCP

Why hasn't everything moved to HTTP/3? gRPC runs on HTTP/2 and usually keeps a long-lived connection between two services, with many calls in flight at once. gRPC over HTTP/3 exists but is not widely deployed yet. Yet services mostly stay on HTTP/2 over TCP, for at least these reasons.

  1. HTTP/3 optimizes for scenarios with packet loss. Loss is common for clients and rare in data centers, so HTTP/3 targets the former.
  2. Connections are long-lived and the handshake cost is amortized. They can also be warmed up before they take traffic, so QUIC's faster handshake saves little.
  3. The cost of running on the CPU. TCP has had decades of heavy tuning in the kernel. QUIC runs in user space on top of UDP, so the application has to do a lot more work. The gap can be large on fast links. One study measured data rates up to 45% lower over HTTP/3 than over HTTP/2, and traced it to receiver-side overhead such as excessive packets and QUIC's user-space ACKs (Zhang et al., WWW '24). Newer Linux kernels can offload UDP segmentation to the network card, which narrows the gap (Linux kernel docs).

For now, HTTP/2 over TCP is still the king where the network is good.

Sources: RFC 9000, RFC 9114, §1.1 and §1.2.