Request collapsing

Request collapsing (also called request deduplication) turns identical requests that arrive at the same time into a single origin request. When a popular entry expires, your origin answers once, not once per waiting client.

Client A

Client B

Client C

GraphPilot

Your origin

Query X

Query X

Query X

Identical in-flight requests collapse into one

Query X, one request

Response

Response

Response

Response

Three waiting clients, one origin request.

How it works

1. Several clients send the same operation at the same time.
2. They all resolve to the same cache key.
3. One of them goes to your origin. The rest wait.
4. The response is stored once and served to all of them.

Nothing is configured for this, and nothing can turn it off.

What it applies to

Collapsing follows the cache, so it covers exactly what the cache covers:

	Collapsed
A MISS on a cacheable operation	Yes, this is the case it exists for.
A PASS with a CACHE_PASS_* reason	No. The cache was never consulted, so there is nothing to wait for.
A response that turns out not to be storable (a PASS with a CACHE_SKIPPED_* reason)	No. No entry is written, so there is nothing to hand to the waiting requests. They are released and go to your origin themselves.
Requests with different variables	No. Different variables mean a different key.
Requests in different buckets	No. A private entry and a public one are different entries.

The not-storable row is worth a second look. An operation whose responses are never stored gains nothing from collapsing: every request reaches your origin. Under a burst, the waiting requests may also reach it one after another rather than all at once, so an uncacheable hot operation can take longer under load than the same operation with a freshness lifetime would. If an operation should be cached, check its gp-cache-reason.

Note

Collapsing happens per edge location, like the cache itself. Ten simultaneous requests at one location become one origin request; the same ten spread across ten locations become ten. See edge locations.

With shielding switched on, the misses from those ten locations meet again at the shield and collapse there, so your origin receives one request.

Why it matters

• Thundering herd protection. When a popular entry expires, the next thousand requests for it do not all become origin requests.
• Cold start protection. A newly deployed service, or a location that has not cached an operation yet, takes one origin request per operation rather than one per client.
• It bounds the worst case. Origin load from cacheable traffic is bounded by the number of distinct operations in flight, not by request volume.

To keep even that one request from holding clients up at expiry, give the entry a stale window; see stale windows and request collapsing.

Related

• When a response is cached
• Cache statuses
• Edge locations