Skip to content

A/B testing with a cookie

Goal: half of your visitors see a new product page layout, the other half the current one, and both halves are still served from the cache: one entry per variant, not one per visitor.

Two things stand in the way by default. A Cookie header makes a request credentialed, which keeps it out of the shared cache, and a Set-Cookie on a response keeps that response out of the cache entirely. A bucket cookie carries no identity, so you tell GraphPilot so, and then partition on its value.

The running example is shop-api. The client assigns each visitor a bucket and keeps it in a cookie, ab_bucket=a or ab_bucket=b. The origin renders the new layout for bucket b.

What you need

Section titled “What you need”
  • A deployed service and its gpilot.toml.
  • A client that assigns the bucket itself and stores it in a cookie. Why the client and not the origin is the first pitfall.
Section titled “1. Say the cookie carries no session”
gpilot.toml
[cache.cookies]
allow = [
  "GAESA", "GCLB", "AWSALB", "AWSALBCORS", "AWSELB", "ARRAffinity", "ARRAffinitySameSite",
  "BIGipServer*", "__cf_bm", "__cflb", "incap_ses_*", "visid_incap_*",
  "ab_bucket",
]

Setting cache.cookies.allow replaces the default list rather than adding to it, so the defaults are restated and ab_bucket added at the end. Matching is case-sensitive: AB_Bucket would not match.

An allowed name has three effects: a Cookie header made only of allowed names does not make the request credentialed, an allowed name in a Set-Cookie does not refuse storing the response, and an allowed name is never written to the cache entry. The cookie still reaches your origin and your client untouched. See what cache.cookies.allow actually does.

2. Partition on the variant

Section titled “2. Partition on the variant”
gpilot.toml
[[cache.policy.vary]]
name           = "x-ab-variant-b"
value          = "in(cookie:ab_bucket, ['b']) || 'false'"
pass_to_origin = "plain"

in(...) is a truth, so the rule derives exactly one of two values: true for bucket b, and false for bucket a, for a visitor without the cookie (through || 'false'), and for any value somebody makes up. Two values mean two cache entries per response, however many cookie values arrive. Your origin receives x-ab-variant-b: true or x-ab-variant-b: false and renders from that header, not from the cookie.

Partitioning on cookie:ab_bucket directly would also work for well-behaved clients, but every value a caller sends would become an entry of its own. See cardinality.

3. Deploy and send three requests

Section titled “3. Deploy and send three requests”
Terminal window
ask() {
  curl -sS -D- -o /dev/null "https://$GRAPHPILOT_HOST/graphql" \
    -H 'content-type: application/json' \
    "$@" \
    -d '{"query":"query ProductPage { product(id: \"p01\") { id name price } }"}' \
    | grep -i '^gp-cache:'
}


ask -H 'cookie: ab_bucket=b'   # variant B
ask -H 'cookie: ab_bucket=b'   # variant B again
ask -H 'cookie: ab_bucket=a'   # variant A
ask                            # no cookie: also variant A
gp-cache: MISS
gp-cache: HIT
gp-cache: MISS
gp-cache: HIT

The second request reads the entry the first one stored. Bucket a gets its own entry, and the visitor without a cookie shares it, because both derive false.

4. Read it in Insights

Section titled “4. Read it in Insights”

In Insights, a request’s Cache tab shows Segmentation Vary, split by x-ab-variant-b. The value is a keyed pseudonym by default, and with only two possible values there are only two pseudonyms: copy one into the Segment filter to see every request of that arm, or use Requests in this bucket on the Cache tab. The Vary-segmented preset lists the requests that were served from a segmented entry.

Pitfalls

Section titled “Pitfalls”
  • Assign the bucket in the client. An allowed cookie is never written to the cache entry, so a cache hit never hands out a Set-Cookie: ab_bucket=.... If your origin assigned the bucket, only the visitors whose first request happened to miss would ever get one. Set the cookie in your client, or from an endpoint that is not cached.
  • Every other cookie still counts. A request that also carries a session cookie, or an analytics cookie such as _ga, is credentialed again: ab_bucket on the allowlist does not vouch for its neighbours. Those requests are cached only if a private policy identifies the caller.
  • A test with more arms needs more rules or a fixed value set. in(...) answers one question per rule. For three arms, give the client a fixed list of values and partition on lower(cookie:ab_bucket) || 'a', accepting that a made-up value gets an entry of its own.
  • Ending the test does not clear the cache. Entries keep their variant until they expire. Purge the affected responses when you switch the winner on for everyone; see purging through the API and CLI.
Section titled “Related”