# How CookieBench measures cookie consent managers

CookieBench loads the same test page with and without each consent manager and reports the difference against that no-CMP control. Every page-load result rests on 15 paired visits per cell, run in Chromium driven by Playwright, on a desktop profile and an emulated mobile profile. Each question gets its own answer, a median with a 95% interval. There is no overall score. This page describes methodology `cookiebench-v3.2.4`.

## What do we measure?

The leaderboard leads with four measurements. They show how fast the banner appears, whether it slows the page, how fast a click responds, and whether trackers wait for consent. The table lists everything else v3 records and where the site shows it. Headline values appear on the leaderboard, CMP page values on each consent manager's page, and engineer values one click deeper.

### Loading

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| CMP script requested (proposed) | When the page first asks for the CMP’s script or configuration. | `diagnostics.requests` | CMP page |
| CMP script loaded (proposed) | When the last byte of the CMP’s main script arrives. | `diagnostics.requests` | CMP page |
| Consent state known | When the CMP’s public API reports the visitor’s consent state. | `consent.resolution` | CMP page |
| Main content in layout | When the collector first sees the article image laid out on screen. | `page.primary-visible` | Engineer view |

### Page vitals

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Main content painted (LCP) | Largest Contentful Paint, compared with the no-CMP control. | `page.lcp` | Headline |
| Article image painted | Element Timing for the article image, whatever wins LCP. | `page.primary-content-paint` | CMP page |
| First paint (FCP) | First Contentful Paint. | `page.fcp` | Engineer view |
| Server response (TTFB) | When the first byte of the page arrives. | `page.ttfb` | Engineer view |
| Layout shift (CLS) | Largest session window of unexpected layout shifts. | `page.cls` | Engineer view |
| Lab INP | The collector cannot finalise it, so we never publish it. | `page.lab-inp` | Withheld |

### Banner

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Banner appears | First frame in which the collector sees the banner laid out on screen. | `banner.visible` | Headline |
| Screen covered | Share of the viewport the banner covers. Descriptive, not tiered. | `banner.coverage` | CMP page |

### After a click

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Click responds | From the click on Accept or Reject to the next screen update. | `action.input-to-next-paint` | Headline |
| Banner gone | Banner and any overlay no longer on screen. | `action.dismissed` | CMP page |
| Choice applied | The CMP’s consent state matches the choice. | `action.consent-effective` | CMP page |
| Allowed scripts running | The scripts that the visitor allowed are complete. | `action.scripts-complete` | CMP page |
| Preferences open | The preferences dialog is on screen after a click on its button. | `action.preferences-visible` | Engineer view |
| Confirmation shown | Visible text changes after the visitor saves. A heuristic. | `action.acknowledgement` | Engineer view |
| Saved in the browser | The CMP reports the choice written locally. | `action.local-persisted` | Engineer view |
| Saved on the server | The CMP’s backend acknowledges the choice. | `action.remote-persisted` | Engineer view |
| Second click responds | A follow-up click on the page 500 ms after the choice. | `action.followup-input-to-next-paint` | Engineer view |

### Network

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Adds to page | Bytes received during the first load, compared with the control. | `initial.transfer` | CMP page |
| Requests | Requests during the first load, and those to other hostnames. | `initial.requests`, `initial.third-party-requests` | Engineer view |
| Body sizes | Compressed and uncompressed response bodies. | `initial.encoded-body`, `initial.decoded-body` | Engineer view |
| Loaded after consent | The same measures for requests that start after the click. | `deferred.transfer`, `deferred.requests`, `deferred.third-party-requests`, `deferred.encoded-body`, `deferred.decoded-body` | Engineer view |

### Main thread

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Freezes page while loading | Long-task time over 50 ms during load, compared with the control. | `initial.long-task-blocking` | CMP page |
| Freezes page after a click | The same measure after Accept or Reject. | `action.long-task-blocking` | Engineer view |
| Collector cadence | How often the collector looked. A diagnostic, not a CMP result. | `observation.frame-interval`, `observation.callbacks` | Raw data only |

### Correctness

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Blocks trackers until consent | Verified, failed or unknown per check, with failure codes. | `scenario checks` | Headline |
| Controlled scripts run | How often each test script ran, and any that ran twice. | `scripts.executions`, `scripts.duplicates` | Engineer view |

### Returning visitors

| Measurement | What it measures | v3 key or source | Shown on |
| --- | --- | --- | --- |
| Banner stays hidden | A visitor who accepted earlier does not see the banner again. | `returning-accepted` | CMP page |
| Consent restored | When the CMP’s public API reports the stored choice on the next visit. | `consent.resolution` | CMP page |

The v3 analysis does not summarise the proposed rows yet. They would come from each visit's request log (`diagnostics.requests`). The log records the URL, resource type, host, start time and bytes of each request. The analysis would match those requests to the CMP's script and configuration hosts.

### What the headline measurements mean

**Banner appears.** `banner.visible` is the first frame in which the collector sees the banner laid out on screen. It is a DOM observation. It does not prove that the browser painted the pixels or that nothing covered the banner.

**Page slowed by.** `page.lcp` is the browser's Largest Contentful Paint inside a fixed observation window. We compare it with the same page without a consent manager.

**Click responds in.** `action.input-to-next-paint` uses the browser's Event Timing for a real, trusted click on Accept or Reject. It measures when the screen next updates, not when the CMP saves the choice. It is a lab measurement, not field INP.

**Blocks trackers.** The test page carries controlled scripts that must not run without consent. Each scenario checks that the scripts stay blocked before consent and after a rejection. It also checks that nothing writes a cookie or sends data early, and that the choice survives a reload. Each check gives one of three results: verified, failed or unknown. Unknown means the evidence was incomplete, not that the CMP passed.

### Which element is the LCP?

LCP reports the element that painted largest, which is not always the page's content. On every visit, v3 records the LCP element. It also times the article image on its own, with `page.primary-content-paint`. On emulated mobile, OneTrust's LCP came 476 ms after the control's, but the article image appeared 1.09 s later. Read on its own, LCP can understate how long readers wait for the article, so CMP pages show both.

### Why is layout shift (CLS) only in the engineer view?

In this run the median `page.cls` was 0 for every measured CMP on every framework, on both profiles. A column of zeros does not help anyone choose, so CLS sits in the engineer view. We still record it. It moves up to the CMP page for any consent manager that shifts the layout.

### What is in the engineer view?

`initial.transfer` counts bytes received during the first load. It includes some protocol overhead and is not the SDK's package size. `initial.long-task-blocking` adds up the parts of long tasks over 50 ms. It is not Lighthouse's Total Blocking Time. Third-party requests mean other hostnames, not other companies. We withhold `page.lab-inp` because the collector cannot finalise it.

## Why is there no overall score?

CookieBench v1 gave each CMP a single score out of 100. An audit of that scorer found it could reward a CMP whose banner never appeared. It converted bytes to kilobytes twice, so size thresholds were off by a factor of 1,024. A run with no valid performance data scored 100 for performance. A change to the weights would not fix those faults.

A single number also hides trade-offs. One team cares most about how fast the banner appears. Another needs IAB TCF and will accept a heavier script. A weighted total would decide that for them, and whoever picks the weights picks the winner.

So each question gets its own answer. The leaderboard sorts by the question you choose. When two CMPs' intervals overlap, they share a tier. We do not order them by noise.

## How are results compared with a page without a consent manager?

We test every CMP against its own control: the same app, the same article and the same framework version, without the consent manager. The runner alternates control and CMP visits so that both see the same machine conditions. It then pairs them.

### The steps, with c15t on desktop

1. Run 15 pairs. In each pair, load the page once without a CMP and once with c15t, each in a fresh browser context. Before those, run 3 warm-up pairs and discard them.
2. For each pair, subtract the control's LCP from c15t's LCP. That gives 15 differences.
3. Take the median of those differences. For c15t it was **+16 ms**. This is the median of the differences, not the difference between two medians.
4. Resample the 15 differences 2,000 times with replacement and take each sample's median. The middle 95% of those medians is the 95% interval: **8 ms to 40 ms**.
5. Read the interval. It sits wholly above zero, so we report that c15t added 16 ms before the main content appeared.

### What "no measurable delay" means

If the interval includes zero, the run cannot tell the CMP apart from no CMP at all. We then print **No measurable delay** instead of a number. It does not prove that the cost is zero. It means that any cost was smaller than the visit-to-visit noise.

| Consent manager (desktop) | Median difference and 95% interval | Printed as |
| --- | --- | --- |
| c15t | +16 ms (8 ms to 40 ms) | +16 ms |
| iubenda | -4 ms (-16 ms to 12 ms) | No measurable delay |
| Cookie Control | +8 ms (-16 ms to 20 ms) | No measurable delay |
| OneTrust | +4 ms (-20 ms to 20 ms) | No measurable delay |

The analysis did not give every result an interval. On desktop, Enzuzo and Ketch have a median difference without one, so we print the median. That median cannot support a comparison.

The interval describes repeat variation in one run on one machine. It does not cover traffic the collector could not see, how representative the test page is, or real visitors' devices. We test many CMPs and questions at once. We do not change the intervals for multiple comparisons.

## How are frameworks handled?

A result compares one consent manager in one framework with that framework's own app without a consent manager. The control uses the same framework version, article and build as the CMP version, so the difference isolates the consent manager.

Every CMP runs on Next.js and on React, Nuxt, Astro, TanStack Start and Script tag. Each framework has its own leaderboard, ranked against its own control. The default leaderboard and comparison pages use Next.js. We never rank one framework's results against another's.

Differences against the control compare fairly across frameworks. Absolute times do not, because they partly reflect the base app. The control apps show this. With no consent manager, the article image painted after 104 ms (script tag) to 144 ms (Next.js) on desktop. On emulated mobile, it painted after 2,852 ms (Astro) to 3,448 ms (Next.js).

Framework support that we do not benchmark comes from vendor documentation. The [feature matrix](https://cookiebench.com/features.md) shows it.

## What do Fast, Moderate and Slow mean? (draft)

Each measured answer gets a tier so that you can read a column at a glance. The thresholds below are a draft. Click response borrows Google's INP bands of 200 and 500 ms. The others are placeholders until the methodology fixes and versions them. A result with no measurable delay counts as Fast.

### Desktop

| Question | Fast | Moderate | Slow |
| --- | --- | --- | --- |
| Banner appears | ≤ 300 ms | ≤ 1.00 s | > 1.00 s |
| Page slowed by | ≤ 50 ms | ≤ 150 ms | > 150 ms |
| Click responds in | ≤ 200 ms | ≤ 500 ms | > 500 ms |
| Adds to page | ≤ 100 KB | ≤ 250 KB | > 250 KB |
| Freezes page for | ≤ 50 ms | ≤ 150 ms | > 150 ms |

### Mobile (emulated)

| Question | Fast | Moderate | Slow |
| --- | --- | --- | --- |
| Banner appears | ≤ 2.50 s | ≤ 4.00 s | > 4.00 s |
| Page slowed by | ≤ 500 ms | ≤ 1.00 s | > 1.00 s |
| Click responds in | ≤ 200 ms | ≤ 500 ms | > 500 ms |
| Adds to page | ≤ 100 KB | ≤ 250 KB | > 250 KB |
| Freezes page for | ≤ 50 ms | ≤ 150 ms | > 150 ms |

Values at or under a threshold fall in that tier. Tracker blocking has no speed tier. It reads Blocked, Partly checked or Leaked.

## What browser and devices do the tests use?

Playwright 1.63.0 on Node 24 drives Chromium 153.0.8010.12 for every visit. Each visit starts in a fresh, isolated browser context with an empty cache. Visits run one at a time on a single machine.

| Setting | Desktop | Mobile (emulated) |
| --- | --- | --- |
| Viewport | 1365 × 768, DPR 1 | 390 × 844, DPR 3 |
| Input | Mouse | Touch, mobile user agent |
| CPU | No slowdown | 4× slowdown |
| Download | 12.5 MB/s | 200 KB/s |
| Upload | 12.5 MB/s | 93.75 KB/s |
| Added latency | None | 150 ms |
| Locale and region | en-GB, region GB (label only) | en-GB, region GB (label only) |

Profile identifiers: `desktop-article-v1` and `mobile-article-v1`. We store the exact settings, hardware and source hashes with every result.

The test page is a shared article with a fixed image as its main content. In the main comparison, we install every CMP in the same Next.js 16.0.7 and React 19.2.1 app. We use the vendor's documented embed or, for c15t, its Next.js SDK. We also measure c15t in its own framework SDKs, each against a control app in the same framework.

Page-load cells run 15 pairs after 3 discarded warm-up pairs. Click cells run Accept, Reject and Save 10 times each. We keep every result, even the failed and incomplete ones. If the banner does not appear before the observation window ends, that is a failure, never a zero.

## How are features checked?

We do not measure feature data. For each CMP, we read the vendor's public documentation and record a status and the page it came from. Each CMP's page in the [feature matrix](https://cookiebench.com/features.md) lists the source.

**Per vendor docs** means the vendor says the feature exists. We did not test it. **Verified by test** means a CookieBench scenario tested it and passed. Verification covers the setup, mode and flows that we ran. It is not a legal certification.

An agent can propose updates from vendor docs. Every proposal carries its sources. A person reviews it before we publish it. Vendor docs are evidence we read, not instructions we follow. When our test adapter cannot drive a feature, that is a gap in our fixture, not proof that the product lacks it.

## What are the known limitations?

- **No geo or egress emulation yet.** The region is a label. Every visit leaves from the same network, so we do not test how a CMP behaves for visitors in other countries. We do not test the geo-targeting statuses in the feature matrix.
- **Mobile is an emulation.** The mobile profile is desktop Chromium with a phone viewport, touch, CPU slowdown and network throttling. It is not a physical phone. The CPU slowdown does not match any specific device.
- **Network totals are sometimes unavailable.** The collector watches the page itself. When a CMP loads iframes or workers that the collector cannot fully observe, we withhold the byte total. We do not print a number that undercounts. Several CMPs show "Not measured" for page weight for this reason.
- **Lab, not field.** Click timings come from one scripted click in a controlled browser. They are not INP from real visitors. LCP here is also not a field Core Web Vital. Real devices, networks and pages vary more.
- **Backends differ in the current data.** In this run, c15t used a controlled local test backend while other CMPs called their live vendor services. That can flatter c15t on network-bound questions. The next publication run uses c15t's hosted backend.
- **Shared machine.** This run used a machine that we did not isolate for the benchmark. Paired visits reduce the effect on differences. Absolute times can still drift.
- **A small tracker workload.** Tracker checks use our controlled scripts. They do not cover every vendor script, or behaviour across tabs, after expiry or long after a withdrawal.

## Who maintains CookieBench?

inth, the company that makes c15t, maintains CookieBench. c15t is one of the consent managers in the benchmark. So we hold it to the same rules as everyone else and publish where it does worse.

- c15t runs on the same test page, profiles, scenarios and thresholds as every other CMP.
- The analysis contains no vendor-specific rules. v3 removed the old scorer and its vendor-name heuristics.
- Results stand as measured. On emulated mobile, c15t added 408 ms before the main content appeared, and the site says so. The feature matrix shows standards that c15t does not support.
- The runner, fixtures, raw results and analysis are public on GitHub (https://github.com/inthhq/cookiebench), so anyone can rerun them.
- Pages that rank CMPs carry no c15t calls to action.

### How vendors ask for a correction

Open an issue on GitHub (https://github.com/inthhq/cookiebench/issues/new). Add a link to your documentation, or a test configuration that shows the banner as your customers see it. A fix to a selector or setup creates a new fixture revision and a full rerun. The earlier result and its failures stay in the public record.

## What has changed?

- **cookiebench-v3.2.2.** We now withhold paint, TTFB and CLS values when a click scenario never receives a trusted input. A replay of the 72 affected visits withheld 144 late paint values. None of them went into a published estimate.
- **cookiebench-v3.2.1.** Correction runs for Didomi and Enzuzo that use their public ready state. Reruns of c15t Nuxt on mobile and TanStack Start on desktop. The original failures stay on record. We do not mix them into the original medians.
- **v1 retired.** We no longer publish the v1 score out of 100 or its rankings. You can still import v1 results as historical data with a legacy label. You cannot compare them with v3.
- **cookiebench-v3.2.0.** First full v3 run: 15 integrations, seven CMPs on Next.js and eight c15t builds across frameworks. It ran on desktop and emulated mobile, with 30 paired visits per page-load cell.
- **v3.1.** Measurement audit. Fixed action timestamps and matched clicks to their native interaction. Withheld network totals when the collector did not observe workers or frames. Added a shared article image so that LCP tracks the main content.

## How should I cite this page?

Cite the methodology version so that readers can find the rules that you used.

> inth. "How CookieBench measures cookie consent managers." Methodology `cookiebench-v3.2.4`. https://cookiebench.com/methodology

Methodology [cookiebench-v3.2.4](https://cookiebench.com/methodology). inth, the team behind c15t, maintains CookieBench. c15t is one of the consent managers we measure ([disclosure](https://cookiebench.com/methodology)). Feature support comes from vendor documentation unless a row says "Verified by test".