How CookieBench measures cookie consent managers

The leaderboard leads with four measurements. They show how fast the banner appears, whether it slows the page, how fast a click responds, and whether trackers wait for consent. The table lists everything else v3 records and where the site shows it. Headline values appear on the leaderboard, CMP page values on each consent manager’s page, and engineer values one click deeper.

What the headline measurements mean

Banner appears. banner.visible is the first frame in which the collector sees the banner laid out on screen. It is a DOM observation. It does not prove that the browser painted the pixels or that nothing covered the banner.

Page slowed by. page.lcp is the browser’s Largest Contentful Paint inside a fixed observation window. We compare it with the same page without a consent manager.

Click responds in. action.input-to-next-paint uses the browser’s Event Timing for a real, trusted click on Accept or Reject. It measures when the screen next updates, not when the CMP saves the choice. It is a lab measurement, not field INP.

Blocks trackers. The test page carries controlled scripts that must not run without consent. Each scenario checks that the scripts stay blocked before consent and after a rejection. It also checks that nothing writes a cookie or sends data early, and that the choice survives a reload. Each check gives one of three results: verified, failed or unknown. Unknown means the evidence was incomplete, not that the CMP passed.

Which element is the LCP?

LCP reports the element that painted largest, which is not always the page’s content. On every visit, v3 records the LCP element. It also times the article image on its own, with page.primary-content-paint. On emulated mobile, OneTrust’s LCP came 476 ms after the control’s, but the article image appeared 1.09 s later. Read on its own, LCP can understate how long readers wait for the article, so CMP pages show both.

Why is layout shift (CLS) only in the engineer view?

In this run the median page.cls was 0 for every measured CMP on every framework, on both profiles. A column of zeros does not help anyone choose, so CLS sits in the engineer view. We still record it. It moves up to the CMP page for any consent manager that shifts the layout.

What is in the engineer view?

initial.transfer counts bytes received during the first load. It includes some protocol overhead and is not the SDK’s package size. initial.long-task-blocking adds up the parts of long tasks over 50 ms. It is not Lighthouse’s Total Blocking Time. Third-party requests mean other hostnames, not other companies. We withhold page.lab-inp because the collector cannot finalise it.

CookieBench v1 gave each CMP a single score out of 100. An audit of that scorer found it could reward a CMP whose banner never appeared. It converted bytes to kilobytes twice, so size thresholds were off by a factor of 1,024. A run with no valid performance data scored 100 for performance. A change to the weights would not fix those faults.

A single number also hides trade-offs. One team cares most about how fast the banner appears. Another needs IAB TCF and will accept a heavier script. A weighted total would decide that for them, and whoever picks the weights picks the winner.

So each question gets its own answer. The leaderboard sorts by the question you choose. When two CMPs’ intervals overlap, they share a tier. We do not order them by noise.

We test every CMP against its own control: the same app, the same article and the same framework version, without the consent manager. The runner alternates control and CMP visits so that both see the same machine conditions. It then pairs them.

The steps, with c15t on desktop

1. Run 15 pairs. In each pair, load the page once without a CMP and once with c15t, each in a fresh browser context. Before those, run 3 warm-up pairs and discard them.
2. For each pair, subtract the control’s LCP from c15t’s LCP. That gives 15 differences.
3. Take the median of those differences. For c15t it was +16 ms. This is the median of the differences, not the difference between two medians.
4. Resample the 15 differences 2,000 times with replacement and take each sample’s median. The middle 95% of those medians is the 95% interval: 8 ms to 40 ms.
5. Read the interval. It sits wholly above zero, so we report that c15t added 16 ms before the main content appeared.

What “no measurable delay” means

If the interval includes zero, the run cannot tell the CMP apart from no CMP at all. We then print No measurable delay instead of a number. It does not prove that the cost is zero. It means that any cost was smaller than the visit-to-visit noise.

The analysis did not give every result an interval. On desktop, Enzuzo and Ketch have a median difference without one, so we print the median. That median cannot support a comparison.

The interval describes repeat variation in one run on one machine. It does not cover traffic the collector could not see, how representative the test page is, or real visitors’ devices. We test many CMPs and questions at once. We do not change the intervals for multiple comparisons.

A result compares one consent manager in one framework with that framework’s own app without a consent manager. The control uses the same framework version, article and build as the CMP version, so the difference isolates the consent manager.

Every CMP runs on Next.js and on React, Nuxt, Astro, TanStack Start and Script tag. Each framework has its own leaderboard, ranked against its own control. The default leaderboard and comparison pages use Next.js. We never rank one framework’s results against another’s.

Differences against the control compare fairly across frameworks. Absolute times do not, because they partly reflect the base app. The control apps show this. With no consent manager, the article image painted after 104 ms (script tag) to 144 ms (Next.js) on desktop. On emulated mobile, it painted after 2,852 ms (Astro) to 3,448 ms (Next.js).

Framework support that we do not benchmark comes from vendor documentation. The feature matrix shows it.

Each measured answer gets a tier so that you can read a column at a glance. The thresholds below are a draft. Click response borrows Google’s INP bands of 200 and 500 ms. The others are placeholders until the methodology fixes and versions them. A result with no measurable delay counts as Fast.

Playwright 1.63.0 on Node 24 drives Chromium 153.0.8010.12 for every visit. Each visit starts in a fresh, isolated browser context with an empty cache. Visits run one at a time on a single machine.

The test page is a shared article with a fixed image as its main content. In the main comparison, we install every CMP in the same Next.js 16.0.7 and React 19.2.1 app. We use the vendor’s documented embed or, for c15t, its Next.js SDK. We also measure c15t in its own framework SDKs, each against a control app in the same framework.

Page-load cells run 15 pairs after 3 discarded warm-up pairs. Click cells run Accept, Reject and Save 10 times each. We keep every result, even the failed and incomplete ones. If the banner does not appear before the observation window ends, that is a failure, never a zero.

We do not measure feature data. For each CMP, we read the vendor’s public documentation and record a status and the page it came from. Every mark in the feature matrix links to that page.

Per vendor docs means the vendor says the feature exists. We did not test it. Verified by test means a CookieBench scenario tested it and passed. Verification covers the setup, mode and flows that we ran. It is not a legal certification.

An agent can propose updates from vendor docs. Every proposal carries its sources. A person reviews it before we publish it. Vendor docs are evidence we read, not instructions we follow. When our test adapter cannot drive a feature, that is a gap in our fixture, not proof that the product lacks it.

• No geo or egress emulation yet. The region is a label. Every visit leaves from the same network, so we do not test how a CMP behaves for visitors in other countries. We do not test the geo-targeting statuses in the feature matrix.
• Mobile is an emulation. The mobile profile is desktop Chromium with a phone viewport, touch, CPU slowdown and network throttling. It is not a physical phone. The CPU slowdown does not match any specific device.
• Network totals are sometimes unavailable. The collector watches the page itself. When a CMP loads iframes or workers that the collector cannot fully observe, we withhold the byte total. We do not print a number that undercounts. Several CMPs show “Not measured” for page weight for this reason.
• Lab, not field. Click timings come from one scripted click in a controlled browser. They are not INP from real visitors. LCP here is also not a field Core Web Vital. Real devices, networks and pages vary more.
• Backends differ in the current data. In this run, c15t used a controlled local test backend while other CMPs called their live vendor services. That can flatter c15t on network-bound questions. The next publication run uses c15t’s hosted backend.
• Shared machine. This run used a machine that we did not isolate for the benchmark. Paired visits reduce the effect on differences. Absolute times can still drift.
• A small tracker workload. Tracker checks use our controlled scripts. They do not cover every vendor script, or behaviour across tabs, after expiry or long after a withdrawal.

inth, the company that makes c15t, maintains CookieBench. c15t is one of the consent managers in the benchmark. So we hold it to the same rules as everyone else and publish where it does worse.

• c15t runs on the same test page, profiles, scenarios and thresholds as every other CMP.
• The analysis contains no vendor-specific rules. v3 removed the old scorer and its vendor-name heuristics.
• Results stand as measured. On emulated mobile, c15t added 408 ms before the main content appeared, and the site says so. The feature matrix shows standards that c15t does not support.
• The runner, fixtures, raw results and analysis are public on GitHub, so anyone can rerun them.
• Pages that rank CMPs carry no c15t calls to action.

How vendors ask for a correction

Report a correction, or open an issue on GitHub. Add a link to your documentation, or a test configuration that shows the banner as your customers see it. A fix to a selector or setup creates a new fixture revision and a full rerun. The earlier result and its failures stay in the public record.

1. cookiebench-v3.2.2

   We now withhold paint, TTFB and CLS values when a click scenario never receives a trusted input. A replay of the 72 affected visits withheld 144 late paint values. None of them went into a published estimate.

2. cookiebench-v3.2.1

   Correction runs for Didomi and Enzuzo that use their public ready state. Reruns of c15t Nuxt on mobile and TanStack Start on desktop. The original failures stay on record. We do not mix them into the original medians.

3. v1 retired

   We no longer publish the v1 score out of 100 or its rankings. You can still import v1 results as historical data with a legacy label. You cannot compare them with v3.

4. cookiebench-v3.2.0

   First full v3 run: 15 integrations, seven CMPs on Next.js and eight c15t builds across frameworks. It ran on desktop and emulated mobile, with 30 paired visits per page-load cell.

5. v3.1

   Measurement audit. Fixed action timestamps and matched clicks to their native interaction. Withheld network totals when the collector did not observe workers or frames. Added a shared article image so that LCP tracks the main content.

Cite the methodology version so that readers can find the rules that you used.

inth. “How CookieBench measures cookie consent managers.” Methodology cookiebench-v3.2.4. https://cookiebench.com/methodology