Put two CDNs in front of the same origin and measure them, and you will get a winner. Whether you've learned anything depends entirely on the test design: most head-to-head numbers are artifacts of unequal configuration, cold caches, or vantage-point luck, and the "winner" is whichever vendor the mistakes happened to favour. The first-benchmark guide teaches the mechanics; this one is about fairness when the result will pick a vendor.
Why most bake-offs are rigged by accident
The classic accidents: one CDN tested through its tuned production config against another's out-of-the-box defaults; one enjoying weeks of warmed cache against a rival deployed yesterday; test agents that happen to sit in the same buildings as one vendor's infrastructure; compression or image optimization on for one side only; and sample sizes small enough that a single congested evening decides the ranking. None of these require bad faith — they require only haste. The discipline is to treat the bake-off like an experiment with one variable: the CDN. Everything else must be equal, and "everything else" is a longer list than it looks.
The fairness rules
Same origin, same objects, same TTLs, same compression settings, same TLS features, same HTTP versions enabled — written down as a parity checklist and verified on both sides by fetching and diffing response headers before any measurement starts. Test each CDN on its own hostname with DNS you control, at full production-like configuration on both (or vendor-recommended settings on both; either is fair, mixing them is not). Run the candidates simultaneously, not sequentially — a week apart measures the internet's weather, not the vendors. And decide the workload honestly: benchmark the asset mix you actually serve, in the geographies you actually sell to, because a CDN that wins on small-object latency in Europe can lose on large-file throughput in Southeast Asia, and only one of those is your business.
Warm-up and cache parity
Cache state is the biggest silent thumb on the scale. A request served from edge memory and one fetched from origin differ by an order of magnitude, so if one candidate's cache is warm and the other's cold, you are comparing cache states, not networks. The protocol: before the measurement window, run identical warm-up traffic through both candidates from the test geographies — enough to populate the specific POPs your agents will hit — then verify with cache-status headers that both sides are serving hits before the clock starts. Decide separately whether you also want a cold-start comparison (miss-path and origin-fetch behaviour is worth knowing, especially if shielding differs), but never let hit lines and miss lines blend into one average. Log cache status with every sample and segment all results by it.
Vantage points and sample sizes
Measure from where your users are, with an eye on agent bias: cloud-hosted agents live unnaturally close to CDN infrastructure and over-reward whoever peers best with that cloud, so prefer a mix that includes backbone or last-mile agents for any decision-grade test — and if all you have is cloud agents, at least use several providers' regions. Cover each target geography with multiple locations, run the test across a minimum of several days including a weekend (evening congestion and weekend patterns are real performance, not noise), and collect enough samples per cell — geography × object type × CDN — that percentiles are stable; hundreds per cell is a floor, not a flourish. Interleave requests to both candidates within each agent rather than testing them in blocks, so any local network event hits both equally.
Scoring and deciding
Score on percentiles per segment — p50 and p90, per geography, per object class — never on a single global average, which lets a vendor's strength in your biggest region launder weakness everywhere else. Declare the decision weights before seeing results (say, 50% priority geographies' p90, 30% large-object throughput, 20% miss-path) so the scoring can't be bent toward a preferred answer afterwards. Demand differences bigger than the noise: if the candidates' distributions overlap heavily, the honest verdict is a performance tie, and the decision moves to price, features and operations — a fine outcome the benchmark has legitimately earned. Publish the method with the numbers internally, because a benchmark whose method can't survive the losing vendor's scrutiny will not survive your own renewal negotiation either; then fold the winner into a proper PoC before signing anything.