Measurement examples: checking performance claims one by one

Most arguments about campaign and partner results are not about numbers. One side has made a claim, and nobody has said out loud what would have to be true for it to hold.

For each claim below: what it asserts, what evidence would support it, and what people usually offer instead. The framework behind them is in measurement and ROI.

What to take away

Claims about what worked and who performed

Each claim below asserts something specific, and each needs its own kind of evidence.

"The campaign drove the sales increase."

"This partner outperformed the others."

Needs: comparable deliverables, comparable audiences, comparable offers, and enough observations to distinguish signal from variation. Usually offered: a single efficiency number per partner, ranked. Partners differ in what they were asked to do, when they posted, and what the offer was. Ranking on one number compares the campaigns, not the people, and then cuts the bottom of a list that was mostly noise.

Take two partners ranked on cost per sale. One posted in launch week with a discount code, the other mid-month with no offer. The ranking measures the timing and the offer as much as the partner, and the difference between them says little about who is better.

"Smaller audiences perform better."

Needs: a like-for-like comparison inside one category, with the same offer and the same measurement. Usually offered: aggregate figures across categories from a party that sells access to smaller audiences. Aggregates across groups can reverse when the groups are examined separately, an effect known as Simpson's paradox. It shows up here constantly, because audience size correlates with category, format and price.

"This creator's audience is the wrong fit."

Needs: a stated definition of fit, and evidence about the audience rather than about the result. Usually offered: a disappointing outcome. A weak result can come from the creative, the offer, the timing or the measurement. Concluding it was the audience requires evidence about the audience. Where that evidence comes from is in influencer discovery.

Claims about the numbers themselves

Partner claims lean on a handful of numbers, and each number hides a definition. The table pairs every claim with the evidence it needs.

The last row quietly wastes the most money. Whether a test could have detected the effect at all has an arithmetic answer, decided before the test runs. The concept is sample size determination, and the computation of a sample size depends on several quantities, some of which have to be assumed in advance. A test that was never large enough is not evidence of no effect.

One sentence before any ranking

Before a partner result is presented, write one sentence: this claim is that X, compared with Y, measured over Z. If any of the three is missing, the ranking is not ready.

Choosing the window or the metric afterward is the quickest way a careful team produces an unreliable ranking. Drawing the boundary around the good data is the Texas sharpshooter fallacy. The wider list of ways results get flattered is in measurement mistakes.

Do the same for the reports you are shown, including a partner's own recap and an agency's summary. Apply the discipline to promotional material using the reading method in campaign case studies.

Claims you should refuse to settle

Some questions cannot be answered with the data anyone has. Whether a campaign changed how people feel about a brand, over a period shorter than a season, from a sample of comments.

Whether one post caused a purchase made three weeks later on a different device. Whether a partner would have performed better with a different brief, when only one brief was used.

The right answer to these is that the data does not support a conclusion, together with a description of what would. That answer is unpopular and it is more useful than a number produced to end a meeting.

Common questions

Is it unreasonable to ask a creator for this level of evidence?

Asking a creator for platform-native analytics viewed live is reasonable. Asking them to prove causation is not: that is the buyer's measurement problem.

What if leadership wants a single number?

Give one, with the comparison and the window attached in the same sentence. A number without them will be quoted for a year without them.

Are platform-reported results useless?

No. They are good diagnostics and weak evidence of effect, because they are produced by an interested party using a definition you did not choose.

How do you compare partners fairly?

Give them comparable briefs, comparable offers and comparable measurement, and then still expect variation. Fair comparison is mostly an experimental design problem, not an analysis problem.

What is the fastest improvement to most reporting?

Writing the expected result down before the campaign. It converts everything afterward from storytelling into a test.