Campaign measurement mistakes: the errors that flatter the numbers and the habits that prevent them

The mistakes that persist in campaign measurement are not the ones that produce obviously wrong answers. They are the ones that produce answers people like, which is why they survive review, get repeated in decks, and end up shaping budgets.

Each item below is a specific error, why it is comfortable, and the habit that prevents it. The positive version of all this is in measurement and ROI.

What to take away

Errors of comparison

Comparing to the previous period. The most common and the least defensible. It attributes to the campaign everything else that changed at the same time. The habit that stops it: state the counterfactual out loud in every report, even when the answer is that you do not have one.

Comparing to a benchmark chosen afterward. There is always a benchmark under which the result looks good. Choose it before, and write it down.

Comparing partners who were not asked to do the same thing. Different deliverables, different offers, different timing, one ranked list. The ranking is mostly measuring the brief.

Treating a platform's figure as a measurement. It is a count produced by an interested party using its own definition. Fine as a diagnostic, weak as evidence.

Errors of arithmetic

An incomplete denominator. Fees are counted; product cost, shipping, production support, agency time and any media spend behind the asset are not. The return figure that results is not wrong so much as answering a different question from the one being asked.

A metric with no denominator at all. Raw counts of anything are uninterpretable without knowing what they are out of.

Reading a rate from a tiny base. A large percentage change on a small number is mostly noise, and it will not repeat.

Concluding no effect from a test that could never have detected one. Whether the test had a chance is arithmetic, and it should be done first. The idea is statistical power, and a badly powered test is not evidence of absence.

Errors of selection

Choosing the window after seeing the results. Extend it until it looks better, or truncate it before the decline. This is the single most common way a careful, honest team produces an unreliable number. Fix it by writing the window into the plan before the campaign.

Reporting the campaigns that worked. Over time, a program's internal history becomes a collection of successes because the failures were never written up. The general problem is publication bias, and it is worse internally than externally because nobody is even trying to correct for it.

Judging partners on the ones you kept. A roster assembled by dropping poor performers will always look like a good roster, and it says nothing about the selection process that assembled it.

Excluding the outlier that hurts and keeping the one that helps. Decide the exclusion rule before, and apply it symmetrically. Selecting the data that agrees with you is cherry picking, and it is usually unconscious.

Errors of interpretation

Scaling up after one strong result. Extreme results are extreme partly by luck and move toward average on repetition. Repeat the test before restructuring the program.

Attributing to the partner what belongs to the moment. A launch, a season, a price change and a competitor's problem all sit inside the campaign period.

Mistaking novelty for effect. A new format performs well partly because it is new, and the advantage decays as the audience gets used to it. The effect has a name, the novelty effect, and it explains a lot of formats that stopped working for no visible reason.

Assuming what worked in one market works in another. Whether a finding transfers is a question in its own right, described as external validity, and the honest default is that it does not until tested.

The error that eats the others

Set a single number as the target and reward people for moving it, and it will move. What it was standing in for will not necessarily come with it. Comments can be bought, clicks can be bought, and a number that is optimized directly stops being informative about the thing it was chosen to represent.

The compact statement of this is Goodhart's law. The practical defenses are ordinary: keep more than one measure, and include at least one that is hard to manipulate.

Review, on a schedule, whether the proxy still tracks the outcome instead of assuming it does. Some manipulation is not yours: purchased engagement is a live problem, covered in fraud and brand safety.

A short prevention list

The plan itself has to come from the decision the campaign is meant to inform, which is in strategy and objectives, and what was actually promised in the deliverable comes from campaign briefs.

Bottom line

Assume every unexamined measurement error is flattering you. Fix the window, the benchmark and the exclusion rule before the campaign, put every cost in the denominator, write down the campaigns that failed, and repeat a result before acting on it. Keep more than one measure, because a single target that people are paid to move stops measuring anything.

Common questions

Which mistake should be fixed first?

Writing the plan, including the window and the failure condition, before the buy. It removes several of the others at once.

Is it dishonest to report a favorable window if that is where the effect really was?

Not if you say the window was chosen after the fact and show the pre-specified one as well. The problem is concealment, not curiosity.

How do you keep a record of failures without punishing people?

Separate the decision from the outcome. A well-designed campaign that did not work is a good decision with a bad result, and treating those as the same thing is what stops anyone recording them.

Should measurement be handled by the same team that runs the campaign?

It can be, provided the plan is written before the results exist. That single step removes most of the conflict.

Is any single metric safe to optimize?

No. Any metric under sustained pressure will be moved by whatever moves it most cheaply, which is rarely the thing you wanted.