How Do I Know When To Stop Running Ads? What Is “Statistical Significance”?

So you’ve started a campaign and you’ve got some data in! It’s time to start optimising your way to profit.

But first – how do you know when you’re safe to cut an ad from your campaign?

Why You Should Care About “Statistical Significance”

If you flip a coin and it comes up “heads”, does that mean the coin will always come up heads?

Of course not.

But one of the mistakes that new affiliates often make is assuming that, essentially, the coin’s always going to come up heads.

Assuming you’re looking for a 0.2% CVR (Conversion Rate), for example, many affiliates will drop any ad that’s showing less than that – regardless of how many times it has been viewed.

That’s a very bad idea. A banner which has no conversions after 500 views can easily soar to having a 0.4% or higher CVR after 5,000 views.

Fortunately, scientists have spent a lot of time working out just how many repetitions of an event you need to be reasonably sure it’ll keep repeating at the same rate.

This is called “Statistical Significance”.

But how do we calculate this?

Well, we can use the same statistical tools that scientists running medical trials or experiments in particle physics use to determine whether their results are valid. And that way, we can be as sure as they are that we’re on the right track.

Don’t worry – it’s less scary than it sounds.

TIP: Don’t use data that had large numbers of impressions on bad placements in it. If in your initial testing round you blacklisted placements that accounted for more than a third of the total traffic, only do these statistical tests using data that you collected AFTER you blacklisted those placements.

Using the Scary Math Calculator

Head to this page.

Now, let’s take your first ad that’s still running.

Go to the “Binomial Confidence Intervals” section. Put the number of conversions it’s had into the ” Numerator (x):” box.

Put the number of impressions it’s had into the “Denominator (N): ” box.

Hit “Compute”.

You’ll end up with something like this. This example assumes you’ve had 12 conversions from 20,000 impressions:

Can’t understand a word? Don’t worry. It’s actually very simple, just written in math-ese rather than affiliate-ese.

What The Math Means

What we’re trying to do here is predict the future Conversion Rate of your ad from the data we have so far. Using statistics, it’s imposs ...