AdSense Experiments Review

Just a few weeks ago Google announced AdSense Automatic Experiments, which is a new tool provided by Google to enable you to optimize your AdSense ads. After logging into my AdSense account and clicking enable in the optimization page, I saw the following:

The ‘automatic experiments’ are the 2 at the bottom. Honestly those 2 ideas didn’t interested me, however it had been a while since I last checked out AdSense Experiments, so I decided to give it a try and set up an experiment of my own.

Starting a new experiment

After clicking to run a new experiment, you have two options: changing ad unit settings, or allowing and blocking ads.

Since I’m not blocking any ads, I figured changing ad unit settings would give me the biggest changes in testing. After clicking that, you name your new experiment and choose the ad unit you’d like to test. You have 2 more options about the ‘experiment setting’ that you want to test: ad unit type or text ad style.

Here are the options if you want to test the ‘ad unit type’. All you can choose from is whether you want to display image, text and image, or text only ads.

If you choose you want to test the ‘text ad style’ you get several options defining the formatting of the text in your ads:

I decided to try a test for both types. One one site I’m testing the ad type, and on another site I’m testing the font style. Keep in mind that you can only run one test per ad unit.

You’ll also see an option asking if you want Google to ‘automatically choose winner’. I was curious what criteria Google would use to choose, so for one experiment I enabled this, and for the other I wanted to be in control.

Results after week 1

I checked results after a couple days, and it said “50% confidence, collecting data.” That makes sense, the test just started.

So I waited and checked back a few more days later, after almost 18k ad impressions:

Still inconclusive? Google must be very, very conservative when it comes to making decisions. 9,000 impressions for each test variant should be enough to move the needle slightly….Keep in mind this test is a text style one, which is a drastic change from default colors to highly contrasting (dark) colors, should make a decent difference one way or another.

Waiting and waiting

I keep checking back every few days, and still saw the same data – 50% confidence, collecting data.

Finally, after over 2 weeks and 40k impressions the confidence went to 80% that the test is a loss. Somehow the quality score is up (how does the text style affect quality?), but overall RPM is down:

The other test I was running is still at 50% confidence.

I decided to end both tests after 2.5 weeks. I’m not going to wait months to get up to the 95% confidence that I’m after. The test that had ‘let google choose’ had not triggered any decisions, rightfully so.

Conclusion

While I admire AdSense for trying to optimize ads for us, I felt as though this was pretty pointless. Yes, I could try again, yet the options to test are so little that I don’t feel I’m ever going to reach statistical significance.

In all my experience, the biggest ad gains have been in testing ad positioning and ad sizes. Only testing the ad type and font gives minimal returns for your effort.

Keep in mind that if your site is running on WordPress, the AmpedSense plugin can automate testing ad position and size.

Have you tried using Google’s AdSense Experiments? Were you disappointed as well?