AB-Testing Ads

Definition of A/B Testing

A/B Testing is the use of a scientific experiment with the target to improve advertising performance. Performance is measured in a control group A and an experimental group B. In the latter group, one variable is altered while all others are held constant. Testing is generally carried out following these steps:

Set up the Experimental Design

  • Formulate a clear research question
  • Define a hypothesis – a possible explanation of why things happen
  • Choose the experimental variable
  • Make a prediction for the future. You can use this guess as benchmark for evaluating your results
  • Split your population in two groups: Control group A and experimental group B – conserving a control group A is very important to make sure that the differences in the outcome between group A and B are the result of the applied changes and not the result of external effects we cannot control. You could even split your population in more than two groups. Nevertheless, the more groups you have, the more difficult it is to get statistically relevant results.
  • Set the time duration – how long the experiment will last – it is recommendable to set a duration of at least some weeks, because setting a too short time period would not generate data to obtain statistically relevant results

Monitor Changes on the Market

  • Implement your changes to the experimental group and start your test.
  • Compare your data streams between the two groups
  • Keep things unchanged while testing – do not apply several changes at the same time, because you cannot control your results. If you apply any additional changes – you will have to start the experiment all over again
  • Interrupt the experiment only for a good reason – because you will not have enough information for relevant decisions if you stop your experiment before! But you can stop your experiment at any time if you need it. Interrupting the experiment without obtaining enough data, you will lose all the money invested in it.

Analysis and Evaluation of Results

  • End your test
  • Check the statistical significance of the results
  • If statistical significance is given and performance of the experimental group was better – apply your changes to the whole campaign and put your experimental group to control group
  • You can delete the experimental changes – your account will use only the keywords, ads, and ad groups in your control group, if the results after experiment in the control group are better than in the experimental group
  • Run a follow-up experiment if your budget gives you this possibility
  • You can extend your experiment – If your experiment did not produce any conclusive results and you think it just needs more time


  • Show the same variation to the visitors coming back to the website
  • Make the A/B test consistent across the whole website
  • Constantly be testing. An experiment is a very good method to improve your accounts
  • Do not run your test groups A and B at different time frames. Remember only to vary one variable. Time would be a second variable
  • Do not surprise regular visitors. All guests must see only the information that belongs to their group (experimental or control group). Try not to experiment with visitors that already converted on your website
  • Do not let your prediction bias the test

AdWords Experiment

This experiment aims to test new keywords. Everything is held unchanged except for an experimental group with new keywords. Through this approach it is possible to comprehend if the former keywords or the newly tested ones achieve more clicks and conversions.

The experiment is set up in the AdWords Account. Navigate to “Campaign” > “Settings” and scroll to the bottom of the tab. In the advanced settings four steps indicate the approach to conduct an experiment. Press the button “specify experiment settings” to start.

In the next section you determine the experiment’s name, the split between control and experimental group, the start time and the duration of the experiment.

Ideally, the name of the experiment should explain its purpose. The share of your auctions belonging to the experimental group is set up carefully. Eventual risks and expenses must be taken into account. In case of doubling your bid, for example, it might make sense not to have too many auctions included otherwise it may be very expensive. After having set up the experimental and control group, this share cannot be changed anymore.

  • Choose whether you would like to start your experiment manually or at midnight on a specific date. If you choose today’s start date, the experiment will start immediately
  • Choose whether you would like to end your experiment after 30 days or on a certain date. At any time after you start your experiment, you can extend it up to three months from that day

When you have saved the settings on this screen, Google will allow you to “start your experiment”.

Before running the experiment, you need to jump back to the AdGroups tab and start to make your changes.

Press the button “add keywords”, choose your AdGroup for that and set your keywords. At the bottom you can see a little option “Add as experiment only keywords” – that means these Keywords will not be considered as control group keywords. You can tick this box and you have created your split test. Save it.

What you see next is a window with all keywords of your campaign. The second column shows the status of your keywords. You can choose the keywords you want to apply the changes via the check box next to the keyword header, and changing the status of your keywords, for example to “control only” etc. Now you have 2 testing groups – control group A with old keywords and experiment group B with new & old keywords.

To enable the new campaign experiment you have created, you will need to jump back to the Campaign > Settings tab and scroll all the way to the bottom again. This time, you want to apply launch changes, and start running the experiment.

Another kind of experiment is a changing of your bid for keywords. Go to Segments > Experiment. After this step you want to see how your different AdGroups are performing on the keyword level.

You can change some option for any or only one keyword. Select a keyword or a group of keywords and press the button “Edit”. Answer “Yes” in upcoming window and change your bidding here.

You can increase your bid by percentages in the experimental group only. You can decide to increase your bid for the duration of the experiment for each one or for all keywords in this group.

To enable the new campaign experiment you have created, you will need to jump back to the Campaign > Settings tab and scroll all the way to the bottom again. This time, you want to apply launch changes, and start running the experiment.

Now you just need to start it and analyze the results. And the next step is actually monitoring your results and lastly – an evaluation!

Finally, we can take a report, download it into Excel, and start to get some interesting insights, such as:

  • Did impression raise/ lower?
  • Did conversions raise/ lower?
  • What about costs and cost per conversion?

From here we can see which areas of your campaign performed better.

Text ads testing

When doing A/B testing there are some details you must keep in mind. You must have a reliable set of data and you must do no changes besides the changes you want to test. When taking conclusions in the end of the experiment, it is crucial to be sure how the consequences are related to the changes made in the beginning of the test.

In text ads, as we already know, there are 4 lines of text. The text you insert on these lines is connecting the user interest to your webpage. Obviously, some texts work better than others. So, it might be important to test what kind of text works better for your ads.

Normally, we have 3 types of text ads inside an AdGroup. An aggressive ad, a non-aggressive ad and a generic ad. As these ads will rotate evenly and will be shown randomly the same amount of times, it means that our “population” is already split in three groups. These groups will see a different ad advertising the same thing, but with different words.

For example, we want to know if one of these ads can have a better CTR with a different headline. So, we pick one of the ads and change the headline (from A to B) and keep the other two ads unchanged. This way, with only one ad changed and the others unchanged, you will be able to evaluate better the performance difference on the changed ad.

When performing this kind of testing, never revolutionize your (in this case) headline. Keep the changes in the same topic, the same content and coherent with the rest of the text. Also remember to change only one thing at a time. Because then, the changes in performance can be related to the different topic or content of the ad or other factors instead of just being related to the better and more appealing words.

Ad extensions are something that can help you to drive attention to your ad, and also makes your ad bigger and more interactive. You have several extension types that you can use. But remember, even if you configure extensions on your campaign, Google will only show them if you achieve a certain ad QS, which means CTR and relevancy, and also if your ad is appearing on the first 3 places.

You can choose among social extensions, location extensions, product extensions, call extensions, seller ratings and sitelinks. Probably not all of them will make sense for the same advertiser, but that is something you need to test.