Definition of A/B Testing
A/B Testing is the use of a scientific experiment with a target to improve advertising performance. Performance is measured in control group A and experimental group B. In the latter group, one variable is altered, while all others are held constant. Testing is generally carried out following these steps:
Set up the Experimental Design
- Formulate a clear research question
- Define a hypothesis – a possible explanation of why things happen
- Choose the experimental variable
- Make a prediction for the future. You can use this guess as a benchmark for evaluating your results
- Split your population in two groups: Control group A and experimental group B – conserving a control group A is very important to make sure the difference in outcome between group A and B are the result of the applied changes, and not the result of external effects we cannot control. You could even split your population in more than two groups. Nevertheless, the more groups you have, the more difficult it is to get statistically relevant results
- Set the time duration – how long the experiment will last – it is recommendable to set a duration of at least a few weeks, because setting too short of a time period would not generate enough data to obtain statistically relevant results
Monitor Changes on the Market
- Implement your changes to the experimental group and start your test
- Compare your data streams between the two groups
- Keep things unchanged while testing. Do not apply several changes at the same time, because you cannot control your results. If you apply any additional changes, you will have to start the experiment all over again
- Interrupt the experiment only for a good reason – because you will not have enough information for relevant decisions if you prematurely stop your experiment. You can stop your experiment at any time if you need it. Interrupting the experiment without obtaining enough data will result in the loss of all the money invested in it
Analysis and Evaluation of Results
- End your test
- Check the statistical significance of the results
- If statistical significance is given and performance of the experimental group was better – apply your changes to the whole campaign and set your experimental group to control group
- You can delete the experimental changes – your account will use only the keywords, ads, and ad groups in your control group, if the results after the experiment are better in the control group than the experimental group
- Run a follow-up experiment if your budget allows for it
- You can extend your experiment – If your experiment did not produce any conclusive results and you think it just needs more time
Hints
- Show the same variation to the visitors coming back to the website
- Make the A/B test consistent across the whole website
- Constantly be testing. An experiment is a very good method to improve your accounts
- Do not run your test groups A and B at different time frames. Remember only to vary one variable. Time would be a second variable
- Do not surprise regular visitors. All guests must see only the information that belongs to their group (experimental or control group). Try not to experiment with visitors that already converted on your website
- Do not let your prediction bias the test
Google Ads Experiment
This experiment aims to test new keywords. Everything is held unchanged except for an experimental group with new keywords. Through this approach it is possible to determine if the former keywords or the newly tested ones achieve more clicks and conversions.
The experiment is set up in the Google Ads Account. Navigate to “Campaign” > “Settings” and scroll to the bottom of the tab. In the advanced settings, four steps indicate the approach to conduct an experiment. Click “specify experiment settings” to start.
In the next section you determine the experiment’s name, the split between control and experimental group, the start time and the duration of the experiment.
Ideally, the name of the experiment should explain its purpose. The share of your auctions belonging to the experimental group is set up carefully. Eventual risks and expenses must be taken into account. In case of doubling your bid, for example, it might make sense not to have too many auctions included, otherwise it may be very expensive. After having set up the experimental and control group, this share cannot be changed anymore.
- Choose whether you would like to start your experiment manually or at midnight on a specific date. If you choose today’s start date, the experiment will start immediately
- Choose whether you would like to end your experiment after 30 days, or on a certain date. At any time after you start your experiment, you can extend it up to three months from that day
When you have saved the settings on this screen, Google will allow you to “start your experiment”.
Before running the experiment, you need to jump back to the ‘Ad Groups’ tab and start to make your changes.
Press the button “add keywords”, choose your ad group for that, and set your keywords. At the bottom, you can see an option “Add as experiment only keywords” – this means these Keywords will not be considered as control group keywords. You can tick this box and you have created your split test. Save it.
What you see next is a window with all keywords of your campaign. The second column shows the status of your keywords. You can choose the keywords you want to apply the changes to via the check box next to the keyword header, and changing the status of your keywords, for example to “control only” etc. Now you have 2 testing groups – control group A with old keywords and experiment group B with new & old keywords.
To enable the new campaign experiment you have created, you will need to jump back to the Campaign > Settings tab and scroll all the way to the bottom again. This time, you want to apply launch changes, and start running the experiment.
You can change some options for several or only one keyword. Select a keyword or a group of keywords and click ‘Edit’. Answer ‘Yes’ in the next window and change your bidding there.
You can increase your bid by percentages only in the experimental group. You can decide to increase your bid for the duration of the experiment for each one or for all keywords in this group.
To enable the new campaign experiment you have created, you will need to jump back to the Campaign > Settings tab and scroll all the way to the bottom again. This time, you want to apply launch changes, and start running the experiment.
Now you just need to start it and analyze the results. The next step is actually monitoring your results and finally – an evaluation!
To do so, we can take a report, download it into Excel, and start getting some interesting insights, such as:
- Did impression increase / decrease?
- Did conversions increase / decrease?
- What about costs and cost per conversion?
From here we can see which areas of your campaign performed better or worse than the control settings.
Text ads testing
When doing A/B testing there are some details you must keep in mind. You must have a reliable set of data and you must do no changes besides the changes you want to test. When making conclusions at the end of the experiment, it is crucial to be sure how the observed differences are related to the changes made in the beginning of the test.
In text ads, as we already know, there are 4 lines of text. The text you insert on these lines is connecting the user’s interest to your website. Obviously, some texts work better than others. So it might be important to test what kind of text works better for your ads.
Normally, we have 3 types of text ads inside an ad group. An aggressive ad, a non-aggressive ad, and a generic ad. As these ads will rotate and be shown randomly the same amount of times, it means our “population” is already split in three groups. These groups will see a different ad advertising the same thing, but using different words.
For example, we want to know if one of these ads can have a better CTR with a different headline. So, we pick one of the ads and change the headline (from A to B) and keep the other two ads unchanged. This way, with only one ad changed and the others unchanged, you will be able to evaluate better the performance difference on the changed ad.
When performing this kind of testing, never revolutionize your (in this case) headline. Keep the changes in the same topic, the same content and coherent with the rest of the text. Also remember to change only one thing at a time. Because then the changes in performance can be related to the different topic or content of the ad or other factors, instead of just being related to the better and more appealing words.
You can choose between social extensions, location extensions, product extensions, call extensions, seller ratings, and sitelinks. Probably not all of them will make sense for the same advertiser, but that is something that must be tested.