Tests in marketing: some basic concepts and tips

Today, I wanted to talk to you about tests. In a world where we are constantly looking for ways to improve, optimize the results of our campaigns and their ROI, maximize the impact of our communications, and be closer to our customers and contacts… testing is imperative! Budgets are getting tighter and tighter and we have to do more with less. So knowing what works best is key!

What’s good now is that we’ve moved away from print testing… always long delays before we could launch the winning versions, and oh boy, was it expensive! Those who started their careers when that was the way to do it will understand me (yes, I am betraying my age with this look at the past).

With today’s technologies, we talk a lot more about A/X Tests as a method of analysis, and with campaign management platforms like Dialog Insight, it’s super easy. You can get the results before you even deploy on a large scale, ensuring you maximize ROI.

Knowing that testing is a must, here are a few tips I’d like to share with you on how to do it more effectively



How to create effective tests

1.       Have a good test plan

First of all, like most things, they need to be well planned, which is why we encourage you to develop what is called a test plan. This will guide you in building effective tests, but more importantly, it will ensure that you follow the metrics to measure the success of your tests. A test plan includes :

  • The purpose of the test
  • A starting hypothesis
  • KPI’s that will determine the winner

Too often, we don’t think about KPI’s or how the analysis of the results will be done, and BANG! The test begins, and the results do not validate what was originally intended since it was not built to get the right information. I strongly invite you to become a member (it’s free!) of the Strategyzer organization, which provides super interesting tools of all kinds (you may recognize the Business Model Generation Canvas). In this case, the organization has created what it calls the Test Card, which you can see below. It is a downloadable resource that meets the requirements of a test plan very well.


Source: strategyzer

Once this first step has been completed, we recommend, as far as possible, to build a test plan which will be accompanied by a matrix. I can already hear you saying “ah no, not a matrix! It’s complicated and cumbersome to manage”. It doesn’t have to be. Just build a table, in Excel for example, that will allow you to track your test results, but more importantly, to see what was tested. It will also help you share the results across teams and keep track of what was tested.



2.     Prerequisites for good tests

Now that your test plan is built, you can start setting them up. But don’t get too excited, because you need to control the environment around the tests, so you can measure the right things again.

The biggest basic principle I can give you on tests, which we often see our clients forget, is to measure the effect of one change at a time, in addition to doing only one test at a time.

If you create a test with different colors AND a different subject, for emails for example, it won’t work. How do you know if one or the other has had a positive or negative effect? Same thing for text, change only one item at a time to make sure you recognize what worked.

The second key aspect to look for in building your tests is targeting. It is a basic statistical principle, but an important one! You need a sufficiently large and representative sample of your target audience to ensure the validity of your conclusions.

Statistical experts recommend tests are done on a sample of at least 1000 people in order to make what is called statistical inference. This means that the conclusion can be statistically extended to the entire sample that meets the criteria of the initial targeting.

However, this obviously depends on the size of your database. Sometimes our sample is smaller so, from a purely statistical point of view, the results are harder to defend (they are statistically invalid). We can still draw trends and revalidate in a second test.


Choice of the type of targeting

Beyond the size of your sample, you can also limit its selection according to other variables. Indeed, we often think of testing the entire database, which is possible but need not always be the case.

For example, you could do tests on a small group of contacts that are part of a segment of your clientele (according to a persona, a level of loyalty, a sector of activity, etc.).

You could also test by comparing two customer segments to each other, and see if a message works better for certain types of customers.

This is why we strongly encourage our clients to make a test plan since these elements will be well established prior.



3.     Different types of tests

Tests can take different forms, here are two below.


Control group or no control group?

The control group is a notion that in my experience is often forgotten or little used. Yet it is essential because it becomes the compass to measure whether the changes we suggest to a campaign really have a positive effect, versus if we hadn’t done anything.

In fact, the control group is the status quo; it represents a portion of the sample that will not receive the tests in order to compare the results with those who will have received them. This practice provides a kind of protection, especially when revenue is involved. Changes should not be made that will ultimately have less effect than if everything had remained the same.

Whether or not you should take this route depends on the objective initially established, since not all tests have an impact on a good part of your database. It will also depend on the number of contacts available to you, since the smaller the sample, the more difficult it is to have a sufficiently large control group. It may also depend on the channel on which the test is made.

Let’s say that the control group is the ideal world, but with that being said, you can have valid tests without a control group.

Finally, I’ll end on the control groups by mentioning that it is important to define them randomly so as not to influence the results and the decision that will be made.


Automated tests

It is also possible to orchestrate tests automatically, and this is what we are seeing more and more with communication management platforms like ours. It’s such an easy and simple way to set up your tests.

As an indication, in our application, we have two main types of A/X tests, either the one that divides the sample in percentage, or the one that limits the portion of the sample. 

The first one is the best known, in the sense that we divide the whole sample in portion, which will be randomly divided in percentage. There could therefore be 3 groups, of which 25% will receive one message, a second 25% will receive another message, and the remaining 50% will receive the third message. Here, the whole sample is tested, and the conclusion of this type of test will rather affect the following mailings. 

We have a second type, which allows us to take a percentage of the sample to which to send a first mailing (say 10%). This percentage will receive the tests randomly and then determine the winning version. Once this is done, it is possible, directly in the application, to send the winning version to the remaining 90% of the sample. 

In short, it’s not a lot of work for something that can only be profitable!



Aspects of a message to be tested

Of course, depending on the channel, you can test a lot of things, here are some ideas!

  • The creative: be it in the visual (colors), the images, but also the editorial aspect. That’s where the fun starts, for me anyway! Because for this kind of test, the door is wide open to creativity.
    • – It can be to measure two different concepts, a photo, the background color, the text, the titles, the positioning of the text versus the images, etc..
  • The order: You can also keep the same visual elements, but change the order of the contents.
  • The offer: this portion can also be very interesting. You can measure a monetary value (or loyalty points) between 10, 12 and 15$ discount for example. It can also be 25, 50 versus 100 points for a loyalty program.
    • – Sometimes, we realize that the redemption/purchase rate is the same with a lower amount, so the profitability is higher since our cost is lower!
  • The subject of the emails is also an interesting element to test even though we often don’t see a huge difference in the opening rates. This can still align the choice and optimize the results.
  • The good old CTA (call to action). Test different action verbs, wording, text. For buttons; color, shape, location, because yes it can impact the click rate and therefore the contact engagement.
  • Deployment period: The time of deployment or the day of deployment can have a big impact on the results of the campaign. Too often we go with preconceived ideas or habits, but we forget to test this portion.




In summary, it is important to have a test plan that will be shared with the right people. It is also important to do one test at a time to optimize our knowledge of key elements. Finally, simplify your life with testing tools like the A/X test tool designed by Dialog Insight.

No excuse to avoid them now! Contact us if you need help with this type of project, we are always available to help your teams.


Article written in collaboration with Roxane Noiseux from Dialog Insight