StoneShot Learning > Email Marketing > A/B Testing

A/B Testing

We’ve created a simple A/B testing solution that allows you to test a second subject line.

All you need to do is enter a second subject when you’re creating an email. At launch, the StoneShot Platform will send to a subset of your list, choose a winner based on the open rate, and use that subject to send to the remainder of the list.

Let’s walk through the steps. On the Summary tab, enter a second subject line as shown below.

You could simply launch here (Schedule button, top right) and the StoneShot Platform will use default options for the A/B test. That would take 30% of your recipient list and pick a winner after two hours. If you’d like to change those options, let’s head into the Advanced tab.

Ok, so here you can tweak the defaults. So if you’re sending to 1,000 professional investors, 30% of your list is 300 contacts. That means 150 people will get Subject A and 150 will get Subject B – and 700 people will get the winner. The winner is selected simply on the highest open rate between A and B.

Bear in mind that making the List Percentage For Test too small could result in unreliable results. Setting 5% with a 1k list wouldn’t give you enough of a comparison.

Launch and Reporting

When you launch you’ll notice “Use Send Time Optimization” is disabled. In short, you can’t use STO as well as doing an A/B test.

Right, so you’ve send the email. Now what.

After sending you’ll see the campaign stats as normal. You’ll also now see an “AB Testing” tab. Click in there and you should see something like this.

This test was sent to a small list and as you can see there’s a marginal different in open rates. In this case, Subject B was the winner.

Best practices

When running A/B tests on the campaign’s subject line, it’s always better to bear in mind a few best practices.

  • Allow enough time for people on the A/B segments to interact with your campaign to give you a realistic picture of what subject line works better. Our suggestion is to never chose less than 2 hours when selecting the testing time explained earlier. If the content of your campaign is not time sensitive, you could even wait for 24 hours before letting the system choose the winning subject line.
  • Don’t be afraid to experiment with your subject lines. A/B testing is the perfect way to test how far you can go with your audience. Our suggestion is to always compare 2 different approaches and tones: 1 more standard and 1 a bit more informal and dynamic. For example, try a cold “Newsletter May 2018” versus a more dynamic title which gives an overview on the content, ie: “A/B testing: all the best practices”. Or maybe go even further and test “A/B testing: all the best practices” and a less formal “Don’t be scared to experiment with A/B testing”.
  • Personalisation vs No personalisation. Ok, using personalised salutation in the subject line could sound too much for the Financial market but maybe you could give it a try in those communications that deserve a “warmer” touch, like an event follow-up to attendees or an email to your most engaged contacts. Try something standard and not personalised versus something like “Hi Daniele, here’s your CPD certificate” or “Hi Daniele, it was great to see you the other day”.

Take a look at our Video on A/B Testing for more information.


How can I be added to the email test list?

Contacts can be added to the test list via Contacts > Lists. The test list is usually named something along the lines of “Test list – Do not delete”.
More detailed instructions can be found on the Managing Lists page.