A/B test
Use the A/B test element to send different variations of a message to part of your audience, measure which variation performs best, and automatically send the winning message to the rest of your audience. There is no manual step to pick the winner.
How A/B testing works [#how-it-works]
When you add the A/B test element to a journey, you define a test audience, a percentage of your total audience that is used to run the test. Set this to any value between 10% and 95%; the default is 30%.
The test audience is split evenly across your message variants. For example, if you create 2 variants and set the test audience to 30%, each variant is sent to 15% of the total audience.
The remaining audience (everyone outside the test audience) does not receive a message during the test. After the test finishes, the remaining audience automatically receives the winning message.
Here is the overall flow:
- Create 2 to 4 message variants (Version A, Version B, and so on) on the same channel.
- Set the test audience size, and choose whether the winner is decided by open rate or click rate.
- Set how long to wait before deciding the winner.
- The element sends each variant to its share of the test audience.
- When the wait time ends, the element evaluates the results and picks the winning variant.
- The remaining audience automatically receives the winning message.
Before you begin [#before-you-begin]
The A/B test element has requirements and constraints that affect where you can place it and how you configure it.
Review the following before you add an A/B test element to your journey:
- The A/B test element is available in Journeys only.
- All variants in a test must use the same channel.
- Add the element after a predefined audience entry point.
You can add the element to all journey entry points. Use the entry point One time for large audiences.
- Each test must include a minimum of 2 and a maximum of 4 variants.
Supporting channels
The A/B test element supports the following channels:
- SMS,
- Mobile Push
- Viber
- RCS
- Apple Messages
- Line.
A single A/B test uses one channel only. Different A/B tests may use different channels.
Add the A/B test element [#add-element]
- In the journey builder, select + to open the Select element panel.
- In the Actions category, select A/B test.
Configure the element [#configure]
Configure the following fields for the A/B test element.
| Field | Description |
|---|---|
| Add test note | Enter a note describing the test. Maximum 160 characters. Example: Testing subject line variations personalized vs generic. This field is optional. |
| Test size of total audience | Set the percentage of the total audience used to evaluate the winning message. The test variants are distributed evenly across this percentage. Default is 30%. |
| Version A, Version B, and so on | Select the channel and build the message for each variant. Enter a name for the variant. Select Add another variant to create more variants, up to the maximum. |
| The winning message will be decided by | Select the success metric used to determine the winner: Open rate or Click rate. |
| Wait time | Set how long to wait before the element decides the winner in minutes, hours, or days. |
For the Click rate option, you must enable the Track URL option for all selected messages.
Monitor the test [#monitor]
While the test is running, the element panel shows a banner indicating progress. Example: Test 60% complete and will finish in 1.2hrs.
Each variant shows the following live stats:
- Recipients
- Sent
- Open rate
- CTR
How the winner is decided [#winner]
The winner is the variant with the best result on the metric you chose (open rate or click rate). In case where there is no clear winner, the winner will be chosen randomly.
Test results [#results]
When the test finishes, the element panel shows a banner confirming the result. Example: Test complete. Variant C had the higher open rate and was sent to the remaining audience.
The winning variant shows a Winner! badge. Each variant shows its final stats:
- Recipients
- Sent
- Open rate
- CTR
Limitations [#limitations]
- Only open rate and click rate can be used to choose the winner.
- Advanced send-time options, such as send-time optimization and delivery time window, are not available on the A/B test element.
Use cases [#use-cases]
- Test two subject lines, personalized and generic, to see which gets more opens.
- Test different message copy or calls to action to see which gets more clicks.