While a standard AB-test is typically used to test a limited amount of designs or versions of a message, multi-variate testing allows to test a large amount of content variations in a single message.
Multi-variate tests can be used for emails as well as pages. It allows creation of multiple variants of one single message by combining variable subject lines, images, and articles. There is no limitation on the number of combinations that can be tested, other than the test audience and performance requirements. The possible content of the different MVT components is defined in the configuration of the component itself. Unlike the A/B test where you need to define each version as a separate email, MVT allows you to define all variants within the same message.
NOTE: MVT components can also be used in templates. In that case, the content of the component is provided from within the Express context. MVT can be used in automation as well.
Every possible combination of these different elements (images, text, and articles) is tested with the goal of finding the best performing combination and using that version to send it to all remaining participants.
When you use multi-variate testing, you will first send out an email blast to two or more segments of random audience contacts and then test the results. The remaining users will receive/view the best version of the message. The 'best version' is chosen if there is a ‘statistical significant difference’ between the different variants, or when a certain time or number of clicks has been reached. You can set this up in the message properties.
Technical Note: The ‘statistical significant difference’ is a statistical calculation that calculates if the result (number of clicks between the different version) is based on coincidence or not. If you only have 2 clicks on an item, the chance that the clicks are coincidental is high. The more clicks you have on an item, the chance becomes lower. For more info see Wikipedia: “Multivariate testing in marketing” or “F-test”.
NOTE: A distinction
must be made between MVT emails with static content (text and articles)
and those with dynamic content (images).
Flows with MVT emails containing static content (with or without images)
are executed as a wave. In the first wave, the email is sent out to part
of the target group and this content is static. When a winning combination
has been found, that combination is sent to the remainder of the target
group.
If the email message only contains images, all email messages are sent
out at the same time. When the winning image or combination of images
has been found, only this image will be returned and shown to the target
group when opening the message (on condition that the images are not cached
by the email client).
Test criteria depend on the number of clicks.
1. Create an MVT email or page using MVT components. The following MVT components are provided:
2. Preview the different MVT variants
3. Select when the 'best version' should be sent or shown in the message properties.
4. The default selection for each MVT variant is 200 contacts. You can override this in the MVT journey, in the MVT Page, or in the Email component Properties, on the 'Best version' tab. If your audience is too small to send all variants, then half of your contacts will be used to send out the first wave of messages and the other half will be targeted with the winning version.
When you activate the MVT journey, the initial random selected contacts will receive the various versions of the MVT email.
Once a winning variant is available for the email, the remaining contacts will be targeted with the winning variant.
For each contact, the MVT page variant shown can be different but it will always look the same when the same contact visits the page again. Until a winning variant of the page is available, the result will always be the winning version for all contacts.
You can also 'Clear multivariate data'. Be careful, because this will delete all gathered multi-variate data. After the journey is tested internally, it is useful to remove all testing data before launching the journey for contacts.
