A/B Winner Campaigns

Set two to four test segments and one final segment with AB Winner campaigns.

While you can test multiple settings or versions of a campaign, it is a best practice to minimize the number of variations in settings for each test. If you have a very large audience, it may be possible to test more settings, as each segment of your list can be larger and more statistically significant. Typically, marketers reserve the final, winning segment as the greatest percentage of their AB Winner or AB Auto Winner Campaign.

The difference between AB Winner and AB Auto Winner campaigns, is that with AB Winner campaigns you will manually select the winning email variant that will be sent to the remainder of your audience.

AB Winner (and Auto Winner) tests are commonly used for subject line testing or other "quick win" tests where you are quickly optimizing based on the test results. For AB Winner, you review the Campaign Summary and Detail Reports to determine the winning version and then manually copy that version for the final segment.

Metrics to consider include:

  • Opens
  • Clicks
  • Pageviews, especially if you are a publisher (requires Personalization Engine to be implemented)
  • Purchases (requires purchase API to be installed)
  • Revenue (requires purchase API to be installed)

For more information on the Purchase API, see Advanced API section in the For Developers documentation pages.

Getting Started

To create a new campaign:

  1. Go to Messaging > Campaigns.
  2. Click Create New from the Campaigns page.
  3. Enter a Campaign Name. You can also rename the campaign later by clicking its name at the top of the campaign builder.
  4. Click AB Test, then select the AB Winner campaign type. For AB Test campaigns, you'll have the option to select the type of test for your campaign.

NOTE: For more information on creating a new campaign, see the Campaign Overview page.

After creating your new campaign and clicking AB Test, then AB Winner type, you'll see the menu on the left which will guide you through next steps. You can always move from section to section if you need to add information in a different order.

NOTE: AB Winner campaigns display the segments above the menu, including a final segment indicated by the check mark. When you add a segment, it will appear with the segment letter and name of campaign.

Click the campaign name next to the paper airplane icon to launch a pop-up in which you can customize the name of each campaign segment.

Audience

You will choose the audience for your email by setting a Your Target Audience and, optionally, Your Suppressed Audience. You can create multiple suppression lists if needed to exclude subsets of Your Target Audience. Please reach out to your Customer Success Manager to enable this feature.

To read more about how to create, manage, grow, and track lists, see the Lists Creation section of the Audience Overview documentation pages.

Selecting Your Lists

On the Audience page, set who will receive your email via Your Target Audience drop down.

Once you choose Your Target Audience, you will have the option to exclude recipients via the Your Suppressed Audience drop down.

The number of valid emails refers to the number of valid emails on a list, minus hard bounces or optouts. See the User Levels Engagement page for more information. If you're using a Natural List, Created will show the date the Natural List was created.

Configure

In this section of the platform, you'll enter the foundational information for your email, including the Template; basic information such as the From Name, Email Subject and Reply-To Email; and choose the Data Feed and/or Ad Plan, if you're using one. This is also where you will set up your AB segments and add tests for any of the basic information.

Build From a Template

On the Configure page, choose your Template if you have not already done so.

If you have selected a template with unsaved changes, you will be prompted to review and save them. Unsaved changes will not be sent.

Any changes made to your template after you schedule your send will not be included. You must unschedule the send, make changes, and then reschedule the send to include updates or changes in your template.

Add Segments

Under Segment, click the gray plus sign to add additional test segments. You can have up to four test segments in AB Winner campaigns and you can also modify the audience percentages for each as well as the percentage for the final segment. The minimum percentage is 1%.

Add the Basics

Tests may be added to any of the Basics, including:

  • The From Name and From Email that will display for recipients.
  • Email Subject for the email.
  • Reply-To Email to which recipients can respond.

Once you add a test, you'll see additional fields for that item which you can edit. For example, in the image below, two different Email Subjects have been entered.

NOTE: Clicking the envelope and plus symbol icon allows you to add a verified email that you have already added to the system for verification.

Add a Data Feed, Promotion, or Ad Plan (optional)

You also have the choice here to include a Data Feed in your email, with or without a test. Data Feeds are used to supply content to not only emails, but website and other online content. The system uses the following feed types, which should contain your product or content URLs:

  • Content Feeds are supplied by the Personalization Engine, which gathers users' interests based on website activity.
  • You provide URL Data Feeds, which may be generated by your content management system. It is recommended that you add these to your feeds page in the platform so that the system can cache them and make them easy to access within your Template or campaign. JSON format is recommended.
  • Merged Feeds contain multiple Content and/or URL Data Feeds, which are merged for you into a single feed.

See the Data Feeds Overview documentation page for more information, and note that your template must be coded to support the display of data from a feed (see Code a Template Using a Data Feed).

To insert a unique, single-use promotion code in each message, configure a Promotion, then select the name of this promotion from the drop-down. Promotion codes will be included in your A/B test sends, however, differentiating Promotions across A/B test variants is not supported.

See the Ad Targeter Setup documentation page for more information on the Ad Plan feature.

Design

The Design page is where you will modify or update content that appears from the Template you have chosen for your campaign. You can create a campaign from scratch, but it is a best practice to start with a Template. When using a Template, you can still manually edit your code for each campaign send, as needed.

Design Your Email Content

On the Design page, you will see the side-by-side code editor. You can modify the HTML from your chosen Template as needed and use the preview window to view and test your content. Click Test Send to quickly send a test email. You will then be prompted to enter the test email address.

NOTE: When using the Test Send button, the opt-out link will not render in the test email.

Code from your Template will be shown in the HTML viewer and as you make changes to the code on the left-hand side, you'll see updates on the right. The Design page allows you to preview your email before making it live.

For AB campaigns, you also have the option to add a test for content creative and copy.

On the HTML viewer, click Text and then to convert the preview to the text version of your email.

Also, above the HTML viewer are your Advanced options:

  • The Zephyr Setup section will run once for every recipient of the email send, enabling dynamic headlines and content.

  • Link Setup allows you to indicate whether you'll be tracking links and/or using Google Analytics to measure what happens on your site after a user opens and clicks within your email.

Using Zephyr

For all AB Tests, you can use HTML to create static data, and Zephyr for dynamic information. Zephyr is the coding language used in email subject lines and campaign Templates to increase customization. For example, Zephyr is used to incorporate data feeds to populate the Template with your latest content and to call variables associated with user's profiles, such as their geolocation to ensure relevancy for each recipient of the email.

See the For Developers documentations pages for more information on using Zephyr for email campaigns. You can also click Zephyr Guide from the Design page to access the Zephyr Syntax Overview documentation.

Viewer Options and Image Library

You can adjust your viewing options by clicking the cog icon at the bottom of the HTML viewer. This is also where the image library is located. Images you upload are available to you for all emails you design. When deleted from an email, they remain in the library for use in other emails.

You can easily bulk upload images by dragging and dropping from desktop or folder into the Image Library.

Schedule

When and how your email will send is critical in getting a response from your audience. There are multiple options that let you control the send date and time to reach your audience when your content is most relevant and valuable, including the Personalized Send Time.

Scheduling Your Campaign

On the Schedule page, set up all options around how and when your email will be sent. If you choose to send On Confirmation your email will send once you click Schedule on the confirmation page. You can also set a Specific Time, which allows you to schedule an exact time and date in the future.

Select Personalized Send Time to optimize the sending schedule for each recipient. 

Note: Tests that use Personalized Send Time show in the Campaign List as scheduled for a specific time. 

Confirmation and Seed Emails

Enter the Confirmation Email for the individual(s) who should receive an email when the campaign completes sending, including a confirmation of completion and total emails sent. To enter multiple confirmation email addresses, separate using a comma.

Enter the Seed Emails to which the email should be sent. Note that this will send to the email address regardless of opt-in status.

Confirmation

The Confirmation section provides you with an opportunity to review a summary of the settings you've chosen for your campaign.

For AB Tests, the confirmation page will provide information for each segment. For AB Winner (and AB Auto Winner) campaigns, the final segment is also shown.

Note that the information for the Final segment is shown as "Determined by Winner" as the test campaigns have not yet been sent.

The tests are highlighted in color for each segment and the percentage for each segment is noted in the title.

TIP: Make it best practice to review all of your campaign details presented on this page prior to clicking Schedule. Try reading this page from the bottom up, or printing to review on paper. Changing the order or method in which you've previously reviewed information can help you to find errors or items that need updating.

If you have scheduled your campaign to send On Confirmation, a pop up will appear to confirm and give you the option to schedule for a later date.

You can always return to edit any section of your campaign prior to the send time.

NOTE:When creating an AB Test, note that each segment -- including final segments for AB Winner or AB Auto Winner campaigns -- will be viewable as separate campaigns on your Campaign page.

The example below shows the three campaigns generated by an AB Winner campaign with two test segments.

Sending the Winning Variant

The winner for the final campaign must be manually selected from the Confirmation screen once both A and B campaigns have been sent out by clicking on the "Set as Winner" button in the Summary section. The winning campaign must be selected within 24 hours of the original send.