A / B Test Scenario [EDP]

This topic describes how to set up an A / B Test in a Campaign that uses the Engagement Data Platform (EDP) as its data source.

In a normal EDP-driven Campaign (without Cells and Splits), the Campaign screen presents the user with five chevrons across the top of the screen that represent the five key steps in a Campaign -- Setup, Content, Review, Status, and Analytics.

When you add Cells and Splits to the Campaign, the Campaign screen presents the user with three additional steps, representing the Cells and Splits workflow -- Setup Cell Tree, Setup Testing Elements, and Setup and Launch Campaign.

To split your Campaign into multiple cells:

  1. In the Tool Bar on the Campaign screen, click Actions > Cells & Splits. A confirmation dialog box is displayed; click yes. The 1. Setup Cell Tree screen is displayed.

  2. The This Cell's Code field is automatically populated with a default cell code consisting of "CELL_" followed by the Object Reference ID for this Campaign. For example: "CELL_15691." Optionally, enter a custom cell code in this field.

Note: If you created this Campaign by copying an existing Campaign, please note that the new Campaign will contain the same Cells & Splits, with the same Cell Codes as the base Campaign. If you don't modify those Cell Codes in the new Campaign, you may unintentionally reuse those Cell Codes, which may cause unexpected results to appear in your Campaign reports. In most cases, you'll want to modify the Cell Codes in the newly copied Campaign.

  1. Click split into testing cells.

  2. Select the A/B Test radio button.

Optionally, if you need to remove the cells and splits from a Campaign, click Actions > Undo Cells & Splits. A confirmation dialog box is displayed; click yes.

EDP-driven Campaigns

For Campaigns that use EDP as the data source, the following A / B Testing features aren't supported:

  • SMS Text Channel

  • Split by Field Values

  • Split by Segment

  • Import cells from a spreadsheet

  • Sorting records by a specified field; the only available sort method is "Random."

  • Some variable elements are not supported (see below for details).

  • The automatic winner feature is supported only for Regular One-off Campaigns.

  • If using the automatic winner feature, the Unique Reads and Web Events winning conditions are not supported.

Dynamic Blocks are partially supported in EDP-driven Split Cell Campaigns. To use Dynamic Blocks, you must select Content as the element by which to split. And, you must insert the same Dynamic Block into the content of every sub-cell.

Split Methods

The next step in the process is to determine the method used to split your Campaign Audience, and to create the desired hierarchy of parent cells and sub-cells.  

  1. From the Split by drop-down menu, select the split method you want to use:

Split by Percentages

Splitting by percentage allows you to assign a specific percentage of the parent Audience to each sub-cell.

  1. From the Split by drop-down menu, select Percentages. 

  2. The Order By drop-down menu allows you to determine how you want to order the recipients within the Campaign Audience. For an A / B Test, the only available option is "Random," so that you don't inadvertently skew your test results.

  3. In the Add Cell text field, enter the desired number of sub-cells you want to create, then click the add button (plus-sign icon). The system adds the indicated number of sub-cells.  

  4. The new cells are given a default cell code consisting of the parent cell code (defined above), and appended with "CELL_" and a counter. For example" Parent_CELL_1." Optionally, enter a custom cell code in the "Cell Code" field.

  5. In the Percentage field next to the sub-cell, enter the desired percentage value.

  6. If you need to remove a cell, click the remove button (X icon) within the cell list.

  7. The system will automatically create a remainder cell for you called the Winner Cell. This cell represents the bulk of your audience, who will receive the winning conditions, once your A/B test results are complete.

Split by Amounts

Splitting by amount allows you to assign a specific record quantity to each sub-cell.

  1. From the Split by drop-down menu, select Amounts.  

  2. The Order By drop-down menu allows you to determine how you want to order the recipients within the Campaign Audience. For an A / B Test, the only available option is Random, so that you don't inadvertently skew your test results.

  3. In the Add Cell text field, enter the desired number of sub-cells you want to create, then click the add button (plus-sign icon). The system adds the indicated number of sub-cells.  

  4. The new cells are given a default cell code consisting of the parent cell code (defined above), and appended with "CELL_" and a counter. For example  "Parent _CELL_1." Optionally, enter a custom cell code in the "Cell Code" field.

  5. In the Amount field next to the sub-cell, enter the desired size of the sub-cell.

  6. If you need to remove a cell, click the remove button (X icon) within the cell list.

  7. The system will automatically create a remainder cell for you called the Winner Cell. This cell represents the bulk of your audience, who will receive the winning conditions, once your A/B test results are complete.

Note: You can't use different split methods within the same parent cell.

  1. The Channel column allows you to identify a marketing Channel that will be used for each sub-cell, or to select Control to designate a sub-cell as a Control Cell. From the drop-down menu in this column, selected the desired Channel for each sub-cell, or select "Control." If you select "Control" for a sub-cell, a confirmation dialog box is displayed; click ok to confirm. Please note that once you make this Channel selection and save the Campaign, you can't later change the Channel for a sub-cell.

Note: The drop-down menu in the Channel column has Email selected, and is disabled. A / B Testing is available only in the email Channel, so this option can't be modified.

  1. Optionally, if you want to rearrange the sub-cells, click on the gray box to the left of the sub-cell row, and drag-and-drop the row into its desired new position.

  2. In the Tool Bar, click save.

Note: When creating the hierarchy of cells for an A / B Test scenario, the platform allows you to create only one layer of "child" cells beneath the main "parent" Campaign. In an A / B Test, you can't further divide those "child" cells into "grandchild" cells. Unlike a Split Cells Campaign (which can create any number of levels), an A / B Test Campaign can have only a single level of child cells beneath the parent Campaign.

Cell Tree Features

The Cell Tree is an important tool to help you keep track of WHAT you're editing. Within the Cell Tree, the currently selected cell is always highlighted in bold. In addition to serving as a navigation tool, the Cell Tree also provides quick access to the following feature:

  • Rename a Cell: Within the Cell Tree, hover your mouse cursor over the cell that you want to rename, and click the "edit" icon. The "Edit Cell Code" pop-up window is displayed. In the "Cell Code" field, enter a new cell code, then click ok.

Winning Options

The next step is to configure how you want the system to determine the "winner" in your A/B Test.

  1. From the "Send Winning Campaign" drop-down menu, select one of the following deployment options:

Manually

If you select Manually, the system won't determine a winner, and won't automatically deploy messages to the Winner Cell; see the Send Winner section below for more details on how to manually send messages to the Winner Cell.

With a manual deployment, you have the option to configure a reminder email notification. The platform will send this notification at the designated interval, if the Winner Cell has not yet been manually deployed. This notification is intended as a reminder, so that you don't forget to deploy the Winner Cell.

To configure the reminder notification:

  1. Enter one or more email addresses for the reminder notification recipient(s). Multiple email addresses must be separated by commas.

  2. Select when to send the reminder notification -- either "x" number of hours after launch, or "x" number of days after launch.

Automatically

Note: The automatic winner feature is available only for Regular One-off Campaigns.

If you select Automatically, the system will determine the winning variable conditions based on the business rules you provide below; in addition, the system will automatically send out messages to the Winner Cell, using the winning variable conditions.

  1. Identify the business rules for defining the winning conditions. From the Based on drop-down menu, select one of the following winning conditions:

    • Unique Clickers

    • Unique Opens

  1. By default, the system will identify the winner as the sub-cell with the most activity of the type you selected in step 2. Optionally, you can define a threshold that states the winning sub-cell must surpass the other sub-cells by a certain percentage. To define this threshold, select "at least" from the "with improvement of" drop-down menu, then enter a percentage amount.

  2. Define the testing period. Once this testing period is finished, the system will begin the process of identifying the winner. The testing period options are:

    • To define a testing window, click the after toggle button. Enter a duration in hours or days (the default is 3 hours). This window begins once all test emails have been sent, and ends once the specific duration has elapsed.

    • To enter a specific date / time, click the start on toggle button. Enter the date and time when the winner calculation process should begin.

  1. Define the tiebreaker rules in the event that no sub-cell can be determined to be the winner. From the If no winner (tie) send drop-down menu, select one of the following options:

    • Random leader -- The system randomly selects one of the tied sub-cells to be the winner.

    • None -- The system will not break the tie; instead, you'll have to manually select the sub-cell to be the winner. See the Send Winner section below for more details on how to manually send messages to the Winner Cell.

  1. Select a specific sub-cell to be the winner in the event of a tie.

  2. Optionally, if you want the system to send out a notification email when the A / B Test is completed, enter one or more email addresses in the When test complete alert field.

  1. In the Tool Bar, click save.

Variable Conditions

The next step is to select what Campaign element or elements are going to be different between each of the sub-cells.

(This is the Drop-down text)

  1. Beneath the cell list, from the Elements to split drop-down menu, select one of the following options:

    • None: All sub-cell values are inherited from the parent cell.  

    • Subject line: Set a subject line for each sub-cell in order to test the effectiveness of a subject line.

    • Content: Define message content for each sub-cell in order to test the effectiveness of different content.

    • Any: This option presents you with a list of all the possible Campaign elements that can be varied, grouped into different categories. Place a check mark next to one or more elements. The possible elements that can be varied are:

General Settings

Email Components

Sending Option

Content / Links

Metadata

To Name

Subject Line

Send Schedule

  1. In the Tool Bar, click save.

Populate Sub-Cells

After you have defined all of the options as described above, the next step is to actually populate the different Campaign variants in each of the different sub-cells.

Sub-cells automatically inherit the characteristics of their parent cell, except for the particular element, or elements, that you're changing. For example, let's say you chose to modify the Subject line. You would enter unique Subject Lines into each sub-cell. These sub-cells would inherit all of the other Campaign elements (From Name, From Address, content, etc.) from the parent cell.

The system supports two different methods of populating the sub-cells in your Campaign. Depending on the complexity of your cell hierarchy, and the number of elements being tested, you may find one method works better than the other. The first method is to select the Campaign element being tested, and to define that element in every sub-cell.  Then you would select a different Campaign element, and repeat that process.

The other method is to focus on a single sub-cell, and to populate all the different variable elements in that sub-cell. Then, you would move on to the next sub-cell and repeat that process.

Both of these methods are described below.

Define a Single Element Across All Sub-Cells

This method allows you to edit multiple sub-cells from the same screen, by first selecting the Campaign element that you want to test.

  1. At the top of the screen, click 2. Setup Testing Elements. This screen allows you to set up the different variants for each sub-cell.

  2. From the Elements to Test drop-down menu, select the element that you want to set up (or use the up-arrow and down-arrow buttons to navigate within this menu). This menu contains all of the Campaign elements that you selected on the previous screen. For example, if you selected Subject Line and Seed List as the elements that you want to test, then you'll see both those items listed in this drop-down menu.

  3. For a sub-cell, select or enter the desired option. If a test element wasn't selected for a particular sub-cell, the message Inherited from parent is displayed next to that sub-cell. The different options for the different Campaign elements are as follows:

    • Content / Links: The system displays a thumbnail preview of the content (when available). Optionally click preview to see a larger preview version of the content. Optionally click edit to edit the content within the Advanced Editor, then click "2. Setup Testing Elements" to return to this screen. Optionally select the desired character encoding method from the "Message Encoding" drop-down menu.

    • Metadata: If you selected Metadata as an element to test, every Metadata field in your account will be displayed within the "Elements to Test" drop-down menu. Select the desired Metadata field. For string fields, enter the desired value (including optionally dragging in personalization fields from the Personalization Pane). For date fields, select the desired date from the calendar pop-up. For pre-defined fields, select the desired value from the drop-down menu. For integer fields, enter the desired number.

    • To Name: Enter the desired "To Name" value (including optionally dragging in personalization fields from the Personalization Pane).

    • Subject Line: Enter the desired Subject Line value (including optionally dragging in personalization fields from the Personalization Pane).

    • Send Schedule: Define the Campaign schedule and / or the Send Time Optimization (STO) options for each sub-cell.

  1. The Campaign screen supports an optional "push" feature, that allows you copy test values from a parent cell down into all of its sub-cells. First enter or select the desired test value in the parent cell, then click the "Copy" icon. A confirmation dialog box is displayed; click ok.

  2. Repeat the above steps as needed for any additional Campaign elements that you need to define.

  3. In the Tool Bar, click save.

Note: For sub-cells that you designated as Control cells, you won't be able to edit anything, as the Control group doesn't actually receive any Campaign messages.

Define All Elements Within a Single Sub-Cell

This method allows you to define all the variable elements within a specified sub-cell.

  1. At the top of the screen, click 3. Setup and Launch Campaign.

  2. This screen displays the Cell Tree within a pane on the left-hand side of the screen. If you want more horizontal screen space, you can optionally dock the Cell Tree at the top of the screen. Within the Cell Tree pane, click the dock icon. The system moves the Cell Tree to a drop-down menu at the top of the screen. Conversely, to move the Cell Tree back to the left-hand pane, click the undock icon.

  3. Select the desired sub-cell, either from the Cell Tree pane, or from the Cell Tree drop-down menu (if you docked the Cell Tree). The system displays a "stripped down" version of the regular Campaign screen, displaying only those Campaign elements that you defined as being variable for the selected sub-cell. The different options for the different Campaign elements are as follows:

    • Content / Links: Click the Content tab view the Advanced Editor, where you can enter the content for this sub-cell. Optionally select the desired character encoding method from the Message Encoding drop-down menu.

    • Metadata: Click the Set Metadata link near the top of the screen. The Metadata pop-up window is displayed, and populated with all of the Metadata fields in your client account. Enter or select the desired Metadata values, then click ok.

    • To Name: In the Envelope section, enter the desired To Name value (including optionally dragging in personalization fields from the Personalization Pane).

    • Subject Line: In the Envelope section, enter the desired Subject Line value (including optionally dragging in personalization fields from the Personalization Pane).

    • Send Schedule: Define the Campaign schedule and / or the Send Time Optimization (STO) options for each sub-cell.

  1. Repeat the above steps as needed for any additional sub-cells that you need to populate.

  2. In the Tool Bar, click save.

After you define all of the cells and sub-cells using either of the above methods, click 3. Setup and Launch Campaign to proceed to the regular Campaign life cycle of Proofing, Auditing, and Launching. You can send proofs from within a sub-cell (which sends proofs for only that sub-cell), or from the top-level Campaign (which sends proofs to all sub-cells). Auditing and Launching, however, must occur from the top-level Campaign, and not from a cell.  

Send Winner

If you opted to manually determine the winning sub-cell, you'll need to select which sub-cell was the winner, then deploy messages to the Winner Cell.

  1. Navigate to and select the Campaign. An A / B Test Campaign that's waiting to send to the Winner Cell audience will have a Campaign status of Pending Winner.

  2. Click 3. Setup and Launch Campaign.

  3. Click the Status tab

  4. The Review & Approve section will display your cell tree with all of your child cells, including the Winner Cell. Next to the Winner Cell, click send winner. The Launch Winner Cell pop-up window is displayed.

  5. From the drop-down menu, select the winning sub-cell. The system will use the variable conditions in this sub-cell (subject line, content, etc.) to build the messages that get sent to the Winner Cell audience.

  6. Click launch winner.