Create Data Export

A Data Export allows you to export Marigold Engage data to an external location. You can define when and how often the export must be executed and if someone needs to be notified about the status of the export. Moreover, you can choose from a range of export locations where the data should be stored.

An Export can be created in two ways:

From the fly-out menu available from the Data Exchange entry in the left navigation bar. Click New and select the Data Export from the drop-down.
In the Imports/Exports Start page, select New > Data Export. A wizard guides you through the creation and configuration process.

Set the General properties

Properties

Folder Path — This is the location in the folder structure where the Data Export is stored.
Name and Description — Make sure to make it as explicit as possible to easily recognize the Export on the Start page.
Labels — The label(s) assigned to this asset. Select one or more labels from the drop-down. (These labels are configured in the Admin configuration.) Users with the proper Access permissions can also create new labels here by typing the new label value in the field.
API-Name — This name is used when the Export is executed via the API. By default the API name is filled out with the name given to the Export.

Schedule

The schedule switch indicates whether this Data Export should have its own scheduling. This is unnecessary if this Data Export will be part of a batch execution of jobs (batch job) – it inherits the scheduling of the parent batch job in that case.

The Schedule section defines the validity period of the export as well as the execution occurrence.

If the export needs to be scheduled, toggle the option on and define the schedule.

Start and End date — You can set a start date and/or end date for the export. For example, your export can start now but needs to run indefinitely.

Periodicity — Indicate when the export should run:

Daily — Select the hours of the day on which the export should run. You can select more than one.
Weekly — Select the day of the week on which the export should run. The export can run more than once a week. You can also set the start time.
Monthly — Select the days of the month on which the export should run. You can also select the time of the day at which the export should start.
Periodically — Set the recurrence of the export, expressed in minutes. For example, the export runs every 10 minutes.

Note: The scheduled time will be executed based on the server's timezone. The currently configured server's timezone is mentioned next to the info icon.

Notifications

A message can be sent either

OnFailure* — When the export fails. At least one notification group/email address is mandatory.
OnComplete — When the export is completed successfully.
OnError — Completed with errors (job was completed but one or more exports generated errors/exceptions).

. The details of the process can be included in the message. Additionally, the instance on which the problem occurred is included in the subject of the message.

To activate the option, simply activate the toggle on the left and enter one or more email addresses. (Multiple email addresses are separated by a semi colon).
You can also select a notification group. These notification groups are created in the Admin Configuration.

Note: Setting one or more notification group(s)/email address(es) for when the export fails is mandatory. The other notifications (on complete, on error) are optional.

When done, press Next.

Set the scheduler (Optional)

Note: The Scheduler section is only visible if there are schedulers configured on your environment. By default, there is 1 scheduler in which case this section is not shown as this default scheduler is used. If there is more than 1 scheduler configured, you will have access to the Scheduler section.

When multiple tasks, imports or exports are running it might be a good idea to use a different scheduler to make sure that long running tasks do not interfere with smaller tasks. Selecting a scheduler is optional and if you keep the default one all tasks/exports/imports will still run but if there is a bigger task, the smaller one will be executed only when the bigger one has finished.

You can choose between 3 different schedulers: the default one, custom scheduler 1 and custom scheduler 2. By selecting different schedulers for your tasks, they will be executed in parallel, without interfering with one another. So if you have long running tasks it might be a good idea to run these on a different scheduler.

Set the Export Options

Export list

List — The name of the Engage list to export. A drop-down list is available for you to choose one from.

Note: When the selected list is a shared list with a filter, only the records complying with the filter will be exported.

Note 2: When the selected list contains a field with an Option List value using the translateValue expression, this export will fail as the translateValue expression is not supported by the export.

Segment — The segment on the list that has to be applied to define the records to export.

Below, an indication is given on how many records will be exported. A filter icon lets you define an additional filter if required.

Note: This filter can use profile data, segments or Engagement and Interest metrics.

Only records belonging to the selected segment and matching the additional filter are exported.

Target

The second step consists of defining where the export file should go. You can choose from the following options: Repository, Azure Blob Storage, Amazon S3 Storage, Google Cloud Storage, Pre-defined, Download Link, FTPS, FTPS implicit, SFTP. Depending on the medium, different settings are defined:

Repository — The repository is the local file system of the job agent. This file system already contains the folder Campaign data in which a sub-folder can be selected. Click the drop down menu to select the sub-folder to store the file.

Azure Blob Storage — Provide the connection string (you can toggle the visibility of the string on and off by clicking on the eye icon) and container, and an optional subfolder. (*)

Amazon S3 Storage — Provide the access key id and secret access key (you can toggle the visibility of both strings on and off by clicking on the eye icon), bucket name, region endpoint code, and an optional subfolder. (*)

Google Cloud Storage — Provide the type, project id, private key id and private key (you can toggle the visibility of both strings on and off by clicking on the eye icon), client email, client id, auth URI, token URI, authprovider X509 certificate URL, client x509 certificate URL, bucket name, and an optional subfolder. (*)

* Note: Details on how to setup Cloud Storage configurations, can be found here.

Pre-defined — When selecting this option the 'Predefined Transport Medium' field is displayed from where you can select a medium from a list of predefined mediums. These mediums are configured already in the Admin Configuration and linked to your organization. When selecting one all corresponding settings are used.

Download Link — With this option selected, a download link becomes available on the Data Exchange > Imports/Exports start page, to locally download the (full or filtered) data export as CSV file.
Note: The 'download link' option is not available by default. It can be activated on request.
- After selecting 'Download Link' from the drop-down, a 'filename' field is available to enter a filename.
- You can enter a filename manually and/or insert personalization fields.
  
  You can personalize the filename using the current date and export system fields.
- It's advised (but not mandatory) to enter your filename with a '.csv' extension.
  Examples :
  - with .csv extension :
  
  - without extension :
- When the Data Export gets executed later on, a download link becomes available.
  
  Clicking it downloads your file which you can store locally.
  Note: The download link expires after 7 days. If you want to be able to download it after the expiration date, the data export needs to be executed again.

SFTP, FTPS, FTPS implicit — The name of the server as well as the user and password to connect to the server is required. The user can provide a specific sub-folder where the file should go. By default, a sub-folder is already selected.

Private Key authentication

For SFTP, besides using a password to authenticate when connecting to the server, it's possible to use a private key instead:

A toggle is available to select either Password or Private key.
When Private key is selected, you can enter (or paste) the private key data in the field Private Key.

Important: We only support PuTTY format private keys for SFTP exports. So please make sure you use the correct format. The PuTTY user manual can be found here.

If the private key requires a password (some servers do), you can enter the password in the Private key password field. This is an optional field.

Note: The data from both fields (private key and private key password) is stored encrypted in the database, and only used when transferring the files.

The Host key is an optional field that can be used as an extra verification step to make sure you’re connecting to the right server.

Note:
When saving the Data Export:
- The content of the private key field is emptied (for security reasons)
- The label for the private key field is updated to Specify a new private key in order to update the existing private key
- The content of the private key password field is emptied (for security reasons)
- The label for the private key password field is updated to Specify a new private key password in order to update the existing private key password

File options

Click the pencil icon to access the File Options Dialog.

The file delimiter can be set to colon, semi colon, tab or pipe delimited. By default the delimiter is set to ;.
It is possible to wrap text with text quotes. Toggle the option on if you want this.
You can also set the first row as the one containing the column names. This option is selected by default.
The Skip BOM option allows you to not take into account the Byte of Order Mark at the beginning of a text stream.
Set the maximum file size, expressed in Megabytes. If the export exceeds this amount, further files will be created with _001, _002 etc appended to the filename (before the file suffix), each not exceeding this size, e.g.: setting this to 10MB will create "export.csv", "export_001.csv", "export_002.csv", "export_003.csv", each less than 10MB. Leave at 0 if no limit is to be set.
Set the Line break either to Windows or Unix. Windows uses carriage return and line feed ("\r\n") as a line ending, where Unix uses just line feed ("\n").
Set the type of encoding to apply to the file. Click the drop-down list to access an extensive list of encoding mechanisms.

Advanced

Overwrite existing target file — If the selected target file already exists, it will be overwritten if the option is selected. If the option is not selected the export will fail upon execution.

If no data to export:

Skip export — In case the selected list, segment and filter does not return any records, the export is not executed. A COMPLETION notification is generated.

Fail export — In case the selected list, segment and filter does not return any records the export is considered as failed. A Failure notification is generated.

Continue export — Exports an empty file or (a file just containing headers if configured) and generates a COMPLETION notification

Click the 'Next' button to go to the Field Mapping step.

Define the Field Mapping

In this section you define the mapping between the fields in the selected list and the columns in the export file.

From the Scope field, select the parent list, a 1:1 or 1:N linked list. This allows you to also export data in these linked lists.

Note: When a linked list is used in the export, this will be visible from the Relation Usage dialog.

Select the field name from the selected list.

Next, set the alias to be used in the export file. A suggestion is made based on the field name but you can change the alias.

Do this for all fields that need to be exported. Once all required fields are mapped, save the export.

You can also use expressions in your export. With expressions you can concatenate fields, convert date fields, format text fields or perform any other operation that is provided through the Personalization Dialog. You can enter the expressions also manually.

To create an expression for the export select Custom from the Scope drop-down list. The field column is then used to put your expression. A Personalization Dialog allows to select expressions to use.

Note : Expressions can include fields from 1:1 linked lists as well.

Note : The order of the fields determines the order of the columns in the export file. You can easily change this through drag and drop of a field. Just hover the line you want to move and click this icon to drag the line to a new position.

To save your configuration:

click 'Save' to save and return to the Imports/Exports Start page.

or, click the arrow next to Save and select 'Save and Execute now' to run the execution and return to the Imports/Exports Start page.

With this second option, you'll be notified immediately that the data export is marked for execution.

History

Once the export is configured a History tab provides details on the execution of the export, the duration, start date and details if an error occurred.