Tasks can be used in different ways to manage data in the database. A user with the necessary rights to create these tasks has access to four types:
- Sql Task — Executes an SQL procedure on the database. This can be used to update data in the database, perform calculations and many more database operations
- Import Task — Imports data from an external source into the database
- Export Task — Creates an export file with data from the database
- Batch Task — Creates one big task from a series of sub-tasks, such as export, import and SQL tasks. The different sub-tasks are executed consecutively.
Note: When multiple tasks are scheduled at the same time, they will be executed one after the other.
Each one of the above tasks requires the use of Registered Stored Procedures. These procedures are created in the Admin configuration. (Check out this topic for more information on Stored Procedures.)
- Export — The Stored Procedure defines data sets to export.
- Import — The Stored Procedure defines how data is migrated from a source table to a production table.
- SQL — SQL is used to transform data. The Stored Procedure contains the actions that modify data held in the tables.
Tasks are executed in the backend and all configured tasks are executed sequentially. This means that even when there are two jobs scheduled to be executed at the same time, one has to finish before the other one can start.
Technical note: We encourage to set up any export or import with a significant size and/or frequency using the Tasks functionality. If real-time behavior is preferred (basically everything that needs to run under 10min.) you should consider using the Marigold Engage API.
Generic system exports
The Export/Import Tasks mentioned above are limited to data found only in lists (Audience Lists, profile extensions, etc) which will be used in building marketing campaigns. Data generated by the platform during the lifecycle of a campaign (opens, clicks, etc) - used for reporting - is inaccessible using normal export routines and tasks.
It is impossible to export ALL data from the Engage platform for use to import/update a Data Warehouse, and there is no justification to do so, either. Instead, Marigold provides a Generic Export Task (maintained, managed, monitored and supported by the Marigold DBAs) which can be setup to export a maximum of 50 organizations.
What is exported?
This task exports "system data", which is data generated by the platform used in campaigns (communications & marketing) that cover nearly all common scenarios for analytics and reporting purposes. The Marigold-developed routine is a tried-and-tested procedure that has proven to be more effective than any custom-built third-party solution. Attempts at creating custom Stored Procedures to access this data have proven problematic in the past, so it is recommended to adopt Marigold's common approach to obtaining platform data.
The purpose of this data export is to make source data available for more complex reporting and analytics tools, permitting organizations to use their preferred choice of dashboards and business intelligence tool sets to gain a deeper insight.
Where is it exported to?
11 files per organization are exported to the client's choice of FTP location (the MEDIUM). It is possible to export just ONE organization or ALL organizations (up to a maximum of 50 organizations) but each organization requires different folders on the FTP store for segregation.
Marigold's own FTP store (SaaSFS) can be utilized, but should be considered temporary as all data it is subject to auto-archiving and quota restrictions.
Clients may provide their own FTP servers for use. Prior to this happening, a request must be raised to whitelist known IP addresses and ports of the client FTP site to permit outgoing connections from Marigold's network infrastructure.
It is commonplace to use both approaches: scheduled exports will use SaaSFS as a target, then clients will retrieve data from this storage to manage themselves.
When is it exported?
Although the job frequency is adjustable, a 24-hour cycle is fairly commonplace.
More frequent exports can be scheduled but very low granularity (e.g.: hourly) can introduce race conditions, causing inaccurate results (for example, an export can report the mail has been receive by an MTA but the soft bounce return code may not have been processed in time to be included in the export) - depending upon the amount of data and the volatility of journeys currently underway.
How much is stored?
On Marigold's own FTP store (SaaSFS), data is retained for 45 days to meet legal compliance, then deleted after that time to conform with GDPR regulations. Data needed to be retained after that time is the responsibility of the client, who must organize transferring data sets to their own managed storage (e.g.: a crontab running rsync) prior to Marigold's automated deletion.
Each client should try not to exceed 1GB of storage space (1GB) on Marigold's SaaSFS; to avoid jobs failing due to insufficient capacity clients are recommended to:
- purchase Marigold's own Archiving system that will auto-prune data against a schedule
- handle truncation themselves (manual, automated or otherwise)
- utilize their own FTP storage outside of Marigold's network, and manage their own capacity and archiving processes.
Sounds great! How do I avail myself of this?
Simply raise a request through your CSM - our trained support staff and experienced DBAs will guide you through the process. Note this is billable work which can take up to four hours to implement and test.
