Email Insights

Background

Marigold constantly checks all traffic that is being sent over our Grid platform against known spamtraps and dangerous domains. The deliverability team is constantly monitoring and identifying troublesome domains to add to a Global Suppression list, which is used to block bad traffic before it’s even sent. This results in an extra layer of security to protect our IP reputation and our sender’s domain reputation.

An increasing number of senders are looking for more than that and are considering third party validation services, not only to validate the domains they’re targeting, but also to validate the actual email addresses in their user lists. Marigold now offers a new service: Email Insights. This service will match the addresses in your user list against a list of known bounced addresses which appeared on our Gird platform so you can identify inactive addresses without having to send anything to them. This will lower any negative impact dead addresses can have on your reputation. If you want to talk numbers: currently more than 30.000.000 hard bounces and more than 2.000.000 soft bounces have been identified and added to the Email Insights Suppression list.

 

Introduction

Email Insights runs a collection of scripts that perform sanity checks and cleanups on user lists. The tool will run several validations based on predefined rules and will try to match email addresses from your user lists against our Email Insights Suppression List. If a match occurs we know the address has recently shown signs of inactivity, so it’s safe to suppress this address. Addresses for which no match is found aren’t necessarily active user, something to keep in mind.

 

What is collected?

Marigold collects hashed versions of email addresses that have bounced on the Grid platform across all Marigold instances. These hashes are stored in a central Email Insights Suppression List without any sender or client information, so the data becomes completely anonymous. An optout mechanism will be available for senders who don’t want us to collect data generated by their platform.

The data is aggregated based on the email hash and whether we received a soft or hard bounce. A counter and last occurrence date is logged per email hash, so not every occurrence of a bounce event is logged.

 

How does the check work?

Match hashed addresses

First we will try to find email addresses which occur in both the sender’s user list and the Email Insights Suppression list. To do this, a hash of the email addresses in the sender’s user list is created and then validated against the Email Insights Suppression List. If a match is found and it adheres to the configured deliverability rules the user record is flagged.

The default rules are:

  • At least 6 soft bounces have been recorded on at least 2 instances in the past 99 months.
  • At least 2 hard bounces have been recorded on at least 2 instances in the past 99 months.

Valid email addresses

If an email address doesn’t have a match in the Email Insights Suppression List, a syntax check is performed. This check is done based on the default email regex which is also performed by the Grid platform before an email is sent. Addresses who don’t pass the syntax check are flagged

Deliverability rules

The next set of checks are divided in several sub-sections and severity levels. Depending on the configured severity level these may generate an optout. Only rules marked with severity level “BLOCK” will generate an optout. When running the tool in Reporting Mode all severity types are reported.

Domain check

The domain part of the email address is checked. Troublesome domains may be identified as:

  • CorporateDomain
  • DangerousAddresses
  • DeadDomains
  • RiskyDomain
  • Spamtrap
  • TemporaryMailbox
  • ThrowawayAddresses
  • Typo

User check

The user part of the email address (before the @) is checked. The following classifications for these might be:

  • DangerousAddresses
  • RoleAccounts
  • ThrowawayAddresses

Email address check

These checks are performed on the full email address. Issues might be classified as:

  • DangerousAddresses
  • Spamtrap
  • ThrowawayAddresses

 

Email Insights Modes

Email Insights runs in 3 modes: Reporting Mode, Optout Mode and Recurring Mode.

Reporting Mode will run all the necessary checks but won’t automatically optout users. It will simply report its findings. If you want the process to also automatically clean up the user list we will run it in Optout Mode. It will also be possible to run Email Insights in On Demand Mode, on a fixed schedule, for instance once a week, to ensure regular maintenance on your user lists. This can be very helpful for customers where organic growth is substantial.

 

Some more details about the different Modes:

Reporting Mode

Reporting Mode can be performed on a either a specific user list or on all user lists available on a sender’s instance. An Email Insights report will be generated but no optouts will be set.

In addition to the checks described earlier, some additional informative checks are performed:

  • Number of records checked
  • Number of duplicates found
  • The top 10 domains found in the user list
  • An overview of the countries for which email addresses have been found

Optout mode

The Optout Mode will update the OPTOUT field on the user record to a specific code. This means these users will be excluded from future mailings. The corresponding OPTOUT_DT and OPTOUT_SOURCE (if available in the user list) fields will also be updated. To be precise, the OPTOUT_SOURCE field will receive the value ‘Email Quality Check Outcome’. An Email Insights report will also be generated.

The OPTOUT field codes that will be used are:

  • 900: Syntax error
  • 902: CorporateDomain
  • 903: DangerousAddresses
  • 904: DeadDomains
  • 905: RiskyDomain
  • 906: RoleAccounts
  • 907: TemporaryMailbox
  • 908: ThrowawayAddresses
  • 909: Typo in address
  • 910: Spamtrap
  • 912: hardbounce
  • 913: softbounce

The recurring run has the option to set a value on a configured 1:1 extended profile instead of using the OPTOUT field. However only 1 value is possible here and not a custom value per rule type

Recurring Mode

The Recurring Mode will schedule an Optout Run once a week. It will run on all user lists and will scan all new or modified records based on the CREATED_DT or MODIFIED_DT which don’t have an OPTOUT specified. New user lists will automatically be picked up during the next run.

If a specific list should be ignored for a specific reason, our DBA team can set a flag so it will be excluded from future runs.

Recurring Mode also has a specific option that allows the updating of a 1:1 extended profile instead of updating the OPTOUT field. This option can be enabled by the DBA team. The following information is needed for this configuration:

  • • ID of the extended profile
  • • Name of the field to update
  • • Value that needs to be set in the field

This option allows one specific value for all rule types and only one field can be updated. The relation to this extended profile will be retrieved from the platform configuration so it should be kept properly up to date in the platform itself.

It is possible to customize the amount of bounces it takes before a user receives an optout. This can be configured by our DBS team and will apply to all user lists. It’s not possible to specify different numbers for different user lists.

To enable Email Insights in Recurring Mode, please contact your TPM or CSM.

 

Appendix 1

Example of a Reporting Mode Run:

 

Appendix 2

  • 900: Syntax errors in the email address (e.g. missing @ sign, strange characters, …): It is impossible to deliver any email to a poorly formatted email address, so there is no point in keeping these in your list. Having a lot of syntax errors in the user list could indicate that the addresses in the list have never been mailed to in the past, so the quality is unknown or Email Quality hasn’t been properly configured.
  • 902: In case the domain, user or email check matches a ‘CorporateDomain’ rule: These are domains owned by big corporations such as Barracuda or McAfee that are linked to anti-spam products/services. We want to avoid sending to those domains as there is a high risk they use the traffic to dig into the data and potentially block ips/domains.
  • 903: In case the domain, user or email check matches a ‘DangerousAddresses‘ rule: These are typically technical role accounts such as postmaster@, abuse@, administrator@, … These addresses aren’t expected to receive marketing email so this might indicate a problem with the acquisition process or list hygiene, which might lead to reputation drops and blacklistings.
  • 904: In case the domain, user or email check matches a ‘DeadDomains’ rule: The domain analysis will determine which of the domains in your lists are no longer active. This could be due to typo’s (gmal.com, hotamil.com, …) or domains that no longer exist. Having a lot of these in your list could indicate a list hygiene issue or acquisition issue.
  • 905: In case the domain, user or email check matches a ‘RiskyDomain’ rule: These are domains that can be potentially linked to an anti-spam product/service or are in general having a high risk of causing reputation damage. (examples: uceprotect.com, example.com, …)
  • 906: In case the domain, user or email check matches a ‘RoleAccounts’ rule: It might be risky to send email to role accounts. Often these types of accounts don’t belong to a single person or are technical mailboxes. They are usually operated by several people which might all have a different opinion on what is or isn’t spam. In many cases the users won’t know if anyone actually used the address to sign up for a mailing, so there is a large risk of people handling the emails as spam. Technical mailboxes usually aren’t used to received commercial emails, so if they do receive this type of mail they might often be handled as spam, either by humans or build in systems.
  • 907: In case the domain, user or email check matches a ‘TemporaryMailbox’ rule: There are many services online who offer temporary mailboxes. These types of email addresses are often used by users want to do a one-time sign up for something but they don’t want to deal with any additional emails later on. These addresses typically disappear after a couple of days, leaving you with a dead email address.
  • 908: In case the domain, user or email check matches a ‘ThrowawayAddresses’ rule: Much the same as TemporaryMailboxes but those addresses are mostly not linked to a service but just random email addresses entered on a signup form (example: emailme@whatever.com).
  • 909: In case the domain, user or email check matches a ‘Typo’ rule: A typo has been detected in the email address, for instance hotmal.com, Gmil.com, Yaho.cmo, … There is no point in having these addresses in your list as these domains won’t be reachable. If these types of addresses are present in a frequently used list it might indicate a hygiene problem as these should have been cleared by the Email Quality process.
  • 910: In case the domain, user or email check matches a ‘Spamtrap’ rule: Spamtraps are email addresses which have been created to catch senders who perform bad practices. There are several types of spamtraps:
    • Typo traps: These are email addresses which can be used to identify senders who don’t implement a double optin system. Since no actual humans operate these boxes these addresses cannot be used sign up for mailings when a double optin is involved, since there is no one to actually confirm the subscription.
    • Recycled traps: Providers turn old, inactive addresses into spamtraps to check if a sender manages the quality of their lists properly. By the time the address is turned into a spamtrap you should've had plenty of time and received plenty of bounces to unsubscribe the user.
    • Honeypots/Pristine traps: Honeypots are email addresses that were never actually operated by a human being. These addresses are hidden inside websites, not visible to the normal user’s eye. If these addresses receive email that means they’ve been “scraped” of a website by an automated process and no one ever gave permission to send mail to that address. Hitting honeypots will result in blacklisting in no time. These types of spamtraps are often found in purchased lists.
  • 912: In case the hash of the email address matches the ‘hardbounce’ rule: Hardbounces typically indicate a permanent issue with an email address. The address might not exist, the domain might not exist, the email address is abandoned, …
  • 913: In case the hash of the email address matches the ‘softbounce‘ rule: Softbounces could indicate issues of a temporary nature. Mailbox full, the server is down, timeout during sending of the email, …