Skip to main content
Sign in

Project Recipe: Dedupe for Marketo

  • Updated

The following is a guideline to follow when using the recipe "Dedupe for Marketo". Please read these guidelines before cloning the recipe.


Overview

This recipe provides the following:

  • Jobs that dedupe lead records in Marketo:
  • A bot to run the above jobs. Note that the bot will be created, but not automated, and the assumption is that your Marketo lead object is being imported on a regular basis by pre-existing bots. 
  • Manual data sources that provide metrics on what records have been deduped.
  • Analytics reports
  • App Factory Apps to view the analytics reports containing the results of deduping

Preparation

  1. Establish data source:  A data source must be created for all your Marketo lead records. If you already have this data source, you do not need to create a new one. The data source needs to contain the following attributes, and may contain more attributes if they're needed for identifying duplicates and/or used to merge data from the non-surviving records into the surviving record:
    • Lead: address, city, company, country, email, firstName, lastName, postalCode, state, unsubscribed
  2. Create new data targets: Create a new data target with any attributes you'll be updating with data merged from the non-surviving records.  Note: If you don’t know exactly what fields you want to use, just create a data target for each object containing only the object id. Then, the data targets can be updated after the project recipe has been cloned and you have planned out which fields you want to update after merging.       

 

Clone

To clone the project recipe, navigate to Projects Recipes and select the project recipe to clone.

1.png

CloneProject01.png

You will need to provide:

  • Project name
  • Prefix and/or suffix for all objects in the project
  • Data source to be used in the project
  • Data target to be used in the project

Examine the cloned project

We recommend you look through all the newly created project elements to familiarize yourself with what has been created.

Look for:

  1. Any job that is marked with errors. These errors must be manually corrected.
  2. Export tasks that write back to Marketo. Make changes by selecting any values that want to write back to Marketo. Note: Any Export task to Marketo has been bypassed for your protection. We recommend that you run the jobs with the bypass enabled so you can review any data changes before writing data back to Marketo.

Review the dedupe logic and make changes as needed

These recipes will need to be modified to reflect the deduplication criteria for your organization. Below is a list of questions to help you develop your dedupe strategy:

  1. Are there any records that you want to leave out of deduplication?
  2. Which fields should be used to identify duplicates? eg. Email for Leads? 
  3. What criteria define the surviving record in a duplicate set?
  4. What fields get merged automatically if there is no value on the surviving record, but a value on one of the non-surviving records in a duplicate set? These fields need to be added to the jobs.
  5. Are there any fields that need special handling when merging records in a duplicate set? eg. For Lead Source, you may want to keep the value from the earliest created lead record.
  6. Are there any fields that should not be merged? eg. deprecated fields.

Run the bot

The first time you run the bots, we strongly suggest you keep the bypass option enabled on all export tasks.

Look at the output from each job, carefully examining your data to make sure it is being transformed correctly and to your standards. Review each export task to verify the mapped attributes are set up correctly.

When you are happy with the job outputs, complete a test run by:

  1. Modifying the merge task by adding a filter to limit records to a single Duplicate Set Ids value.
  2. Modify the export task to set a maximum number of records updated. This option is in the Advanced Configuration section of the export task template. 
  3. Remove the Bypass option on the merge and export tasks.
  4. Run the bot. When the bot is complete, review the records updated by looking at the job outputs and reviewing the attributes: op_merge_success, op_merge_errors, op_export_success and op_export_errors. Please also review the records in Marketo to verify the changes were made correctly.

Finalize the project

Final steps are:

  1. Edit the merge and export task to remove the restrictions implemented during testing.
  2. Edit the Dedupe MDS step to remove the filter that blocks writing records to the MDS. This will provide data for the analytics charts that are part of this recipe.
  3. Edit the Bot to schedule the frequency and enable automation.