Skip to main content
Sign in

Project Recipe: Dedupe for Dynamics

  • Updated

The following is a guideline to follow when using the recipe "Dedupe for Dynamics". Please read these guidelines before cloning the recipe.


Overview

This recipe provides the following:

  • Jobs that dedupe records in Dynamics:
    • Account-to-Account
    • Contact-to-Contact
    • Lead-to-Lead
    • Lead-to-Contact (this job does not "qualify" the leads into an existing contact, it just fills the contact record with the merged data that resulted from the duplicate detection, and can update a custom field with the matching contact id)
  • Bots to run the above jobs. Note that the bot will be created, but not automated, and the assumption is that your Account, Contact and Lead objects are being imported on a regular basis by pre-existing bots. 
  • Manual data sources that provide metrics on what records have been deduped.
  • Analytics reports
  • App Factory Apps to view the analytics reports containing the results of deduping

Preparation

  1. Establish data sources:  A data source must be created for all three Dynamics objects: Accounts, Contacts and Leads. If you already have these data sources, you do not need to create new ones. The data sources need to contain the following attributes, and may contain more attributes if they're needed for identifying duplicates and/or used to merge data from the non-surviving records into the surviving record:
    • Accounts: merged, accountid, address1_city, address1_country, address1_line1, address1_postalcode, address1_stateorprovince, description, name, websiteurl
    • Contacts: merged, _parentcustomerid_value, address1_city, address1_country, address1_line1, address1_postalcode, address1_stateorprovince, description, donotemail, emailaddress1
    • Leads: merged, address1_city, address1_country, address1_line1, address1_postalcode, address1_stateorprovince, companyname, description, donotemail, emailaddress1, firstname, lastname
  2. Create new data targets: Create a new data target for each object with any attributes you'll be updating with data merged from the non-surviving records. You will need two data targets for contacts, one for the C2C dedupe and one for L2C dedupe. Note: If you don’t know exactly what fields you want to use, just create a data target for each object containing only the object Id. Then, the data targets can be updated after the project recipe has been cloned and you have planned out which fields you want to update after merging.       

 

Clone

To clone the project recipe, navigate to Projects Recipes and select the project recipe to clone.

1.png

 

1.png

 

You will need to provide:

  • Project name
  • Prefix and/or suffix for all objects in the project
  • Data sources to be used in the project
  • Data targets to be used in the project

Examine the cloned project

We recommend you look through all the newly created project elements to familiarize yourself with what has been created.


Look for:

  1. Any job that is marked with errors. These errors must be manually corrected.
    Note: When cloning, the dedupe task in the C2C, L2L, and L2C jobs may show an error, typically due to an attribute used in the task that is missing from the data source. To clear the error, look at the task configuration to see where there is an attribute missing, and fill it in where possible. Then  click Save. This should clear the error.
  2. Export tasks that write back to Dynamics. Make changes by selecting which values to write back to Dynamics. Note: Any Export task to Dynamics has been bypassed for your protection. We recommend that you run the jobs with the bypass enabled so you can review any data changes before writing data back to Dynamics.

 

Review the dedupe logic and make changes as needed

These recipes will need to be modified to reflect the deduplication criteria for your organization. Below is a list of questions to help you develop your dedupe strategy:

  1. Are there any records that you want to leave out of deduplication?
  2. Which fields should be used to identify duplicates? eg. company name and country for accounts? Email address for Leads? Email address and AccountId for Contacts?
  3. What criteria defines the surviving record in a duplicate set?
  4. What fields get merged automatically if there is no value on the surviving record, but a value on one of the non-surviving records in a duplicate set? These fields need to be added to the jobs.
  5. Are there any fields that need special handling when merging records in a duplicate set? eg. For Lead Source, you may want to keep the value from the earliest created lead record.
  6. Are there any fields that should not be merged? eg. deprecated fields or fields maintained by a managed package are often omitted from the merge logic.

Run the bot

The first time you run the bots, we strongly suggest you keep the bypass option enabled on all export tasks.

Look at the output from each job, carefully examining your data to make sure it is being transformed correctly and to your standards. Review each export task to verify the mapped attributes are set up correctly.

When you are happy with the job outputs, complete a test run by:

  1. Modifying the merge task by adding a filter to limit records to a single Duplicate Set Ids value.
  2. Modify the export task to set a maximum number of records updated. This option is in the Advanced Configuration section of the export task template. 
  3. Remove the Bypass option on the merge and export tasks.
  4. Run the bot. When the bot is complete, review the records updated by looking at the job outputs and reviewing the attributes: op_merge_success, op_merge_errors, op_export_success and op_export_errors. Please also review the records in Dynamics to verify the changes were made correctly.


Finalize the project

Final steps are:

  1. Edit the merge and export task to remove the restrictions implemented during testing.
  2. Edit the Dedupe MDS step to remove the filter that blocks writing records to the MDS. This will provide data for the analytics charts that are part of this recipe.
  3. Edit the Bots to schedule the frequency and enable automation.
    Note: the Lead-to-Contact deduplication bot should be run after the Contact-to-Contact and Lead-to-Lead deduplication bots, and after a re-import of both Leads and Contact data sources.
  4. When Dynamics merges records, it marks the non-surviving record by setting the attribute "merged=yes". For ease of data governance, please consider adding a filter-purge step in a bot to the Account, Contact and Lead data sources to purge merged records after each import, or after running these dedupe and merge jobs. Do this by setting the filter to purge records where "merged=yes".