MS Dynamics online 365 Integration with Sitecore

In this blog, let us understand how to integrate MS Dynamics CRM data into Sitecore CMS using Sitecore Connector Module.

MS Dynamics is a CRM system to manage business entities such as Contacts, Products, Events etc and to establish the relationship among these entities. This provides an API layer with set of API methods to read and write CRM data.

Sitecore is an experience platform for building and managing advanced web applications. The platform provides the capabilities such as

Multisite, Multilanguage sites, Users and Roles management, Content workflow, Personalization, Search, Marketing Automation, EXM, Forms, AB testing, Analytics, Commerce etc.

What is the purpose of integrating the CRM data into Sitecore?

Once the CRM contacts are integrated into Sitecore, the capabilities (Personalization, Promotion, Marketing Automation campaign, AB testing, Email campaign, Customer behavioural analytics) of Sitecore can be applied on the contacts to have contextual and personalized experience. This way, customers can be well connected with brands/products. Since all these digital capabilities are not part of the CRM, contacts will be integrated with Sitecore.

Ways to integrate CRM data with Sitecore

CRM data can be integrated with Sitecore using following ways.

MS Dynamics system exposes an API layer with set of API methods to read/write data. This API can be consumed to read the required data and push it to Sitecore xDB using Xconnect Service. This involves lot of custom coding in establishing secure connection to the API, reading the data, applying data filters, pushing the data to xConnect etc.

In order to import/export data with external systems, Sitecore has come up with a framework module called Data Exchange Framework (DEF). Using this module, data from external sources (CRM, DB, Filesystems [Json/Xml]) can be integrated with Sitecore using few predefined configurations. Sitecore has further simplified the CRM integrations by developing CRM connectors on top of DEF. In this blog, we are going to use Sitecore Connector for MS Dynamics module to integrate MS Dynamics with Sitecore.

Why data should be imported into xDB?

Experience Platform is designed to use xDB as its visitor profile data repository, most of the Sitecore capabilities function based on xDB data.

High level Architecture

The above diagram shows the high-level view of various components and modules involved in the integration.

To explain it in simple way, Sitecore connector will have two sets of configurations

 1. Source provider (Dynamics provider)

 2. Target provider (xConnect provider)

 Each provider has its own set of configurations such as Service endpoints, Value accessors, Field settings, value mapping sets etc. First, need to configure the providers.

Pipelines are the set of functionalities that should be performed by making use of source and target providers to import the data from MS Dynamics. The actual logic present in the pipelines.

Pipeline batches are the trigger points which executes sets of pipelines to perform the import operations.

Once the import pipeline batch runs, it pulls the data from MS dynamics based on the provider settings (field mapping, data filtering) and pushes the data into xdb through xConnect.

Now, we have understood how the data integration happens in a high level. Let us start understanding the actual steps that should be followed to set up the integration.

Step1:  Install Sitecore XP 9.1.1 on your system, make sure it is up and running with xConnect and solr.

Reference: https://dev.sitecore.net/Downloads/Sitecore_Experience_Platform/91/Sitecore_Experience_Platform_91_Update1.aspx

Step2: Install the module “Sitecore Connect for Microsoft Dynamics 365 for Sales 2.1.0”. Install the supporting modules in the same order given in the installation document.

Reference : https://dev.sitecore.net/Downloads/Dynamics_CRM_Connect/2x/Sitecore_Connect_for_Microsoft_Dynamics_365_for_Sales_210.aspx

https://dev.sitecore.net/~/media/8CF086484FC742F997BDC2A8EDAA770D.ashx

Step3: Generate the xConnect collection module for the MS dynamics data (contacts and interactions) and deploy it in xConnect.

• Open Sitecore content editor and navigate to the path “sitecore/system/Settings/Data
• Exchange/Providers/xConnect/Collection Models/Connect for Dynamics/Connect for Dynamics Collection Model” and select the item.
• Click on “Convert Model to Items”, this generates the collection items (firstname, lastename, job title etc. along with the facets) and then click on “Convert Model to Json”, this would generate the collection model and downloads the Json file.

• Deploy the Json file in the following xConnect Folders

          xConnect Main:  \App_Data\Models\

          xConnect Indexer:  \App_Data\jobs\continuous\IndexWorker\App_Data\Models\

Step4: Add MS dynamics specific connection string entry in the Sitecore connectionstring.config file

Ex. <add name="MSDynamicCRM" connectionString="authentication type=2;Username=#####@####.onmicrosoft.com; Password=######;Url=https://######.crm8.dynamics.com/XRMServices/2011/Organization.svc"/>

: replace # with corresponding values.

Step5: Navigate to the path “Sitecore/system/Data Exchange”, right click and create a tenant for dynamics connect.

This would take some time and create a tenant for Sitecore connect. A tenant will have all the required setting items for Source (dynamics) and Target (xConnect) providers, default Value accessors, Value mapping sets, Pipelines and Pipeline Batches. 

 Let’s pause the steps and understand the purpose of these setting items in the generated Tenant. 

DataAccess

This section contains the configurations related to the data access such as data fields that should be read from MS dynamics and data fields that should be updated in xconnect (vice versa), value reader and value writers’ settings etc.

Fields in ‘Value accessor sets’ represents the fields in the MS dynamics and xConnect. 

Note: By Default, all these Source and Target accessors will be generated as part of the tenant creation, based on the needs, fields can be added/removed/disabled.

End Points

This section contains the configurations related to service endpoints for both MS dynamics and xConnect.

Pipelines

Pipelines contains categories of data transfer folders; each folder contains the set of actual pipeline logic for the data transfer. Based on the needs, additional pipelines can be added or existing pipelines can be removed.

Pipeline Batches

The pipeline batches are the trigger points to initiate the various categories of data transfer. Each Pipeline batches are associated with actual pipeline categories folder in the ‘pipelines’ section. As shown in the diagram, the ribbon ‘Run Pipeline Batch’ command should used to run pipeline batches.

Tenant Settings

This section contains the common settings like filter expressions and linking related entities configurations that can be applied for all the data that is going to be transferred.

Value Mapping sets

It contains the field value mapping configurations between Source (Dynamics) and Target (xConnect) systems. The value mapping sets are used by the pipelines for value mapping.

We can visualize the relationship between all these configurations as shown below. 

All these components can be extended with the custom implementations and can be enabled/disabled with the checkboxes on each item level.

Now we understood each configuration in the Sitecore connector tenant. Let’s continue with the steps.

Step6: Configure CRM and xConnect endpoints

Make sure that the connection is getting established on both the source and target providers by selecting the provider and clicking on ‘Run Troubleshooter’.

Step7: Run the pipeline batch ‘Dynamics Contacts to xConnect Sync’ (make sure to clear off the ‘Last Run Finished’ date field before running the pipeline). The result of the run can be verified by viewing the log file in the field ‘Log’ on pipeline batch item.

Step8: Verify the imported CRM Data on xDB using following queries.

select * from [sc911_Xdb.Collection.Shard0].xdb_collection.InteractionFacets

select * from [sc911_Xdb.Collection.Shard0].xdb_collection.Interactions

select * from [sc911_Xdb.Collection.Shard0].xdb_collection.ContactIdentifiers

select * from [sc911_Xdb.Collection.Shard0].xdb_collection.ContactFacets

select * from [sc911_Xdb.Collection.Shard0].xdb_collection.Contacts

select * from [sc911_Xdb.Collection.Shard1].xdb_collection.InteractionFacets

select * from [sc911_Xdb.Collection.Shard1].xdb_collection.Interactions

select * from [sc911_Xdb.Collection.Shard1].xdb_collection.ContactIdentifiers

select * from [sc911_Xdb.Collection.Shard1].xdb_collection.ContactFacets

select * from [sc911_Xdb.Collection.Shard1].xdb_collection.Contacts

Note: ContactFacets table will have all the contacts data imported from CRM.

If the data is imported successfully, the contacts will be loaded on sitecore’s experience profile dashboard.

Happy Learning 😀