Introduction to the Data Migration Framework in Microsoft Dynamics by Ruben Barron Senior Consultant - Blue Horseshoe Solutions. 1
Table of Contents Intro to the Data Migration Framework (DMF)... 3 DMF Process Overview... 3 Prepare Source Data... 3 Define type of AX entity to be used... 3 Fields in the Source file are mapped from the source file to the staging table... 4 Transferring Data to the Staging Table... 4 Data transferred from the Staging tables to the final AX Target destination.... 4 DMF Components... 4 Accessing the DMF tool.... 4 Set Up Target Entities... 5 Set Up - Source Data Format... 6 Set Up File Location (data migration framework parameters)... 8 Processing Groups... 9 Entities... 9 Moving Data from Source Files to Staging Tables... 14 Moving Data from staging table to the Final AX destination.... 16 2
Intro to the Data Migration Framework (DMF) For any organization embarking into a new MS Dynamics AX 2012(AX 2012) implementation loading legacy data from difference sources into AX2012 is a critical component to have a good start and a successful implementation. Microsoft introduced the new Data Migration Framework (DMF) as an extension of AX 2012 to address the challenge of loading data to AX 2012. We consider that the DMF tool represents a great step forward for data upload and with great potential that can be used to complete data migration for the most common data entities needed for an implementation as well as a framework that can be customized to load other required entities. DMF Process Overview The DMF process can be summarized in the following diagram and includes preparing source data, defining the type and order of AX entities to load, mapping the source file to the staging table, transferring from the source file to the staging table and finally transferring the data to the final destination in AX. Data Migration Process Prepare Source Data Define type of AX entity to be used Mapping and Validation of Source File fields to Staging AX fields Transferring Data from Source file to the Staging Table Transfer of data from Staging to target AX destination. Prepare source files to be uploaded. Define AX entity(entities) to be used for upload. Example of entities include Customers, Vendors, Sales Orders, Journal Entries, etc. Mapping and validation before data is transferred from source file to staging Transfer data from source file to Staging table. Data can be validated at this stage too. Transfer data from Staging to final target AX destination. AX records are created or updated DMF Process Overview Prepare Source Data During this step, source data files are created and prepared to be uploaded to the AX application. A typical source file will have a header row with field names identifying each data column. The number of fields for each file will depend on the requirements for data upload. Thus it is important to note that some of these field names are required for each entity. The DMF version only supports flat files. Future versions of the DMF tool will support data from ODBC and AX destinations. Define type of AX entity to be used During this step we define the entity or entities that will be loaded to AX. These DMF entities are predefined relationships that come in the application and include common AX entities such as customers, vendors, products, etc. and available AX fields. 3
Fields in the Source file are mapped from the source file to the staging table During the mapping stage, the DMF tool will allow you to map each field from the source file to an AX field in the in the staging table for the corresponding DMF entity. The DMF tool uses the field name to map them against the available fields in the AX Staging table. Users have the ability to review and manually map these fields. Transferring Data to the Staging Table Once fields from the source file have been mapped to AX fields in the staging table data should be ready to be transferred to the staging table. This staging table acts as a temporary area and intermediate location before data is transferred to the final AX destination. While the data resides on the staging table, the DMF can apply some basic validation for referential data integrity. Data transferred from the Staging tables to the final AX Target destination. Once data has been reviewed and validated in the stating table, the data is then moved to the target AX entity and AX records are created. If data is reloaded the DMF tool will try to update the AX fields. DMF Components Accessing the DMF tool. After installation, the DMF tool can be accessed from the AX client and shows as a module in the AX application. You will see a common, setup and periodic areas as part of the DMF module. Data Migration Framework in AX 2012 4
Set Up Target Entities There is some setup that needs to be configured prior to use the DMF tool. Target Entities - The DMF tool comes with a set of predefined entities that can be used to load the most common entities in AX. The following list includes the entities supported out of the box: Bill of materials (BOM) Contacts Contact addresses Customer Customer addresses Dimension Employees Employee addresses Inventory journal Ledger balances Main account Open invoices (accounts payable) Open invoices (accounts receivable) Open sales orders Open purchase orders Price disc (agreements) Product Project Unit of measure Vendor Vendor address The list of available entities can be accessed from the DMF> setup > Entities area. These DMF entities come already preconfigured and can be defined as a mapping relationship between the fields in the staging table and the target destination. These DMF entities also include procedures to move data from the staging table to the final destination creating objects and fields in AX. It is noteworthy to say that these entities can be used to create new custom entities with additional fields or logic. The DMF tool also comes with a custom entity wizard to create a DMF entity from scratch. 5
Set up Data Entity Sample Set Up - Source Data Format Source format allow the DMF tool to identity the type of source data that will be imported into AX. In the source format, a user specifies the different parameters that define the source file to be imported. Some of the parameters that can be specified include the type of source(file, ODBC, AX table), file format, first row header, row column delimiters and few other described in the following table. A user can define more than one source format to be used in the DMF upload process based on specific needs. Example a source format can be created for flat files that are comma delimited. Another source format can be created for flat files with fixed length and a different source format can be created for data that is planned to be imported from an ODBC connection. 6
Setup - Source Data Formats FIELD GENERAL TAB Source Name Type File Format First Row Header Row Column Delimiter Text Qualifier Regional Settings DESCRIPTION Identifies the different source that can be configured in the application. Types include File, AX and ODBC. DMF currently supports only Flat files. Future versions of the DMF will support AX tables and ODBC connections. Options include character delimited or fixed length. Character delimited includes characters such as commas, semicolon, pipes, etc. Flag to indicate if firs row of the file includes a header row. Indicates which characters will be used to determine end of the line for the source file. Defaults to {CR}{LF) Indicates character to encase text and commas. Example 8230 E Main Ave, Suite A Regional settings for file. We normally default this to code 1252 for English. A very large selection of regional setting come with the application to cover all world regions/languages depending on customer requirements. It also includes list for Unicode and other specific formats. 7
Multiple Value Separator Delimited used to indicate multiple entries. Example multiple addresses, multiple email, multiple phones 317-509-90900 ; 508-999-1212 Table 1.1 Setup Source Format General Setup There is an Application tab that is part of the Source Format configuration and that include several setting to configure dimensions and how data is going to be processed. Setup Source format Application tab Set Up File Location (data migration framework parameters) There is a required configuration that includes defining a shared folder in which files will be located for the DMF tool. The DMF admin user needs to have admin access to this folder in order for the DMF to properly work. Setup DMF Parameters (shared directory) 8
Processing Groups Processing Groups are used in DMF as the main driver that will allow users to configure entities, move data to the stating table, validate data and copy data from the staging to the final AX destination. These processing groups can be reused to reimport data and also keep a history log of transactions and errors. Processing Group Creating a Processing Group Entities After creating a processing group, the first thing to configure is an entity. These entities are used to indicates what DMF entity to be used( example Vendor, customer) for processing, order in which entities will be processed if more than one is selected, data source to use, source file to use for processing and if AX business logic will be taking into consideration when uploading. The user will also use this entity to map and validate the source fields against the staging AX fields. Settings for Entity configuration New Entity: A user will select the New menu from the processing group to create a new entity or entities. Sequence: Indicates the order in which entity or entities will be processed to load data from source to staging area. You can have more than one entity to be processed. An example could be to load vendors, products and customers in one single load. While this is option is available, we prefer to keep it simple and only run one entity per processing group to keep things simple to debug and also allow the processing group to be reused for future loads. Entity Name: The entity name is used to select the DMF entity that will be used for processing data. This entity contains the mapping relationship with all the available fields that will be mapped from the source file to the staging tables. 9
Source Format: This source data formats are configured in the setup and specifies the type of source data that will be loaded. Run Business Logic: There are two setting that can be used to enable the check for AX business logic. Sample File Path: This path is used to indicate the location of the source file to be used for the mapping and validation of source data. Note: If a large file is used you can use just a sample of data to do perform the mapping and validation. Processing Groups Entity Configuration Generate Source Mapping: The next step is to map the source file against the staging table. If you use AX column names as your header the DMF tool will automatically try to do the mapping. If a different name is used, a user will have to manually configure the mappings. It is probably recommended to generate the source template file using the DMF tool tp speed up the mapping process. Still the user will need to review and approve the mapping. Use the Generate Source File to create a template with AX fields. 10
Modify Source Mapping: During this stage the user can review and modify the mappings from the source to the staging area. If any of the fields on the Source file is not mapped, a user has the option to manually link the field to desired field on the staging table using the graphical interface or on the table view. Processing Group Entity Mapping Source to Staging 11
Validate: After the mappings have been reviewed, the user can then validate. The validation process ensures that all fields in the source file are correctly mapped to the staging table. If an error is encountered a user can go back to the mappings and make corrections in the mappings. Processing Group Validation View Target Mapping: This option can be used to validate the mappings from the staging to final Ax target destination. Please note that the staging only contains fields from the source file. Fields that are not used are nor displayed and will not be used in your migration. 12
Processing Group View Target Mapping Preview Source File: Once mappings have been validated, a user has the option to preview the source file before the file is copied to the Staging Table. This is a good check that show how data aligns with each AX field after mapping. Please note that is not data validation during this stage.. Processing Group Preview data source file. 13
Moving Data from Source Files to Staging Tables Once data has been mapped and validated for the Entities, the data is ready to be transferred from the source file to the staging table. Get Staging Tables: Data is ready to be copied to the staging table. A user selects the Get Staging Table menu option to start the process. During this stage the user has the option to run the job on demand or batch it. This batch option is probably useful when processing large files. Processing Groups Moving data to Staging table 14
You will see a message indicating that the data has been copied to the staging table. If errors were encountered during the load, these errors can be viewed in the log file in the execution history of the processing group menu. Processing Group Copying data to Staging 15
Moving Data from staging table to the Final AX destination. Once data has been transferred to the staging table and validated, the data is ready to be transferred to the final destination in AX by selecting the Copy data to target. A message will be displayed informing of the processing and creating the AX records if the processing was successful. If records are reloaded, the AX records will be updated. Processing Group Copying Data from Staging Table to Target 16
Processing Group Created Vendor Records. 17