Software Development Kit (SDK) Data Migrator Guide

Transcription

1 Software Development Kit (SDK) Data Migrator Guide Sage ERP MAS 500

2 Copyright Sage Software, Inc. All rights reserved. Sage, the Sage logos, and the Sage product and service names mentioned herein are registered trademarks or trademarks of Sage Software, Inc. or its affiliated entities. All other trademarks are the property of their respective owners.

3 Contents Chapter 1: Overview... 1 Introduction to Data Migrator...1 Using Data Migrator...2 Setting Up Migration Links...2 Migrating Data...2 Chapter 2: Data Migrator Architecture... 3 High Level Data Flow...4 Supported Data Migrator Configurations...5 Data Migrator Processes & Communication...6 Processes and Communication Details...8 Migration Links...8 Role of Assisted Company Setup in Migration...11 Migrate <Step Name>...12 Staging Table Architecture Details...16 Migration Calling Architectural Overview...17 Migration Interface: dmmigrateintfc...17 Migration Caller Interface: IOwner...18 Migration Interface Enumerated Types...18 Extraction Architecture Details...21 Heterogeneous Data Sources Issue...21 Getting Extraction Data in Sets...22 Extraction COM Interface: IExtract...22 Extraction Stored Procedure Interface...24 Insert Architecture Details...25 Background Information and Overview...25 Insert Stored Procedures Interface...27 Migration Parameter Behavior and Metadata...28 Sage ERP MAS 500 SDK Data Migrator Guide iii

4 CONTENTS Chapter 3: Data Migrator Customization...33 Calling the APIs Determine the API Name Determine the Passing Data to the API Maximizing API Performance Return Values Populating Migration Parameters Getting API Error Logs Example of Calling the APIs Creating a New Source System Migration Create the Custom Extract Procedure Defining a Migration Link For Your Source System Add Metadata for Your Extractions Adding Functionality to Existing Extractions and Insertions Sample Add-on Extraction Metadata Adding a Staging Table to a Migration Step Chapter 4: API Reference...49 Balances General Ledger Account Balances Inventory Balances Inventory Balances Actual Cost Method Cash Banks Bank Accounts Tender Types Customers Customer Classes Customer Custom Fields Customer Items Customer Payment Terms Customer Price Groups Customers Sales Territories General Commission Classes...57 Processing Cycles Product Categories Reason Codes General Ledger Allocations Journals General Ledger Accounts Account Segments Budgets General Ledger Custom Fields Natural Accounts Sales Tax Accounts iv Sage ERP MAS 500 SDK Data Migrator Guide

5 CONTENTS Global...62 Sales Tax Classes...62 Sales Tax Codes...62 Sales Tax Schedules...63 History...64 Closed AP Invoices...64 Closed AR Invoices...65 Closed Purchase Orders...66 Closed Sales Orders...67 General Ledger Transactions...67 Inventory History...67 Inventory...68 Inventory...68 Inventory Item Class...69 Inventory Items...70 Warehouse Bin...72 Warehouses...72 Items...73 Freight Classes...73 Item Custom Fields...73 Noninventory Item Class...73 Noninventory Items...74 Product Price Groups...74 Purchase Product Line...75 Sales Product Line...75 Manufacturing...75 Employees...75 Operations...76 Routings...76 Tools...77 Work Centers...77 Open Orders...77 Purchase Orders...77 Sales Orders...78 Open Transactions...78 Bank Account Transactions...78 Open AP Invoices...79 Open AR Invoices...79 Pricing...81 Customer (Contract) Pricing...81 Matrix Pricing by Product Price Group...82 Matrix Pricing by Customer Group...83 Price Sheets...83 Product Group Pricing...83 Purchasing...84 Buyers...84 Purchase Order Custom Fields...84 Purchase Order Departments...85 Sales...85 Sales Transaction Custom Fields...85 Shipping Methods...86 Sage ERP MAS 500 SDK Data Migrator Guide v

6 CONTENTS Sales Organizations Sales Commissions Sales Teams Sales Persons Vendors Vendor Classes Vendor Custom Fields...88 Payment Terms Vendor Items Vendors vi Sage ERP MAS 500 SDK Data Migrator Guide

7 Overview 1 Use this guide to learn what drives migration, how to build your own migration for any legacy system you want to upgrade, and how to use Data Migrator APIs to load data on an ongoing basis. This guide provides detailed information on the technical aspects of Data Migrator, including: Schema and metadata that drive migration APIs used in Data Migrator Calling the Data Migrator APIs Modifying and creating extractions and insertions Introduction to Data Migrator Data Migrator, a task in Assisted Company Setup, offers a straightforward means of upgrading from Sage ERP MAS 90 and 200 to Sage ERP MAS 500. This migration feature replaces the need to write custom routines to transfer customer, vendor, and inventory information and to transfer the data to flat files for use with the import programs. Migrations are available for most of the data entry tasks, including all of the critical and voluminous items. Data Migrator automates many aspects of upgrading a back-office system to Sage ERP MAS 500. With the features available in Assisted Company Setup, Data Migrator provides the following features: A guided list of steps to set up a new company in Sage ERP MAS 500, including access to the appropriate migration, import, and data entry forms Expert schema mapping from Sage ERP MAS 90 and 200 to Sage ERP MAS 500 Staging tables to hold data between the source and Sage ERP MAS 500 Integrated access to Microsoft Excel for changing or adding to the data to be inserted Reporting capabilities that list records that do not meet the business rule requirements of Sage ERP MAS 500 An opportunity to fix failed records Metadata-driven architecture that supports customized extraction and insertion logic Sage ERP MAS 500 SDK Data Migrator Guide 1

8 CHAPTER 1 Using Data Migrator Setting Up Migration Links Migrating Data If your migration involves automatic extraction of data from a source system, you must first define a migration link that refers to the source system. For example, a migration link may indicate that the source data is Sage ERP MAS 90 Level 3.70, company ABC, and is found on the centralized server called Accounting in the MAS90 shared folder. By giving this migration link a name, the user can refer to the particular Sage ERP MAS 90 information without having to know where it is or how to extract it. You do not need to set up a migration link if you are going to use Data Migrator only as an insertion tool, entering data into the staging tables yourself, and then running the insertion process. For more information, refer to the Assisted Company Setup Help. Data Migrator consists of three separate activities: 1 Extracting the data from the source system (this is optional and used only for Sage ERP MAS 90 and 200) 2 Modifying, deleting, or adding records to those extracted 3 Inserting the records into Sage ERP MAS 500 The first and third steps are controlled with the Processing Options tab on the Migrate form. You can input data manually by going to the tab and using Excel to populate the staging tables, or you can use Data Transformation Services (DTS) in SQL. Processing options allow the choice of how to handle certain records. 2 Sage ERP MAS 500 SDK Data Migrator Guide

9 Data Migrator Architecture 2 There are several perspectives from which to discuss the Data Migrator architecture including the following: A high level view of the data flow The supported topologies The various processes and communication between them A functional view of the various processes in Data Migrator Sage ERP MAS 500 SDK Data Migrator Guide 3

10 CHAPTER 2 High Level Data Flow The figure below illustrates how Data Migrator supports the concept of moving data from a source system to the staging table holding area in the Sage ERP MAS 500 database. Data in the staging tables can be examined and modified using tools such as Microsoft Excel. The data can then be moved into the Sage ERP MAS 500 permanent tables and accessed through the front-end forms. Sage ERP MAS 500 Client PC Current Accounting System Database Sage ERP MAS 500 Staging Tables Sage ERP MAS 500 Permanent Tables Figure 1 - Data Migrator High Level Data Flow 4 Sage ERP MAS 500 SDK Data Migrator Guide

11 DATA MIGRATOR ARCHITECTURE Supported Data Migrator Configurations The figure below illustrates the five supported configurations. The first two configurations indicate that a source database, including a Sage ERP MAS 200 SQL database, can reside on any server in the network that has access to the Sage ERP MAS 500 client and server. (Sage ERP MAS 200 data is in a database pair: one database, MAS_xxx, where xxx is the company ID, and MAS_SYSTEM, which contains information common to all Sage ERP MAS 200 companies.) The third configuration, shown in the figure below, depicts the support for Sage ERP MAS 90 and 200. The fourth configuration indicates that any source system accessible using a connection string can be supported with customization. In particular, the common code can be used with an ODBC or OLE DB provider that can communicate using ActiveX Data Objects (ADO). Even if the common access code cannot support getting the source system data, you can still exploit the migration architecture, including the Assisted Customer Setup (ACS) forms, and use your access mechanism to put the data into the Sage ERP MAS 500 staging tables using the Sage ERP MAS 500 connection. The fifth configuration indicates that you can put Sage ERP MAS 90 or 200 data on the same server as SQL and the Sage ERP MAS 500 database. This configuration avoids network traffic and increases system performance. Figure 2 - Supported Configurations Sage ERP MAS 500 SDK Data Migrator Guide 5

12 CHAPTER 2 The SQL commands required to move data between SQL Servers is different than those required when the databases reside on the same server. Also, some connection properties and behaviors are different for the ProvideX ODBC driver, other ODBC drivers, and OLE DB. Sage has built a common piece of code to handle many of these differences. The third configuration in the figure shown above indicates that Sage ERP MAS 90 and 200 files can be located on any server in the network, including the Sage ERP MAS 500 database server. Because Sage ERP MAS 90 and 200 databases are files, passing of data between them and SQL server does not depend on where they are in the network, as long as both servers can be accessed by the client PC. For information on the methodology used to obtain data in all four cases, see Processes and Communication Details. Data Migrator Processes & Communication There are three main recurring processes in Data Migrator: the extraction, the insertion, and the Assisted Company Setup (ACS) form interaction (which contains two parts). The main ACS form and migration form are the first two pieces in the process: taking requests from the user, and calling the appropriate extractions and insertions, based on the current ACS step. The main processes and their primary responsibilities include: Assisted Company Setup Helps the user set up a Sage ERP MAS 500 company. Lets the user pick a migration source system. Determines if a migration is available for a particular ACS step. Launches the migration processing form. Migrate <Step Name> This is the primary migration interaction form. <Step Name> is the name of the current ACS step (for example, Sales Tax Accounts). Based on user choices, this migration interaction form runs the extraction and insertion and allows staging table data manipulation. A migration progress dialog box displays when either extraction or insertion is running. Extraction Places all data related to a particular step into the Sage ERP MAS 500 staging tables associated with that step. Insertion Applies the Sage ERP MAS 500 business rules to data in a step's staging tables and inserts the records into Sage ERP MAS 500. Depending on the report setting, information may be stored about the successes, warnings, and failures of each attempted record. The communication between the migration process and the extraction and insertion processes is done through two well-defined interfaces. All extractions use the same interface and all insertions use the same interface. The migration process uses metadata to determine what extractions and insertions to call for a particular step. The details of this process are provided later in this document. 6 Sage ERP MAS 500 SDK Data Migrator Guide

13 DATA MIGRATOR ARCHITECTURE The diagram below depicts a data flow of the migration process. User-interface interaction Assisted Company Setup Step related migration data Migrate System, link and step information Extract, insert, and staging table manipulation requests (using user-interface interaction) Migrate Step Step related extract, insert, and parameter data Data M igration System Tables Staging table data Excel Source System Database or Files Sage ERP MAS 500 connection, continue flag, number of records processed, success, failure Insertion Report Log Step entity related data (all records that can be migrated) Extraction Source link, Sage ERP MAS 500 connection, success, failure Data to Migrate Step form, m odifications returned to staging table Staging table data meeting step s Sage ERP MAS 500 business rules Source system data mapped to staging table layout Staging Tables Sage ERP MAS 500 Database Figure 3 - Data Migrator Data Flow Diagram Sage ERP MAS 500 SDK Data Migrator Guide 7

14 CHAPTER 2 Processes and Communication Details Migration Links This section describes the primary migration processes, the metadata that drives them, and the communication between the processes. Migration links are specific to a particular source system and version, for example, Sage ERP MAS 90 Level They are restricted to a single company, so if you want to migrate two Sage ERP MAS 90 companies, you need to either define two migration links, or migrate the first company's data, modify the migration link to point to the second Sage ERP MAS 90 company, and then migrate the second company's data. Migration links are one of four types: Linked server A linked server is a SQL Server object that allows the execution of commands against OLE DB data sources. These can be used to communicate with Sage ERP MAS 200 SQL databases that reside on another server. Same server database Another database located on the same server as the Sage ERP MAS 500 application databases. ProvideX ODBC driver The ODBC driver for Sage ERP MAS 90 and 200 files. Other connection string supported providers These include ODBC and OLE DB providers. If a migration link's source system is one of the natively supported products, the migration link type is determined automatically. Selecting either Sage ERP MAS 90 or 200 results in a ProvideX ODBC type. Selecting Sage ERP MAS 200 results in either a linked server or a database link, depending on whether the source database is on the same server as Sage ERP MAS 500. Custom migration source links support a connection string provider. If the connection string provider is supported by ADO, the common code can be used to obtain the source data; otherwise, another means needs to be employed to put the data into the staging tables. This does not, however, preclude utilizing the communication between the migration process and your extract logic. Migration link data are stored in tsmmigratelink, which is described more fully in the Schema Browser. Migration links require a Migration Source system (for example, an entry in tsmmigratesource). 8 Sage ERP MAS 500 SDK Data Migrator Guide

15 DATA MIGRATOR ARCHITECTURE Linked Servers Linked servers exist in SQL server at the server level they are not database specific objects; however, Sage ERP MAS 500's migration links are database-specific objects (stored in tsmmigratelink). Because linked servers are not database specific, migration links are not company specific. After defining a migration link in one company, the migration links are available in all Sage ERP MAS 500 companies in that database. In environments with multiple Sage ERP MAS 500 databases on the same server, if you created one database per company, linked servers may exist on the machine that are not known to a particular database (company). To get the migration link definition into additional databases requires redefining it in each database. When you save the migration link in another database on the same server, you will be prompted with a message stating that the migration link already exists (because it is at the server level, not the database level). You must overwrite the migration link to save the definition in the current database. An alternative is to name the migration links differently from one another. Using Migration Links in Assisted Company Setup For an illustration of how a migration link is used, see Figure 4 How Migration Links are Used. In ACS, selecting a product version for which migration links exist, results in the lower drop-down list having the available migration links appear. In this example, selecting Sage ERP MAS 90 ABC points to the network server, Tahoe, where the Sage ERP MAS 200 SQL MAS_ABC company resides. After a migration link is selected, step-sensitive help becomes available and the Migrate button becomes available or unavailable based on the availability of a migration for the current step. Sage ERP MAS 500 SDK Data Migrator Guide 9

16 CHAPTER 2 If your source system is not one of the supported source systems, you will find that ACS can be used for moving your data into Sage ERP MAS 500. By selecting source system Other and version Any, you can use all of the migration steps; however, there will not be an available extract. You can, however, access the staging tables and use Excel to obtain your source data manually. The data you put into the staging tables using Excel will be available to the insert step. Figure 4 How Migration Links are Used For a description of how to define migration links for custom extractions, see Chapter 3 - Data Migrator Customization. 10 Sage ERP MAS 500 SDK Data Migrator Guide

17 DATA MIGRATOR ARCHITECTURE Role of Assisted Company Setup in Migration Assisted Company Setup (ACS) supports migrations from predefined source systems. These can be out-of-the-box migrations, or custom migrations. The schema related to ACS migration metadata is shown in the figure below. Selecting a migration source on the Migration System tab in the Assisted Company Setup form is allowed if that product is defined as a migration source in tsmmigratesource. The bottom of the figure shown below illustrates the relationship between the tab and the database table. Selecting a valid migration system relays to ACS to check for the migration system for any step. Selecting a Source Name migration link relays to ACS where the data is located. This step, source system, and migration link information is used by the Migrate <Step Name> process. A phase, such as Inventory, can have multiple steps, and each could have a migration from several source systems (for example, Sage ERP MAS 90 and 200). tsmsetupphase SetupPhaseKey tsmsetupstep SetupStepKey tsm M igratestep SetupStepKey MigrateSourceSystem MigrateSourceVersionNo tsmmigratesource MigrateSourceSystem MigrateSourceVersionNo Figure 5 - ACS Migration Schema Sage ERP MAS 500 SDK Data Migrator Guide 11

18 CHAPTER 2 Migrate <Step Name> Extraction and Insertion The Migrate <Step Name> process is called by ACS to run a particular step's extractions and insertions. The form's options allow for running either the extract or insert process or both. It is here that the reporting option is also specified. The following information is communicated to this process: Migration link Target or current company in Sage ERP MAS 500 Current step Each step can support multiple, serialized extraction and insertion steps. For example, the extraction might consist of two steps and the insertion might consist of one step. The name, type, and sequence number of each step is stored in the tsmmigratestepextract and tsmmigratestepinsert tables. The type of an extract or insert refers to the software component's type (for example, stored procedure or DLL). This, determines how it is called by ACS. For an illustration of the schema metadata that determines the extraction and insert processes, see Figure 6 - Migrate <Step Name> Metadata. By calling DLL routines in a late bound fashion, they can be named in metadata and unknown until they are called. This is the technique Sage ERP MAS 500 uses to call extraction routines, which are written as Visual Basic COM objects exposing a standardized interface expected by this process. For more details, see Migration Calling Architectural Overview, Extraction Architecture Details, and Insert Architecture Details. In addition to extraction and insertion metadata, there is also metadata for migration parameters. 12 Sage ERP MAS 500 SDK Data Migrator Guide

19 DATA MIGRATOR ARCHITECTURE Migration Parameters Migration parameters help fix extracted data that fails to meet the Sage ERP MAS 500 business rules. For example, the source system may not require the existence of records in foreign key relationships, such as ZIP Codes that do not exist in the ZIP Code table. If the field is not required in Sage ERP MAS 500, you can be prompted to fail the records that have bad references or blank out the bad references. Another commonly needed parameter is for missing general ledger accounts. If Sage ERP MAS 500 requires an account that is not a supported concept in the source system, then a parameter allows you to choose to set the missing general ledger account to the suspense account. Alternatively, the parameter could allow selection of an existing Sage ERP MAS 500 general ledger account. For more information about parameter definitions and usage, see Migration Parameter Behavior and Metadata. tsmsetupstep SetupStepKey tsmmigratestep SetupStepKey MigrateSourceSystem MigrateSourceVersionNo tsmmigratestepextract SetupStepKey ProcessNo tsmmigratestepinsert SetupStepKey ProcessNo tsmmigratestepparam SetupStepKey ParamNo tsmmigratestepparamuse SetupStepKey ParamNo MigrateSourceSystem Migrate SourceVersionNo Figure 6 - Migrate <Step Name> Metadata After insertion is run, if any reporting was requested, the Migration report appears. The report can be used to refer to the staging tables as an aid to correct failed records. Sage ERP MAS 500 SDK Data Migrator Guide 13

20 CHAPTER 2 NOTE For large data sets, bringing the data to your PC and moving it between Excel and the Staging Table grid can take a significant amount of time. Excel allows only rows in a spreadsheet. Staging Table Data The extraction routines place data into staging tables. Before the insert process is run, all of the extracted data is available for review. Running insertion removes successful records from the staging tables, leaving only those records that failed the insert step's business rules. Use the tab in the Migrate <Step Name> form to access the step's data. The figure below shows a sample of the tab. When you select a table from the list, its data is brought to your machine and the total number of rows in the table displays. You can add or delete rows, clear all of the staging tables' data, edit value directly in the grid or edit the data in Excel. Returning modified data from Excel immediately puts the changes into the staging table in the Sage ERP MAS 500 database. tsmsetupstepstage tsmsetupstep StageTableName SetupStepKey Figure 7 - Staging Table Data 14 Sage ERP MAS 500 SDK Data Migrator Guide

21 DATA MIGRATOR ARCHITECTURE Migration Progress Form The Migration Progress form (illustrated in the figure below) provides feedback while extracts and inserts occur. The Migration Progress form also indicates the number of records to process, the number processed, and the number of failed records. On extraction, if there are any data issues, a report is provided to describe the issue. For example, Sage ERP MAS 90 commissions have the concept of a base commission, while Sage ERP MAS 500 does not; therefore, Sage ERP MAS 90 records with nonzero base commissions are reported, informing you that their base commissions will not be brought into Sage ERP MAS 500. During insertion, which is an asynchronous process, the Done button is replaced with the Stop Processing button, allowing you to stop insertion. Extraction is not asynchronous; therefore, Not Responding may appear in the title bar of the Migration Progress form during extraction. Figure 8 - Migration Progress Form NOTE Information is not recorded about all records, but only for records that were requested. This helps increase the insertion performance by avoiding writing data that will not be used. Migration Error Log When the Insert process is running for a migration step, it logs successes and failures as specified in the reporting option and warning flag. If, for example, you request a report of all failed records, the insert step records information about failed records. The report appears after the insertion is finished and contains the following information: Step Name This is located in the report header. Entity ID Depending on the migration step, this indicates the natural key of the entity being reported on. Several different entities may be involved in a migration; therefore, an understanding of them is required to be able to determine the entity ID. For example, during the Sales Tax Code migration, the entity ID results may be similar to the following: Entity ID CA CA OC The first line's entity ID, CA, refers to the sales tax code staging table's natural key (STaxCodeID), while the second line's entity, CA OC, refers to the sales tax class staging table's compound natural key (STaxCodeID, STaxClassID). The pipe character ( ), is used to separate the different columns in the entity ID of compound key entities. Sage ERP MAS 500 SDK Data Migrator Guide 15

22 CHAPTER 2 Status Indication of failure, success or warning for the reported row. Column Name of the staging table column related to the entry. This column and the Value column are only populated if a failure or warning occurred, making this column useful for reviewing a report for failed records. Value Data in the staging table column referred to in the previous column. For example, if this is a duplicate record, then the entity ID displays here, because it is the value that caused the duplications. Empty columns that require data appear blank here. Comment The reason for the failure or warning. Successful records will not populate this column. Staging Table Architecture Details The use of staging tables in migration provides a location where source system data can be placed before it is validated against the rigorous business rules of the Sage ERP MAS 500 application. Sage ERP MAS 500 demands a highly reliable, consistent set of data that adheres to strict referential integrity, data type, and value range rules. It would be extremely challenging to move data out of an existing system and simultaneously require that it conform to the business rules of Sage ERP MAS 500. Additionally, the Sage ERP MAS 500 databases are highly normalized and provide maximum flexibility. This normalization has a tradeoff it is difficult for a nontechnical person to understand. Because of these challenges, Sage provides staging tables as the target for source system data extraction. Staging tables resemble their Sage ERP MAS 500 counterparts, but are different in a few significant ways: NOTE The third level was removed from the normalization; the customer address and common address related fields were pulled out to keep the abstraction transparent to any populating of the staging tables. This avoids confusion and a need to fully understand our customer address schema. Surrogate key values do not exist in the staging tables. These are created during insertion. Fields that are numerically enumerated values in the Sage ERP MAS 500 permanent tables have been changed to varchar fields (the SQL Server variable character length field type) in the staging tables. For example, if the field in Sage ERP MAS 500 supports values (0, 1, and 2), which represent (Active, Inactive, Deleted), then the staging table equivalent would be a varchar(8) long enough to support the longest string value equivalent of the enumerated value 1, Inactive. By doing this, the entry of any of the six acceptable values is allowed (0, 1, 2, Active, Inactive, and Deleted). It is expected that extractions will populate these fields with the string value equivalents (for example, Active). The values are converted to the corresponding numeric value on insertion. Where it improves understandability, otherwise normalized data is flattened into the parent table. For example, the Sage ERP MAS 500 customer table references a customer address table, which references our common address table. This three-level normalization allows all addresses to be kept in a single table for referral from different entities, adding addition entity-only related attributes to the joining table (in this case, the customer address table). For example, customer addresses contain a reference to a sales person, which is not applicable to vendor address; they more likely relate to buyers. If, in this case, a source system only supports one customer address, there is no need to populate two staging tables. Even in this case, full functionality has not been removed a customer address table is provided for additional addresses related to a customer. Staging tables do not enforce referential integrity, uniqueness, or any other business rule related consistency. Only the natural SQL Server data type restrictions are enforced. 16 Sage ERP MAS 500 SDK Data Migrator Guide

23 DATA MIGRATOR ARCHITECTURE All Sage ERP MAS 500 staging tables contain three columns used by the migration processes. Any new staging tables you define must contain these columns to be processed in our migration environment: NOTE Successful rows are deleted from the staging table as they are processed. ProcessStatus Indicates the insertion status of the staging table record. 0 Not processed 2 Failed the insertion process. If a parent record's validation fails, often it will have ProcessStatus 2, while its child records have ProcessStatus 0. Child records are not considered because the parent record is known to be bad. RowKey A unique number identifying this row. This is defined as an identity column so that its value cannot be changed. Example: A staging table has three columns. There are two records in the staging table and the permanent table does not allow duplicates: NOTE It is not required that a row exist in tsmmigratesession to utilize the Sage staging tables, but the SessionKey in the staging table must match that passed into the insertion routine to have the correct data considered for insertion. ID Description Effective Date LowPrice Low priced items 01/01/2002 LowPrice Low priced items 01/01/2002 Technically, the first row is good and the second is a duplicate. When the first row is processed, if deletion is attempted based on its ID, both rows are deleted and duplication is not noticed; therefore, a column that is guaranteed to be unique is needed. SessionKey Migration supports multiple users running the same migration step at the same time. To enable migration to differentiate between two users' data in the staging tables, the concept of the SessionKey is available. Table tsmmigratesession contains the information for a particular session. Migration Calling Architectural Overview This section describes the migration interface that is provided through the Assisted Company Setup (ACS) migration processes. Details of the extraction and insertion functionality and related interfaces are provided in the following sections. Migration Interface: dmmigrateintfc dmmigratintfc.dll contains the migration interfaces needed for extraction. Its inclusion in DLL type extractions and insertions is required. The interfaces it contains are: IExtract The extraction COM interface. For more information about this interface, see Extraction Architecture Details. IInsert The insertion COM interface. For more information about this interface, see Insert Architecture Details. IOwnerMigrate COM Interface supported by extraction callers, for example ACS, to enable communication of progress. dmmigratintfc.dll includes several enumerated types for various parameters. Sage ERP MAS 500 SDK Data Migrator Guide 17

24 CHAPTER 2 Migration Caller Interface: IOwner NOTE Although this interface is required, at this time, only OnProgress is supported (although Sage migrations do not utilize it). OnProgress causes information to be written into the detailed progress grid on the Migration Progress form. A migration caller supports the following interface, allowing called extractions or insertions to respond back through the available IOwner methods. Public Sub OnError (ByVal EventSource As String, ByVal ErrorCode As Long, ByVal Source As String, ByVal Description As String, ByVal HelpFile As String, ByVal HelpContext As Long, ByVal IDofInterfaceWithError As String, pbcancel As Boolean) Public Sub OnFinish (ByVal EventSource As String) Public Sub OnProgress (ByVal EventSource As String, String, ByVal PercentComplete As Long, ByVal ProgressCountHigh As Long) ByVal ProgressDescription As ByVal ProgressCountLow As Long, Public Sub OnQueryCancel (ByVal EventSource As String, pbcancel As Boolean) Public Sub OnStart (ByVal EventSource As String) Migration Interface Enumerated Types ExtractCallReturnValsEnum Extract DLL or stored procedure return values. Constant Value Description UnknownFailure 0 Error trapped but not enumerated. Success 1 Procedure was successful. This does not mean that all data is correct, only that the process was successful in doing its job of attempting to extract data. Extracting zero records can still result in success. GeneralFailure 2 Nonspecific, fatal error. BadParameters 3 Some parameter is in error, such as required but not supplied. VersionError 4 Unsupported product version. ProductError 5 Unsupported product. TempTableError 6 An issue occurred with a temporary table. For example, it could not be created, or was expected to exist and does not. ConnectionError 7 Unable to connect to the Sage ERP MAS 500 database or the source database given the connection and/or connection information. SourceTableNotFound 8 A source system table, expected to exist, does not. SuccessWithInfo 9 Extraction was successful, but data issues were discovered and recorded in the migration error log. 18 Sage ERP MAS 500 SDK Data Migrator Guide

25 DATA MIGRATOR ARCHITECTURE ExtractSourceType Indicates what type of source system connection is expected, allowing access to heterogeneous data sources. Certain extraction parameters are interpreted according to this value (for more information, see Start Function). UseDatabase, UseLinkServer, and UseODBC_ProvideX are all supported by the Sage ERP MAS 500 extractions. Constant Value Description UseNeither 0 No connection information supplied. UseLinkServer 1 The connection is expected to be through a linked server, defined on the Sage ERP MAS 500 server (can point to any product supported by SQL Server linked servers and supporting SQL syntax). UseDatabase 2 The connection is expected to be to a SQL Server database on the same server as the Sage ERP MAS 500 application database. UseODBC_ProvideX 3 Sage ERP MAS 90 ProvideX ODBC driver connect type. UseConnectString 4 A free form string used to connect to any provider. This is stored encrypted, so passwords can be safely included. InsertCallReturnValsEnum Insertion DLL or stored procedure return values. Constant Value Description UnknownFailure 0 Error trapped but not enumerated. GeneralFailure 2 Nonspecific, fatal error. Success 1 Procedure was successful. This does not mean that all data is correct, only that the process was successful in doing its job of attempting to insert data. BadParameters 3 Some parameter is in error, such as required but not supplied. InsertDuplicateActionEnum Determines what to do when a duplicate is encountered. Constant Value Description DoNotInsert 0 Fail records that are duplicates. ReplaceExisting 1 Replace duplicate record with this record. Sage ERP MAS 500 SDK Data Migrator Guide 19

26 CHAPTER 2 InsertInvalidGLAcctActionEnum Determines how the occurrence of invalid general ledger accounts should be handled during insertion. Constant Value Description DoNotInsert 0 If the general ledger account field violates a business rule, do not insert this record. This includes bad general ledger accounts or fields with missing general ledger accounts when required. UseSuspenseAcct 1 Use the suspense account in place of the bad value. InsertInvalidReferenceActionEnum Indicates how invalid references in staging table data should be handled upon insertion. Constant Value Description DoNotInsert 0 Fail records that have invalid foreign key references. UseBlankReference 1 Replace invalid foreign key references with a blank reference. This is only done if the foreign key reference is not required. InsertReportOptionEnum Determines what records to log during the insertion process for a migration step. Constant Value Description All 0 Log both records that succeeded and records that failed insertion. None 1 No migration logging on insertion. Good 2 Only log successful records. Bad 3 Only log failed records. InsertWarningActionEnum Determines what records to log during the insertion process for a migration step. Constant Value Description ExcludeWarnings 0 Exclude warnings from the migration log. IncludeWarnings 1 Include warnings on the migration log. This is only applicable if InsertReportOptionEnum is not equal to None. 20 Sage ERP MAS 500 SDK Data Migrator Guide

27 DATA MIGRATOR ARCHITECTURE Extraction Architecture Details Extraction routines move source system data to the Sage ERP MAS 500 staging tables. There are two primary technical challenges that shaped the architecture developed for migration extracts. First, because the source data can be in different databases, including file-based structures, extraction requires data retrieval from various data sources. Second, without being able to bookmark the current spot in the extraction data, the technique of getting sets of data (of a predefined size) in a looping construct cannot be used. That is, you cannot get the first 100 records, and then get the next 100 until you are done. The implications of each of these issues are discussed below. The discussion of extraction architecture below is also heavily influenced by the supported configurations (for information on supported configurations, see Figure 2 - Supported Configurations). Heterogeneous Data Sources Issue At first glance, having source system data in different data structures seems like a simple problem to solve because OLEDB and ODBC drivers exist for many data sources. These drivers are not available to stored procedures; however, Data Migrator must support both stored procedure and DLL extracts. To support communication to a remote server through stored procedures, infrastructure was created to use linked servers; however, SQL Server's 6.5 compatibility mode does not support linked server query syntax. To resolve this issue, the following is performed in the Sage extract processes when using linked servers: Move to the MASTER database, which supports the OPENQUERY SQL syntax for linked servers because it is in 8.0 compatibility mode. Create a temporary table copy of the source system tables using the link source SQL statement. By putting the data into a temporary table, the location is always known and a database name does not need to be specified to reference it. Return to the Sage ERP MAS 500 application database. These are important considerations when building your own extracts. Regardless of the link type (Linked Server, database located on the Sage ERP MAS 500 server, ProvideX ODBC, or Connection String), Sage s extractions always place the source data into temporary tables and refer to these when populating the staging tables. Sage ERP MAS 500 SDK Data Migrator Guide 21

28 CHAPTER 2 Getting Extraction Data in Sets Long processes should provide progress feedback to the user. This includes extracting what may be many thousands of records from a source system. Because there is no way of tagging individual source system records, you cannot stop extracting at some record, mark the current position in the data, inform the user of the progress, and then continue picking up where you left off. (Certainly, there are ways to continue, including knowing the source entity's unique key and using some looping structure.) To help address this issue, Sage decided that by providing a COM-based extraction interface, we can inform the user when a lengthy process is about to start and also inform the user of milestones within the extraction. Alternatively, we can create an asynchronous process, but some data connectivity providers are not asynchronous. As a result, Not Responding may appear in the title bar of the active migration window. To make extraction as quick as possible, we decided that all extraction processes will pull all rows at once, in a set-oriented manner. Sage s extractions must support the extraction COM interface expected by ACS. ACS also supports an extract interface for stored procedures; choose either mechanism for your custom extractions. (Technically, the interface is expected by the Migrate <Step Name> process, which is a DLL called by ACS to perform the extractions and insertions.) There are two interface types that must be supported in an extraction environment: the extraction process must support the extraction interface, allowing the caller to start the extraction routine; and the caller must support the migration communication interface allowing the extraction to communicate status back to the caller. Extraction COM Interface: IExtract To support ACS's migration extraction call, an extraction DLL must support the dmmigrationintfc.iextract interface discussed below, while an extraction stored procedure must support the interface discussed in the Register section. Sage ERP MAS 500 extract DLLs written in Visual Basic expose the interface using Visual Basic's Implements statement. Unfortunately, Implements does not support events; therefore, to mimic the ability to receive events, the caller (ACS) passes a handle (to itself) to the extract class as an IOwnerMigrate interface. This interface supplied by ACS allows the extraction object to communicate status back to ACS. Sage ERP MAS 500 has a COM object that exposes the extraction interface expected by ACS. The extraction interface has three public routines, listed below. Public Function Start (oconn As ADODB.Connection, ByRef bcancel As Boolean, ByVal lsessionkey As Long, ByVal scompanyid As String, ByVal esourcetype As ExtractSourceType, ByVal sappsourcename As String, ByVal ssyssourcename As String, ByVal sappdbname As String, ByVal imigratesourcesystem As Integer, ByVal sversionno As String) As ExtractCallReturnValsEnum Public Sub Register(ByVal Callback As IOwnerMigrate) Public Sub UnRegister() 22 Sage ERP MAS 500 SDK Data Migrator Guide

29 DATA MIGRATOR ARCHITECTURE Start Function The Start function is called to perform the extraction. For information about enumerated values, see Migration Interface Enumerated Types. Parameter oconn bcancel Description A connection to the Sage ERP MAS 500 application database with the current user's connection information. Note: This connection has the following properties: SET ANSI_NULLS ON SET ANSI_WARNINGS ON Returned to the caller if this process was cancelled. Currently not used. lsessionkey The unique key for this user's session, used to identify their data in the staging tables. scompanyid Company ID of the destination company in Sage ERP MAS 500. esourcetype Determines the type of connection to make to the source system. See Migration Interface Enumerated Types. sappsource ssyssourcename sappdbname imigratesourcesystem sversionno If esourcetype = UseDatabase or UseLinkServer, then this is the application database for the source system. If esourcetype = UseConnectString or UseODBC_ProvideX, then this is the decrypted connection string. If esourcetype = UseDatabase or UseLinkServer, then this is the system database for the source system (which may not be applicable). Unused otherwise. Name of the Sage ERP MAS 500 application database. Valid entries from tsmmigratesource.migratesourcesystem: 0 Other 1 Platinum for Windows 2 Sage ERP MAS 200 SQL 3 Sage ERP MAS 200 client/server 4 Sage ERP MAS 90 Note: Only values 2, 3, and 4 are supported by our extractions. Valid entries from tsmmigratesource.migrateversionno. Sage s extractions support the following levels: 3.6, 3.61, and 3.7 in conjunction with the supported migration source systems supported (shown above for tsmmigratesource.migratesourcesystem) that is: Sage ERP MAS 200 client/server Level 3.6 Sage ERP MAS 200 client/server Level 3.61 Sage ERP MAS 200 client/server Level 3.7 Sage ERP MAS 200 client/server Level 3.71 Sage ERP MAS 200 SQL Level 3.6 Sage ERP MAS 200 SQL Level 3.61 Sage ERP MAS 200 SQL Level 3.7 Sage ERP MAS 200 SQL Level 3.71 Sage ERP MAS 90 Level 3.6 Sage ERP MAS 90 Level 3.61 Sage ERP MAS 90 Level 3.7 Sage ERP MAS 90 Level 3.71 Sage ERP MAS 500 SDK Data Migrator Guide 23

30 CHAPTER 2 Register Register is used by the caller to pass a reference to itself to the extraction object, allowing it to call the IOwner methods mentioned in Migration Caller Interface: IOwner. UnRegister Releases references to caller. Extraction Stored Procedure Interface The following section describes the interface for extraction stored procedures. For information about enumerated values, see Migration Interface Enumerated Types. CREATE PROCEDURE <ProductName><EntityDescription>Ext (@_isessionkey INTEGER OUTPUT) Parameter The unique key for this user's session, used to identify their data in the staging Company ID of the destination company in Sage ERP MAS Indicates the product to be extracted from. The expected values are listed in Migration Interface The version number of the migration source system. This is a string because it can have letters in it, for example 5.0e. Integer indicating the extraction return state. Expected values are listed in the enumeration section under Extraction Return Values. For template examples of using the extraction interface, see Chapter 3 - Data Migrator Customization. 24 Sage ERP MAS 500 SDK Data Migrator Guide

31 DATA MIGRATOR ARCHITECTURE Insert Architecture Details This section discusses the interfaces and coding style of the migration insertion routines. It also discusses why this architecture was chosen. If you are only interested in the insertion interface, see Insert Stored Procedures Interface without missing prerequisites. Background Information and Overview Insertion routines exist to move data from staging or temporary tables into Sage ERP MAS 500 after validating the records against the insert step's business rules. There are two primary technical issues to recognize when considering the architecture developed for migration inserts. First, because it is likely that large sets of data will be processed, do not bring the data to the client machine. Second, because the process may be lengthy and you may want to cancel it (if, for example, the failed record count becomes too large), the architecture needs to support processing data in chunks. Large Data Set Issue As mentioned in the extraction architecture section, long processes should provide progress feedback to the user. Additionally, because the insertion procedures will be taking large sets of data and moving them into Sage ERP MAS 500, the data should not travel more than is necessary. Because the extraction process described above has already put the data into staging tables located in the Sage ERP MAS 500 application database, Sage strongly suggests the use of stored procedures to validate and insert the data into Sage ERP MAS 500, which will avoid bringing the data to the client machine. Data Migrator uses stored procedures for insertion. Insertion Processing Time Inserting data is more involved than extracting data because each record must be validated before performing the actual insertion. Validations can be quite extensive. Due to the relatively long, per-record processing time for an insertion (when compared to the extraction process' per-record processing time), it is preferable to provide useful feedback to the user. Because you may want to cancel the insertion, an asynchronous behavior is necessary. Data Migrator launches the insertion stored procedures in an asynchronous manner using ADO. This technique allows the Migration Progress form to receive your Cancel button click action and interrupt the insertion. To keep you informed of the insertion progress, the insertion stored procedure interface includes a continue flag, which lets the stored procedure return to the caller, but tell it (for example the Migration Progress form) that there are more rows to insert. The interface also includes a cancel flag, which lets the caller tell the stored procedure that the insertion should be cancelled. To provide feedback, a call to an insertion stored procedure will, in general, return some type of status information approximately every 10 seconds. When you examine the example template, you will notice that it compares the stored procedure start time with the current time and works on a subset of rows from the staging tables hopefully a set that it can process within the 10 second limit. The insertion should stop if 10 seconds has transpired, no more rows are available, or a critical error happened. Sage ERP MAS 500 SDK Data Migrator Guide 25