Minimize Project Risk with an Effective, Actionable, and Repeatable Data Migration Strategy Yosh Eisbart NIMBL 2012 Wellesley Information Services. All rights reserved.
In This Session Understand the key components of a data migration strategy and when to begin defining your strategy Hear about the mistakes some customers have made and how you can avoid them Learn about some of the tools available to you that can be used during data migration some of which may be included in your SAP license! See example templates, project plans, and job-aids your team can use in support of your data migration strategy Discover what s available from SAP as part of SAP Best Practices for Data Migration 1
What We ll Cover The composition of a data migration strategy Common data migration deliverables during a project Kick start your strategy with examples and templates Discuss some of the tools available to support data migration Wrap-up 2
Some Quick Quotes Data Migration is never as easy as it sounds! Excerpts from analyst reports on data migrations during ERP implementation projects More than 80% of data migration projects run over time and/or over budget. Cost overruns average 30%. Time overruns average 41%. Philip Howard, SAP Data Migration Using Business Objects Information Management Software (Bloor Research, July 2007). 80% of organizations will underestimate the costs related to the data acquisition tasks by an average of 50% T. Friedman, Best Practices Mitigate Data Migration Risks and Challenges, (Gartner Research, May 2009). Data migration is an opportunity to support data quality By 2005, Fortune 1000 enterprises will lose more money in operational inefficiency due to data quality issues than they will spend on data warehouse and customer relationship management initiatives T. Friedman, Gartner Research, April 2004 3
Definition of Terms What is a data migration? The translation of data from one system to another, whether manual or automated, so that it can be used in another application or system And a data conversion strategy? A group of decisions centered on the data migration goals that determine the process, tools to implement, and resources to be applied What about a data quality plan? Define the metrics that track and monitor data quality during the data migration process 4
Data Migration Strategy Components Context The type of data being migrated and its usage (ERP, BW, CRM) Storage Where data resides today and tomorrow Data flow Tools How data enters and moves through the organization Software and/or manual processes used to move data Ownership Person(s) responsible for managing data Continuous monitoring Processes for ensuring data quality 5
Context in Detail The context of your data determines the types of tools required to perform the data migration Customer data: Names, addresses, phones numbers, etc. Financial data: Dates, balances, accounts, etc. Supply chain data: Materials, BOMs, suppliers, etc. For example, we need to understand the data being moved to build the correct algorithm and business rules during data migration Convert St. to Street and VP to Vice President Recognize the same customer with multiple ship-to addresses Understanding the context will enable you to leverage a data migration as a data quality exercise, as well 6
Storage Every data migration strategy must consider where the data currently resides System architecture can sometimes determine the method or frequency of data extraction, transformation, etc. Centralized or distributed, homogenous or heterogeneous Software vendor and hardware platform can also impact connectivity options Is there a relational database driver available? Does data reside only within flat files? Is the system in a highly secured environment? Determine what type of storage will be used as target Database, data warehouse, content server, etc. 7
Data Flow With the type of data and a location defined, next we focus on the data flow through an organization Hitting a moving target data can move in (transactional data) and out (archival processes) of systems during a project Must determine the best location to perform extraction to meet the desired data migration goals In essence, you are documenting the information supply chain Understanding the data flow answers the following questions: What are my opportunities to extract the best data possible? Does it make sense to extract data before other transformational activity occurs? 8
Example Data Flow CRM Source: SAP Understanding the lifecycle of data within our applications can better prepare us to build our data migration strategy Mapping the data flow also provides an opportunity to decide when and where to perform data cleansing tasks before migration 9
Tools Must define threshold between manual and automated data migration Manual data load is the physical entry of data using normal SAP transactions or reports Automated data loads would be considered Batch Data Communications (BDCs), a Legacy System Migration Workbench (LSMW), IDocs, or other data load tools The context and storage of data will drive tool selection Complexity of business rules and data cleansing requirements will also significantly impact the tools leveraged during a data migration We ll cover some of the tools available to you later in this session 10
Ownership Ownership resides with the person(s) who manages information and activities that encompass data creation, capture, maintenance, decisions, reporting, distribution, and deletion Not an easily completed task; assigning data stewards Sometimes the question of ownership can be answered by who has the authority to approve the creation and deletion of data Once stakeholders of the source data have been defined, they must be included in the plans for extraction, transformation, and load into the new system Not establishing ownership over data can lead to: Issues with data integrity Impacts to user adoption The data in the system isn t correct; I m not using it 11
Continuous Monitoring Take a lesson from the principals of Total Quality Management (TQM) Continuous quality of data post-migration is measuring, analyzing, and then improving data accuracy Several factors contribute to answering the question, How often should I profile my data after data migration (cleansing)? How often is the data used? Is the data mission-critical, routine operations, end of month? What is the cost of monitoring the data (people + technology)? Is the assessment process entirely manual, somewhat automatic, or completely automatic? 12
What We ll Cover The composition of a data migration strategy Common data migration deliverables during a project Kick start your strategy with examples and templates Discuss some of the tools available to support data migration Wrap-up 13
Data Migration and ASAP Defining your data migration strategy should start early in the project, during the Project Preparation phase of SAP s ASAP methodology Per ASAP, the data migration strategy not only covers the migration of legacy system data, but should also include a plan for archiving data in SAP In practice, very few companies think about archiving during initial implementation The ASAP roadmap from SAP provides guidance on when and how to collect data conversion requirements Download the latest roadmap: (http://service.sap.com/ roadmaps *) Online version (www.sdn.sap.com/irj/scn/asap-methodology) * Requires login credentials to the SAP Service Marketplace 14
Project Preparation Conduct a data migration workshop Typically a two-day workshop Explain data migration capabilities and tools from SAP Understand source systems, priorities, and technical infrastructure Foundation for data migration strategy Perform data profile (audit) Identify source systems and profile quality of data Analyze data to detect anomalies and understand gaps between legacy systems and SAP SAP provides templates as part of ASAP roadmap 15
Blueprint Perform a data quality assessment Understand the availability and quality of existing data Define the complexities and risks associated with migration What kind of downtime will be required? Will we need to run in dual maintenance mode? More extensive deep dive than data profiling exercise referenced on previous slide Construct a data migration design document Which interfaces will be manual vs. automated? Include detailed object/data element mapping, business rules, input and output file formats, cleansing logic, etc. More importantly, should contain error handling and escalation process 16
Realization Execute legacy data migration Iterative development and test cycles Focused on analysis of data, refinement of business rules, deployment of programs Test cycles enable the migration team to: Improve data quality to an acceptable production level Develop a detailed cutover sequencing plan Exercise data reconciliation and validation processes required to support the production cutover Provide final data quality report Summary of data quality metrics for all the business objects Any outstanding data quality issues that need to be resolved 17
Final Preparation Provide data migration sequencing to cut-over planning manager Include automated and manual data conversions Identify items that can be run in parallel Define key checkpoints for stakeholder verification Support go-live simulations 1 through NN Establish process with Basis team for client backup and deletion Provide a consistent pre-production environment where team members can execute the data migration plan Capture start and stop times to understand duration and total data load times Practice error handling and notification processes 18
Go-Live and Support Execute the data migration cut-over checklist Define location for central tracking of all data conversions Example spreadsheet on next slide Location should be accessible by all team members Ideally stored in SAP Solution Manager Provide regular status updates Document errors and/or exceptions for future resolution Defined issues management process? Ideally, you are entering data conversion issues in your incident management tool Ensure functional validation and reconciliation is completed 19
Example Data Migration Cut-Over Checklist Examples are provided as part of your session materials Also included in ASAP roadmap from SAP 20
What We ll Cover The composition of a data migration strategy Common data migration deliverables during a project Kick start your strategy with examples and templates Discuss some of the tools available to support data migration Wrap-up 21
Why Leverage ASAP? Phased, deliverable-oriented methodology that ideally streamlines implementation projects Minimizes risk because it s based on experiences of other customers and SAP Lessons learned = reduced total cost of implementation How is this attained? The methodology supports project teams with templates, tools, questionnaires, and checklists, including guidebooks and accelerators Includes templates for all team members, including PMO Best utilized when accessed via SAP Solution Manager Offline versions also available 22
What s Available for Data Migrations? Data Migration Workshop Findings Document Data Audit Report Findings Data Migration Scoping Questionnaire Data Migration Scope Assessment Gap Analysis Template Data Migration Strategy Document Test Data Management Document Data Migration Project Plan (.MPP) Data Definition Object Template Functional Data Conv. Template Technical Data Conv. Template Data Quality Assessment Document Manual Data Migration Design Document Data Governance Templates Data Security Design Template Data Testing Strategy Template Legacy Data Archive Document Data Retention Questionnaire Data Archiving Plan (.MPP) Archive Documentation Checklist Data Migration Cut-Over Checklist Data Migration Reconciliation Guidebook Migration Quality Tracking Sheet 23
Exploring ASAP Roadmap 24
What We ll Cover The composition of a data migration strategy Common data migration deliverables during a project Kick start your strategy with examples and templates Discuss some of the tools available to support data migration Wrap-up 25
Data Migration Is LSM Workbench Legacy System Migration Workbench (LSMW) Transaction Code LSMW SAP application area BC-SRV-DX-LSM Good to know for reporting issues with SAP http://scn.sap.com/docs/doc-26158 SAP Education: Customer Training Data Transfer Course BC420 LSMW is part of the WebAS with SAP Basis Release 6.20 and higher This includes SAP R/3, SAP ERP 5.0, or SAP ERP 6.0+ 26
When Is LSMW Best Used? Ideal for initial and periodic data transfer from non-sap systems to an SAP system Integrated Transaction Recorder Consistency check against customizing Support of the below SAP standard import techniques: Batch input programs (including own recordings) Direct input programs BAPIs, IDocs More Importantly, provides reusable mapping 27
How It Works.READ-file LSMW creates several files to be utilized during conversion process Data is stored in Z tables prior to load.conv-file 28
Steps Within LSMW Wizard Activities to be completed for an LSMW to run Maintain object attributes Convert data Maintain source structures Create batch input session Maintain source fields Run batch input session Maintain structure relations Start direct input program Field mapping and conversion rules Start IDoc generation Start IDoc processing Fixed values and translations Create IDoc overview Specify files Start IDoc follow-up Assign files Frame program for periodic Read data data transfer Display read data 29
What Can Be Read by LSMW? Data from spreadsheet-based applications Any data from flat files Text files,.csf files,.csv files, etc. Data from host systems on SAP application servers Fixed record length, end of record indicator, or hexadecimal length field at the beginning With and without separator Different non-unicode code pages, Unicode Hierarchical data in multiple files (relational structures, tables from relational databases) Hierarchical data in one flat file with identifying field values 30
What Data Can Be Migrated? Standard batch input, standard direct input Precondition: Application program given Entries in SXDA* tables Batch input recording Recording a transaction Flat target structure Methods for business objects ( BAPIs ) All BAPIs for which an ALE-IDoc interface has been generated and released (by SAP development) All BAPIs for which an ALE-IDoc interface is generated and released by customer (using transaction BDBG) IDocs All released IDocs 31
Process in Detail Organize the data migration project Project, subproject, object Document the data migration project Define data conversion per object Select target Business object (e.g., customer master, material master) Method (e.g., create, change) Import technique (e.g., batch input, IDoc) Define source Structure of the source data (segments, fields, file attributes) Define relations Between source and target on record or field level 32
Process in Detail (cont.) Define rules Read data (technical conversion) Convert data (conversion of values) Import data: With a standard import technique Batch input Direct input BAPI IDoc technique LSMW does not write data into the database tables of the applications Transport the data migration object 33
What LSMW Will Not Do Extract data from the legacy system It s not meant to replace an ETL or data loading utility Transfer data from SAP into another system Work with.xml files Get involved with the data management SIG with ASUG to voice support for.xml files! 34
What Else Is Available to Me? Multiple third-party vendors with certified software-based tools Winshuttle: Data import and upload utility BackOffice Associates: Boring Go Live data migration Find more on SAP EcoHub http://ecohub.sap.com What else is available from SAP? SAP Best Practices for Data Migration v3.32 Methodology + tools + templates Methodology and templates are free http://service.sap.com/bp-datamigration * Tools require additional license Unless you already own SAP BusinessObjects Data Services * Requires login credentials to the SAP Service Marketplace 35
Components of SAP BP for Data Migration SAP BusinessObjects Data Services SAP Best Practices for Data Migration SAP BusinessObjects Edge BI with Data Management/ Integration Kit Source: SAP 36
Benefits vs. LSMW? Predefined content for SAP ERP and SAP CRM Provides content for everything you would find in SAP Best Practice Package Data extraction capabilities Ability to perform extraction, mapping, cleansing, and load from within a single toolset Too many connectors and capabilities to list Graphical tool for data migration design Built-in data profiling tools Web-based administration Operational dashboards (flash-based) 37
Process Overview from SAP Source: SAP 38
Predefined Objects for SAP ERP Cost Element Characteristic Class Activity Type Activity Price Customer Master Vendor Master Material Master Material Master Classification Material Inspection Type Material Customer Replenishment Purchase Info Record Inspection Method Master Inspection Characteristics Inspection Plan Object Dependency Source List Work Breakdown Structure Functional Location Sales Order Equipment BOM (Bill of Materials) Open Deliveries Routing Org Management SD Pricing Purchase Order Inventory Balances Internal Order Accounts Payable/Receivable HCM Best Practices-Based Exchange Rates Cost Centers Profit Centers Standard Hierarchy Purchasing Requisitions Contracts Scheduling Agreements Service Master Reference Operation Set Order Reservation Planned Independent Requirements 39
Predefined Objects for SAP CRM Business Partner General/Sold-To Business Partner Contact Persons Business Partner Competitors Business Partner Employees Business Partner Relationships Target Groups Product Material Product Service Product Individual Objects Installed Bases Pricing Conditions Sales Orders Service Orders Sales Contracts Service Contracts Leads Opportunities Quotations Complaints Business Partner Hierarchies Plant Prospect Channel Partner Consumer Enterprise Org. Structure BP Product Relationships Material Business Activities Financing Contracts Service Plans Pool Contracts Framework Agreements Product Service Letters Tasks Confirmations Price Agreements 40
Data Migration Process with Content from SAP Data extraction SAP BusinessObjects Data Services tool helps connect to most known legacy systems to extract legacy data (master and transaction) Data mapping Functionality supports all the major master data business objects required to configure SAP Business All-in-One as the target system Includes support for SAP ERP, as well Data validation Contains predefined content for business logic validation and data quality improvement Dashboards identify and visualize data errors early in the process 41
Data Migration Process with Content from SAP (cont.) Data load Contains program to activate the IDocs needed for the data migration process Leverage standard IDoc monitoring tools in addition to SAP BusinessObjects Reconcile data Right data to support the business process 42
Video SAP BusinessObjects Data Services in Action 43
How Do We Purchase BP for Data Migration? Package names Large Enterprise Customers (LE) Data Migration Starter Pack SME Customers SAP BusinessObjects Edge BI with Integration Kit Or SAP BusinessObjects Edge BI with Data Management SAP Best Practices for Data Migration package is available to download or order as a DVD http://service.sap.com/bp-datamigration * * Requires login credentials to the SAP Service Marketplace 44
What We ll Cover The composition of a data migration strategy Common data migration deliverables during a project Kick start your strategy with examples and templates Discuss some of the tools available to support data migration Wrap-up 45
Where to Find More Information SAP Standards for Data Integrity and Transactional Consistency http://service.sap.com/supportstandards * Media Library Data Integrity and Transactional Consistency Online ASAP roadmap www.sdn.sap.com/irj/scn/asap-methodology Data migration template ASAP Methodology for Implementation (link) 1 Project Preparation 1.4 Data Management 1.4.4 Data Migration Approach and Strategy Document LSMW overview http://scn.sap.com/docs/doc-26158 A Simple LSMW Checklist (SAP Community Network, August 2008). www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/10717 SAP Note 1559332 on BP for Data Migration https://service.sap.com/sap/support/notes/1559332 * * Requires login credentials to the SAP Service Marketplace 46
7 Key Points to Take Home Converting data to SAP is an experience every customer has participated in leverage the content created by others Begin defining your data migration strategy during Project Preparation start early Do your best to include data archiving as part of your data conversion strategy Leverage LSWM wherever possible for initial data loads, as it s a tool that s included with your SAP license Step up to the SAP BP for Data Migration if you need help with data extraction and need operational dashboards Explore your data migration options on the SAP EcoHub No need to reinvent the wheel explore SAP-provided content as part of the standard or enterprise maintenance before creating content from scratch 47
Your Turn! How to contact me: Yosh Eisbart YoshEisbart@benimbl.com Twitter: @NIMBLNews Continue the conversation! Post your questions in the Project Management Forum on Insider Learning Network* *bit.ly/projectmanagementforum 48
Disclaimer SAP, R/3, mysap, mysap.com, SAP NetWeaver, Duet, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP. 49