How to Manage the Conversion Legacy Data to Oracle Financials 1 Converting data from a legacy system to Oracle Financials consists of more the Extraction-Translation-Loading of data. While it is true that a developer can begin coding and quickly produce results, there may be significant problems in the data and the storage of that data that are noticed only after mega-rows of data have been converted. To avoid the surprises involved with converting data for go-live, it is wise to consider other factors than just the program itself. One of the most difficult hurdles in preparing the conversion is the involvement of the stakeholders. It is essential to start with, and to have an agreement for, the Statement of Work to be performed. So let s start by looking at the contents of the SOW. 1 Bill Pry (Priority Data Delivery, Inc.) 12/22/2003 1
Statement of Work The work to be done on the contract must be stated to include Overall Scope Business Arrangements Environment Roles and Responsibilities Infrastructure Deliverables Project Plan Data Issues Technical Issues The Conversion The SOW will require 2-3 weeks to prepare, and must be accepted by the Customer before work begins. It would be anticipated that the customer would pay for this preparation, with the acceptance of the plan as the deliverable. The proposal must be accepted prior to continuing the work. The overall scope is a list of the subject areas to be considered, for example: AR Customers Open invoices PO Vendors Vendor Sites Hierarchy HR Employees INV Items Inventory BOM Bills and Components FA Assets and Books AP Open Payables General Ledger Accounts Detail Balances Summary Balances CRM Installed Base Warranty Service Requests 2
It will be very easy to underestimate the amount of work to be done. Subject matter experts in each area should be consulted. If subject area conversion experts are not available, don t make the mistake of estimating the time necessary before acquiring the conversion documentation for the specific subject area. Add 3 days to the preparation of the SOW for each subject area after reviewing the documentation in those cases where a subject area expert is not available. 3
Business Arrangements An understanding of business arrangements, to anticipate Scope Creep is necessary. There are many issues that will only be found when the data is examined in detail. For that reason, provide in advance for: Provision for unexpected requirements Provision for support after Go Live Hidden requirements such as the need to set up flexfield key values, category values, and category sets must be determined and assigned to the customer. The method of loading these values must be decided (Oracle Forms entry by staff, by Excel VBA macros, or by specialized programming to load the tables directly.) 4
Environment It is best that the conversion team be located near the customer contact staff so that questions may be addressed quickly. Other issues include: Facilities required for conversion team (terminals, communications ports, dial-in facility, telephones, long distance facility, fax, analog phone lines, power outlets) Work area and facilities required for conversion team Network support DBA support 5
Roles and Responsibilities The conversion will not succeed without heavy involvement from the customer. There will be a tendency for the customer staff to throw the data over the wall rather than to take co-ownership of the conversion process. Consider the following areas: Customer infrastructure involvement (Roles such as subject area expert, legacy system, internal systems liaison, customer system dba and coordinator) Application setup support for each subject area Conversion results user testing Executive champion 6
Infrastructure It is necessary that an instance be available for trial loading of data so that the instance can be restored without impact to other users such as development and training. There also needs to be an adequate user and technical library. It is essential that the customer purchase the Oracle Financial Technical Library and Oracle Metalink support. The conversion will fail without it. Requirements for production instance, data instance, test instance Standards for error reporting for data Oracle documentation requirements (Technical manuals, User manuals) Conversion work impact to other development and system availability Code repository such as PVCS Standards (like the Oracle Development standards) Formalized code reviews: preliminary (flowchart), intermediate, and final 7
Deliverables Deliverables include the milestones defined for each subject area, and other documentation: Reporting requirements (weekly status, etc.) Documentation requirements Definition of subset of data for proof of concept testing. Definition of subset of data for user training. Customer acceptance criteria for data Conversion completion criteria and measurement 8
Project Plan The conversion project plan should be documented with MS Project. The following data is required: Timing estimates for data collection, modification, loading Estimates include 1. Data source discovery 2. Data mapping 3. Script development 4. Unit testing 5. Acceptance of data subset 6. Test full load 7. Acceptance of data load 8. Data audit Hours of operation Staffing requirements (skill sets, etc.) Project Plan (MS Project) Staffing requirements including rollon, rolloff schedule Personnel Schedules (days, hours) There is an inherent order for conversions, so that each subject area builds on the one before it. GL must be first. Naturally BOM requires Item definition. PO Hierarchy requires employees. Be sure to consider that order when creating the project plan. The appendix gives a list of questions that prompt for completeness of the processes for each subject area. These questions should be placed in a spreadsheet with columns for each subject area to specify the status and comments about each question. This is an excellent tool for review during weekly status meetings. Each subject area should provide specific plans for failure recovery. A sample spreadsheet is given in the appendix. 9
Data Issues The customer must be dedicated to the task of identifying data, reducing redundant data, correcting data and creating data. This one issue is 80% of the work. Again: the work to interface good data will take 20% of the time; 80% of the effort is involved with cleansing the data. Source for data Data mapping, legacy to Oracle Volume of data in each subject area Estimates for storage of data in staging area and in Oracle Financials tables Subject area specifics (How many kinds of price lists, how are organization items to be populated, etc.). Write a short description of each area. A sample form is given in the appendix. Data validation procedures The source for data and customer expert for that data must be stated. The data fields required for each import may be stated after the SOW is approved, but the date for data availability must appear on the Project Plan. 10
Technical Issues The customer may have a preference for tools used for data transformation, translation, and loading. Specification of data mapping, data cleansing, data verification tools Technical considerations (Tools to be used for data manipulation) Data loading procedures (Special sql processing to load import tables, Data entered via Oracle Forms, Data entered via Excel VBA) The conversion programs should include the following details: Metrics Number of records available Number of records converted Display periodic statistic of number of records/unit time in log Display periodic estimate of time remaining Logging Write to /usr/tmp and to utl_file.put_line(utl_file.log, xxx ) Provide internal procedure to write to log Log transactions to table for type, identifier, message, timestamp Provide external procedure for transaction log Include program name so that multiple programs may use the same log table Input Stage input data to a staging table Provide column for processed Y/N and for timestamp Processing Provide for periodic commit Examine explain to determine where indexes are needed, especially in large input files Provide defaults for missing relationships such as customer, item or contact 11
The Conversion The appendix gives a sample spreadsheet for the dependencies in the conversion. This is the detailed plan for the conversion execution roadmap. If this spreadsheet is placed in a common area, all who are involved with the conversion can track the progress of the conversion. By using this tool from the first trial conversion until the final production conversions, all the kinks can be ironed out and the final conversion will progress as planned. 12
Appendix Oracle Financials Overview Template Information Request Form Weekly Status Report Form Tasks to be Complete for Each Subject Area Failure Recovery Execution Sequence Error Explanation Spreadsheet 13
Oracle Financials Conversion Interface Table Definition Work Table Definition Data Source Work Table Load Tool Interface Table Load Tool Interface Table Load Status Oracle Financial Table Load Status client Functional Contact Client Technical Contact Functional Contact GL Considerations Timing Considerations Conversion Steps Comments: 14
Request for Information Requested by: Date requested: Date received: Request given to: Date required: Date provided: Conversion Project: (check applicable areas) A/R Cust A/R Open Inv A/P Open INV Master BOM O/E Price O/E Open GL Accts GL Bal Assets Fixed HR Empl Information requested: 15
Conversion Project Weekly Status Report MM/DD/YY Planning Who Topic Note Hours Research Who Topic Note Hours Documentation Who Topic Note Hours Overhead Who Topic Note Hours Accomplishments Accomplishment Plans Plan 16
Tasks for each subject area Have you completed the program flowchart? Intermediate code review template completed? Final code reviewed completed? Final code review template completed? Any open or new design issues? Any change requests? Is your staging load code ready? Is your staging load code reviewed? Tested the staging load scripts? Have you written failure recovery procedure steps for the staging load? Have you completed writing your staging load count scripts? Have you tested your staging load script? What privileges do you need on APPS? Which schema would you use to run the validation scripts? What privileges would you need on APPS? Have you identified the indexes on the staging tables? Have you created the index creation scripts? Have you ensured nologging and the correct tablespace in the creation of the indexes? Have you estimated the staging table index creation timings? Can you ensure that you're using synonyms to the source tables in the staging load scripts? Have you written 11i validation scripts (count match etc.)? Has your 11i validation script been reviewed? Have you executed 11i validation scripts? Do you need to disable any base 11i indexes (only non unique) for your inserts? What is the max volume you've tested for the staging load? Do you have expected execution timings for the staging load? Do you have the total tablespace sizing information defined (for data and index)? What is your performance test plan? What is the largest volume test you've performed? Have you registered your concurrent programs? Have you completed concurrent request groups? What are your risks in terms of code quality? What are your risks for the execution? What is the level of your confidence in your unit tested code? What other scenarios do you need? What is your plan for processing for error records? Do you see a need for backup during your migration? Can your migration handle backup (or any other downtime) during migration? Is your staging load code ready? Generate trace file for your staging load. Generate trace file for your 11i load. Review / Validation of the trace file. Code changes completed based on trace review? Validated the performance tuned code execution timings? Place your validation scripts in PVCS, update PVCS document Have you reviewed your code for Snapshot too old error (ORA-1555) 17
Failure recovery spreadsheet example ID Conversion Step Expected # of Threads Recovery Steps Maximum Expected Recovery Time 1 Create stage table 1 Rerun script 1 min 2 Create stage table 1 Rerun script 1 min id index 3 Load stage table 1 Rerun concurrent program; duplicate keys will be ignored; total stage time est is 24 hours. However, because the time is determined by the size of the join that occurs after the cursor starts, the total time for a complete run is 24 hours 24 hours 18
Execution sequence spreadsheet % Com plete Complet ion Date Owner Step ID Exec Seque nce Prerequisite (Col E Step ID) Conversion Item Migration Step Method 1 1 Setup 3 1 Setup Create database links; verify db links in APPS Create User ID lookups package 85 6 Setup - SA Pull SA ids into temp SQL*Plus 86 7 85 Setup - IB Pull IB ids into temp SQL*Plus 10 1 HR Staging 20 3.1 10 HR Count matching 30 3.2 20 HR Staging / 11i index creation 40 3.3 30 HR HR 11i Load 50 3 2,3 CCO Staging 60 3.1 50 CCO Count matching 70 3.2 60 CCO Staging / 11i index creation Toad 80 3.4 70 CCO 11i Load (Non Employee) Conc program 90 3.3 40,70 CCO 11i Load (Employee) Request jtf roles for IB 95 90 JTF setup display for specific users Conc program 81 3.3 TZ Staging SQL*Plus 82 5.1 81 TZ Count matching SQL*Plus 19
Error explanation spreadsheet ID Business Entity Example Business Key Error Summary active end date 6/28/2003, 8/18/2003, 8/41/2003 Contracts bill_to_address_id 1334633, 129650 (28 occurrences) Contracts instance_id 122423407, 73041552,73365382 missing value in csi_ip_accounts Contracts instance_id 14061, 33907, 33923 status 10002,10005,10007, 20