Management & Technology Consulting. Management & Technology Consulting 11/4/2011 1

Transcription

1 Management & Technology Consulting Management & Technology Consulting 11/4/2011 1

2 Management & Technology Consulting 11/4/2011 2

3 Objectives: Like for Like Technical Upgrade of Oracle Applications Retrofit existing reports & customizations for compatibility and usability on R12 Provide training and coordinate testing across 4 major regions North America, Asia, India and Europe Management & Technology Consulting 11/4/2011 3

4 Scope: 26 Ledgers GL, FA, AP, AR, TAX, IE, CE, PO, INV, OM 40,000 User Base 955 Custom Objects Required Analysis 3TB Database Global training and testing 7 ½ Months Management & Technology Consulting 11/4/2011 4

5 Steering Committee: 15 Project Manager: 2 Technical Lead: 2 2 Leads (Training & Testing) 9 Functional Analysts 2-3 DBAs 12 Developers Global Business SMES: 50 Management & Technology Consulting 11/4/2011 5

6 Decide upgrade approach and scope Determine timeline & resources Maximize Testing Robust Cutover Planning Testers should have a strong knowledge of business processes Issue Strategy Communication Management & Technology Consulting 11/4/2011 6

7 Training Approach Phase 1 Onsite Classroom Training Provided by Oracle University Functional Analysts and Business Core Team Phase 2 Train the Trainer Provided by Jibe Consulting Delta training of mandatory R12 functionality Phase 3 End User Training Provided by Amazon Business Core Team Brown Bag Sessions, Onsite Training, Web Conferences Management & Technology Consulting 11/4/2011 7

8 Training delivered prior to test cycles to ensure maximum benefit End user training delivered prior to UAT/Go-Live On-Site training was the most successful Management & Technology Consulting 11/4/2011 8

9 Turnover within the organization led to training gaps Follow up and obtain training plans from all areas Small Changes = Big Questions Provide some level of training for all new/changed functionality Face Time Business users who were offered onsite training/ web sessions were the most successful in navigating through the new functions Video Recordings Timing Deliver training to coordinate with testing and Go-Live Management & Technology Consulting 11/4/2011 9

10 3 Test Cycles TC1 Standard Oracle Functionality Core Business Team TC2 UAT Introduced interfaces and customizations Business SMEs All business processes, custom objects and reports End Users Management & Technology Consulting 11/4/

11 Challenges Educating testers on how and what to test Differences between unit testing and end to end testing Volume of testing How to measure progress Solution Conducted testing planning sessions with entire team Held planning work sessions with individual teams Assigned a finance owner to each end to end scenario Management & Technology Consulting 11/4/

12 Global Reference Scenario Name Category Week 1 Week 2 Week 3 Target Test Completion Date Responsible AR-US-001 Autoinvoice Import Program Vendor Returns 1 7/12/2011 Kala Jones AR-US-002 Payables Open Interface Import Vendor Returns 1 7/12/2011 Kala Jones AR-US-003 Prepayment Matching Program (Prepayments Matching Program) Vendor Returns 1 7/12/2011 Kala Jones AR-US-004 AMZ Load Vendor Return Data From FATE Vendor Returns 1 7/12/2011 Kala Jones AR-US-005 AMZ Customers Interface load from Vendor Returns Vendor Returns 1 7/12/2011 Kala Jones AR-US-006 Customer Interface Vendor Returns 1 7/12/2011 Kala Jones AR-US-007 AMZ Load AR Vendor Returns Invoices Vendor Returns 1 7/12/2011 Kala Jones AR-US-008 AutoInvoice Master Program Vendor Returns 1 7/12/2011 Kala Jones AR-US-009 View/Change Payment Terms Vendor Returns 1 7/12/2011 Kala Jones AR-US-010 AMZ Vendor Return Create Reclass Journal Vendor Returns 1 7/12/2011 Kala Jones AR-US-011 AMZ Vendor Returns Current State Summary Report Vendor Returns 1 7/12/2011 Kala Jones AR-US-012 AMZ Vendor Returns Current State Detail Report Vendor Returns 1 7/12/2011 Kala Jones AR-US-013 AMZ AR Aging Report - Customer/Product Summary AR Unit Test 1 7/8/2011 Andrew Nienhaus AR-US-014 AMZ AR Invoice Aging Detail Report AR Unit Test 1 7/8/2011 Andrew Nienhaus Management & Technology Consulting 11/4/

13 Blocks of time scheduled by category Color coding helped testers identify when they were needed Management & Technology Consulting 11/4/

14 Total Pass Total Fail Total Incomplete Attempted Total Scripts Module # % # % # % # % Accounts Payable % 132 5% 0 0% % Accounts Receivable % 48 7% 0 0% % Cash Management % 1 1% 3 3% 87 97% Fixed Assets % 0 0% 46 19% % General Ledger % 86 17% 68 13% % Inventory % 0 0% 0 0% % Order Management % 0 0% 0 0% % Purchasing % 5 1% 0 0% % ebusiness Tax % 18 69% 0 0% % TOTAL % 290 6% 117 3% % Metrics were summarized for reporting Drilldown by process and country Management & Technology Consulting 11/4/

15 3 Week Test Cycle Objectives Test standard Oracle functionality Business teams identified test scenarios & created test scripts Identify End to End Scenarios Planning of future test cycles Management & Technology Consulting 11/4/

16 Test Cycle 1 - Challenges Challenges Managing Expectations Users not aware of what was a customization Solution Functional Analysts present in the testing rooms Knowledge Transfer Unexpected Issues with Standard Functionality Custom Folders Custom Application ASLs no longer supported Deleted all custom folders Created all new responsibilities Converted all ASL items to requisition templates Issues Logging Process Multiple entries for the same issue Entries which did not clearly state the problem Process Owner Assisted with reporting metrics and logging issues Management & Technology Consulting 11/4/

17 5 Weeks 3 Weeks Unit/Transaction Testing 2 Weeks Integrated/End to End Testing 2343 Unit Test Scripts Executed 59 End to End Scenarios Management & Technology Consulting 11/4/

18 Objectives Identify and resolve system issues Identify training gaps Complete generation/rework of test scripts Exit Criteria All Blocker issues resolved All Critical issues identified and resolution ETA determined Management & Technology Consulting 11/4/

19 Challenges Global Coordination Testers outside of the US struggled with lack of support Expectations were not clear Testing Interfaces Initial testing was not successful Frustration Solution Bi-Weekly testing status calls Weekly calls with Functional Analysts Daily communication of blocking issues and progress All interfaces thoroughly tested by FA prior to user testing Interfaces scheduled to ensure data preparation was complete Interfaces tested globally Management & Technology Consulting 11/4/

20 3 week test cycle Unit testing and Integrated testing End Users responsible for testing Global participation 4636 Test Scripts Majority were in AP and AR Performance testing Management & Technology Consulting 11/4/

21 Objectives Resolve any remaining issues Ensure business and system readiness for Go-Live Complete performance testing Exit Criteria All Blocker Issues Resolved All Critical Issues Identified Needed to be resolved by month-end Management & Technology Consulting 11/4/

22 Challenges Functionality Issues New testers found new issues Customizations and reports Performance tested the first time Issues Reports did not complete due to large Performance Issues volume of data Reports did not complete due to large volume of data Solution Additional resources added Encouraged as much testing as possible Identified key reports and did performance tuning Worked with Oracle for solutions Performance Testing Few resources and limited time to complete Attempted load testing during UAT Ran several time consuming processes and monitored execution Management & Technology Consulting 11/4/

23 Performance Issues Test, test, test and test some more Tax and AP Data Issues Multiple data fixes required Test both new and upgraded data PO/AP Reconciliation Significant changes that were not tested by all geographies Create Accounting Completely new concept, ensure users understand the changes and implications to the month end process Management & Technology Consulting 11/4/

24 Agenda Customization Planning Approach Development Challenges Performance Issues Concurrent Manager R12 Patch Monitoring and Tuning Management & Technology Consulting 11/4/

25 Retrofit 900+ customizations in time for TC2 testing. Development time P1 customizations ~2 months. P2 customizations ~3.5 months Minimize business downtime due to R12 patch upgrade Management & Technology Consulting 11/4/

26 Inventory Inventory all the customizations Identify What can retire What will upgrade cleanly What needs to be re-implement in R12. E.g. check printing Resource planning for the development team Finalize the list of customizations that will need to be retrofitted. Data model changes tool used Document id Cross reference with the customization list Assign level of effort to broken customizations Level of effort used: Low(8), medium(16) and high (40 hrs.) Load resources appropriately Assign resources to business modules. E.g. designated developer for bank integration Provide technical training to the developers Begin Development phase Management & Technology Consulting 11/4/

27 Management & Technology Consulting 11/4/

28 Data model comparison report Help understand the data model changes Download document id Management & Technology Consulting 11/4/

29 Check Print development Start early. Long development cycle Long test cycle with the banks Print issues Report output shifted across printers even though they were the same make and model Issue was due to the print drivers not being up to date for the printer. Some of the printers were using altogether different drivers as a result of changes over time. Used pdftops to stabilize the shifting of the output Custom Application (Amazon.com) associated to Custom Responsibilities E.g. AMZ Purchasing Manager associated to Amazon.com instead of seeded Purchasing application Caused breakage of self service forms in Payments, iprocurement and iexpense LOV errors in certain standard reports E.g. Create Accounting Solution: Re-do responsibilities and remove amazon.com association. ~1200 responsibilities impacted Consideration for R12 upgrade Management & Technology Consulting 11/4/

30 BI Publisher reports are memory hungry compared to traditional Reports 6i Several month end reports failed to complete. E.g. Payables Posted Invoice/Payments Register 45,000 pages of pdf output for one month Started as 16 GB of report output. Solution: Oracle support provided a stripped down version of the XML Output Pre-allocated 2GB memory on the concurrent program definition. Set Options = -ms2048m -mx2048m 3 properties need to be set to True for the report template under XML Administrator Responsibility Management & Technology Consulting 11/4/

31 Auto-Select and Build Payments programs erroring out after hitting batches of 20K+ invoices Solution: Profile AP: Gather PPR Table Stats set to Yes at site level Performance patch released by Oracle support Create Accounting process running longer 20+ hrs. impacting month close activities Solution: Bug raised with Oracle support and patch received Created a daily run schedule for Create Accounting job for each geography Assigned multiple processors to Create Accounting process to decrease run time. Caution: optimize the number of processor based on server and concurrent manager load. Screenshot on next page Management & Technology Consulting 11/4/

32 Performance Issues (cont.) Recommend doing a load test during UAT testing to uncover performance issues Management & Technology Consulting 11/4/

33 Amazon uses 30+ custom managers to process jobs under Concurrent Manager Used to segregate quick and long running jobs. E.g. Print PO s and Alerts(quick jobs) vs. Invoice import (long running) Separate managers for high volume jobs. E.g. Print AR Invoices. Daily load of > 10K jobs Pain Points Managers were constantly backed up. Several hundred jobs sitting in pending state everyday 2 hrs. average wait time before jobs gets picked for processing Global users being impacted by spill over from other geographies Impacted several time sensitive activities E.g. month end reports, PO printing SLA s Created negative user experience. Users constantly requesting IT to bump up the priority of jobs Management & Technology Consulting 11/4/

34 Observations: All workers not being utilized. E.g. between workers would be used instead of the full 20. Would use the full allotted only at times. Solution: Reduced the number of concurrent managers from 30 to 18 Updated cache size of the managers to twice the number of workers Decreased sleep seconds of quick running jobs to 20 Increased sleep seconds to 120 for long running jobs Summary Key is to minimize the number of managers Optimize: Number of workers/processes Sleep seconds Cache size. Thumb rule is twice the number of workers Management & Technology Consulting 11/4/

35 5 ½ Days Downtime Upgrade 3TB of database Management & Technology Consulting 11/4/

36 Major Cutover Activities Management & Technology Consulting 11/4/

37 Useful tool to communicate with the IT and Business management Helps understand the long activities and focus on reducing the time of those activities Helps the team not to get confused with >5,000 lines of detailed cutover plan. Was used in conjunction with the detailed cutover plan Contingency built into major activities. Relieves pressure if any activity gets delayed Management & Technology Consulting 11/4/

38 Biggest contributors to the 5 ½ days of downtime R patch = 55 hrs. Oracle data fixes for tax, AR invoices and AP =15 hrs. 10 data fixes totaling up to 15 hrs. of downtime Long running due to the number of transactions being updated. E.g. 86 million rows in ap_invoices_all Missing Establishment_id in ZX tables Ensure Oracle provides a script that can be run with parallel workers Post-Upgrade reports = 8 hrs. Issue was the runtime of the reports. Month end reports ran for >5 hrs Management & Technology Consulting 11/4/

39 Technical upgrade is the process of upgrading 11i environment to R Involves upgrading database to 11gR2 (done as a precursor project) Laying technical stack for R12 Running R patch Running R patch Post upgrade patches including one-off s identified by project team Recommendation: Plan on running at least 5 tech upgrades Practice and practice more to make it a repeatable process Identify long running jobs and tune them Management & Technology Consulting 11/4/

40 Business shut down activities Process all transactions Submit all accounting transactions Verify all interfaces are clean Must be coordinated across all business teams Smoke Testing Key Interfaces and processes Provide assurance to business and management that system was ready Ramp Up Slowly Management & Technology Consulting 11/4/

41 Erratic runtime of AP/AR jobs. Oracle database optimizer would change the sql execution plan and cause jobs to run longer This was due to: Oracle dynamic sampling Lack of database statistics Use of temporary tables as part of upgrade Solution: Pin SQL queries to a specific plan by using SQL profiles Procedure: Monitor long running queries in OEM Implement a better explain plan if recommended by Oracle OEM SQL profile the plan Export and use for next run Management & Technology Consulting 11/4/

42 Technical Upgrade Process flow and tuning exercise Management & Technology Consulting 11/4/

43 Management & Technology Consulting 11/4/

44 Oracle database name can only be in A-Z or 0-9 character format Rapidwiz failed to run during R12 prod upgrade. Did not like _ underscore in FIN_P db name Solution: Changed the name to FINP. Caused the tnsnames file to be changed for external systems. Diligently follow R12 upgrade guide (Part No. E ) Excellent suggestions in the technical and functional areas Good tips in pre-upgrade functional steps that help reduce downtime. E.g. Submit Accounting missed in 11i and caused one of job to run for 7 hrs. instead of 3 hrs. during the tech upgrade Management & Technology Consulting 11/4/