American Express Data Center Migration Presentation to the TPF Users Group Washington, DC March 11 th, 2014
Agenda Context Point of Departure New Data Center Set-Up New Data Center Validation Migrations Results
Point of Departure - CAS/TPF Environment Three Processors (2) IPC1, (1) MDC > IPC1 Production and Dev/Test are split > Dev/Test provides a production back-up platform > MDC for Disaster Recovery Production and Dev/Test share Six DASD Sub Systems (3) IPC1, (3) MDC > Production and Dev/Test are striped across all Sub Systems > West and ADC share some volumes with TPF Four Tape ATLs (2) IPC1, (2) MDC > Production and Dev/Test share all tape > West and ADC share Tape ranges with TPF 100% IP network as of March 2012 Shared Console environment across LPARs > Sharing limited by LPARs being fixed to separate processors > Windows servers running ware and SuperVision
Point of Departure: 2012 IGS TPF Install and Testing Plan --- March > Provide WWCAS a Detail Timeline and Migration Plan EMC/IGS detail install plan --- March > EMC to reflect back IGS/AMX Requirements for IPC1 and MDC hardware, Enginuity microcode, all scripts, EMC Product Suite for TPF, and detail copy design > EMC to reflect back IGS/AMX Requirements for IPC2 hardware, Enginuity microcode, EMC Product Suite for TPF and copy all scripts, and detail copy design Replication Network Detail Planning --- March Update WWCAS on Replication planning provided & Timeline --- April IGS/EMC IPC1/MDC Migration Methodology Test plan --- April Channel Extension install and zoning IPC1, MDC, IPC2 --- May EMC configuration and Product Suite for TPF changes for Test Plan --- May Verify 7.0.0.4 and Cascaded SRDF in the IPC1/MDC environment --- June
Point of Departure: CAS/TPF DASD Environment IPC1 PHX I9, I10 MDC MSP M16 AR2 CLN R1 AX AY AZ AT R2 Local Box TimeFinder Copy Local or Remote Box or SRDF/DM Copy Red or Black, SRDF or TimeFinder, local or remote, suspended copy TPF Records striped across 3 AX using SRDF/DM Immediate Tape file copies WK/AZ ATE
EMC Migration Method Validation: June 30, 2012 IPC1 PHX MDC MSP AR21 SHR P R21 AY Temp AX AZ SHR P R2 Temp AX R2 AT CLN R1 Temp ATE I9, I10 M16 * Set-up DR Environment to validate IPC2 requirements * This can be made permanent for the Migration Method WK/AZ
Set-up IPC2 Environment: 2012 EMC configuration input to IGS Hardware Management --- July > IPC1/MDC/IPC2 Microcode upgrade and patches Copy Network install IPC1, MDC, IPC2 --- July Channel Extension install and zoning IPC1, MDC, IPC2 --- July EMC DASD and BIN File upgrades IPC1, MDC --- August EMC delivery of all scripts --- September IBM install W9 processor & Tape --- October IGS install SuperVision Servers --- October EMC DASD and BIN File install IPC2 --- October > IPC1/IPC2 IOPS 300k -- Projected > IPC2 Double bandwidth of IPC1 (RFs in AX) > IPC1/MDC/EMC Product Suite for upgrade to version 7.1.0
Introduction of IPC2 Hardware: October 31, 2012 IPC1 PHX MDC MSP AR21 CLN R1 AY R2 AX AZ AT ATE I9, I10 M16 24 WK/AZ 48 WK1 W9 IPC2
TPF Systems Validation: 2012 Fully Functional W9 Environment --- October - November > First /TPF copy Cascaded SRDF/DM from MDC AR21 to IPC2 R2 TimeFinder Clone from IPC2 R2 to IPC2 AX Clone (R1) SRDF/DM from IPC2 AX R1 to IPC2 R2 SRDF/DM from MDC / R1 to IPC2 / R2 TimeFinder / IPC2 R2s to IPC2 / Clones > First /TPF IPL Hardware Verification, Processor, DASD, Tape TPF Testing/Validation --- November - December > TPF DB Utilities Test Cases Tape copy delay timings and copy start > Application Utilities Test Cases > Repeatable TPF Migration Copy Function Developed and Documented Validate Swapair as depicted in Detail Plan
IPC2 Validation: December 31, 2012 IPC1 PHX I9, I10 MDC MSP AY AZ AT M16 AX AR21 WK/AZ ATE R2 WK1 48 24 IPC2 W9
WWCAS IPC2 Validation: January July, 2013 TPF Stand-alone Network Start --- January > Validate Network connectivity > Begin TPF Migration Copies per WWCAS requests Additional Processor Install/Validation --- February Network Connectivity Validation --- February -- May > West, East, International, CAS Transactions testing in IPC2 > Individual GAN, STRATUS, traffic generators > Outage Timing Runs, IPC2 DR like verification exercise Production Network Migration Testing --- June & July (after MG6 & MG7) > Final Tape Copy Validation, remote shared tape > Final West Validation, remote shared DASD > Final TPF Migration Validation > Final Swapair Validation with AX to s
DC2015 Risk Platforms Migration Calendar DC2015 Migration Calendar- 2013 January February March April May June July August September 4 11 18 25 1 8 15 22 1 8 15 22 29 5 12 19 26 3 10 17 24 31 7 14 21 28 5 12 19 26 2 9 16 23 30 6 13 20 27 MG 4.2 MG 4.3 MG 4.4 MG 5A MG 5B MG 6A MG 6B MG 6C MG 7A MG 7B MG 8A MG 8B MG 8C Midrange Midrange Midrange Midrange Midrange Mainframe (DIPJ) Mainframe (PPRD/EPRD) Midrange Midrange Mainframe (West/LPRD) Midrange Midrange Mainframe (EAST/zData) Midrange Midrange Mainframe (CAS/TPF) Apr 26 Institutional and OPEN 4.2 Mar 22 Risk Rules Engine (WinARES) UW (CRUSE, icruse) ROP, AboutRisk 4.3 Jun 7 Credit Bureau Reporting (CBR), Credit and Collections (WCC), Global Account Monitor (GAM), Global Acquisition (GNA), Card Number Generation (CANS), Risk Assessment (GRMS) 4.4 5A 5B Apr 12 Credit Review Online (irisk ), Enterprise MIS, AMC, Enterprise Model Mgr, Risk Change Mgmt & BNR portals, CDIT 6A 6A 6A 6B 6B Jun 16 IDN Analytics, Big Data 6B 6B Jul 19 IDN Warehouse, IDN 6C Jul 12 Credit Bureau Reporting (CBR), Credit and Portal, 7A 7A Collections (WCC), Global Account Monitor (GAM), Global Acquisitions (GNA), ibureau 7A 7A Aug 16 Sphere 7B Aug 9 Credit Bureau Reporting (CBR), Credit and Collections (WCC), Global 8A 8A (Global Decision Engine, Global Risk Fulfillment 8B Account Monitor (GAM), Global Acquisitions (GNA) Card Number Generation (CANS), Risk Assessment (GRMS) Primary Migration Event 8A 8A 8C 8C Application Freeze- Primary Migration Date (2 wks prior,3 days post) Aug 30 Authorizations (CAS) Contingency Migration Event Region IPCW IPCE EPRD PPRD Migration Date EST 7/13/2013 8/11/2013 6/8/2013 & 6/9/2013 6/8/2013 & 6/9/2013 Application Freeze- Contingency Migration Date Outage Time 22:00 14:00 EST 00:01 16:00 EST 14:00 06:00 EST 14:00 06:00 EST Outage Duration 16 hours 16 Hours 16 Hours 16 Hours Local Time 00:01 16:00 MST 00:01 16:00 EST 19:00 11:00 GMT 04:00 20:00 UTC 12
Mainframe Migration Downtime (MST-PHX Times) Move Group 6A May 11-12 Move Group 6B June 8-9 Move Group 7A July 13-14 Soft Launch (DIPJ) International (EPRD, PPRD) zlinux LPRD UROC EAST zlinux Move Group 8C Aug 31 Saturday May 11 START Saturday June 8 START Saturday July 13 WEST zlinux Move Group 8A Aug 10-11 START CAS/TPF START Sunday July 14 START TIME 18:00 START TIME 12:00 START TIME 23:00 START TIME 00:01 (midnight) START TIME 00:01 Saturday August 10 (midnight) END Sunday - May 12 END Sunday June 9 END Sunday July 14 START TIME Saturday August 10 21:00 END TIME* 2:00 END TIME* 12:00 END END TIME* Sunday July 14 16:00 END END TIME* Saturday August 10 13:00 START START END TIME* 4:00 END TIME* Sunday August 11 13:00 END START START TIME END END TIME (MSTPHX) Saturday August 31 2:00 Saturday August 31 6:00 (*) Includes contingency window of approximately 4 hours
CAS/TPF Migrations: June August, 2013 CAS/TPF Migration Dry Run Testing -- June -- July West Migration to IPC2 -- July 12, 2013 > Remote Shared DASD Automation CAS/TPF Migration to IPC2 -- August 31, 2013 > Run EMC Product Suite for in IPC1 until commit to IPC2 > Back-up the EMC Control Tables > Sync in IPC2 via from IPC1 and IPL in IPC2 > Activate SRDF/DM copy from MDC to IPC2 > Ensure all copies are very close to in-sync Cycle TPFA in IPC1 to 1052 state Start a new cycle and stop copies > When all copies are 100% in-sync Perform Pend/Drop on TPFA Suspend SWAPair Split Suspend Delpair Clip IPC2 Volsers
Summary of Credit Authorizations (CAS) Mainframe Migration Move Group 8C Aug 31 Authorizations Pre-Migration START Saturday August 31 Migration Infrastructure Hardware and data replication enhancements have been implemented DATA Integrity Process in place to ensure all updates are applied to the new hardware Integration Developed integrated migration plan working with all vendor partners Outage Time Scheduled 60 minutes starting on Saturday @ 02:00MST Change Restrictions All changes will be reviewed weekly with development teams for freeze impacts Repeatable Developed a repeatable testing process to validate data, software, and hardware in the new datacenter Stand-In Distributed platforms provide authorizations during planned outage Impacts Servicing, ATM, PIN@POS functions will be unavailable during the migration change window START TIME END 2:00 Saturday August 31 Outage Duration 60 mins Post Migration Validate Connectivity All teams will be engaged to validate network and intersystem connectivity Validate Data All teams will be engaged to validate data integrity Validate Apps All teams will be engaged to validate capabilities Fallback Processes and scripts ready to execute in case of fallback decision Uplift Upgraded disk arrays used for authorizations to avoid additional cost and outages 15
Move Group 8C Move Day Timeline TPF/CAS on August 31 1:50AM CHECK POINT Confirm all is ready for shut down 00:01AM CHECK POINT All Pre-Work Confirmed 2:00AM Shut down TPF 2:40AM CHECK POINT Confirm ready to start TPF 3:00AM Start TPF in IPC-2 3:15AM CHECK POINT Confirm TPF is up and available 4:30AM BSD VP Sign-off Migration Complete 5:00AM Migration GO/NO GO Checkpoint decision complete TPF/CAS Migration 1:00AM Open Migration Bridges 1:30AM Quiesce batch and Drain copy data to IPC-2 2:05AM Drain and Stop 2:25AM Split IPC-2 data from the copy process 2:15AM Complete data synchronization 3:00AM Systems, BSD, and Business verification of migrated data and network connectivity 6:00AM Migration Complete 6:00AM 10:00AM CHECK POINT Monitor/TPF health Additional Check Points: 9/1 3:00am, 8:00pm 9/2 8:00am, 8:00pm Saturday, August 31
August 2013 TPF Performance IPC1 Mills per Msg: 4.16 MIPS per Msg: 2.16 ECB s in Use: 173.15 VFA Efficiency: 76.7% DASD Service Time: 0.973ms TPF Auth Time: 0.182 seconds
TPF Migration: August 31, 2013 IPC1 PHX I9, I10 AY AX AT MDC MSP M16 AR21 AZ WK/AZ ATE R2 WK1 48 24 W9, W10 IPC2
CAS/TPF Migration Success: August 31, 2013 To minimize downtime and customer impact, the CAS and IBM teams completed a total of four dry runs in preparation for the migration. Automation and scripting were used extensively and the duration of the actual CAS outage was minimized to only 27 minutes (2:02am-2:29am MST). As planned, GAN stood-in for CAS during that time period and Payment Awareness provided stand-in logic to ensure payments were not interrupted. All technical, application and business validation activities were completed at 3:51am MST with no significant problems identified. The CAS/TPF Mainframe environment consisted of: 46 applications 1 LPAR 6,300 MIPS 120 Terabytes of storage
CAS/TPF Migration Commit: August 31, 2013 CAS/TPF Migration -- August 31 Stop running EMC Product Suite for in IPC1 > Run EMC Product Suite for in IPC2 > Set-up CAS/TPF DR copies Create IPC2 to MDC Groups from IPC2 s (no clear) Create MDC SRDF Groups from s to s Create MDC TimeFinder Clone Sessions Create IPC2 TimeFinder Clone Sessions Create IPC1 SRDF Sessions > Commit Decision for CAS/TPF to remain in IPC2 > Clean-up & build new CAS/TPF DR copies Clean-up any SRDF/TimeFinder session not needed Delpair SRDF TPFA Terminate IPC1 TimeFinder Clone Sessions Remove IPC1 TimeFinder and SRDF Groups > Initiate CAS/TPF DR copies from IPC2 to MDC
TPF Migration Commit: August 31, 2013 IPC1 PHX I9, I10 MDC MSP AZ AX M16 AT AR21 WK/AZ R2 AY ATE WK1 48 24 R1 W9,W10 IPC2
CAS/TPF Migration Success: August 31, 2013 CAS/TPF Migration completed and BAU --- August 31, 2013 >Normal operation in IPC2 / Normal DR support in MDC TPF Production back-up copies created in IPC2 TPF DR copies in MDC created from IPC2 TPF Test Systems copies in IPC1 created from IPC2 /data copies in IPC2 from IPC1 to support Production TPF issues only
TPF Migration Success: August 31, 2013 IPC1 PHX I9, I10 AT AZ AX AY M16 MDC MSP WK/AZ ATE R1 WK1 48 24 W9,W10 IPC2
September 2013 TPF Performance IPC2 Mills per Msg: 3.04 (-26.9%) MIPS per Msg: 2.32 (6.8%) ECB s in Use: 92.72 (-46.5%) VFA Efficiency: 76.5% (- 0.2%) DASD Service Time: 0.467ms (-51%) TPF Auth Time: 0.068 seconds (-62.6%)
March 2014 TPF Performance CAS Authorizations TPF Time 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 Series1
Questions