Perforce Best SAP. Claudia Loff, SAP AG Thomas Kroll, SAP AG Walldorf, Germany, 2006/08/14

Transcription

1 Perforce Best SAP Claudia Loff, SAP AG Thomas Kroll, SAP AG Walldorf, Germany, 2006/08/14

2 Presenters and the other team members Claudia Loff Claudia is project lead of the "Perforce Improvement Project" in the Perforce team within SAP NetWeaver Lifecycle Management department. Before joining SAP in 1997, Claudia was a developer and project lead for more than 6 years at two midsized software companies. Prior to working with the Perforce team at SAP, she held several positions as developer and project lead within SAP. Claudia holds a masters degree of computer science from the University of Karlsruhe, Germany and took "business administration for engineers" class at University of Hagen, Germany. Thomas Kroll Thomas graduated and finished his studies of computer science at the BA of Mannheim, Germany for IBM, in He joined SAP in October 2001 where he was in charge of the C/++ build infrastructure on Windows platforms used for the SAP products like R/3, SAP GUI & Business One. End of 2004 he took over the responsibility for the Perforce team and Perforce infrastructure at SAP. Delcho Christian Haydar Sherry Milchev Schmitt-Plank Asan Wu SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 2

3 PART I Presented by Thomas History & key figures Hardware configuration Monitoring Infrastructure introduction

4 PART II Presented by Claudia Loff Codeline Concept & Consolidation Authorization Management, Mass Handling Lessons Learned (Improvement Project)

5 PART I Presented by Thomas History & key figures Hardware configuration Monitoring Infrastructure introduction

6 History of Perforce within SAP Evaluation of SCM systems First Perforce server at SAP Productive move from ClearCase into Perforce Tool infrastructure development & maintenance Major Hardware investments & performance improvements P4MS PerforceManagementSystem running on SAP NetWeaver 80+ Perforce server instances Upgrade of all instances 4,600+ Registered Perforce users SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 6

7 SAP's Global R&D Organization 4.57% Canada Montréal North America Palo Alto and satellites 45.79% Germany Walldorf and satellites Hungary Budapest Bulgaria Sofia Israel Ra anana 13.52% 17.74% China Shanghai 3.69% Location Bulgaria India Israel Overall Outbound Inbound GB 0.68 GB GB 0.18 GB GB GB 9.31 TB 2.74 TB Monthly transfered volume via P4 protocol India Bangalore 6.55% SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 7

8 Key figures User/License growth USERS SERVERS LEVELS FILES/REV CURRENT +11.6% users last year Perforce users & licenses 4,700 4,200 3,700 3,200 2,700 2,200 1,700 1, /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/ /03/ /06/ /09/ /12/02 SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 8

9 Key figures Server growth USERS SERVERS LEVELS FILES/REV CURRENT +11,6% users last year ~8 new servers per year Perforce servers by by by by by by * by SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 9

10 Key figures Project & codeline growth USERS SERVERS LEVELS FILES/REV CURRENT +11,6% users last year Perforce projects (2nd) & codelines (3rd) ~8 new servers per year ~2,000 new projects last year Projects Codelines by by by by by by by SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 10

11 Key figures File & revision growth USERS SERVERS LEVELS FILES/REV CURRENT +11,6% users last year Perforce files & revisions ~8 new servers per year ~2.000 new projects last year >10 Mio files added last year Files Revisions/Activity by by by by by by by SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 11

12 Key figures Current overview USERS SERVERS LEVELS FILES/REV CURRENT +11,6% users last year ~8 new servers per year ~2.000 new projects last year >10 Mio files added last year ~27 p4 calls per second Active Users / Licenses Server instances Depots (1st level) Projects (2nd level) Codelines unique Codelines overall (3rd level) Current overview Files Revisions Filesystem consumption Average number of p4 calls per day 4, ,143 6,498 55,787 46,972, ,482, TB 2,303,174 SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 12

13 PART I Presented by Thomas History & key figures Hardware configuration Monitoring Infrastructure introduction

14 Hardware setup Cluster node 1 & 2: FSC PrimePower 450N CPU: 4 x 1320 MHz Memory: 32GB OS: SunOS 9 HA ensured by PRIMECLUSTER Cluster node 3 & 4: FSC PrimePower 650 CPU: 8 x 675 MHz Memory: 32GB OS: SunOS 9 Shared Storage (SAN): 2 x EMC Clariion CX500 (mirrored) Size: 3TB each SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 14

15 HA Setup Location I (ROT) Location II (WDF) Cluster Node 1 Cluster Node 2 Cluster Node 3 Cluster Node 4 2 Fibre Channel SAN switches 2 Fibre Channel SAN switches EMC I Mirroring filesystem per service EMC II Redundant 2 Gbit/s Fibre Channel connections Redundant Gbit/s Ethernet connections SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 15

16 Backup strategy Two times a week an incremental backup per Perforce service: stop p4d filesystem snapshot start p4d Incr. backup snapshot Downtime: 1-2 minutes per service On the weekend full backup of each Perforce service stop p4d checkpoint filesystem snapshot start p4d Full backup snapshot Downtime: depending on size, approx. 2 hours Time for complete restore from backup: Maximum loss of history: ~3 hours 2 workdays SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 16

17 PART I Presented by Thomas History & key figures Hardware configuration Monitoring Infrastructure introduction

18 Monitoring Performance Monitor Nagios Cluster/HW Real time monitoring of p4 servers (based on debug level 3 logs) display all currently running processes on all instances graphical & tabularly representation per server (TOP10) notification on reaching warning and error values find bottlenecks (e.g. commands, clients etc.) Monitoring of OS parameters CPU & memory consumption Monitor filesystem usage => increase of disk space Warnings in case a service is not responding Low-level monitoring by PRIMECLUSTER Failover in case of problems (e.g. filesystem, HW failure) restart on other node SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 18

19 Monitoring via P4PM SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 19

20 Monitoring via P4PM SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 20

21 PART I Presented by Thomas History & key figures Hardware configuration Monitoring Infrastructure introduction

22 Complete infrastructure at a glance protections & triggers server instances P4MS PerforceManagementSystem self services & web representation P4SAP backend Configuration Consolidation Transport several scripts Performance Monitor P4SAP External tools like JavaCorrectionWorkbanch (DTI) or developer dashboard SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 22

23 PART II Presented by Claudia Loff Codeline Concept & Consolidation Authorization Management, Mass Handling Lessons Learned (Improvement Project)

24 Complete infrastructure at a glance inside SAP infrastructure Fix directory structure protections & triggers server instances P4MS PerforceManagementSystem self services & web representation //depot/project/codeline P4SAP backend Configuration Consolidation Transport several scripts Performance Monitor P4SAP External tools like JavaCorrectionWorkbanch (DTI) or developer dashboard SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 24

25 Production process (development phase) Dev Submit changes to dev after local test nightly build, central deployment to test system and test Release change for transport to export Maturity cons delivery Export Collect changes of dev (gate to patch) Transport (copy) content to patch in regular intervals (e.g. weekly cycle) using a timestamp export patch last call collect, synchronize Patch Small group of developers is allowed to apply fixes for stabilization approval by Quality Manager(s) dev develop t Transport (copy) content to cons Cons (consolidated code line) ready to be used by dependent components/products SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 25

26 Depot/Directory Structure depot project codeline SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 26

27 Dependencies between changes dev export c1... Call method A Call method A... c2 Method A {...}... Method A {...}... c3... Call method A... Some fixes... Call method A... Some fixes Bundle 42* c1 c2 c3 * For developers, a bundle is the answer to all questions; see also: D. Adams The Hitchhiker's Guide to the Galaxy SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 27

28 Transports between the code lines Integration manual transport (p4 integrate) Tools provided by Perforce (p4.exe, P4Win and P4V) Ignores dependencies between changes Used to create identical codelines = full copy e.g. initial creation of (remote) maintenance codelines out of development codelines (P4MS) Perforce@SAP Consolidation controlled integration Tools developed within SAP based on p4 integrate takes dependencies between changes into account allows bundling of changes bundle = set of changes that have to be transported together (e.g. c1 and c2) shows status, user and timestamp of a transport N (new), P (consolidation in process), E (error), W (waiting for dependent change), C (consolidated) No multi-server so far SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 28

29 infrastructure Fill Process Runs every 5 minutes for each Perforce server collect changes of consolidation codelines and write them into the database P4SAP one change list in P4 = one consobject per branch(!) with status N in P4SAP preparation for consolidation frontend (SelfService) determine dependencies and write them into P4SAP protections & triggers P4SAP backend //depot/project/codeline server instances P4SAP SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 29

30 infrastructure Consolidation Processes Preparation Process Steps Look for the requested changes in P4SAP and determine target branch Determine dependencies - Any unconsolidated predecessors or bundle siblings? -> Set status to W (waiting) Transport P4 integrate set status to C Post Processing Look for successors in status W aiting and start consolidation for them send notification via to requester (P4MS) SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 30

31 PART II Presented by Claudia Loff Codeline Concept & Consolidation Authorization Management, Mass Handling Lessons Learned (Improvement Project)

32 Authorization concept/user management Perforce standard user and group management; protection table more than 80 Perforce Servers more than 4000 users additional processes via SAP infrastructure Authorization concept with central user and group management Central user/group management via database P4SAP using update of user data from SAP Corporate Portal Set password on all p4servers (P4MS), password-check Consolidation rights Protection table: keep it small for good performance! -> P4 authorization concept (=> more than > 100 entries) SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 32

33 Mass Handling Local small transports Synchronized milestones => synchronized transports (250+ projects at the same time/timestamp) Central responsibility for transports (central Quality Management) Mass transport via Perforce Management System (P4MS) background jobs (CONSOLIDATION, INTEGRATE, COPY, PATCH) scheduled and monitored via job monitor notification is send to creator after the job is finished Mass consolidation may last very long: up to 9h for 236 changes in 229 projects Mass integration may last very long: 1 hour on a local server, up to 9.5 hours for 159,155 files in 154 projects for remote depots SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 33

34 PART II Presented by Claudia Loff Codeline Concept & Consolidation Authorization Management, Mass Handling Lessons Learned (Improvement Project)

35 Improvement project (lessons learned) Challenges and Mitigation Performance, Performance, Performance, Perforce Cluster Performance monitor (identify bad calls; push better solutions) Authorization concept (keep protection tables small) Migrate to new version of operating system and database Separate Perforce servers for maintenance codelines - Better performance for development codelines due to less codelines - Less need to integrate from dev to maintenance codeline (remote depot) Increase productivity and agility (more codelines, mass handling,...) Perforce Management System (P4MS) Multi-Perforce Server Environment Perforce Management System (P4MS) Submit template (track change lists via service request number) Complex (intransparent) dependency networks for change lists Graphical representation SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 35

36 Questions? Q&A SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 36

37 Details For your reference SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 37

38 Real life example: Codelines for NW04s (working titles) correction validation patch Service Pack 11 Service Pack 10 Service Pack 09 Service Pack _SP_COR 645_VAL_COR 645_COR consolidation by developer 645_SP_TEST consolidation by Final Assembly consolidation by Final Assembly NW04S_08_COR NW04S_08_REL consolidation by reponsible NW04S_07_COR NW04S_06_COR NW04S_07_REL NW04S_06_REL NW04S_05_COR NW04S_05_REL 645_SP_REL 645_VAL_REL 645_REL NW04S_04_COR NW04S_04_REL Several p4 servers perforce3645 copy by Perforce Team ordered by Final Assembly or integrate by Responsible/dev copy by Perforce Team ordered by Final Assembly SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 38 Delivery by Final Assembly Customer (SAP Service Marketplace)

39 infrastructure Modules&Configuration Modules P4SAP backend Data base P4SAP (MaxDB ) - Configuration - Consolidation objects (created from change lists of Perforce servers) LINUX environment for processes SelfServices (web server) Perforce Management System (P4MS) New SelfService and main tool for Perforce Team, now Running on SAP NetWeaver Configuration Meta data for projects Name, creator, description... Perforce Server Codelines - P4 Trigger ensures that the first 3 level of a Perforce directory can be created by Perforce Team, only Meta Data for consolidation (consolidation branches) - development branch in Perforce server - Definition of source and target codeline (per project) - naming convention: <PROJECT>_<CODELINE> Authorization, User and Group Data SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 39

40 infrastructure Consolidation Processes Task transport change list(s) in a fix sequence and along defined tracks Triggered by SelfService (web server) Perforce Management System (P4MS) Command line (used by tools like build environment) Restrictions Source and target codeline have to be on the same Perforce server SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 40

41 Perforce Authorization concept (extract) All codelines which are part of one Release are handled with the same concept. 645-admin (layer/comp resp) 645_COR_<layer>-cons 645_TEST_<layer>-cons Final Assembly, only Final Assembly, only Final Assembly, only Codeline 645_SP_COR 645_SP_TEST 645_SP_REL 645_VAL_COR 645_VAL_REL 645_COR 645_REL NW04s_XX_COR NW04s_XX_REL write(submit) write (submit) 645-patcher open (edit) 645-write (Q-team) Maintain group members open group 645-dev * //*/*/645*_COR/... write group 645-write * //*/*/645*_COR/... write group 645-patcher * //*/*/645_SP_COR/ dev SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 41

42 Mass Consolidation using P4MS Job type(s) CONSOLIDATION Select criteria User, project, codeline, collection Input data Changes Prerequisite Change lists are filled into database P4MS Restriction Source and target codeline have to be on the same Perforce server Consolidation for collections with many projects may have a duration of up to 9h (236 changes in 229 projects) SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 42

43 Mass Integration using P4MS Job type(s) INTEGRATE COPY PATCH integrate sources create an identical copy in the target branch (delete files, if needed) integrate change list(s) Select criteria Codeline Collection Change list Project Timestamp Input data target codeline P4MS knows on which Perforce server the target codeline for a project is Prerequisite Existance of target codeline For new codelines all meta data in the database P4SAP is created on a fingertip by Perforce Team Integration for collections with many projects may have a duration of more than 1 hour on the same server up to 9.5 h for transports to different servers (154 projects, 159,155 files) SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 43

44 Improvement project (lessons learned) Planned improvements Perforce Cluster Upgrade to Perforce or higher (p4 obliterate) Speed up technical Perforce access for colleagues in remote locations LDAP connection infrastructure Service Queue - Integration of p4 processes in automatic workflows - Migrate SelfServices from Webserver to SAP WebDynpro resp. webservices Change tracking between different servers SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 44

45 Copyright 2006 SAP AG. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages. SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 45

46 Copyright 2006 SAP AG. Alle Rechte vorbehalten Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten. Microsoft, WINDOWS, NT, EXCEL, Word, PowerPoint und SQL Server sind eingetragene Marken der Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix und Informix Dynamic ServerTM sind Marken der IBM Corporation. ORACLE ist eine eingetragene Marke der ORACLE Corporation. UNIX, X/Open, OSF/1 und Motif sind eingetragene Marken der Open Group. Citrix, das Citrix-Logo, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, MultiWin und andere hier erwähnte Namen von Citrix-Produkten sind Marken von Citrix Systems, Inc. HTML, DHTML, XML, XHTML sind Marken oder eingetragene Marken des W3C, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA ist eine eingetragene Marke der Sun Microsystems, Inc. JAVASCRIPT ist eine eingetragene Marke der Sun Microsystems, Inc., verwendet unter der Lizenz der von Netscape entwickelten und implementierten Technologie. MaxDB ist eine Marke von MySQL AB, Schweden. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, und weitere im Text erwähnte SAP-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und anderen Ländern weltweit. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch SAP AG gestattet. Bei dieser Publikation handelt es sich um eine vorläufige Version, die nicht Ihrem gültigen Lizenzvertrag oder anderen Vereinbarungen mit SAP unterliegt. Diese Publikation enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP -Produkts. SAP entsteht aus dieser Publikation keine Verpflichtung zu einer bestimmten Geschäfts- oder Produktstrategie und/oder bestimmten Entwicklungen. Diese Publikation kann von SAP jederzeit ohne vorherige Ankündigung geändert werden. SAP übernimmt keine Haftung für Fehler oder Auslassungen in dieser Publikation. Des Weiteren übernimmt SAP keine Garantie für die Exaktheit oder Vollständigkeit der Informationen, Texte, Grafiken, Links und sonstigen in dieser Publikation enthaltenen Elementen. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts. SAP haftet nicht für entstandene Schäden. Dies gilt u. a. und uneingeschränkt für konkrete, besondere und mittelbare Schäden oder Folgeschäden, die aus der Nutzung dieser Materialien entstehen können. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit. Die gesetzliche Haftung bei Personenschäden oder Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab. SAP AG 2006, Perforce Best SAP / C. Loff & T. Kroll / 46