Data Center Automation with SUSE Manager Federal Deployment Agency Bundesagentur für Arbeit Data Center Automation Project

Transcription

1 Data Center Automation with SUSE Manager Federal Deployment Agency Bundesagentur für Arbeit Data Center Automation Project Ralf Bueker DSE Bundesagentur für Arbeit

2 What is a DSE? DSE means Designated Support Engineer Designated to one customer Onside 4 days a week Working embedded in the customers IT team Takes care of and escalates support issues Interface for the customer to SUSE Assist the customer in ongoing project 2

3 The Customer: Federal Employment Agency The Federal Employment Agency (Bundesagentur für Arbeit: BA) is the largest provider of labor market services in Germany, with more than 800 branch offices nationwide. BA is a governmental organization based in Nuremberg, Germany Employs ~100,000 people IT- staff ~2,100 internal and external admins and engineers BA drives ~100 different tailored applications (VAM, ALG2, ZELOS. ALEGRO ) for their customers 3

4 The Task The BA software systems needs to be changed and upgraded according to social laws Obtaining the laws requires a lot of development and testing A test server should match perfectly in existing infrastructure. No solution from scratch The allocation of test HW had been identified as a bottleneck (~6 weeks waiting period for appropriate HW) 4

5 DCA Challenges No common API / Authorization / CMDB New data added in every step of the process No consistent server classification (templates) Complex software stack needs to be applied Solution should match for Linux and Windows Servers should seamlessly match in the current admin/ update/ monitoring infrastructure For Linux: ARS (Remedy) tool (end user GUI) UC4 (task management) Soap Service SUSE Manager / Python script VMware ESX 5

6 6 Workflow

7 SUSE Manager and the Python script SUSE Manager Configuration XML and Schema AutoYaST customization Cobbler to create profiles and boot ISO ESX Server communication Monitoring installation 7

8 SUSE Manager Configuration Create Distribution Create profile(s) Create configuration channel Create software channel Create activation key(s) /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api Install DCA Python scripts 8

9 SUSE Manager Profile / AutoYaST

10 SUSE Manager Configuration Create Distribution Create profile(s) Create configuration channel Create software channel Create activation key(s) /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api Install DCA Python scripts 10

11 11 Configuration Channel

12 12 Configuration Files

13 SUSE Manager Configuration Create Distribution Create profile(s) Create configuration channel Create software channel Create activation key(s) /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api Install DCA Python scripts 13

14 14 Software Channel

15 SUSE Manager Configuration Create Distribution Create profile(s) Create configuration channel Create software channel Create activation key(s) /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api Install DCA Python scripts 15

16 16 Activation Key

17 SUSE Manager Configuration Create Distribution Create profile(s) Create configuration channel Create software channel Create activation key(s) /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api Install DCA Python scripts 17

18 Python Script: Class CreateServer Main program Installation flow control Reports back to soap service (back channel) Other helper functions Error Handler Logger Function 18

19 XML and Schema Parameters and settings can easily be checked and restricted by an appropriate XML schema. All parameters used during the installation from different programs can be collected and mapped in the schema The schema rules deliver a readable and evaluable description of data and objects for all different integrated applications Python delivers with lxml a very good library to handle xml and schema 19

20 20 Schema

21 Python Script Class XmlInputHandler XML_input XmlParseInput XmlValidateInput XmlGetEntry apply ".//" to search string to enforce search from root XmlGetSubTree FormatXML Used to create index for multivalued Attributes needed in more than one output xml (nic, disc) 21

22 22 Input file

23 Python Script Class PrepareIsoInstallation MountEsx Mount ESX Drive for iso Images XmlGetKopts Collects information for Installation Kernel Options GetDistro Gets the SUSE Manager distro from server class provided CopyToEsx Obvious 23

24 Python Script Class EsxHandler 1 / 2 VMware vcenter provides an API for its services ( For DCA this API has been published and partly customized as a webservice. Sending an xml to the webservice triggers requested function. Install server Clone install server Delete server PowerOn esx_data (publish returnvalues from ESX) 24

25 Python Class EsxHandler 2 / 2 SendToEsx: used to send xml to ESX Server InputStringDelete, InputStringVmCreate, InputStringCloneCreate, InputStringPowerOn, InputStringQueryTask: creates xml input for ESX Tasks FormatXml: parses answers from ESX (not necessary xml ) check_esx_output: processes ESX output ExtractTaskId: Gets ESX task ID for status query getmac: gets the randomly created MAC address(es) for the new virtual machine from ESX output 25

26 26 ESX Input

27 Python Class SumaHandler 1/2 OpenSumaConnection (key = client.auth.login(username, password, timeout) GetServerSumaId ((client.system.getid(key, servername))) DeleteServerFromSuma (client.system.deletesystem(systemid)) RebootServer (client.system.schedulereboot(key, sumaid, now) RunRemoteBashScript client.system.schedulescriptrun(key, sumaid, "root", "root", 5, script, now) 27

28 Python Class SumaHandler 2/2 OsaRefresh subprocess.call ([curl, -Sks, " + self. suma_server + "/rhn/systems/details/overview.do?sid=" + str(sumaid) + "&ping=1"]) list_system_events (client.system.listsystemevents(key, msumaid)) get_server_dmi (client.system.getdmi(key, sumaid)) CloseSumaConnection (client.auth.logout(key)) 28

29 Python Class AutoYaSTHandler AutoYaSTGetAutoYaSTfile: gets appropriate autoyast file template from /var/lib/rhn/kickstarts/upload/ AutoYaSTGetParameterList: get the parameter list from the AutoYaST template AutoYaSTSetMetaData: creates a dictionary from AutoYaST parameter and values from XML input file AutoYaSTCreateNewAyfile : creates a specific AutoYaST file for the new server 29

30 Class CobblerHandler 1/2 Cobbler creates the iso image for the installation. To allow automatic boot boot.template has to be rewritten. This is implemented in a class with following functions: CobblerBuildProfile: creates profile profile_id = server.new_profile(token) server.modify_profile(profile_id, 'name', bcpname, token) server.modify_profile(profile_id, 'distro', bcpdistro, token) server.modify_profile(profile_id, 'kopts', bcpkopts, token) server.modify_profile(profile_id, 'kickstart', bcpkickstart, token) server.save_profile(profile_id, token) 30

31 Class CobblerHandler 2 / 2 cobbler_build_iso_template: customizes the /etc/cobbler/iso/boot.template, so that the new iso is booted automatically Build_iso_template cobbler_build_iso: build iso image and copies to ESX server (currently nfs share) Could be done by xmlrpc (see: pydoc cobbler.api on SuMa Server), currently uses cmdline cobbler buildiso commandline on Suma Server cobbler_delete_profile_if_exist: obivous 31

32 32 Boot.template

33 Class BootstrapMonitor 1 / 3 FirstPing: waits till the new server is pingable for the first time SumaReg: queries SUSE Manager if the new Server is already registered SumaHandler.GetServerSumaId WaitForBootstrapEnd: SumaHandler.RunRemoteBashScript ConfigDone: uses the same as above to see if all configuration from the channel is completed SumaHandler.RunRemoteBashScript 33

34 Class BootstrapMonitor 2 / 3 CheckConfigLog: analyzes configuration on the new server (return value remote script cat /var/adm/autoinstall/logs/bootstrap.sh.log had any errors LastBoot: SumaHandler.RebootServer FinalUp: SumaHandler.get_server_dmi RemoteCleanUp: cleans all the remains from installation from the server SumaHandler.RunRemoteBashScript 34

35 Your Server is ready! Sending a success message to the SOAP Webservice SOAP webservice forwards message to UC4 Next Step in workflow can be triggered (I.E. DB installation) Clean up the SUSE Manager and DCA environment (Deleting the profiles of the new Server, iso files ) 35

36 Operating Experience DCA has been introduced to the internal customer The SUSE Manager implementation won against the VMware cloning approach Several hundred DCA installations have already performed successfully Stress tests have been performed (we delivered 20 Server in 30 minutes instead of 6 weeks ) The customer wants to develop DCA further A cloud software like SUSE Cloud should replace the homegrown SUSE Manager Python implementation 36

37 Lessons Learned 1 / 2 Local caches and infrastructure play an important role when doing live provisioning in a productive environment Error handling is vital part of process management and should be considered carefully A schema is a very useful to control and parameters from participating applications Automatic logging of every single step in the automatic process is vital! 37

38 Lessons Learned 2 / 2 We showed the customer: complex installation and configuration can be described flexible in SUSE Manager server classification can be mapped in SUSE Manager An existing administration concept can be represented in SUSE Manager Thanks to the API SUSE Manager can flexibly used in a complex environment for several tasks 38

39 Corporate Headquarters Maxfeldstrasse Nuremberg Germany (Worldwide) Join us on: 39

40 Unpublished Work of SUSE LLC. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.