High Availability Services with SAS Grid Manager

Transcription

1 White Paper High Availability Services with SAS Grid Manager August 29, 2011

2 Table f Cntents Intrductin... 4 Scpe... 4 Terms... 4 Service Resurce Redundancy... 5 Active-Passive Redundancy... 5 Active-Active Redundancy... 7 Service Resiliency... 8 EGO Service Cntrller... 8 Managing HA Services with Platfrm RTM fr SAS... 8 Lading HA Cnfiguratin... 9 Defining HA Applicatins Applying HA Cnfiguratin Mnitring and Cntrlling HA Applicatins Service Relcatin IP Lad Balancer (Hardware) EGO Service Directr (Sftware) EGO Service Directr withut Dynamic Updates f Crprate DNS Server EGO Service Directr with Dynamic Updates f Crprate DNS Server Allwed Cmparisn Hardware VS Sftware SAS Cnfiguratin fr HA Services Cnfiguring Virtual Hstnames Initial SAS Deplyment Mdifying an Existing SAS Deplyment SAS Service Startup Scripts Helper Scripts Defining SAS Services in EGO Service Cntrller Cnfiguratin Verifying HA Service Cnfiguratin Testing Service Resurce Redundancy Testing Service Resiliency and Relcatin Appendix A: EGO Service Cntrller Manual Cnfig EGO Service Cntrller Definitins Mnitring and Cntrlling HA Services Appendix B: EGO Service Directr Cnfiguratin

3 EGO Service Cntrller Definitin EGO Service Directr DNS Server Cnfiguratin Withut Crprate DNS Updates Crprate DNS Dynamic Updates fr Lcatin f EGO Service Directr DNS Only Crprate DNS Dynamic Updates fr Lcatin f all EGO HA Services

4 Intrductin - High Availability Services with SAS Grid Manager Intrductin In a SAS envirnment, there are certain services that need t always be available and accessible. These services are vital t the running applicatins and their ability t prcess SAS jbs. Example services include: SAS Metadata Server SAS Object Spawner Platfrm Prcess Manager Platfrm Grid Management Service Web Applicatin Tier Cmpnents High Availability fr these services can be achieved using the EGO capabilities prvided by the Platfrm Suite fr SAS which is a third party cmpnent included with the SAS Grid Manager prduct. Althugh Platfrm Suite fr SAS can be licensed withut SAS Grid Manager the EGO High Availability capabilities require a SAS Grid Manager license. In rder fr SAS Grid Manager t prvide High Availability fr SAS services, the fllwing aspects must be addressed: Service Resurce Redundancy Prvide alternate resurces fr executin f essential services. Allwing critical functins t execute n multiple physical r virtual ndes eliminates a single pint f failure fr the grid. Service Resiliency Prvide a mechanism t mnitr the availability f services n the grid and t autmatically restart a failed service n the same resurce r n an alternate when necessary. Service Relcatin Prvide a methd fr allwing access t services running n the grid withut requiring clients t have any knwledge f the physical lcatin f the service. Scpe This dcument addresses the requirements fr implementing High Availability (HA) services running in a SAS grid envirnment using the EGO capabilities f Platfrm Suite fr SAS which is included with SAS Grid Manager. Cnfiguratin examples are included that prvide details fr cnfiguring essential SAS services t be Highly Available in the grid. Terms High Availability (HA) A system design prtcl and assciated implementatin that ensures a certain abslute degree f peratinal cntinuity during a given measurement perid. Availability refers t the ability f the user cmmunity t access the system, whether t submit new wrk, update r alter existing wrk, r cllect the results f previus wrk. If a user cannt access the system, it is said t be unavailable. Failver The capability t mve ver autmatically t a redundant r standby resurce upn the failure r abnrmal terminatin f a previusly active applicatin r service. Switchver/Migrate The capability t mve ver t a redundant r standby resurce upn manual user request. Fault Tlerance The capability t cntinue prper peratin after a failure has ccurred. Platfrm Enterprise Grid Orchestratr (EGO) Cmpnent f Platfrm Suite fr SAS that enables High Availability. It is autmatically enabled when Platfrm Suite fr SAS is installed and cnfigured. IP Lad Balancer A hardware device that is used t distribute IP cnnectins acrss multiple hsts. Virtual Hstname A name r alias that is substituted fr the actual hstname t be accessed. Dmain Name System (DNS) System fr translating human readable hstnames int numerical identifiers assciated with netwrking equipment fr the purpse f lcating and addressing these devices. Fully Qualified Dmain Name (FQDN) The unambiguus name fr a hst which specifies the system hstname alng with all dmain levels up t and including the tp level dmain. 4

5 High Availability Services with SAS Grid Manager Service Resurce Redundancy Service Resurce Redundancy The first cnsideratin fr making any service Highly Available within the grid is t ensure that redundant resurces are available fr executin f the service. Resurces within the grid might becme unavailable in the event f a fault. Sme failures that can impact grid resurces include: Pwer utages Failed CPU, RAM, Disks, Netwrk Interface Cards, etc Over-temperature related shutdwn Security breaches Applicatin, middleware, and perating system failures Grid resurces culd als pssibly be taken ffline manually by an administratr fr maintenance, but n matter the reasn fr a resurce being ffline, Service Resurce Redundancy allws alternate resurces within the grid t be used aviding service utages which wuld disrupt the grid s ability t execute jbs. Fr the grid t be truly HA, any critical services must be allwed t execute n tw r mre ndes within the grid, and nte that nly the LSF server ndes are eligible t be executin resurces fr HA services. Fr SAS services there are tw pssible redundancy schemes t cnsider: Active-Passive Redundancy The service will execute n nly ne nde at any given time but alternate (standby) resurces are available fr executing the service. Active-Active Redundancy The service will execute n multiple ndes cncurrently with each service instance prviding full capabilities t cnsumers. Active-Passive Redundancy When using an active-passive (cld standby) redundancy scheme multiple ndes are capable f executing the service but nly ne instance f the service is executing at any given time. The nrmal executin nde, referred t as the primary; the first alternate nde, is the secndary; ptinal secnd alternate, wuld be the tertiary. By default the service will be started n the primary resurce when it is available and the alternates remain in cld standby; meaning the service is nt cncurrently executing n multiple resurces and it will nly be started n an alternate when the primary becmes unavailable. If a service is started n an alternate resurce and the primary is recvered and becmes available, the service will nt be autmatically mved back t the primary (nn-revertive failver); the recvered primary becmes the first alternate (secndary) and remains in cld standby. NOTE: The example cnfiguratins presented in this dcument will nly include primary and secndary resurces, but if tertiary resurces are required refer t Appendix A fr cnfiguratin details. 5

6 Service Resurce Redundancy High Availability Services with SAS Grid Manager A active-passive redundancy scheme is pictured belw in Figure 1 which includes tw critical services cnfigured acrss three grid ndes. Figure 1: Example Active-Passive Resurce Redundancy Scheme Metadata Server Primary Secndary Grid Nde 1 Grid Nde 2 Grid Nde 3 Primary Secndary Object Spawner The Metadata Server is cnfigured t primarily run n Grid Nde 1, but can failver t Grid Nde 3. The Object Spawner has Grid Nde 2 as its primary resurce, but can als failver t Grid Nde 3. This is a simple example but illustrates hw under nrmal cnditins the grid resurces wuld be fully used, but prviding Resurce Redundancy can increase the availability f vital services within the grid if resurces becme unavailable allwing the grid t cntinue executing jbs withut disruptin. When deciding which grid resurces will be used as alternates fr critical services it is imprtant t cnsider the ptential utilizatin f resurces within the grid when a service is started n an alternate nde. Fr example, utilizing the scheme shwn abve, if Grid Nde 2 becmes unavailable Grid Nde 3 will be executing the Object Spawner. If instead Grid Nde 1 were chsen as standby fr the Object Spawner, Grid Nde 1 wuld then be prviding bth services. Executing bth services might increase the lad n Grid Nde 1 t at r abve 100%, which quite pssibly culd degrade the grid s ability t execute jbs. Again, a simple example, but it is imprtant t develp an ptimal redundancy scheme fr the critical grid services such that the grid can cntinue t functin after a fault has ccurred. Fr the grid t prvide redundancy fr any given service, the applicatin prviding the service must be able t execute n each nde cnfigured t prvide the service. This requires that all necessary sftware be installed r smehw accessible and apprpriately licensed, plus all required applicatin cnfiguratin and data must be available and prperly synchrnized. Using the grid s shared strage fr the SAS cnfiguratin simplifies much f the redundancy setup, and the examples presented in this dcument assume this apprach is used. 6

7 High Availability Services with SAS Grid Manager Service Resurce Redundancy Active-Active Redundancy In sme deplyments fr certain SAS services an active-passive redundancy scheme isn t the preferred cnfiguratin. Fr example, a SAS Object Spawner which implements lad balancing prvides Resurce Redundancy fr the service in an active-active cnfiguratin; meaning the service executes n multiple resurces cncurrently. This type f redundancy scheme can still use sme aspects f the HA functinality prvided by the SAS Grid Manager t imprve service availability. It is pssible that sme services still use active-passive redundancy while thers are cnfigured fr activeactive redundancy. An example f such a mixed redundancy scheme is shwn belw in Figure 2. NOTE: Althugh mst f the discussin and examples in this dcument will be based n a activepassive redundancy scheme, any knwn differences in cnfiguratin fr an active-active scheme will be nted where applicable. Figure 2: Example Mixed Resurce Redundancy Scheme Metadata Server Primary Secndary Grid Nde 1 Grid Nde 2 Grid Nde 3 Grid Nde 4 Instance 1 Instance 2 Instance 3 Object Spawner As with the previus example, the Metadata Server is cnfigured t primarily run n Grid Nde 1, but in this layut it wuld failver t Grid Nde 2. The Object Spawner will have three cncurrent instances running n Grid Ndes 2, 3, and 4. Again this is a simple example but it illustrates hw active-passive and active-active redundancy can be cmbined t increase service availability while als achieving the desired resurce utilizatin. 7

8 Service Resiliency High Availability Services with SAS Grid Manager Service Resiliency T increase service availability there needs t be sme facility t mnitr executin f the services within the grid and t autmatically restart failed services r mve them t an alternate resurce when necessary. The resiliency, r fault tlerance, f services within the grid is handled by the EGO Service Cntrller. Each service that will be mnitred within the grid must be defined in the EGO Service Cntrller cnfiguratin. NOTE: It is nt required fr a service t have Resurce Redundancy t take advantage f the resiliency prvided by the EGO Service Cntrller. Even when a service will nly execute n a single nde within the grid the EGO Service Cntrller can be used t prvide executin resiliency fr the service. T add such a service t the EGO Service Cntrller cnfiguratin set bth the Primary Hst and Failver Hst t be the same nde. NOTE: All machines used t run services nly, thse that d nt cntribute cmputing pwer t the grid, fr which SAS Grid Manager will prvide High Availability are cunted in the ttal number f cres fr the SAS Grid Manager license. A machine dedicated t running the SAS Metadata Server fr which SAS Grid Manager will prvide High Availability is a gd example. EGO Service Cntrller The EGO Service Cntrller is a Platfrm EGO cmpnent that is always executed n the EGO master hst (Grid Cntrl Server) and is respnsible fr starting and mnitring cnfigured services. It ensures that all the defined services are running by detecting failures and restarting service instances. The EGO Service Cntrller reads cnfiguratin files fr the services that must be instantiated. These cnfiguratin files are XML dcuments which cntain the service definitin that prvides details such as hw t start and stp the service, parameters describing types f resurces the service may require, and s n. There is ne service definitin file dedicated t each service t be managed by the EGO Service Cntrller. These files can be created and edited using Platfrm RTM fr SAS, but in deplyment scenaris where this utility cannt be used the service definitins can als be created manually. The examples in this sectin were generated using Platfrm RTM fr SAS versin 2.06; fr instructins n manually creating EGO Service Cntrller definitins refer t Appendix A. Managing HA Services with Platfrm RTM fr SAS Platfrm RTM fr SAS is a web-based peratinal dashbard fr the SAS grid envirnment. It prvides capabilities fr cnfiguring, mnitring, and cntrlling many aspects f the grid envirnment including HA services. Defining an HA Applicatin inside RTM will create the necessary definitin fr the service s that the EGO Service Cntrller will begin mnitring and managing the executin f the service within the grid. The exact steps fr creating HA Applicatins within RTM are detailed in the fllwing sectins. 8

9 High Availability Services with SAS Grid Manager Service Resiliency Lading HA Cnfiguratin Befre defining HA Applicatins inside RTM, an HA cnfiguratin must be created fr the cluster that is t be managed. The actin within RTM t be perfrmed is *Lad HA Cnfiguratin Frm Cluster which will read in any existing EGO Service Cntrller definitins and create the initial cnfiguratin inside RTM. Even if n EGO Service Cntrller definitins exist this peratin is still required, but the new cnfiguratin will be empty with N Cnfigured Applicatins. Referring t the figure belw, perfrm the fllwing steps t lad the HA cnfiguratin: 1) Select cnfig frm the tab list at the tp f the screen. 2) On the left-hand menu, under the Grid Management sectin select HA. 3) In the peratin pane, select *Lad HA Cnfiguratin Frm Cluster and then answer yes when prmpted fr cnfirmatin

10 Service Resiliency High Availability Services with SAS Grid Manager Once the cnfiguratin has been laded, the peratin pane will refresh and a table shwing the newly created cnfiguratin will appear. Defining HA Applicatins Once an HA cnfiguratin has been created, t edit the cnfiguratin and define HA Applicatins click n the Cnfiguratin Name link. The RTM HA cnfiguratin is initially created with N Cnfigured Applicatins as shwn in the HA Applicatins table belw; t create a service definitin required fr the EGO Service Cntrller, select the Add link n the far right f the header rw in the HA Applicatins table. 10

11 High Availability Services with SAS Grid Manager Service Resiliency The Applicatin Edit frm will be displayed, this frm is used t create the EGO Service Cntrller definitin fr the service. The imprtant settings t be entered in this frm are described belw: Applicatin Name Shrt name, r alias, fr this service. Primary Hst / Failver Hst Select the primary and secndary grid ndes used fr executin f this service. Executin Cmmand [Start] / [Stp] Cmmands t start and stp the service; these cmmands must be executable n bth the primary and failver hst using the exact same cmmand line. Executin User and Passwrd User and passwrd fr executin f the service; the specified user must have the necessary permissins t start and stp this service. Dependency Optinal; specifies the Applicatin Name f any ther EGO services that must be running befre this service can be started. Once all the desired settings have been entered int this frm, click n the create buttn t submit the infrmatin and then answer yes when prmpted fr cnfirmatin t cmplete adding the service definitin. 11

12 Service Resiliency High Availability Services with SAS Grid Manager The imprtant service definitin settings fr sme critical SAS services are discussed belw. Refer t Defining SAS Services in EGO Service Cntrller Cnfiguratin fr details n hw t better determine the exact settings t be used. SAS Metadata Server Fr the SAS Metadata Server, the key settings fr the Applicatin Edit frm are as fllws: Applicatin Name Any service alias; e.g. sasmeta Primary Hst / Failver Hst Select the primary and secndary grid ndes cnfigured t execute the Metadata Server. Executin Cmmand [Start] / [Stp] Start path t level cnfig@/sasmeta/metadataserver/metadataserver.sh start Stp path t level cnfig@/sasmeta/metadataserver/metadataserver.sh stp path t level cnfig@ is replaced with the full directry path n the nde where the SAS cnfiguratin level is lcated. Start (Windws) T Be Added Stp (Windws) T Be Added Executin User and Passwrd e.g. sas Dependency Nne SAS Object Spawner Fr the SAS Object Spawner, the key settings fr the Applicatin Edit frm are as fllws: Applicatin Name Any service alias; e.g. sasbjspawner Primary Hst / Failver Hst Select the primary and secndary grid ndes cnfigured t execute the Object Spawner. NOTE: When cnfiguring Object Spawner fr active-active redundancy using lad balancing, create multiple HA Applicatins, ne fr each nde that will execute the Object Spawner, and set bth the Primary Hst and Failver Hst t be the same nde. Executin Cmmand [Start] / [Stp] Start path t level cnfig@/objectspawner/objectspawner.sh start Stp path t level cnfig@/objectspawner/objectspawner.sh stp path t level cnfig@ is replaced with the full directry path n the nde where the SAS cnfiguratin level is lcated. Start (Windws) T Be Added Stp (Windws) T Be Added Executin User and Passwrd e.g. sas Dependency Applicatin Name fr SAS Metadata service; e.g. sasmeta 12

13 High Availability Services with SAS Grid Manager Service Resiliency Applying HA Cnfiguratin Once all the desired applicatins have been defined inside RTM, the cnfiguratin must be applied t the cluster fr the settings t take effect. In the tp prtin f the peratin pane, select the link *Apply HA Cnfiguratins T Cluster. Once the HA cnfiguratin has been applied, the peratin pane will be redisplayed with a Status message. If the cnfiguratin was crrectly applied t the cluster, a message similar t the ne shwn belw shuld be displayed. 13

14 Service Resiliency High Availability Services with SAS Grid Manager Mnitring and Cntrlling HA Applicatins RTM als includes mnitring and cntrl capabilities fr HA services. T see the current status r t change the state f any defined service, perfrm the fllwing: 1) Select HA frm the tab list at the tp f the screen. 2) On the left-hand menu, under the Dashbards sectin select Applicatins

15 High Availability Services with SAS Grid Manager Service Resiliency The Applicatin List table shws the current Status f each defined service and ther varius bits f imprtant infrmatin including the Running Hst, r the nde which is currently executing the service within the grid. Frm this screen a service can be stpped, started, restarted, r migrated. Migrating a service will manually switchver the service t the alternate redundant nde. 15

16 Service Relcatin High Availability Services with SAS Grid Manager Service Relcatin Services that are bth redundant and resilient are technically Highly Available as they can recver frm faults and transitin t alternate executin resurces when necessary. The service will be available a high percentage f the time, but exactly hw wuld a client determine where the service is currently lcated if it can be mved arund between different grid ndes? It makes little difference if the service is HA if it can t be accessed by clients after it is restarted n a secndary resurce. In rder t lcate an HA service within the grid there must be sme mechanism t allw access t a running service withut requiring clients t have knwledge f which grid nde is currently executing it. This aspect f High Availability is quite pssibly the mst imprtant and the mst cmplex. Fr Service Relcatin within the grid there are tw different appraches t cnsider: IP Lad Balancer (Hardware) hardware based slutin which requires the additin f an IP Lad Balancing switch. EGO Service Directr (Sftware) sftware based slutin that utilizes the EGO Service Directr included with Platfrm Suite fr SAS. With either slutin the fundamental cncepts are the same: Define a virtual hstname fr the services that are t be HA within the grid. Any client wishing t access an HA grid service must use the virtual hstname. The virtual hstname is reslved t the current physical lcatin f the service within the grid. The methd used t reslve the virtual hstname t the service s current physical lcatin is cmpletely hidden frm the client; it is exactly the same as any ther hst t which a cnnectin is t be made. Hwever, the detail f hw virtual hstnames are reslved is where the tw slutins differ. IP Lad Balancer (Hardware) The IP Lad Balancer apprach requires additinal hardware t be installed and cnfigured fr the grid, thus this is frequently referred t as the Hardware apprach. In sme deplyments the required hardware may already be present in the netwrk allwing the grid t leverage existing infrastructure. Figure 3 belw shws a lgical netwrk diagram with a lad balancer cnfigured with a single virtual hstname (e.g. gridha.example.cm). In this cnfiguratin the clients wuld cnnect using the virtual hstname gridha.example.cm, this hstname reslves t the IP address f the lad balancer switch. Based n the lad balancer cnfiguratin, the switch wuld then decide which physical hst is available t respnd t client requests. When services are failed ver r migrated t redundant resurces by the EGO Service Cntrller the lad balancer is respnsible fr detecting the change and ruting new cnnectins t the crrect destinatin. 16

17 High Availability Services with SAS Grid Manager Service Relcatin Figure 3: IP Lad Balancer Lgical Netwrk Client WAN IP Lad Balancer LAN Virtual Hstname: gridha.example.cm Grid Nde 1 Grid Nde 3 The fllwing example scenari describes hw a service wuld be relcated during failver. A client attempts a cnnectin t gridha.example.cm prt number The IP Lad Balancer has been cnfigured t mnitr prt 8561 n Grid Nde 1 and Grid Nde 3 and redirect cnnectins t either prt; i.e. cmmn Rund Rbin scheduling. The SAS Metadata server is currently executing n Grid Nde 1 listening n prt 8561, while Grid Nde 3 is the secndary resurce and currently is nt executing the service. The IP Lad Balancer will redirect all incming cnnectins fr prt 8561 t Grid Nde 1. Grid Nde 1 is taken ffline by a fault r manually by an administratr. The IP Lad Balancer detects the disruptin and traffic destined fr prt 8561 is mmentarily disrupted as there is n destinatin available. EGO Service Cntrller starts up the SAS Metadata Server n Grid Nde 3. IP Lad Balancer detects that prt 8561 is nw in-service n Grid Nde 3. The IP Lad Balancer will redirect all incming cnnectins fr prt 8561 t Grid Nde 3. The client establishes a new cnnectin t the SAS Metadata Server nw executing n Grid Nde 3. 17

18 Service Relcatin High Availability Services with SAS Grid Manager EGO Service Directr (Sftware) The EGO Service Directr is a Platfrm EGO service that functins as a lcating mechanism fr ther Platfrm EGO services. The EGO Service Directr cntains a stand-alne DNS server, which can serve as the authritative name server fr the EGO DNS subdmain and respnd t DNS queries fr EGO HA services. The EGO Service Directr relies n the EGO Service Cntrller t prvide lcatin infrmatin and state change ntificatins f service instances. Each service which is defined in the EGO Service Cntrller cnfiguratin will be assciated with a DNS recrd which includes a virtual hstname, r alias, fr the service. When a service instance begins executin, enters the RUN state, the EGO Service Directr will add a crrespnding DNS recrd with the service s current lcatin infrmatin. When a service instance transitins frm RUN t any ther state, the EGO Service Directr will delete the assciated DNS recrd. When the lcatin f the EGO Service Directr DNS server r any ther EGO HA service is changed, the crrespnding DNS recrd in the DNS database is updated with the IP address fr the new lcatin. In rder fr clients t recgnize this address change in a timely manner it is crucial that every attempt t reslve an EGO service s virtual hstname receive the mst up t date infrmatin. Fr this reasn, the Time-T-Live (TTL) setting fr EGO virtual hstnames and perating system DNS caching mechanisms can negatively impact the verall time required t realize EGO service relcatin. The TTL setting fr EGO virtual hstnames shuld be tuned t a marginally lw duratin as the ttal failver time is directly prprtinal, (e.g. ttal failver time = time t detect service failure + time t initialize redundant service + time fr client t realize service relcatin). The virtual hstnames fr the EGO HA services that are dynamically updated by the EGO Service Directr will always be in a subdmain f the crprate DNS dmain. Fr instance, if the crprate dmain were example.cm all virtual hstnames fr EGO HA services by default wuld be in the subdmain eg.example.cm. The name f the EGO subdmain can be cnfigured as required, but the dynamic virtual hstnames fr the EGO services must always be inside the EGO subdmain. The hst prtin f the virtual hstname fr an EGO service will be the Applicatin Name assigned t this service in the EGO Service Cntrller definitin. S fr example, if the EGO Service Cntrller were t define an applicatin named sasmeta, the EGO Service Directr will track the lcatin f this service and dynamically update the virtual hstname sasmeta.eg.example.cm. The fllwing example scenari describes hw a service wuld be relcated during failver. The SAS Metadata server is currently executing n Grid Nde 1, while Grid Nde 3 is the secndary resurce and currently is nt executing the service. A client attempts a cnnectin t sasmeta.eg.example.cm. The client s DNS query fr the virtual hstname sasmeta.eg.example.cm will return the IP address f Grid Nde 1. Grid Nde 1 is taken ffline by a fault r manually by an administratr. EGO Service Cntrller infrms EGO Service Directr f the service state change and the EGO Service Directr dynamically deletes the DNS recrd fr sasmeta.eg.example.cm. Any DNS queries fr the virtual hstname sasmeta.eg.example.cm will return Hst Unknwn as there is currently n destinatin available. EGO Service Cntrller starts up the SAS Metadata Server n Grid Nde 3. EGO Service Cntrller infrms EGO Service Directr f the service state change and EGO Service Directr dynamically adds a DNS recrd fr sasmeta.eg.example.cm with the IP f Grid Nde 3. DNS queries fr the virtual hstname sasmeta.eg.example.cm will nw return the IP address f Grid Nde 3. Clients can recnnect t the SAS Metadata Server nw executing n Grid Nde 3. 18

19 High Availability Services with SAS Grid Manager Service Relcatin When utilizing the EGO Service Directr fr Service Relcatin within the grid there is ne key decisin that will determine which f the pssible setups can be used: Will the EGO Service Directr be allwed t send dynamic updates t the crprate DNS server? N The nly pssible cnfiguratin is described belw in EGO Service Directr withut Dynamic Updates f Crprate DNS Server. Yes There are tw pssibilities; refer t EGO Service Directr with Dynamic Updates f Crprate DNS Server Allwed belw fr mre details. The EGO Service Directr cannt be cnfigured with Platfrm RTM fr SAS, please refer t Appendix B fr detailed cnfiguratin instructins and examples fr each f the setups described belw. 19

20 Service Relcatin High Availability Services with SAS Grid Manager EGO Service Directr withut Dynamic Updates f Crprate DNS Server In this cnfiguratin the EGO Service Directr DNS is setup t master a subdmain, r subzne, which will be used t reslve the lcatin f virtual hstnames fr the HA services within the grid. The crprate DNS server will frward all queries fr any f the virtual hstnames within that subdmain t the EGO Service Directr DNS server. It is imprtant that the crprate DNS server be cnfigured with multiple Name Server recrds fr the EGO subdmain, ne fr each f the redundant ndes that can pssibly execute the EGO Service Directr DNS server. The detailed flw f a DNS query fr an EGO service virtual hstname is shwn belw in Figure 4. Figure 4: EGO Service Directr N Crprate DNS Updates (2) Frward query (2a) 1st Name Server (1) DNS query Service Directr DNS Server *(2b) 2nd Name Server Crprate DNS Server (4) Respnse (3) Respnse Update lcatin f EGO services Client Service Directr (5) Access Service Cntrller cntrl & mnitr Service Hst B EGO kernel Service Hst A (Master) Hst n In the example abve, when the query is frwarded t the EGO Service Directr DNS, step 2, the flw includes sub-steps 2a and 2b. This is shwn t highlight the fact that the EGO Service Directr itself is cnfigured t be HA s there will be redundant resurces cnfigured t execute the service and this means the crprate DNS server must have multiple Name Server recrds cnfigured fr the EGO subdmain. On initial startup and after an EGO Service Directr failver r switchver, the crprate DNS server can t pssibly knw which Name Server is active and s it may be required t frward the query t multiple servers befre receiving a respnse. The figure nly depicts tw Name Servers fr the EGO subdmain, but if there are mre redundant ndes cnfigured fr the EGO Service Directr, the query culd quite pssibly need t be frwarded t each f these ndes (i.e. max frwards = number f EGO Service Directr redundant ndes). 20

21 High Availability Services with SAS Grid Manager Service Relcatin EGO Service Directr with Dynamic Updates f Crprate DNS Server Allwed The EGO Service Directr s jb is t dynamically update the lcatin f all the services under the cntrl f the EGO Service Cntrller, and allwing these updates t be sent directly t the crprate DNS server will prduce mre efficient DNS queries. The tw distinct pssibilities fr these updates are detailed in the fllwing sectins. Crprate DNS Updates: EGO Service Directr DNS Server Lcatin nly Since the EGO Service Directr itself is cnfigured t be a resilient service managed by the EGO Service Cntrller, its lcatin wuld be dynamically updated if the service were t failver r be manually switched t a different hst. Allwing the lcatin f the EGO Service Directr DNS server t be dynamically updated in the crprate DNS server will eliminate the ptential inefficiencies f having multiple Name Server recrds fr the EGO subdmain. Fr this cnfiguratin t be pssible, the virtual hstname f the EGO Service Directr DNS server must reside inside the crprate dmain, e.g. if the crprate dmain were example.cm and the EGO subdmain eg.example.cm, the EGO Service Directr virtual hstname wuld be smething like egnameserver.example.cm. Just as with the previus cnfiguratin, the EGO Service Directr DNS is setup t master a subdmain which is used t reslve the lcatin f virtual hstnames fr all EGO services within the grid, the nly difference in this setup is that the crprate DNS server is cnfigured with nly a single Name Server recrd fr the EGO subdmain. This Name Server recrd is cnfigured using the virtual hstname f the EGO Service Directr DNS server which will be dynamically updated in the crprate DNS with the current lcatin f this service. Details fr a DNS query are shwn belw in Figure 5. 21

22 Service Relcatin High Availability Services with SAS Grid Manager Figure 5: EGO Service Directr Crprate DNS Updates fr EGO Service Directr DNS Server (1) DNS query Service Directr DNS Server (2) Frward query (3) Respnse Crprate DNS Server (4) Respnse Update lcatin f EGO services Update lcatin f Service Directr DNS Server Client Service Directr (5) Access Service Cntrller cntrl & mnitr Service Hst B EGO kernel Hst A (Master) Service Hst n 22

23 High Availability Services with SAS Grid Manager Service Relcatin Crprate DNS Updates: Lcatin f All EGO HA Services In bth f the previus setups, the crprate DNS server must delegate reslutin fr the EGO virtual hstnames t the EGO Service Directr DNS server, but in mst scenaris the EGO subdmain will nly cntain a small handful f hstnames. The efficiency f the DNS queries culd be substantially imprved if the crprate DNS server knew the latest lcatin infrmatin fr the EGO services and culd respnd with authrity t queries fr the EGO virtual hstnames. Details fr a DNS query with this setup are shwn belw in Figure 6. Figure 6: EGO Service Directr Crprate DNS Updates fr All HA Services (1) DNS query Service Directr DNS Server Crprate DNS Server (2) Respnse Update lcatin f EGO Services Client Service Directr (3) Access Service Cntrller cntrl & mnitr Service Hst B EGO kernel Hst A (Master) Service Hst n The EGO Service Directr DNS server is shwn in the figure as it is still required, but it wuld nt receive any dynamic updates and wuld nt be invlved with any DNS queries. In this setup the EGO subdmain is mastered by the crprate DNS server s it isn t necessary t invlve the EGO Service Directr DNS when querying an EGO virtual hstname. This wuld be the mst efficient setup as far as DNS queries are cncerned, but there is ne ptential pitfall with this cnfiguratin. As previusly nted, the DNS Time-T-Live fr the EGO virtual hstnames must be set lw such that clients will recgnize lcatin changes fr services and nt be delayed by DNS name caching, but in this cnfiguratin the EGO Service Directr DNS server is nt in cntrl f reslving these hstnames. This means that the TTL fr the EGO virtual hstnames is entirely cntrlled by the crprate DNS server and must be tuned crrectly r failver times fr EGO HA services culd be unacceptably high. The details fr tuning TTL in the crprate DNS servers are beynd the scpe f this dcument as varius DNS server sftware will have differing methds, but if using the abve setup make certain t cnsider this issue in the cnfiguratin. 23

24 Service Relcatin High Availability Services with SAS Grid Manager Cmparisn Hardware VS Sftware Bth f the appraches fr Service Relcatin have their pluses and minuses, and exactly which f these slutins might be best fr any given grid deplyment really must be decided n a case by case basis. Table 1 belw cntains a pint by pint cmparisn f the tw slutins: Table 1: Prs and Cns f Service Relcatin Alternatives IP Lad Balancer (Hardware) Advantages Single IP address; n IP address change during failver, clients and dependent services can maintain r reestablish cnnectins after failver. Quick failver; switching incming traffic t the failver hst is nearly instantaneus. Minimal risk fr enterprise netwrk disruptin as the cnfiguratin changes are lcalized t the LAN segment cntaining the IP Lad Balancer. Less sftware cmplexity; fewer third party cnfiguratin steps required fr SAS deplyment. N cnflicts with crprate IT plicies. Disadvantages Mre expensive; additinal hardware t purchase and maintain. EGO Service Directr (Sftware) Disadvantages Multiple IP addresses; IP address change during failver requires sme clients and dependent services t be restarted. Ptentially slwer failver; DNS Time-T-Live, prpagatin f DNS updates, and crprate DNS setups can cause unexpected failver delay. Attempts t minimize failver time with imprper DNS cnfiguratin tuning culd cause netwrk delays fr the entire enterprise. Increased sftware cmplexity; additinal third party cnfiguratin steps required fr SAS deplyment. Pssible cnflicts with crprate IT plicies; many enterprises have strict plicies disallwing usage f secndary DNS servers. Advantages Less expensive; n additinal hardware t purchase and maintain. EGO Service Directr is included with SAS Grid Manager license csts. Full fault tlerance requires an HA slutin fr the IP Lad Balancer; culd increase ttal cst and add cmplexity t the verall slutin. EGO Service Directr leverages the EGO functinality t prvide full fault tlerance with zer added expense. Bth slutins are ttally viable and the right ptin must be determined based n the specific custmer s IT requirements. 24

25 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services SAS Cnfiguratin fr HA Services In rder t take full advantage f the High Availability f services within the grid all clients and dependent services must be crrectly cnfigured. The necessary cnfiguratin changes are t make certain that all references t EGO HA services use the virtual hstnames created fr the services. Cnfiguring Virtual Hstnames As discussed in the previus sectin, depending n the slutin used fr Service Relcatin the virtual hstname fr an HA service can either be the hstname assigned t the IP Lad Balancer r the alias fr the service within the EGO subdmain. N matter which slutin is used t relcate a service all SAS cnfiguratin must use the apprpriate virtual hstname, never the IP address r the real hstname f a single grid nde. Initial SAS Deplyment When cnfiguring HA during the SAS deplyment the virtual hstnames fr the HA services can be entered directly int the Deplyment Wizard. The sectins belw prvide sme helpful tips when initially deplying imprtant SAS HA services. Examples are prvided fr bth Service Relcatin slutins. IP Lad Balancer (Hardware) When using an IP Lad Balancer fr HA Service Relcatin use the hstname assigned t the IP Lad Balancer in the SAS cnfiguratin when referencing any f the HA services. 25

26 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager SAS Metadata Server When entering the SAS Metadata Server cnnectin infrmatin use the hstname f the IP Lad Balancer fr Hst Name as shwn belw. This is the nly difference frm a nrmal SAS deplyment fr cnfiguring SAS Metadata Server as HA. 26

27 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services SAS Object Spawner NOTE: T cnfigure the SAS Object Spawner as HA, the Prmpting Level must be set t Custm, else the dialg shwn belw will nt be presented. Enter the Object Spawner infrmatin using the hstname f the IP Lad Balancer fr Hst Name. Prceed t verify that all SAS lgical servers being deplyed that utilize this Object Spawner are cnfigured with Hst Name being crrectly set the same as the Object Spawner, (using the hstname f the IP Lad Balancer). A few examples are included belw; nt all the example cmpnents, r pssibly additinal cmpnents, may be included with any specific deplyment. 27

28 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager Mdify the Server Cmpnent Name t highlight the fact that this Spawner is nt assciated with a single server. This is nt required, but recmmended; the name entered can be anything desired. The SAS Wrkspace Server must have Hst Name set the same as the Object Spawner. 28

29 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services The Metadata Utility Wrkspace Server must have Hst Name set the same as the Object Spawner. The SAS Pled Wrkspace Server must have Hst Name set the same as the Object Spawner. 29

30 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager The SAS Stred Prcess Server must have Hst Name set the same as the Object Spawner. 30

31 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services EGO Service Directr (Sftware) When using the EGO Service Directr fr HA Service Relcatin a different virtual hstname is created fr every service which is defined in the EGO Service Cntrller cnfiguratin. This allws the services t be independently switched t alternate resurces, but des require the specific virtual hstnames t be crrectly referenced in the SAS cnfiguratin. Sme example SAS cnfiguratins are included belw. SAS Metadata Server When entering the cnnectin infrmatin use the EGO subdmain virtual hstname fr Hst Name f the SAS Metadata Server as shwn in the example belw. This is the nly difference frm a nrmal SAS deplyment when cnfiguring SAS Metadata Server as HA. NOTE: In rder fr the deplyment wizard t cmplete successfully the Hst Name entered here must crrectly reslve, but since the EGO Service Cntrller cannt start the service until after the deplyment has been cmpleted, this presents a bit f a dilemma. One simple slutin is t temprarily add an entry int the lcal server s netwrk hsts file fr the virtual hstname f the SAS Metadata Server making certain t remve these entries nce the deplyment is cmpleted. A preferable slutin wuld be t add a DNS A recrd fr the virtual hstname f the SAS Metadata Server inside the EGO subdmain assigning the IP address f the deplyment hst; refer t Appendix B fr mre details n this pssibility. 31

32 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager SAS Object Spawner NOTE: T cnfigure the SAS Object Spawner as HA, the Prmpting Level must be set t Custm, else the dialg shwn belw will nt be presented. Enter the Object Spawner infrmatin using the EGO subdmain virtual hstname fr Hst Name. NOTE: In rder fr the deplyment wizard t cmplete successfully the Hst Name entered here must crrectly reslve, but since the EGO Service Cntrller cannt start the service until after the deplyment has been cmpleted, this presents a bit f a dilemma. One simple slutin is t temprarily add an entry int the lcal server s netwrk hsts file fr the virtual hstname f the SAS Object Spawner making certain t remve these entries nce the deplyment is cmpleted. A preferable slutin wuld be t add a DNS A recrd fr the virtual hstname f the SAS Object Spawner inside the EGO subdmain assigning the IP address f the deplyment hst; refer t Appendix B fr mre details n this pssibility. Prceed t verify that all SAS lgical servers being deplyed that utilize this Object Spawner are cnfigured with Hst Name being crrectly set the same as the Object Spawner, using the EGO subdmain virtual hstname. A few examples are included belw; nt all the example cmpnents, r pssibly additinal cmpnents, may be included with any specific deplyment. 32

33 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services Mdify the Server Cmpnent Name t highlight the fact that this Spawner is nt assciated with a single server. This is nt required, but recmmended; the name entered can be anything desired. The SAS Wrkspace Server must have Hst Name set the same as the Object Spawner. 33

34 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager The Metadata Utility Wrkspace Server must have Hst Name set the same as the Object Spawner. The SAS Pled Wrkspace Server must have Hst Name set the same as the Object Spawner. 34

35 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services The SAS Stred Prcess Server must have Hst Name set the same as the Object Spawner. Mdifying an Existing SAS Deplyment T Be Added 35

36 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager SAS Service Startup Scripts The scripts used t startup the SAS server prcesses will need t be adjusted t accmmdate the HA services. There are several cnsideratins t take int accunt when making adjustments t these scripts, the fllwing sectins discuss the necessary mdificatins and give details fr making these changes. T simplify management f the service dependencies it is highly recmmended that all services running within the grid be managed by the EGO Service Cntrller. Fr services that will nt be HA, meaning they can nly execute n a single resurce, allwing the EGO Service Cntrller t manage these services can still prvide service resiliency even withut resurce redundancy. When defining nn-ha services within the EGO Service Cntrller cnfiguratin select the single executin nde fr bth the Primary Hst and the Failver Hst. The discussin that fllws, alng with the prvided examples, assumes all SAS services are being managed by the EGO Service Cntrller. Helper Scripts The examples included in this sectin make references t utility scripts which have been published t assist with integrating the suggested HA cnfiguratin changes; these scripts are available fr dwnlad frm: A general descriptin and an example usage fr the helper scripts are shwn belw. UNIX Scripts eg_server.sh This script will be used when interacting with the EGO Service Cntrller t start, stp, and status managed services. /shared/cnfig/eg_server.sh start sasbjspawner server_wrap.sh This script culd be used when creating the EGO Service Cntrller definitin; wrapping the Executin Cmmand [Start] t keep the service s start functin frm exiting. This may be necessary if the nrmal server cntrl script start functin (typically will frk daemn prcesses and exit) results in the EGO Service Cntrller declaring the service as failed. /shared/cnfig/server_wrap.sh /shared/cnfig/lev1/sasmeta/metadataserver/metadataserver.sh Windws Scripts T Be Added Defining SAS Services in EGO Service Cntrller Cnfiguratin All SAS services will need t be defined in the EGO Service Cntrller cnfiguratin. T easily determine the full list f services t be added including the executin user that shuld be used t cntrl the services and the executin cmmands that shuld be used t start and stp each service, refer t the sas.servers script in the SAS cnfiguratin level directry. UNIX Example The relevant excerpts frm an example sas.servers script are shwn belw. SERVERUSER=sas #***** # Cmmands passed t the server cntrl scripts #***** STARTCMD=start STOPCMD=stp 36

37 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services start_metadata_server() { $METADATA_SERVER_DIR/MetadataServer.sh $STARTCMD >/dev/null 2>&1 } start_bject_spawner() { $OBJECT_SPAWNER_DIR/ObjectSpawner.sh $STARTCMD >/dev/null 2>&1 } stp_metadata_server() { $METADATA_SERVER_DIR/MetadataServer.sh $STOPCMD >/dev/null 2>&1 } stp_bject_spawner() { $OBJECT_SPAWNER_DIR/ObjectSpawner.sh $STOPCMD >/dev/null 2>&1 } start_servers () { METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer METADATA_SERVER_LOGS=/shared/cnfig/Lev1/SASMeta/MetadataServer/Lgs METADATA_SERVER_CONTEXT=SASMeta if [ -x "$METADATA_SERVER_DIR/MetadataServer.sh" ]; then { start_metadata_server; #ech back frm start meta 1 if [ "$?" -eq 0 ]; then { $Lgmsg "SAS Metadata Server 1 is UP"; } else { $Lgmsg "SAS Metadata Server 1 is NOT up" $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; } fi } else { $Lgmsg "Culd nt access SAS Metadata Server 1 script" $Lgmsg "SAS Metadata Server 1 is NOT up" $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; } fi OBJECT_SPAWNER_DIR=/shared/cnfig/Lev1/ObjectSpawner OBJECT_SPAWNER_LOGS=/shared/cnfig/Lev1/ObjectSpawner/Lgs OBJECT_SPAWNER_CONTEXT=NULL if [ -x "$OBJECT_SPAWNER_DIR/ObjectSpawner.sh" ] then 37

38 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager } { start_bject_spawner; if [ "$?" -eq 0 ]; then $Lgmsg "SAS Object Spawner 1 is UP"; else $Lgmsg "SAS Object Spawner 1 is NOT up"; fi; } else { $Lgmsg "Culd nt access SAS Object Spawner 1 script" $Lgmsg "SAS Object Spawner 1 is NOT up" } fi stp_servers () { OBJECT_SPAWNER_DIR=/shared/cnfig/Lev1/ObjectSpawner OBJECT_SPAWNER_LOGS=/shared/cnfig/Lev1/ObjectSpawner/Lgs if [ -x "$OBJECT_SPAWNER_DIR/ObjectSpawner.sh" ]; then stp_bject_spawner; fi; METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer METADATA_SERVER_LOGS=/shared/cnfig/Lev1/SASMeta/MetadataServer/Lgs if [ -x "$METADATA_SERVER_DIR/MetadataServer.sh" ]; then stp_metadata_server; fi; } Windws Example T Be Added The functin start_servers will cntain a blck f cde fr starting each service and there will be a separate start_<server_name> functin that actually initializes the service. Likewise, the functin stp_servers will cntain a blck f cde fr stpping each service and there will be a separate stp_<server_name> functin t actually shutdwn the service. Each service listed in the sas.servers script will need t be smehw added t the EGO Service Cntrller cnfiguratin. There are three pssibilities fr this integratin: Independent Service All service transitins (start, stp, and restart) can be perfrmed withut cnsideratin fr the current state f any ther service. Dependent Service One r mre service transitins (start, stp, and restart) must cnsider the current state f anther service. Service Grup The service is bundled with ne r mre ther services that will be transitined tgether as a grup. Independent Service When a service is cmpletely independent all that is required is t create an EGO Service Cntrller definitin (refer t Defining HA Applicatins fr instructins) using the values frm the sas.servers script. Use the value f SERVERUSER fr the Executin User, the cmmand line that wuld be 38

39 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services executed by the start_<server_name> functin as the Executin Cmmand [Start], and the cmmand line frm the stp_<server_name> functin as the Executin Cmmand [Stp]. Dependent Service The scripts fr starting up the SAS servers must accunt fr the pssible dependencies between the varius services; these can be generally classified in tw categries: Startup Dependency The service cannt initialize until the dependency service is available. Restart Dependency The service nt nly requires the dependency service t be available at initializatin but als requires restarting if the dependency service is restarted. Startup Dependency When defining services within the EGO Service Cntrller cnfiguratin dependencies can be specified and these will be enfrced when the services are started; meaning the services will always be initialized in dependency rder. What the EGO Service Cntrller desn t accunt fr hwever, is the time required t fully initialize the required services. Fr example, when the SAS Metadata Server is cnfigured as a dependency fr Object Spawner, the EGO Service Cntrller will always start the Metadata Server befre the Object Spawner, but if the Metadata Server takes time t initialize befre it begins listening fr cnnectins, when the Object Spawner is started by the EGO Service Cntrller it might nt be able t cnnect t the Metadata Server as expected. Fr this type f startup sequencing the EGO Service Cntrller dependencies d nt prvide an adequate slutin and a different apprach must be cnsidered. NOTE: When timing cnstraints are nt a cnsideratin the startup dependencies can be handled by specifying all dependencies when creating the EGO Service Cntrller definitin fr the service. T handle the sequencing f startup dependencies the recmmended slutin is t create a startup script fr the tp-level service which prvides the dependency and have that script als start all the dependent services. In rder fr this t functin crrectly the EGO Service Cntrller definitins fr each f the dependent services must be manually edited t change <sc:starttype> frm AUTOMATIC t MANUAL as this is nt a supprted peratin in Platfrm RTM fr SAS (refer t Appendix A: EGO Service Cntrller Manual Cnfig fr mre details). Changing each f the dependent services s that they are nt autmatically started by the EGO Service Cntrller will allw this tp-level script t start these services nly after their dependency is fully initialized. Once the services are started the EGO Service Cntrller will mnitr their executin and restart them if necessary, prviding the expected Service Resiliency. The EGO Service Cntrller definitin fr the tp-level service will cntinue t have <sc:starttype> f AUTOMATIC, but it must be mdified t change the Executin Cmmand [Start] and Executin Cmmand [Stp] t utilize this newly created script that will sequence the dependencies. T create the tp-level script t sequence the startup dependencies fllw the steps belw: 1. Start with a cpy f the sas.servers script frm the SAS cnfiguratin level directry. 2. Decide which services will need t be gruped; it s pssible that nt all services listed in the sas.servers script have the same dependency. Fr any service that isn t dependent, remve all references t that service frm the new script. If multiple dependency grups are necessary create multiple cpies f the sas.servers script and mdify each cpy as needed. 3. Fr the tp-level service, leave the existing start_<server_name> and stp_<server_name> functins untuched alng with the crrespnding blcks within the start_servers, stp_servers, and server_status functins. 4. Fr each dependent service, make the fllwing mdificatins: Lcate the related blck fr the service within the start_servers functin. Replace the entire blck with smething similar t the fllwing: "${@CMD_EGOSERVER@}" 39

40 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager Where is the full path t the eg_server.sh helper script described abve in Helper Scripts is the EGO Service Cntrller Applicatin Name fr the service t be started. Lcate the related blck fr the service within the stp_servers functin. Replace the entire blck with smething similar t the fllwing: "${@CMD_EGOSERVER@}" is the full path t the eg_server.sh helper script described abve in Helper Scripts is the EGO Service Cntrller Applicatin Name fr the service t be stpped. Lcate the related blck fr the service within the server_status functin. Replace the entire blck with smething similar t the fllwing: "${@CMD_EGOSERVER@}" is the full path t the eg_server.sh helper script described abve in Helper Scripts is the EGO Service Cntrller Applicatin Name fr the service t be queried. UNIX Example The relevant excerpts frm an example mdified sas.servers script are shwn belw which demnstrates hw t sequence the startup f the SAS Metadata Server and the Object Spawner. start_servers () { METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer METADATA_SERVER_LOGS=/shared/cnfig/Lev1/SASMeta/MetadataServer/Lgs METADATA_SERVER_CONTEXT=SASMeta if [ -x "$METADATA_SERVER_DIR/MetadataServer.sh" ]; then { start_metadata_server; #ech back frm start meta 1 if [ "$?" -eq 0 ]; then { $Lgmsg "SAS Metadata Server 1 is UP"; } else { $Lgmsg "SAS Metadata Server 1 is NOT up" $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; } fi } else { $Lgmsg "Culd nt access SAS Metadata Server 1 script" $Lgmsg "SAS Metadata Server 1 is NOT up" $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; } fi # Start Object Spawner /shared/cnfig/eg_server.sh start sasbjspawner 40

41 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services # Start any additinal dependent EGO services } stp_servers () { # Stp Object Spawner /shared/cnfig/eg_server.sh stp sasbjspawner # Stp any additinal dependent EGO services METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer METADATA_SERVER_LOGS=/shared/cnfig/Lev1/SASMeta/MetadataServer/Lgs if [ -x "$METADATA_SERVER_DIR/MetadataServer.sh" ]; then stp_metadata_server; fi; } server_status() { $Lgmsg "SAS servers status:" METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer if [ -f $METADATA_SERVER_DIR/server.pid ]; then { pid=`cat $METADATA_SERVER_DIR/server.pid` kill -0 $pid >/dev/null 2>&1 if [ $? -eq 0 ]; then $Lgmsg "SAS Metadata Server 1 (pid $pid) is running..." else $Lgmsg "SAS Metadata Server 1 is stpped" fi } else $Lgmsg "SAS Metadata Server 1 is stpped" fi # Status Object Spawner /shared/cnfig/eg_server.sh status sasbjspawner # Status any additinal dependent EGO services } Windws Example T Be Added Restart Dependency T handle restart dependencies a service wuld either need t detect when its required services are restarted and restart itself r the initializatin f the tp-level service must accunt fr restarting each f its dependents. Since it can t be assumed a service wuld be able t detect the restart f a dependency, the recmmended apprach is the latter. T manage the restart dependencies use the same apprach as fr sequencing startup dependencies described in the previus sectin. It is imprtant t nte when and hw the tp-level service will be restarted, as this can be easily 41

42 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager cnfused with the restart functinality prvided by the sas.servers script. The EGO Service Cntrller will use the service s Executin Cmmand [Start] t initialize the service, including when restarting the service after it has failed n the current hst and after a failver t an alternate hst. Since the EGO Service Cntrller definitin cnfigures the Executin Cmmand [Start] t execute the mdified sas.servers script with start as the argument, the service is nt being restarted, frm the script s perspective it will always execute start functinality. S, if any service has a restart dependency n anther service the sequencing scripts must make certain the dependent service is stpped befre the start f the tp-level service can begin. Using the previusly mdified cpy f sas.servers as a starting pint, make the fllwing additinal mdificatins: 1. Decide which services have the restart dependency; it s pssible that sme services have nly a startup dependency while thers actually have the restart requirement. It is imprtant nt t just assume that all services need t be restarted when any dependency is restarted; if the restart isn t required it intrduces unnecessary additinal service dwntime, s dn t d it. 2. Lcate the blck in the start_servers functin that starts the tp-level service; immediately befre this blck, fr each service that must be restarted add a cmmand t stp the dependent service; smething similar t the fllwing: "${@CMD_EGOSERVER@}" is the full path t the eg_server.sh helper script described abve in Helper Scripts is the EGO Service Cntrller Applicatin Name fr the service t be stpped. UNIX Example Excerpts frm an example mdified sas.servers script are shwn belw. start_servers () { # If Object Spawner is running, it must be stpped /shared/cnfig/eg_server.sh stp sasbjspawner # Stp any additinal dependent EGO services METADATA_SERVER_DIR=/shared/cnfig/Lev1/SASMeta/MetadataServer METADATA_SERVER_LOGS=/shared/cnfig/Lev1/SASMeta/MetadataServer/Lgs METADATA_SERVER_CONTEXT=SASMeta if [ -x "$METADATA_SERVER_DIR/MetadataServer.sh" ]; then { start_metadata_server; #ech back frm start meta 1 if [ "$?" -eq 0 ]; then { $Lgmsg "SAS Metadata Server 1 is UP"; } else { $Lgmsg "SAS Metadata Server 1 is NOT up" $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; } fi } else { $Lgmsg "Culd nt access SAS Metadata Server 1 script" $Lgmsg "SAS Metadata Server 1 is NOT up" 42

43 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services } fi $Lgmsg "The remaining SAS servers will NOT be started as a result." return 1; # Start Object Spawner /shared/cnfig/eg_server.sh start sasbjspawner # Start any additinal dependent EGO services } Windws Example T Be Added Service Grups In certain deplyment scenaris it may be required r desirable t keep a grup r cluster f services always executing tgether n the same nde, while High Availability is still needed fr these services. This will be an exceptin t the previus recmmendatin f defining all SAS services individually within the EGO Service Cntrller cnfiguratin as the gruped services are nt desired t be cntrlled independently. T supprt this scenari, the service gruping will be defined within the EGO Service Cntrller cnfiguratin as a single service and then wuld be cntrlled by a single grup cntrl script. This new script is similar t the sequencing script frm the previus sectin, the difference is that nne f the services being cntrlled are individually defined in the EGO Service Cntrller cnfiguratin; meaning all the services referenced within the new script will be started exactly as they wuld be in a nrmal SAS deplyment rather than using EGO specific cmmands. T create a service grup, fllw the steps belw: 1. Start with a cpy f the sas.servers script frm the SAS cnfiguratin level directry. 2. Fr any service that will nt be part f the service grup remve all references t that service frm the new script. If multiple service grups are necessary create multiple cpies f the sas.servers script and mdify each cpy as needed. It is nt necessary t mdify any f the ther lgic within the script, simply minimize the list f services being cntrlled by this script t thse services being gruped. 3. Create a new EGO Service Cntrller definitin fr the service grup with the Executin Cmmand [Start] and Executin Cmmand [Stp] bth cnfigured t execute this new script. Verifying HA Service Cnfiguratin Testing Service Resurce Redundancy The fllwing test scenaris can be used t verify that services are able t be executed n each f the redundant ndes. The Service Startup and Shutdwn tests shuld be cmpleted successfully befre beginning the EGO HA cnfiguratin. Service Startup and Shutdwn 1. On ne f the redundant ndes cnfigured t execute the service use the cmmand line specified in the EGO Service Cntrller definitin as Executin Cmmand [Start] t startup the service. If this is active-active redundancy, repeat this step fr each f the redundant ndes. 43

44 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager 2. Once service startup has been verified, use the cmmand line specified in the EGO Service Cntrller definitin as Executin Cmmand [Stp] t shut dwn the service. If this is activeactive redundancy, repeat this step fr each f the redundant ndes. 3. Verify access t the service using a SAS client. Fr example, t verify Metadata Server the SAS Management Cnsle culd be used t cnnect. 4. If this is active-passive redundancy, repeat steps 1 and 2 fr each f the redundant ndes. Testing Service Resiliency and Relcatin The fllwing test scenaris can be used t verify that the EGO Service Cntrller will crrectly detect service failure and restart services as required, and that when service executin mves t an alternate resurce the service will be accessible by clients and dependent services as expected. Service Restart Restart n Current Nde Verify access t the service using a SAS client. Fr example, t verify Metadata Server the SAS Management Cnsle culd be used t cnnect. Kill the service prcesses r use the cmmand line specified in the EGO Service Cntrller definitin as Executin Cmmand [Stp] t shut dwn the service. Verify the service is crrectly restarted n the same nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Re-verify access t the service using a SAS client. Failver t Alternate Nde NOTE: These scenaris d nt apply t services using active-active redundancy. Shutdwn the primary nde while it is currently executing the service. Verify the service is crrectly started n the standby nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Verify the Service Relcatin t the secndary nde by accessing the service using a SAS client. Fr example, t verify the relcatin f Metadata Server the SAS Management Cnsle culd be used t cnnect. If there is a tertiary resurce, shutdwn the standby while it is currently executing the service (while leaving the primary nde ffline). Verify the service is crrectly started n the tertiary nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Verify the Service Relcatin t the tertiary nde by accessing the service using a SAS client. Bring the primary nde back nline and shutdwn the service n the secndary r tertiary by killing the service prcesses r using the cmmand line specified in the EGO Service Cntrller definitin as Executin Cmmand [Stp] t shut dwn the service. Verify the service is crrectly restarted n the primary nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Verify the Service Relcatin t the primary nde by accessing the service using a SAS client. Service Switchver/Migratin Using Platfrm RTM fr SAS NOTE: These scenaris d nt apply t services using active-active redundancy. 44

45 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services RTM includes capabilities t manually switchver, r migrate, HA services t a redundant nde. T access the HA Applicatin migratin functinality, perfrm the fllwing: 1) Select HA frm the tab list at the tp f the screen. 2) On the left-hand menu, under the Dashbards sectin select Applicatins. 1 2 Verify the service Status is ACTIVE and that the Running Hst is the primary nde. T migrate the service t the secndary nde, perfrm the fllwing: 1) Check the actin bx in the far right clumn next t the service t be migrated. 2) Frm the Chse an actin pull-dwn menu, select Migrate. 3) Click n the g buttn Enter the LSF Administratr passwrd if necessary and answer yes when prmpted fr cnfirmatin t begin the service migratin. Once the migratin has been submitted, the peratin pane will be redisplayed with a Status message. If the migratin was initiated successfully, a message similar t the ne shwn belw shuld be displayed. 45

46 SAS Cnfiguratin fr HA Services High Availability Services with SAS Grid Manager Once the service begins executing n the secndary nde, the next peratin pane refresh will update with the rw fr the particular service highlighted in yellw as shwn belw; this is nrmal and emphasizes the fact that this service is nw executing n a resurce ther than its cnfigured primary. Verify the Service Relcatin t the secndary nde by accessing the service using a SAS client. Fr example, t verify the relcatin f Metadata Server the SAS Management Cnsle culd be used t cnnect. NOTE: T return service executin t the primary nde, repeat the service migratin steps. Using egsh cmmand line NOTE: These scenaris d nt apply t services using active-active redundancy. T execute the cmmands shwn belw the user must have LSF administratr privilege. The egsh utility requires a separate lgin and t change service states r make any changes the lgin must be an administrative user. T lgn using the default administratr accunt execute the fllwing: egsh user lgn -u Admin -x Admin Verify that the service is currently executing n the primary nde; use the service list cmmand t verify the service is in the STARTED state and the RESOURCE is crrectly listed as the primary nde. egsh service list l Manually clse the primary resurce. egsh resurce is the hstname f the primary nde. Stp and then restart the service. Make certain t pause after stpping the service t give sufficient time fr service shutdwn. Use the service list cmmand t verify the service is in the DEFINED state befre attempting t restart. egsh service egsh service is the Applicatin Name assigned t this service in the EGO Service Cntrller definitin. The service shuld be restarted n the secndary nde; use the service list cmmand t verify the service is in the STARTED state and the RESOURCE is crrectly listed as the secndary nde. Verify the service is crrectly started n the standby nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. 46

47 High Availability Services with SAS Grid Manager SAS Cnfiguratin fr HA Services Verify the Service Relcatin t the secndary nde by accessing the service using a SAS client. Fr example, t verify the relcatin f Metadata Server the SAS Management Cnsle culd be used t cnnect. If there is a tertiary nde cnfigured fr the service, manually clse the secndary resurce, leaving the primary als clsed. egsh resurce is the hstname f the secndary nde. Stp and then restart the service again. Make certain t pause after stpping the service t give sufficient time fr service shutdwn. Use the service list cmmand t verify the service is in the DEFINED state befre attempting t restart. The service shuld be restarted n the tertiary nde; use the service list cmmand t verify the service is in the STARTED state and the RESOURCE is crrectly listed as the tertiary nde. Verify the service is crrectly started n the tertiary nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Verify the Service Relcatin t the tertiary nde by accessing the service using a SAS client. Once the service startup has been verified, prepare t restart the service back n the primary nde by first pening bth the primary and secndary resurces. egsh resurce egsh resurce are the hstnames f the primary and secndary ndes respectively. Stp and then restart the service again. Make certain t pause after stpping the service t give sufficient time fr service shutdwn. Use the service list cmmand t verify the service is in the DEFINED state befre attempting t restart. The service shuld be restarted n the primary nde; use the service list cmmand t verify the service is in the STARTED state and the RESOURCE is crrectly listed as the primary nde. Verify the service is crrectly started n the primary nde. If the service has any restart dependent services, verify that all the dependents are crrectly restarted. If the service is part f a service grup, verify that all the grup services are available. Verify the Service Relcatin t the primary nde by accessing the service using a SAS client. 47

48 Appendix A High Availability Services with SAS Grid Manager Appendix A: EGO Service Cntrller Manual Cnfig NOTE: The examples belw make reference t LSF and EGO specific envirnment variables; it is assumed the LSF envirnment setup has been surced in the cnfiguratin envirnment. EGO Service Cntrller Definitins T utilize the EGO Service Cntrller fr service resiliency a definitin must be created fr each service t be managed. As previusly discussed, Platfrm RTM fr SAS is the easiest methd f creating these definitins, but in sme deplyment scenaris RTM might nt be available. RTM als desn t supprt sme advanced EGO Service Cntrller definitin ptins that might be needed t meet specific deplyment requirements; tertiary redundant server r additinal envirnment variables are tw examples. In these situatins the service definitins can be manually edited and added t the EGO Service Cntrller cnfiguratin. The EGO Service Cntrller definitins are lcated in the fllwing cnfiguratin directry: ${EGO_ESRVDIR}/esc/cnf/services There is a template EGO Service Cntrller definitin file, service.xml.tmpl, lcated in the abve directry, the cntents are shwn belw: <?xml versin="1.0" encding="utf-8"?> <sc:servicedefinitin xmlns:sc=" xmlns:eg=" xmlns:xsi=" xmlns:xsd=" xsi:schemalcatin=" ServiceName="Dummy"> <sc:versin>1.2</sc:versin> <sc:descriptin></sc:descriptin> <sc:mininstances>1</sc:mininstances> <sc:maxinstances>5</sc:maxinstances> <sc:pririty>10</sc:pririty> <sc:maxinstancesperslt>1</sc:maxinstancesperslt> <sc:maxinstancesperhst>1</sc:maxinstancesperhst> <sc:needcredential>false</sc:needcredential> <sc:cntrlplicy> <sc:starttype>manual</sc:starttype> <sc:maxrestarts>10</sc:maxrestarts> <sc:hstfailverinterval>pt60s</sc:hstfailverinterval> <sc:dependency type="onstart">db</sc:dependency> </sc:cntrlplicy> <sc:allcatinspecificatin> <eg:cnsumerid>/sampleapplicatins/eclipsesamples</eg:cnsumerid> <!-- The ResurceType specifies a "cmpute element" identified by the URI used belw - -> <sc:resurcespecificatin ResurceType=" <eg:resurcegrupname>cmputehsts</eg:resurcegrupname> <eg:resurcerequirement>linux86</eg:resurcerequirement> </sc:resurcespecificatin> </sc:allcatinspecificatin> <sc:activitydescriptin> <eg:attribute name="hsttype" type="xsd:string">linux86</eg:attribute> <eg:activityspecificatin> <eg:cmmand>sleep 60</eg:Cmmand> <eg:executinuser>sampleosuser</eg:executinuser> 48

49 High Availability Services with SAS Grid Manager Appendix A <eg:wrkingdirectry>/tmp</eg:wrkingdirectry> <eg:umask>0777</eg:umask> </eg:activityspecificatin> </sc:activitydescriptin> <sc:activitydescriptin> <eg:attribute name="hsttype" type="xsd:string">ntx86</eg:attribute> <eg:activityspecificatin> <eg:cmmand>sleep 60</eg:Cmmand> <eg:executinuser>sampleosuser</eg:executinuser> <eg:wrkingdirectry>c:\</eg:wrkingdirectry> <eg:umask>0777</eg:umask> </eg:activityspecificatin> </sc:activitydescriptin> <sc:activitydescriptin> <eg:attribute name="hsttype" type="xsd:string">all</eg:attribute> <eg:activityspecificatin> <eg:cmmand>sleep 60</eg:Cmmand> <eg:executinuser>sampleosuser</eg:executinuser> <!-- executable t cntrl/shutdwn service instance --> <eg:jbcntrller>/path/cmd</eg:jbcntrller> <!-- time in secnds t wait fr jb cntrl t take effect, service instance will be killed if it is already terminated after wait perid --> <eg:cntrlwaitperid>pt10s</eg:cntrlwaitperid> <eg:wrkingdirectry>/tmp</eg:wrkingdirectry> <eg:umask>0777</eg:umask> </eg:activityspecificatin> </sc:activitydescriptin> </sc:servicedefinitin> The template itself shws sme f the advanced cnfiguratin pssibilities that can t be accmplished thrugh the RTM user interface. The imprtant sectins f the file are highlighted in bld, and these settings are further discussed belw. ServiceName Set this t a shrt alias fr the service. If the EGO Service Directr is used fr Service Relcatin this will be the hst cmpnent f the virtual hstname fr the service. Descriptin Set this t a shrt descriptin fr the service. MaxInstances Set this 1 when using active-passive redundancy scheme r t the desired number f cncurrently executing services when using active-active scheme. StartType Change this t AUTOMATIC if this service is t be autmatically started by the EGO Service Cntrller. Dependency If there are ther services defined t be managed by the EGO Service Cntrller that must be started befre this service, include ne Dependency clause fr each f these services with the value equal t the ServiceName frm the ther service definitin. If n dependencies are needed, delete r cmment ut the example line. CnsumerID Change this value t /ManagementServices/LSFManagementServices which is the value required fr RTM t recgnize this definitin. ResurceGrupName Update this t ManagementHsts if the service is t be executed nly n LSF master candidate ndes, r leave the setting f CmputeHsts if the service can execute n any grid cmpute nde. ResurceRequirement This setting can be used as written t allw the service t be executed n any grid nde meeting specific hsttype, but mst ften shuld be updated t list 49

50 Appendix A High Availability Services with SAS Grid Manager the specific redundant ndes which are cnfigured t execute the service. An example is shwn belw: <eg:resurcerequirement> select('@hsta@' '@hstb@') rder('@hstb@') </eg:resurcerequirement> is the FQDN f the primary executin hst is the FQDN f the secndary. If a tertiary hst is required, as fllws: select('@hsta@' '@hstb@' '@hstc@') rder(1*'@hsta@' + 2*'@hstB@' + 3*'@hstC@') If using an active-active redundancy scheme list all the ndes that will cncurrently execute the service. Fr example, if the SAS Object Spawner is cnfigured fr lad balancing acrss fur ndes the select statement wuld lk like the fllwing: select('@hsta@' '@hstb@' '@hstc@' '@hstd@') The rder clause can be mitted in this case as all ndes will execute the service cncurrently. ActivityDescriptin The template includes a fairly cmplex example f hw the service can be cnfigured with different parameters fr starting and stpping which depend n the hsttype f the executin nde. This culd be required in hetergeneus grid envirnments, but in general the ActivityDescriptin frm the template can ften be simplified t smething similar t the fllwing: <sc:activitydescriptin> <eg:attribute name="hsttype" type="xsd:string">all</eg:attribute> <eg:activityspecificatin> <eg:cmmand>@start_command@</eg:cmmand> <eg:jbcntrller>@stop_command@</eg:jbcntrller> <eg:executinuser>@user_account@</eg:executinuser> <eg:envirnmentvariable name="ego_app_type">rtm_ha</eg:envirnmentvariable> <eg:envirnmentvariable name="ego_service_ver">@app_ver@</eg:envirnmentvariable> <eg:envirnmentvariable name="@var_name@">@var_value@</eg:envirnmentvariable> <eg:cntrlwaitperid>pt30s</eg:cntrlwaitperid> <eg:umask>0022</eg:umask> </eg:activityspecificatin> </sc:activitydescriptin> with the cmmand line required t start the service. with the cmmand line required t stp the service. with the user name n the server that shuld be used fr executin. Add the RTM required envirnment variables EGO_APP_TYPE and EGO_SERVICE_VER with the sftware versin fr the service being defined. If any specific envirnment variables are needed t execute Cmmand r JbCntrller, these can be defined as shwn, ne EnvirnmentVariable clause per required setting. with the name f variable with the assciated value. Mnitring and Cntrlling HA Services Once the services have been defined in the EGO Service Cntrller cnfiguratin, the cmmand line utility egsh can be used t mnitr and cntrl the services. The egsh utility requires a separate lgin and t change service states r make any changes the lgin must be an administrative user. T lgin using the default administratr accunt execute the fllwing: # egsh user lgn -u Admin -x Admin 50

51 High Availability Services with SAS Grid Manager Appendix A Anytime a new service definitin is created r mdificatins are made t an existing definitin the EGO Service Cntrller must be recnfigured t pick up the changes. This is accmplished by restarting the LIM n the EGO master hst using the cmmand: # egsh eg restart -f NOTE: The restart cmmand will start up any services that have StartType set t AUTOMATIC regardless f the state the service was in when the restart was executed. The fllwing sessin capture shws sme useful cmmands fr wrking with services. # egsh service list SERVICE STATE ALLOC CONSUMER RGROUP RESOURCE SLOTS SEQ_NO INST_STATE ACTI sbatchd DEFINED /Cluster Intern ServiceD DEFINED /Managem Manage sasbjsp DEFINED /Managem Manage res DEFINED /Cluster Intern WebServi DEFINED /Managem Manage sasmeta DEFINED /Managem Manage # egsh service start sasmeta Starting service <sasmeta>. Run <service view> t ensure startup # egsh service list SERVICE STATE ALLOC CONSUMER RGROUP RESOURCE SLOTS SEQ_NO INST_STATE ACTI sbatchd DEFINED /Cluster Intern ServiceD DEFINED /Managem Manage sasbjsp DEFINED /Managem Manage res DEFINED /Cluster Intern WebServi DEFINED /Managem Manage sasmeta STARTED 230 /Managem Manage t71110-r 1 1 RUN 1189 # egsh service view sasmeta Basic Service INFO Service Name : sasmeta Service Descriptin: "SAS Metadata Server" state pri maxinstance mininstance maxperslt maxperhst credflag STARTED The Allcatin Requirement INFO Name : sasmeta Cnsumer Name : /ManagementServices/EGOManagementServices Resurce Grup: ManagementHsts Requirement : select('gridnde1.example.cm' 'gridnde3.example.cm') rder('gridnde3.example.cm') minslts maxslts EXCLUSIVE TILE 0 1 N 1 The Assciated Activity Specificatins all: Cmmand : /shared/cnfig/lev1/sasmeta/metadataserver/metadataserver.sh start Executin User : sas Umask : 022 Wrking Dir : InstanceCntrl Cmd: /shared/cnfig/lev1/sasmeta/metadataserver/metadataserver.sh stp CntrlWaitPerid : 30 ENV<0> : EGO_APP_TYPE=RTM_HA ENV<1> : EGO_SERVICE_VER=9.2 RLIMIT-CPU : - RLIMIT-FSIZE : - RLIMIT-DATA : - RLIMIT-STACK : - RLIMIT-CORE : - RLIMIT-RSS : - RLIMIT-NOFILE : - 51

52 Appendix A High Availability Services with SAS Grid Manager RLIMIT-AS : - The Cntrl Plicy INFO StartType: AUTOMATIC Service Dependency: NO DEPENDENCY MaxRestarts MaxHeartBeatInterval MaxMissedHeartBeats HstFailverInterval The Allcatin Situatin Allcatin ID:232 RESOURCE TYPE SLOTS gridnde3.example.cm X86_64 1 Service Instances INFO STATE ACTIVITY_ID RESOURCE SEQ_NO RUN 1191 gridnde3.example.cm 1 # egsh service stp sasmeta Service <sasmeta> has been stpped successfully # egsh service list SERVICE STATE ALLOC CONSUMER RGROUP RESOURCE SLOTS SEQ_NO INST_STATE ACTI sbatchd DEFINED /Cluster Intern ServiceD DEFINED /Managem Manage sasbjsp DEFINED /Managem Manage res DEFINED /Cluster Intern WebServi DEFINED /Managem Manage sasmeta DEFINED /Managem Manage # egsh service view sasmeta Basic Service INFO Service Name : sasmeta Service Descriptin: "SAS Metadata Server" state pri maxinstance mininstance maxperslt maxperhst credflag DEFINED The Allcatin Requirement INFO Name : sasmeta Cnsumer Name : /ManagementServices/EGOManagementServices Resurce Grup: ManagementHsts Requirement : select('gridnde1.example.cm' 'gridnde3.example.cm') rder('gridnde3.example.cm') minslts maxslts EXCLUSIVE TILE 0 1 N 1 The Assciated Activity Specificatins all: Cmmand : /shared/cnfig/lev1/sasmeta/metadataserver/metadataserver.sh start Executin User : sas Umask : 022 Wrking Dir : InstanceCntrl Cmd: /shared/cnfig/lev1/sasmeta/metadataserver/metadataserver.sh stp CntrlWaitPerid : 30 ENV<0> : EGO_APP_TYPE=RTM_HA ENV<1> : EGO_SERVICE_VER=9.2 RLIMIT-CPU : - RLIMIT-FSIZE : - RLIMIT-DATA : - RLIMIT-STACK : - RLIMIT-CORE : - RLIMIT-RSS : - RLIMIT-NOFILE : - RLIMIT-AS : - The Cntrl Plicy INFO StartType: AUTOMATIC 52

53 High Availability Services with SAS Grid Manager Appendix A Service Dependency: NO DEPENDENCY MaxRestarts MaxHeartBeatInterval MaxMissedHeartBeats HstFailverInterval The Allcatin Situatin N Allcatin Service Instances INFO Number f service instances:<0> 53

54 Appendix B High Availability Services with SAS Grid Manager Appendix B: EGO Service Directr Cnfiguratin NOTE: The examples belw make reference t LSF and EGO specific envirnment variables; it is assumed the LSF envirnment setup has been surced in the cnfiguratin envirnment. EGO Service Cntrller Definitin Utilizing the EGO Service Directr t prvide HA Service Relcatin requires that the EGO Service Directr itself be HA. The required service definitin is already present fr the EGO Service Cntrller in the file: ${EGO_ESRVDIR}/esc/cnf/services/named.xml The fllwing mdificatins are required t prperly enable the EGO Service Directr: Set the service t be autmatically started: Change the fllwing line: <sc:starttype>manual</sc:starttype> T: <sc:starttype>automatic</sc:starttype> Update the ResurceRequirement definitin t specify the grid ndes that will be used t execute the EGO Service Directr. Change the fllwing: <eg:resurcerequirement>select('linux86' 'X86_64' SOLX8664')</eg:ResurceRequirement> T: <eg:resurcerequirement> select('@hsta@' '@hstb@') rder('@hstb@') </eg:resurcerequirement> is the FQDN f the primary executin hst is the FQDN f the secndary. If a tertiary hst is required, as fllws: select('@hsta@' '@hstb@' '@hstc@') rder(1*'@hsta@' + 2*'@hstB@' + 3*'@hstC@') Duble check that the value f ESD_CONF envirnment variable is set t the crrect lcatin. This might be crrect as is, but a safe change is t replace the current setting with the fllwing: <eg:envirnmentvariable name="esd_conf">${ego_esrvdir}/esd/cnf</eg:envirnmentvariable> The cmpleted xml file shuld lk similar t the fllwing example: <?xml versin="1.0" encding="utf-8"?> <sc:servicedefinitin xmlns:sc=" xmlns:eg=" xmlns:xsi=" xmlns:xsd=" xsi:schemalcatin=" ServiceName="ServiceDirectr"> <sc:versin>1.2</sc:versin> <sc:descriptin>ego: EGO Service Directr</sc:Descriptin> <sc:mininstances>1</sc:mininstances> <sc:maxinstances>1</sc:maxinstances> <sc:pririty>10</sc:pririty> 54

55 High Availability Services with SAS Grid Manager Appendix B <sc:maxinstancesperslt>1</sc:maxinstancesperslt> <sc:maxinstancesperhst>1</sc:maxinstancesperhst> <sc:needcredential>false</sc:needcredential> <sc:cntrlplicy> <sc:starttype>automatic</sc:starttype> <sc:maxrestarts>1</sc:maxrestarts> <sc:hstfailverinterval>pt1m0s</sc:hstfailverinterval> </sc:cntrlplicy> <sc:allcatinspecificatin> <eg:cnsumerid>/managementservices/egomanagementservices</eg:cnsumerid> <!--The ResurceType specifies a "cmpute element" identified by the URI used belw--> <sc:resurcespecificatin ResurceType=" <eg:resurcegrupname>managementhsts</eg:resurcegrupname> <eg:resurcerequirement> select('@hsta@' '@hstb@') rder('@hstb@') </eg:resurcerequirement> </sc:resurcespecificatin> </sc:allcatinspecificatin> <sc:activitydescriptin> <eg:attribute name="hsttype" type="xsd:string">all</eg:attribute> <eg:activityspecificatin> <eg:cmmand>${ego_top}/7.0/scripts/egsrvlader.sh named -u lsfadmin - f</eg:cmmand> <eg:executinuser>rt</eg:executinuser> <eg:envirnmentvariable name="esd_conf"> ${EGO_ESRVDIR}/esd/cnf </eg:envirnmentvariable> <eg:umask>0022</eg:umask> </eg:activityspecificatin> </sc:activitydescriptin> </sc:servicedefinitin> EGO Service Directr DNS Server Cnfiguratin T cntrl the behavir f the EGO Service Directr DNS server there are 3 imprtant cnfiguratin files: EGO Subzne Definitin File ${EGO_ESRVDIR}/esd/cnf/named/namedb/db.@EGO_SUBDOMAIN@ DNS Server Cnfiguratin ${EGO_ESRVDIR}/esd/cnf/named/cnf/named.cnf EGO Service Directr Plugin Cnfiguratin ${EGO_ESRVDIR}/esd/cnf/esddefault.xml Fr each f the pssible EGO Service Directr DNS setups, these three files must be prperly cnfigured t crrectly match the crprate DNS server cnfiguratin. The fllwing sectins give example cnfiguratins which can be used as a starting pint fr cnfiguring the EGO Service Directr DNS server. All the examples belw are based n the fllwing envirnment assumptins: The crprate dmain name is example.cm. The eg subdmain is eg.example.cm. The virtual hstname fr the EGO Service Directr DNS Server is egnameserver. The Time-T-Live fr EGO virtual hstnames is set t 0, when pssible. 55

56 Appendix B High Availability Services with SAS Grid Manager Withut Crprate DNS Updates EGO Subzne Definitin File $ORIGIN. $TTL 0 ; 0 secnds eg.example.cm IN SOA egnameserver. rt. ( 1715 ; serial ; refresh (3 hurs) 900 ; retry (15 minutes) ; expire (1 week) 0 ; minimum (0 secnds) ) NS egnameserver.eg.example.cm. ; ; Add A recrds fr the sub zne as required $ORIGIN eg.example.cm. sasmeta A sasbjspawner A TTL fr the subdmain is set t 0 secnds in this example; this setting shuld be tuned as required. The A recrds fr the sasmeta and sasbjspawner are examples nly and nt required. Setting these aliases t the IP f the service s primary executin hst will allw the virtual hstname fr the service t reslve prir t the definitin being added t the EGO Service Cntrller cnfiguratin; which may be required t cmplete the SAS deplyment wizard. DNS Server Cnfiguratin Remve r cmment the lines in the glbal ptins sectin which enable query frwarding: // }; // frward first; Update the name f the key that will be used t permit dynamic updates, change the default lines: key eg. { algrithm HMAC-MD5.SIG-ALG.REG.INT; secret "@EGO_SECRET_VALUE@"; }; T: key key.eg.example.cm { algrithm HMAC-MD5.SIG-ALG.REG.INT; secret "@EGO_SECRET_VALUE@"; }; Update the subzne definitin t be used fr the EGO virtual hstnames, change the fllwing: zne "eg." IN { type master; file "db.eg"; allw-update { key eg.; }; }; T: zne "eg.example.cm" IN { type master; file "db.eg.example.cm"; allw-update { key key.eg.example.cm; }; frwarders { }; }; 56

57 High Availability Services with SAS Grid Manager Appendix B EGO Service Directr Plugin Cnfiguratin <?xml versin="1.0" encding="utf-8"?> <ESDDefaultPluginCnfiguratin> <ESD_EGO_NAMESERVER>egnameserver</ESD_EGO_NAMESERVER> <ESD_EGO_DOMAIN>eg</ESD_EGO_DOMAIN> <ESD_CORP_DOMAIN>example.cm</ESD_CORP_DOMAIN> <ESD_EGO_KEY name="key.eg.example.cm">@ego_secret_value@</esd_ego_key> </ESDDefaultPluginCnfiguratin> The ESD_EGO_KEY will be used t sign dynamic update requests sent t the EGO Service Directr DNS server. The name and value must crrespnd t the key previusly defined in the DNS server cnfiguratin abve. Crprate DNS Cnfiguratin The crprate DNS server must be changed t delegate a subzne fr the EGO virtual hstnames. Multiple Name Server entries shuld be added fr this subzne, ne fr each redundant hst that is cnfigured t pssibly execute the EGO Service Directr based n the EGO Service Cntrller definitin. The Name Server recrds will utilize the actual crprate hstnames fr the EGO Service Directr hsts. Crprate DNS Dynamic Updates fr Lcatin f EGO Service Directr DNS Only EGO Subzne Definitin File $ORIGIN. $TTL 0 ; 0 secnds eg.example.cm IN SOA egnameserver.example.cm. rt. ( 1715 ; serial ; refresh (3 hurs) 900 ; retry (15 minutes) ; expire (1 week) 0 ; minimum (0 secnds) ) NS egnameserver.example.cm. ; ; Add A recrds fr the sub zne as required $ORIGIN eg.example.cm. sasmeta A sasbjspawner A TTL fr the subdmain is set t 0 secnds in this example; this setting shuld be tuned as required. The A recrds fr the sasmeta and sasbjspawner are examples nly and nt required. Setting these aliases t the IP f the service s primary executin hst will allw the virtual hstname fr the service t reslve prir t the definitin being added t the EGO Service Cntrller cnfiguratin; which may be required t cmplete the SAS deplyment wizard. DNS Server Cnfiguratin Remve r cmment the lines in the glbal ptins sectin which enable query frwarding: // }; // frward first; Update the name f the key that will be used t permit dynamic updates, change the default lines: 57

58 Appendix B High Availability Services with SAS Grid Manager key eg. { algrithm HMAC-MD5.SIG-ALG.REG.INT; secret "@EGO_SECRET_VALUE@"; }; T: key key.eg.example.cm { algrithm HMAC-MD5.SIG-ALG.REG.INT; secret "@EGO_SECRET_VALUE@"; }; Update the subzne definitin t be used fr the EGO virtual hstnames, change the fllwing: zne "eg." IN { type master; file "db.eg"; allw-update { key eg.; }; }; T: zne "eg.example.cm" IN { type master; file "db.eg.example.cm"; allw-update { key key.eg.example.cm; }; frwarders { }; }; EGO Service Directr Plugin Cnfiguratin <?xml versin="1.0" encding="utf-8"?> <ESDDefaultPluginCnfiguratin> <ESD_EGO_NAMESERVER>egnameserver</ESD_EGO_NAMESERVER> <ESD_EGO_DOMAIN>eg</ESD_EGO_DOMAIN> <ESD_CORP_DOMAIN>example.cm</ESD_CORP_DOMAIN> <ESD_EGO_KEY name="key.eg.example.cm">@ego_secret_value@</esd_ego_key> <ESD_CORP_KEY name="key.example.cm">@corp_secret_value@</esd_corp_key> </ESDDefaultPluginCnfiguratin> The ESD_EGO_KEY name and value must crrespnd t the key previusly defined in the DNS server cnfiguratin abve. The ESD_CORP_KEY wuld be used when updating the virtual hstname lcatin fr the EGO Service Directr DNS Server, the name and value must be set t crrespnd with the crprate DNS cnfiguratin. Since the virtual hstname is within the crprate dmain, the ESD_CORP_KEY must be permitted fr updating the crprate DNS dmain. This culd pssibly nt be required, depending n the plicy fr dynamically updating the crprate DNS. When either key is nt defined in this file the dynamic update requests will nt be signed. Crprate DNS Cnfiguratin The crprate DNS server must be changed t delegate a subzne fr the EGO virtual hstnames. The Name Server fr this subzne shuld be set t the virtual hstname f the EGO Service Directr DNS Server, egnameserver.example.cm in this example. In rder fr this cnfiguratin t wrk successfully this virtual hstname must statically reslve within the crprate DNS. This means that an initial A recrd needs t be present in the crprate DNS fr this virtual hstname, ften referred t as a glue recrd. The IP fr this recrd shuld be set t the address f the primary executin hst fr the EGO Service Directr, but nce the EGO Service Directr DNS server is initialized a dynamic update with the crrect lcatin infrmatin will be sent t the crprate DNS server. 58

59 High Availability Services with SAS Grid Manager Appendix B Crprate DNS Dynamic Updates fr Lcatin f all EGO HA Services EGO Subzne Definitin File Nt used in this setup. DNS Server Cnfiguratin N subzne will be defined fr the EGO Service Directr DNS server, s cmment ut r remve the default definitins: //key eg. { // algrithm HMAC-MD5.SIG-ALG.REG.INT; // secret "@EGO_SECRET_VALUE@"; //}; And: //zne "eg." IN { // type master; // file "db.eg"; // allw-update { key eg.; }; //}; EGO Service Directr Plugin Cnfiguratin <?xml versin="1.0" encding="utf-8"?> <ESDDefaultPluginCnfiguratin> <ESD_EGO_NAMESERVER>egnameserver</ESD_EGO_NAMESERVER> <ESD_EGO_DOMAIN>eg</ESD_EGO_DOMAIN> <ESD_CORP_DOMAIN>example.cm</ESD_CORP_DOMAIN> <ESD_EGO_KEY name="key.example.cm">@corp_secret_value@</esd_ego_key> <ESD_CORP_KEY name="key.example.cm">@corp_secret_value@</esd_corp_key> </ESDDefaultPluginCnfiguratin> It is nt a typgraphic errr abve; the value fr bth the ESD_EGO_KEY and ESD_CORP_KEY must be set t the secret keys crrespnding t the crprate DNS cnfiguratin. The ESD_CORP_KEY will be used when updating the crprate dmain, (virtual hstname f the EGO Service Directr DNS server), and the ESD_EGO_KEY is used when updating the EGO subdmain, (EGO service virtual hstnames). Since defining the EGO subdmain in the crprate DNS cnfiguratin is ptinal in this setup, further discussed belw, the ESD_EGO_KEY wuld have the same value as ESD_CORP_KEY if n EGO subdmain exists, else it must crrespnd t the secret fr the EGO subdmain in the crprate DNS cnfiguratin. In this setup, bth f these keys are ptinal depending n the crprate DNS dynamic update plicies, but the settings must crrectly align with the crprate DNS cnfiguratin. Crprate DNS Cnfiguratin The crprate DNS server shuld be changed t add a subzne fr the EGO virtual hstnames. The Name Server entries fr this subzne wuld be the same as the crprate zne since the crprate DNS server wuld be serving as master fr this new zne. The TTL fr hsts in the new zne shuld be set t 0 r sme marginally lw value. In this setup, it isn t abslutely required t add the new EGO subzne t the crprate DNS server, prvided that the TTL fr EGO virtual hstnames can be set as required. Just as with the previus cnfiguratin, the virtual hstname fr the EGO Service Directr DNS server, egnameserver.example.cm in this example, must statically reslve within the crprate DNS when the EGO Service Directr initializes. 59

60 SAS INSTITUTE INC. WORLD HEADQUARTERS SAS CAMPUS DRIVE CARY, NC TEL: FAX: U.S. SALES: SAS and all ther SAS Institute Inc. prduct r service names are registered trademarks r trademarks f SAS Institute Inc. in the USA and ther cuntries. indicates USA registratin. Other brand and prduct names are trademarks f their respective cmpanies. Cpyright 2011, SAS Institute Inc. All rights reserved