Cray System Management Workstation 4.1 Release (SMW)

Transcription

1 TM Using Cray Management Services (CMS) S

2 2009, 2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR or DFARS 48 CFR , as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR or DFARS 48 CFR , as applicable. Cray, LibSci, PathScale, and UNICOS are federally registered trademarks and Active Manager, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray Fortran Compiler, Cray Linux Environment, Cray SeaStar, Cray SeaStar2, Cray SeaStar2+, Cray SHMEM, Cray Threadstorm, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XMT, Cray XR1, Cray XT, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h, Cray XT5m, CrayDoc, CrayPort, CRInform, Cray ECOphlex, Libsci, NodeKARE, RapidArray, UNICOS/lc, UNICOS/mk, and UNICOS/mp are trademarks of Cray Inc. Origin is a trademark of Silicon Graphics, Inc. MySQL is a trademark of MySQL AB or Sun Microsystems, Inc. All other trademarks are the property of their respective owners. Version 4.0 Published February 2009 Supports limited availability (LA) release of the Cray System Management Workstation (SMW) 4.0 release. Version 4.0 Published May 2009 Supports general availability (GA) release of the Cray System Management Workstation (SMW) 4.0 release. Version 5.0 Published March 2010 Supports the Cray System Management Workstation (SMW) 5.0 release.

3 New Features Using Cray Management Services (CMS) S This revision of Using Cray Management Services (CMS) supports the 5.0 release of System Management Workstation (SMW). Chapter 3, Managing Log Files on page 19 now describes how logs are generated and deleted. Setting Up a Remote MySQL Server for CMS Logs on page 25 provides an updated procedure to more easily set up a remote MySQL server for CMS logs. Using the Log API on page 26 is a new section that describes the log application program interface (API). Table 5 describes new mzparse and mzproxy commands.

4

5 Contents Page Introduction [1] 9 System State Service [2] mzsd System State Cache Daemon mzsd-client Client Cache Daemon System State API Claims Functions Labels Functions Reservations Functions Nodes Functions mz2attr Command Creating a Label Listing Nodes with the Specified Label Deleting a Label Displaying Node Attributes ALPS Interface Using CMS with Other Batch Systems Restarting System State Daemons Restarting the Server Daemon on the SMW Restarting the Client Daemon on the SDB Node Restarting Client Daemon Not Stopped Before a System Reboot Managing Log Files [3] Log Manager Commands Using Log Manager Viewing Logs Searching for Information in Existing Logs Counting or Summarizing Messages Setting Up Log Definitions Listing Current Definitions S

6 Using Cray Management Services (CMS) Page Creating New Definitions Deleting a Definition Restarting System Log Daemons Setting Up a Remote MySQL Server for CMS Logs Using the Log API Appendix A CMS Commands 29 A.1 Administrator Commands A.2 Miscellaneous Commands Appendix B Using CMS APIs in an Application 31 Procedures Procedure 1. Synchronizing mzsd-client and mzsd in the SDB and SMW Procedure 2. Setting up a remote MySQL server for CMS logs Procedure 3. Building on a remote machine Examples Example 1. Creating a label Example 2. Listing nodes with the specified label Example 3. Deleting a label Example 4. Displaying node attributes Example 5. Searching for job reservation information Example 6. Watching all events Example 7. Watching a specific node Example 8. Watching multiple nodes Example 9. Ignoring a specific node Example 10. Watching for a specific event or program Example 11. Continuously checking the logs Example 12. Sorting messages Example 13. Searching for a specific message Example 14. Searching for a node and an event Example 15. Summarizing messages based on a query Example 16. Summarizing messages before setting up a definition Example 17. Getting a summary for a specific message Example 18. Getting a count of a specific message Example 19. Listing all log definitions Example 20. Creating a notify script Example 21. Creating a definition to use a script S

7 Contents Page Example 22. Creating a definition to ignore a specific message Example 23. Creating a definition to archive a specific message Example 24. Creating a definition to archive a specific event Example 25. Deleting a log definition Tables Table 1. CMS API Claims Functions Table 2. CMS API Labels Functions Table 3. CMS API Reservations Functions Table 4. CMS API Nodes Functions Table 5. CMS Log Manager Commands Table 6. Message Sorting Criteria Table 7. CMS Administrator Commands Table 8. CMS Miscellaneous Commands S

8

9 Introduction [1] Cray Management Services (CMS) is the administration framework that integrates both hardware and software environments to provide monitoring of and administrative functionality for Cray XT compute nodes. CMS provides software tools for tasks such as collecting log information, managing system assets, and coordinating information from all parts of the system. CMS tools provide a hierarchy of control and authority that originates on the System Management Workstation (SMW), delegating control to local agents such as Hardware Supervisory System (HSS) cabinet and blade controllers. CMS software tools enable you to easily manage and monitor hundreds of processors in a Cray XT system as one or more logical computers. This manual explains how the CMS software automates system management tasks and assists you in performing the manual tasks. S

10 Using Cray Management Services (CMS) 10 S

11 System State Service [2] The Cray Management Services (CMS) system state service optimizes the collection of system data and provides the infrastructure for future CMS enhancements. The components are: A pair of daemons (mzsd and mzsd-client) An application programming interface (API) to allow applications to send and receive system information (Deferred implementation until the SMW 5.1 release) The mz2attr command, which provides an interface to system data kept by the state daemons 2.1 mzsd System State Cache Daemon The mzsd is the system state cache daemon that runs on the System Management Workstation (SMW). It provides a cache of information about the nodes on the system for fast retrieval of node attributes, active reservations, and claims. This daemon is started automatically to create and populate the cache. The system state cache daemon supports concurrent queries and updates, and synchronization support for multiple reader/single writer locks. 2.2 mzsd-client Client Cache Daemon The mzsd-client daemon is the client cache daemon that runs on the SDB node. An mzsd-client database exists on the SDB node for storage and recovery when access to the SMW is not available. Permissions are set up by Cray installs, and tables are created by mzsd-client. The databases on the SDB node are used only if the mzsd (the system state cache daemon) is not accessible on the SMW. When mzsd recovers, the data in the SDB node database is forwarded by mzsd_client to mzsd on the SMW. S

12 Using Cray Management Services (CMS) 2.3 System State API CMS also provides a set of application programming interface (API) functions that allow ALPS to send application placement data to the CMS state daemon. The APIs also allow ALPS and other batch systems to retrieve system physical attributes and state information from the CMS state daemons. Using these API functions, batch systems can receive active reservations, claims, and labels for nodes on the system. See Appendix B, Using CMS APIs in an Application on page 31 or individual function man pages for detailed information about the API functions Claims Functions Claims are portions of a reservation that runs a specific application. Multiple claims can be made for a reservation over time until all claims are unassigned and the reservation is deleted. Claims functions are listed in the following table. Table 1. CMS API Claims Functions Function mz_assign_claim mz_free_claim mz_free_claimlist mz_get_all_claims mz_get_claim mz_unassign_claim Task Performed Assigns an Application Level Placement Scheduler (ALPS) claim to reserved nodes Frees all memory associated with an mz_claim_t data structure Frees a GList of mz_claim_t data structures Gets the list of all active ALPS claims Get a specific ALPS claim Unassigns (deletes) a specific ALPS claim from reserved nodes 12 S

13 System State Service [2] Labels Functions Labels make it possible to associate a text string with specific nodes. Nodes can have multiple labels. Labels functions are listed in the following table. Table 2. CMS API Labels Functions Function mz_create_label mz_delete_label mz_free_label mz_free_labellist mz_free_labelnamelist mz_free_labelnodes mz_get_all_labelnames mz_get_all_labels mz_get_labelnodes Task Performed Creates a new ALPS label and associates the label to a list of nodes Deletes a specific ALPS label Frees all memory associated with an mz_label_t data structure Frees a GList of mz_label_t data structures Frees a GList of mz_labelname_t data structures Frees all memory associated with an mz_label_nodes_t data structure Gets the list of all defined ALPS labels Gets the list of all defined ALPS labels and the nodes associated with each label Gets the list of all nodes assigned to a specific ALPS label Reservations Functions Reservations dedicate the requested resources to a specific batch job; the resources are dedicated to that job throughout its lifetime. Claims are assigned within a specific reservation. Reservations functions are listed in the following table. Table 3. CMS API Reservations Functions Function mz_create_reservation mz_delete_reservation mz_free_reservation mz_free_reservationlist mz_get_all_reservations mz_get_reservation Task Performed Creates a new ALPS reservation Delete a specific ALPS reservation Frees all memory associated with an mz_reservation_t data structure Frees a GList of mdb_reservation_t data structures Gets the list of all active ALPS reservations Gets a specific ALPS reservation S

14 Using Cray Management Services (CMS) Nodes Functions Nodes are the group of processor(s), memory, and network components acting as a network end point on the system interconnection network. A NUMA node is a multicore processor and its local memory. Multisocket compute nodes have two or more NUMA nodes. Nodes functions are listed in the following table. Table 4. CMS API Nodes Functions Function mz_free_node mz_free_nodelist mz_get_all_nodes mz_get_free_nodes mz_get_labelnodes mz_get_node mz_get_reserved_nodes Task Performed Frees all memory associated with an mz_node_t data structure Frees a GList of mz_node_t data structures Gets a list of all nodes and associated information Gets a list of nodes that have at least one unreserved NUMA node Gets the list of all nodes assigned to a specific ALPS label Get the mz_node_t data structure for a specific node Gets a list of nodes that have at least one reserved NUMA node 2.4 mz2attr Command Deferred Implementation: The mz2attr command will be available in the SMW 5.1 release. The mz2attr command provides a way to read component attributes and labels, and display system data kept by the state daemons: Read, create, or delete labels Assign nodes to labels View node attributes 14 S

15 System State Service [2] Creating a Label You can create a label for specific nodes. Example 1. Creating a label The following example creates test_label on component r11, then verifies the creation by listing all labels. smw:~ # mz2attr -c test_label -C r11 smw:~ # mz2attr -l MyLabel1 MyLabel2 label_1 newlabel orange test_label Listing Nodes with the Specified Label smw:~ # mz2attr -n test_label test_label affinity: 0 You can also list the nodes with the specified label; the output displays all node attributes. Example 2. Listing nodes with the specified label This example lists the nodes associated with test_label and displays all node attributes. Component Attr Value c0-0c0s0n0 NID 0 STATE 7 -> "STATE_READY" STATE_CHANGE_TIME ARCH 0 TYPE 0 PROCESSOR_SPEED 2600 MHz PHYSICAL_PARTION 0 ADMIN_STATE 0 NODE_SERVICE_TYPE NETWORK_ID 0 NETWORK_ADDRESS 0 NETWORK_ORDINAL 0 NETWORK_TYPE 0 NETWORK_REV 0 NETWORK_TOPOLOGY 0 NUMA_NODE_ORDINAL 0 NUMA_NODE_SOCKET_ORDINAL 0 NUMA_NODE_MEMORY_SIZE_MB 2 NUMA_NODE_HUGEPAGE_MEMORY_MB 0 NUMA_NODE_CACHE_SIZE_MB 0 NUMA_NODE_NUMBER_OF_CORES 2 NUMA_NODE_RESERVATION_STATE ALPS_FREE 2 -> "GENERIC_SERVICE" S

16 Using Cray Management Services (CMS) Deleting a Label When you no longer want resources tagged with a specific label, you can delete the label. Example 3. Deleting a label The following example deletes test_label, then verifies the deletion by listing all labels. smw:~ # mz2attr -d test_label smw:~ # mz2attr -l MyLabel1 MyLabel2 label_1 newlabel orange Displaying Node Attributes You can specify the list of node attributes to read. Example 4. Displaying node attributes This example specifies the STATE attribute for all nodes; in addition, it specifies the -S option so that the output is in a form that makes it easier for other programs to parse the output. In this example, note that the output is truncated for readability. sparky$ mz2attr -S -r STATE c0-0c0s0n0:nid:0 c0-0c0s0n0:state:7 c0-0c0s0n3:nid:3 c0-0c0s0n3:state:7 c0-0c0s1n0:nid:4 c0-0c0s1n0:state:7 c0-0c0s1n3:nid:7 c0-0c0s1n3:state:7 c0-0c0s2n0:nid:8... c2-0c2s6n1:nid:345 c2-0c2s6n1:state:7 c2-0c2s6n2:nid:346 c2-0c2s6n2:state:7 c2-0c2s6n3:nid:347 c2-0c2s6n3:state:7 c2-0c2s7n0:nid:348 c2-0c2s7n0:state:7 c2-0c2s7n1:nid:349 c2-0c2s7n1:state:7 c2-0c2s7n2:nid:350 c2-0c2s7n2:state:7 c2-0c2s7n3:nid:351 c2-0c2s7n3:state:7 16 S

17 System State Service [2] See the mz2attr(8) man page for complete information about all command options. See Cray XT System Management for more information about setting and viewing node attributes. 2.5 ALPS Interface Deferred Implementation: The ALPS interface will be available in the SMW 5.1 release. The Application Level Placement Scheduler (ALPS) communicates with mzsd-client to send reservation or claim information that is forwarded to the SMW to update the mzsd server daemon. Forwarding this information from the SDB node to the SMW allows a persistent store of the data. ALPS notifies CMS of application create or start and destroy or stop. The state daemon also stores the information associated with a reservation, including account ID, start and end time of the reservation, execution host name, batch job identification, and user information. Multiple node assignments could be made during the lifetime of a reservation. The node assignment information such as the start and end time of the assignment, the name of the application, and exit status is also stored. Example 5. Searching for job reservation information Using this example, you can list all jobs for the last 24 hours. smw:~ # mzjob See the mzjob(8) man page for more information. 2.6 Using CMS with Other Batch Systems You can use CMS to get node lists and attributes and to track jobs on the system (and their usage) with any supported batch system. See Appendix B, Using CMS APIs in an Application on page 31 for complete instructions. S

18 Using Cray Management Services (CMS) 2.7 Restarting System State Daemons Restarting the Server Daemon on the SMW CMS persistent processes manage state information. The SMW boot process starts these services. You can also manually restart these services if the SMW has not been rebooted since the installation: smw:~ # /etc/init.d/cray-mzsd restart smw:~ # Note: During a software upgrade, the daemon is automatically restarted when the RPM is upgraded. Note: If mzsd is not stopped on the SMW before a Cray XT system reboot, restarting mzsd on SMW after the Cray XT system reboot is not sufficient to restart the daemon. See Restarting Client Daemon Not Stopped Before a System Reboot on page 18 for the restart procedure Restarting the Client Daemon on the SDB Node The mzsd-client on the SDB node starts when the SDB is booted. You can also manually restart these services if the SDB node has not been rebooted: sdb:~ # /etc/init.d/cray-mzsd-client restart Restarting Client Daemon Not Stopped Before a System Reboot If mzsd on the SMW was not stopped before a Cray XT system reboot, restarting mzsd on the SMW is not sufficient to restart and synchronize the state daemons. Use the following procedure to stop and start mzsd-client on the SDB node to synchronize it with mzsd on the SMW after mzsd is restarted. Procedure 1. Synchronizing mzsd-client and mzsd in the SDB and SMW 1. Stop mzsd-client on the SDB node: sdb:~ # /etc/init.d/cray-mzsd-client stop 2. Stop mzsd on the SMW: smw:~ # /etc/init.d/cray-mzsd stop 3. Restart mzsd on the SMW: smw:~ # /etc/init.d/cray-mzsd restart 4. Restart mzsd-client on the SDB node: sdb:~ # /etc/init.d/cray-mzsd-client restart 18 S

19 Managing Log Files [3] The Cray Management Services (CMS) log manager provides the capability to collect, analyze, and display messages from the system. The data is collected from a variety of sources and loaded into the CMS database on the SMW. You can view and search the database for events of interest. Notification of events on the event router uses the mzlogmanagerd daemon; notification of events in the syslog uses the mzsyslogd. Messages of importance from the cabinet, router, and blade controllers are stored in the cabinet controller's Hardware Supervisory System (HSS) database and are forwarded to the SMW via syslog or events. Daemons mzsyslogd and mzlogmanagerd filter the messages based on the log definitions so that messages can be ignored, archived, or an action taken (like notification). By default, messages are stored in the CMS log database for searching. The log filtering of noncritical messages is configurable using log definitions, so that messages of interest can be added to the SMW database. Log tables are generated on a per-day basis, allowing a daily table to be dropped after the specified number of days have passed. The daily tables are dropped rather than deleting messages from a single, larger log table. Changing the number of days before deleting logs can be controlled by changing the /etc/sysconfig/mazama daystoclearlogs=30 configuration parameter. By default, logs are deleted after 30 days. This chapter describes using the log manager on Cray XT systems. S

20 Using Cray Management Services (CMS) 3.1 Log Manager Commands The CMS log manager (mzlogmanager) provides several commands to accomplish logging tasks, as shown in the following table. Table 5. CMS Log Manager Commands Command mzlogmanagerd mzsyslogd mzlslog mzlog mzlogdef mzjob mzparse mzproxy Task Performed Runs and filters a given event, determining whether a definition is set up to ignore, notify, or archive the event. The event is then added to the CMS database and the event can be searched with mzlslog. Filters and stores syslog messages in the CMS database. Searches the logs according to the options stated on the command line and displays the results. Adds messages to the CMS log database; access limited by the mzauth. Defines log addition, deletion, and archiving policies as the logs are processed; access limited by mzauth. Reads the persistent store of the jobs run on the system and allows administrators to find particular reservations. Analyzes log messages and adds them to the CMS log database. Provides a proxy for CMS daemons between the SDB node and the SMW; forwards log traffic to the SDB node from the SMW or from the SDB node to the SMW. Currently, only mzlog, mzlslog, and mzlogdef require authentication. Users root and crayadm have access by default. 3.2 Using Log Manager You can use the log manager to do all of the following tasks: View and search logs Set up log definitions The examples in this section show typical uses of log manager functions. For a detailed explanation of options and additional examples, see individual man pages. 20 S

21 Managing Log Files [3] Viewing Logs You can use the log manager to view events as they occur. You can ignore a node, watch only for specific events, count messages, or summarize messages. Example 6. Watching all events smw:~ # mzlslog -el Example 7. Watching a specific node To watch all events as they occur on node c0-0c0s0n3: smw:~ # mzlslog -e1 -Cc0-0c0s0n3 Example 8. Watching multiple nodes To watch all events as they occur on nodes c0-0c0s0n3 and c0-0c0s1n4: smw:~ # mzlslog -e1 -Cc0-0c0s0n3,c0-0c0s1n4 Example 9. Ignoring a specific node To watch all events as they occur except on node c0-0c0s0n3: smw:~ # mzlslog -e1 -C"-c0-0c0s0n3" Example 10. Watching for a specific event or program To watch for event ec_console_log: smw:~ # mzlslog -e1 -Pec_console_log Example 11. Continuously checking the logs To continuously check the logs every second, enter the following mzlslog command. Note that the -FtHCPm option controls the format of which fields are displayed and order of the fields. To kill the output, enter ctrl-c. smw:~ # mzlslog -e1 -FtHCPm -v Entry Time Hostname Component Program Message :57:27 c0-0 ec_l1_heartbeat :57:36 c0-0 ec_l1_heartbeat :57:45 c0-0 ec_l1_heartbeat ctrl-c Searching for Information in Existing Logs You can use the log manager to search for specific information in existing logs. Example 12. Sorting messages You can sort messages based on any two criteria you choose from the list of fields: entry time, origin time, hostname, component, level, program, or message. You use single letters to indicate the sort, as shown in the following table: S

22 Using Cray Management Services (CMS) Table 6. Message Sorting Criteria Option t O H C l P m Description time origin time hostname component level program message This example sorts messages based on level, then Program: smw:~ # mzlslog --sort lp Example 13. Searching for a specific message This example shows you how to search the last two day's logs for any segfault or general protection errors. smw:~ # mzlslog -tnow-2d -q"*segfault*,*general protection rip*" The comma acts as an OR operator. Example 14. Searching for a node and an event Using this example, you can search component c3-0c2s5n0 for messages less than two days old with event ec_console_log event if it contains a string in the order *LustreError*: smw:~ # mzlslog -tnow-2d -Cc3-0c2s5n0 -Pec_console_log -q"*lustreerror*" Counting or Summarizing Messages You can use the log manager to simply count or summarize messages. Example 15. Summarizing messages based on a query This example provides a summary of components and programs for the last hour: smw:~ # mzlslog -tnow-1h -FCP --summary count component program c0-0c0s0n3 ec_console_log 399 c0-0 ec_l1_heartbeat 278 r11-0 ec_l1_heartbeat 50 c0-0c1s1 ec_mesh_link_failed 5 sshd 5 erd 1 c0-0c2s1s3 ec_mesh_link_failed 22 S

23 Managing Log Files [3] Example 16. Summarizing messages before setting up a definition You can use a summary or count to see how frequent a message has been before setting up a notify or archive log definition. You can also use a summary or count to see nodes that have sent a message commonly (which may help identify a hot spot). For example, to see a summary of memory errors: smw:~ # mzlslog -tnow-2d -q"*single*bit*,*multi*bit*,*multi*dram*" -FC --summary Example 17. Getting a summary for a specific message This example gets a summary of Machine Check Exception messages (by default the search is for the last 24 hours): smw:~ # mzlslog -FwC -q"*machine Check Exception*" --summary smw component count main-smw c3-0c0s2n2 87 main-smw c1-0c0s5n1 58 main-smw c3-0c1s5n0 13 Example 18. Getting a count of a specific message This example gets a count of Machine Check Exception messages (by default the search is for the last 24 hours): smw:~ # mzlslog -FwC -q"*machine Check Exception*" --count Number of matches: Setting Up Log Definitions You can use the log manager to set up log definitions to reduce data, notify you of events, or archive messages of specific interest Listing Current Definitions Example 19. Listing all log definitions You can list all current log definitions using the mzlogdef --list. smw:~ # mzlogdef --list Creating New Definitions You can use the log manager to create definitions that notify you of events or archive specific messages. The default notify script is notify.sh. The location and script name listed in /etc/sysconfig/mazama can be changed when a specific log definition is created. S

24 Using Cray Management Services (CMS) Example 20. Creating a notify script This example creates script /opt/mazama/scripts/segfault.sh to notify you of all segfaults or general protection errors coming from syslog kernel: % cat /opt/mazama/scripts/segfault.sh #!/bin/sh =$1 logdefid=$2 message=$3 echo "$logdefid $message" /usr/bin/mailx -s"smw Segfault message" $ % chmod 755 /opt/mazama/scripts/segfault.sh Example 21. Creating a definition to use a script This examples creates the definition that tells the log manager to use the script /opt/mazama/scripts/segfault.sh to notify you: smw:~ # mzlogdef -c -snotify -Ncrayadm@localhost -Osegfault.sh \ -q"*general protection rip* *segfault*" -Pkernel Example 22. Creating a definition to ignore a specific message You can decide which events you would like to watch or ignore. This example uses mzlogdef to ignore ec_mesh_link_failed messages. smw:~ # mzlogdef --create -s ignore -Pec_mesh_link_failed Example 23. Creating a definition to archive a specific message This example creates a definition to archive messages with a Machine Check Exception in the message field to the file /opt/mazama/archive/machine_exception.log: smw:~ # mzlogdef -c -s archive -A machine_exception -q"*machine Check Exception*" Example 24. Creating a definition to archive a specific event This example creates a definition to archive event ec_l0_health to the file /opt/mazama/archive/health.log: smw:~ # mzlogdef -c -s archive -A health -Pec_l0_health Deleting a Definition You can also delete definitions you no longer need. Use mzlogdef --list to get a complete list, then delete by definition ID. Example 25. Deleting a log definition This example deletes definition ID 37 from the list: smw:~ # mzlogdef -d -i37 24 S

25 Managing Log Files [3] 3.4 Restarting System Log Daemons CMS persistent processes manage log information. The SMW boot process starts these services. You can also manually restart these services if the SMW has not been rebooted since the installation: smw:~ # /etc/init.d/cray-mzlogmanagerd restart smw:~ # /etc/init.d/cray-mzsyslogd restart smw:~ # Note: During a software upgrade, the daemons automatically restart when the RPM is upgraded. 3.5 Setting Up a Remote MySQL Server for CMS Logs my_db_server % vi update.sh #!/bin/sh export SMW_NAMES="mysmw mysmw.domain" Using the following procedure, you can set up a remote MySQL server for CMS logs. Procedure 2. Setting up a remote MySQL server for CMS logs 1. Set up the database permissions from the MySQL server. From the MySQL server, add the hostname that refers to the SMW in the /etc/hosts or the name server. for SMW in ${SMW_NAMES} ; do echo "grant all privileges on \ `mazama%\ `.* to 'mazama'@'${smw}' identified by 'mazama';" \ mysql -uroot -p <root DB passwd> echo "grant all privileges on \ `mz%\ `.* to 'mazama'@'${smw}' identified by 'mazama';" \ mysql -uroot -p <root DB passwd> done my_db_server %./update.sh 2. Perform the following steps from the SMW. a. Set up the tables: smw % mysql -h ${HOST} -umazama -pmazama < /opt/cray/x2/mazama/default/scripts/mazama2.sql smw % mysql -h ${HOST} -umazama -pmazama < \ /opt/cray/x2/mazama/default/scripts/mazama-noncluster.sql b. Edit the log_host configuration parameter in the /etc/sysconfig/mazama file. Change the value from log_host=localhost to log_host=msyql db servers name S

26 Using Cray Management Services (CMS) That section of the file would read as follows: ## Type: string ## Default: localhost ## Config: "" # # Default host for mysql mazama Log database # By default it is the same as the # general mazama database. log_host=msyql db servers name 3.6 Using the Log API The CMS log application program interface (API) allows Cray tools and remote tools to forward data to the log manager to further aggregate logs on the system. The CMS log API provides an interface to the CMS log manager from any C application program. Using these basic functions allows C applications to send error information or other information directly to the CMS log. 26 S

27 Managing Log Files [3] Basic functions for different levels of information are as shown: #include <stdio.h> #include <mz/mz_log_api.h> int main(int argc, char **argv) { mz_log_debug(1,"hello this is a Debug"); mz_log_debug(0,"hello this is a debug 0 showlines"); mz_log(1,"hello this is a just log"); mz_log(0,"hello this is a just log, 0 showlines"); mz_log_info(0, "This is a info message."); mz_log_info(1, "This is a info message with lines."); mz_log_info(1,"hello this is a info"); mz_log_info(0,"hello this is a info, 0 showlines "); mz_log_info(0,"hello this is a info message."); mz_log_info(0,"with line info message."); mz_log_warn(1,"hello this is a warn"); mz_log_warn(0,"hello this is a warn, 0 showlines"); mz_log_err(1,"hello this is a error"); mz_log_err(0,"hello this is a error, 0 showlines"); mz_log_critical(1,"hello this is a critical"); mz_log_critical(0,"hello this is a critical, 0 showlines"); mz_log_alert(1,"hello this is a alert"); mz_log_alert(0,"hello this is a alert, 0 showlines"); mz_log_emergency(1,"hello this is a emergency"); mz_log_emergency(0,"hello this is a emergency, 0 showlines"); } return 0; S

28 Using Cray Management Services (CMS) 28 S

29 CMS Commands [A] A.1 Administrator Commands Cray Management Services (CMS) administrator commands are invoked from the SMW for CMS command authorization, and the collection of syslog messages sent to the SMW and event log information from the event router. The following table lists the CMS administrator commands and their functions. Table 7. CMS Administrator Commands Command intro_cms_api.8 intro_log_api mz2attr mzauth mzdbbackup mzdbrestore mzinit mzjob mzlog mzlogdef mzlogmanagerd mzlslog mzparse mzsd-client mzsyslogd mzupdate Description Provides an introduction to the application program interface to the CMS state daemon Provides an introduction to the C application program interface to the CMS log manager (Deferred implementation) Reads or writes component attribute values Sets the capabilities and authorization for a user's access to programs from a specific host Makes a backup of the CMS database or specific tables in the database from the SMW Restores the entire CMS database or specific tables from a backup Initializes the CMS database Reads the persistent store of the jobs run on the system Adds messages to the CMS log database Defines log addition, deletion, and archiving policies Filters events from the event router Searches the logs and displays the results Analyzes log messages Provides a client cache daemon that runs on the SDB node Filters syslog messages that are sent to the SMW and stores them in the CMS database Updates the administrative environment S

30 Using Cray Management Services (CMS) A.2 Miscellaneous Commands Table 8. CMS Miscellaneous Commands Command mazama.5 mzproxy.7 Description Describes the configuration file for the Cray Management Services (CMS) Provides a proxy for Cray Management Services (CMS) daemons between the SDB node and the SMW 30 S

31 Using CMS APIs in an Application [B] You can use the Cray Management Services (CMS) C API to get node lists and attributes, send/request nodes for reservations (jobs) and claims (sub-jobs while the nodes are reserved), and for tracking the jobs on the system and their usage with any supported batch system. If your system's batch scheduler does not send CMS the job information, however, jobs are not added to the CMS logs or history, and the information cannot be viewed using the mzjob(8) command. Using the following procedure, running on the SDB node or the SMW, you can build the CMS environment on remote machines without the CMS tree to test CMS C API functionality. The following RPMs must be installed on your system to build: lsb-cray-mazama-devel lsb-cray-mazama-libs Procedure 3. Building on a remote machine 1. Install the two RPMs that correspond to the client and server RPMs installed on your System Management Workstation (SMW). smw:~ # rpm -Uvh lsb-cray-mazama-devel-#.#.#-#.####.####.#.x86_64.rpm smw:~ # rpm -Uvh lsb-cray-mazama-libs-#.#.#-#.####.####.#.x86_64.rpm 2. Load the Mazama module. smw:~ # module load mazama smw:~ # S

32 Using Cray Management Services (CMS) 3. Create a makefile. The following is simply an example, you need to modify it for your application and file names. GLIB_INC=$(shell pkg-config --cflags glib-2.0) GLIB_LIB=$(shell pkg-config --libs glib-2.0) MAZMAM_INC=$(shell pkg-config --cflags mazama) MAZAMA_LIB=$(shell pkg-config --libs mazama) CFLAGS+=-Wall -g ${MAZMAM_INC} ${GLIB_INC} LDFLAGS+=-Wall -g -DCI_VERSION=\"1.2.0\" -pthread ${GLIB_LIB} LDFLAGS+${MAZAMA_LIB} CC=gcc all: mztest mztest: mztest.o $(CC) -o mztest mztest.o ${LDFLAGS} 32 S