Support Operations Handbook

Support Operations Handbook Cray Platinum Diamond Support Cray Inc. Version 2.2 February 11, 2014

Table of Contents About Cray Customer Service.4 About This Document...........5 Document Version History... 4 About Cray Customer Service... 5 1. Cray Platinum and Diamond Support... 7 1.1 Platinum and Diamond Response... 7 1.2 24X7 Online Self Services... 7 1.2.1 Software Upgrades, Updates, and Patches... 7 1.2.2 Case Management Tool... 7 1.2.3 Documentation... 7 1.3 General Services... 8 1.3.1 Proactive Field Notices... 8 1.3.2 On-site Spare Parts... 8 1.3.3 On-site Maintenance Tools... 8 1.3.4 Training... 8 1.3.5 Manuals and Documentation... 8 2. Cray Platinum and Diamond Support Resources... 9 2.1 District Service Manager... 9 2.2 Cray Field Support Engineers... 9 2.2.1 System Support - Hardware... 9 2.2.2 System Support - Software... 10 3. Support Tools... 11 3.1 CrayPort... 11 Registering for CrayPort... 11 3.2 Defect Management... 11 3.2.1 Bugzilla... 11 3.2.2 Bug Severity... 12 3.3 Field Notices... 12 3.4 Email Notifications... 12 Platinum-Diamond SOHB 2.2-021114 Page 2 of 19 www.cray.com

3.5 Spare Parts... 12 3.6 Cray Documentation... 12 4. Case Lifecycle... 14 4.1 Defining a Case... 14 4.2 Case Priority Definition... 14 4.2.1 Case Priority 1: Critical... 15 4.2.2 Case Priority 2: Urgent... 15 4.2.3 Case Priority 3: Major... 15 4.2.4 Case Priority 4: Minor... 15 4.3 Definition of Case Status... 15 Table 1. Case Status... 16 4.4 Spare Parts Management... 16 4.5 Monitoring Case Progress... 16 4.6 Escalation Guidelines... 16 4.7 Case Stages... 17 5. Cray Support Contact Information... 18 Table 2. Cray Contact Information... 18 6. Glossary... 19 Platinum-Diamond SOHB 2.2-021114 Page 3 of 19 www.cray.com

Document Version History Rev Description Originator Date 1.0 Document creation Paula C Salovich 08/23/2012 2.0 Added About this document Paula C Salovich 02/08/2013 and release document to Cray.com 2.1 Deleted response and resolution Paula C Salovich 09/02/2013 guidelines, created Document Version History, cleaned up Table of Content 2.2 Deleted repeated document version history. Changed descriptions of field support engineering. Paula C Salovich 2-11-14 Platinum-Diamond SOHB 2.2-021114 Page 4 of 19 www.cray.com

About Cray Customer Service Cray Customer Service strives to maintain high standards of performance. Cray service teams, backed by Cray support systems, are highly skilled and experienced in the tradition of high performance computing and are equipped to provide world-class support for mission-critical systems. Cray support engineers, field engineers, and field analysts undergo a full range of product, technology, and support system training. They are qualified to deliver services that include installation, troubleshooting, system maintenance, system administration, and system optimization for some of the world s most impressive high performance systems. Cray support and field engineers are backed by experts from the Cray Engineering and Development organization as well as third party technology partners. Support and field engineers are closely linked to Cray hardware and software product support groups, and have access to Cray s in-house systems to reproduce hardware and software problems, which aids system troubleshooting and problem diagnostics. Cray Customer Service offers a variety of service levels to ensure coverage of all customer needs. This document focuses on Cray Platinum and Diamond Support; however Cray Customer Service recommends customers gain a full understanding of all Cray service offerings. Cray Silver Support is tailored to customers with technical expertise and capabilities. These customers are fully trained by Cray to take on the responsibility of maintaining the operations and availability of their own systems with limited contact and intervention from Cray. This is a low cost alternative for customers and support is provided remotely. For customers that require a greater level of engagement from Cray Customer Service, the next level offered is Cray Gold Support. This level of support includes all the benefits and service offerings of Cray Silver Support. Additionally, it provides customers the assurance of having an experienced Cray technical support resource on site, if required, the next business day after an issue is reported. The highest levels of support Cray offers are Platinum and Diamond, which include 4-hour and 2-hour hour response times respectively. Each customer will have dedicated and experienced Cray technical resources at their disposal during the contracted coverage and response times. Platinum and Diamond Support also offer a single point of contact into Cray to resolve system issues and procedural questions. Platinum-Diamond SOHB 2.2-021114 Page 5 of 19 www.cray.com

About this document The Support Operations Handbook (SOHB) has been developed to describe the processes, policies and practices of the Cray Platinum and Diamond support level. It is also intended to provide the processes and procedures that will ensure a successful working relationship between Cray and its customers. Cray Platinum and Diamond support is available for the following Cray products: Cray XE/XK/XC Series Systems. Cray Sonexion Storage Systems. YarcData Urika Systems. This document is not intended to replace any current legal agreements between Cray and the customer. This document is subject to review, change and update upon the creation of new services, processes, changes in existing services or policies or support level structuring, without notice. Cray customers may copy or transmit this document for internal use only. Platinum-Diamond SOHB 2.2-021114 Page 6 of 19 www.cray.com

1. Cray Platinum and Diamond Support Cray Platinum and Diamond Support is a comprehensive hardware and software maintenance that provides assistance available 7 days a week, 24 hours per day. This includes both on-site support and remote support when appropriate. Platinum Support offers a four hour response time and Diamond offers one to two hour response. Cray field support engineers are assigned to every client under a Cray Platinum or Diamond Support agreement, as well as a Cray District Service Manager (DSM), who acts as the focal point for all servicerelated communications and will have the authority to act on Cray s behalf in matters regarding the support of the system. Cray Platinum and Diamond Support also include a spare parts kit located on the customer site to cover the majority of hardware problems. The following services are available with a Cray Platinum or Diamond agreement: 1.1 Platinum and Diamond Response With Cray 7x24 support, issues can be reported directly by the customer, by Cray field support engineers working with the customer, or by the customer s DSM. Response time will depend on the customer s agreement with Cray, with options such as 4 hours, 2 hours and 1 hour response. In all cases, the customer s DSM is responsible for the overall account and all open issues. 1.2 24X7 Online Self Services All Cray customers with active service contracts receive access to Cray s customer service portal, CrayPort. CrayPort is the tool with which customers can report and manage system maintenance issues, find relevant product documentation, receive Field Notices, and download software. 1.2.1 Software Upgrades, Updates, and Patches The latest releases of software are provided via the customer s CrayPort account. Updates include major releases, patches, documentation updates, and related release notes. 1.2.2 Case Management Tool Customers under an active service contract use CrayPort to access Cray s case management tool. The case reporting tool is a self-service system that allows customers to log issues through cases 24x7. Customers must enter all the information related to the issue being reported, capturing all necessary information required to help diagnose a problem. 1.2.3 Documentation Cray provides complete documentation of all software and hardware products delivered with Cray systems. All system documentation is kept current and made available to customers with each release of system software, this includes third-party products provided by Cray. Platinum-Diamond SOHB 2.2-021114 Page 7 of 19 www.cray.com

1.3 General Services 1.3.1 Proactive Field Notices The e-mail configurable Field Notices deliver information about Cray s software and hardware by providing documentation, information updates, product updates, security updates, critical notifications, and breaking news. 1.3.2 On-site Spare Parts Cray provides an onsite inventory of a number of critical parts. Customers can opt for a more comprehensive onsite parts inventory for an additional fee. To purchase these parts, customers must contact their Sales Account Manager. 1.3.3 On-site Maintenance Tools Cray provides an on-site electrostatic discharge (ESD) tool kit to help maintain the hardware. 1.3.4 Training Cray provides a variety of training courses, detailed in the following link: http://www.cray.com/customersupport/training/schedule.aspx. 1.3.5 Manuals and Documentation All user and administrative documentation for software and hardware products sold by Cray can be found at CrayDoc, http://docs.cray.com/. Platinum-Diamond SOHB 2.2-021114 Page 8 of 19 www.cray.com

2. Cray Platinum and Diamond Support Resources Cray field support engineers are assigned to every client under a Cray Platinum or Diamond Support agreement, as well as a Cray District Service Manager (DSM). 2.1 District Service Manager Cray District Service Manager (DSM) is the focal point for all service-related communications and will have the authority to act on Cray s behalf in matters regarding the support of the system. The DSM also represents the customer s interests within Cray as the customer s advocate in service delivery matters and is responsible for arranging support under the SOW as 24-hours-a-day, seven-days-a-week coverage. Alternate DSM will be designated in those instances when the customer assigned DSM is absent. The DSM will perform the following activities: 1. Escalation and management of critical situations ensuring each incident reaches a satisfactory conclusion. The DSM will maintain focus through the channels involved in the particular escalation. 2. Manage on-site Cray support personnel engaging additional resources when needed. The DSM also identifies and manages specialist support or additional resources required by the customer. For assistance requests in specialist areas, the DSM can use their network of contacts and tools to find resources and oversee the delivery of any contracted services. 3. Conduct service delivery review meetings providing incident record analysis addressing any customer concern by providing improvement initiatives. 4. Define with each customer a process for delivering a service improvement plan, directed at the customer system operational requirements and Cray service capabilities. 5. Provide service delivery reporting as reasonably required by the customer. Content and report frequency will be specified by the customer and agreed to by Cray. 6. Work with the customer to understand the customer s system incident management process and ensure that the process works efficiently with the Cray support process. 2.2 Cray Field Support Engineers Under a 7x24 Cray support agreement customers will have dedicated field resources supporting their accounts. These resources are fully trained and experienced Cray technical resources. Field support engineering activities include the following: 2.2.1 System Support - Hardware On-site field support engineers may provide support in systems engineering by analyzing hardware failure information, performing hardware repairs and firmware upgrades for Cray-provided products. Field support engineers will also manage the on-site spares inventory and monitor operation of the cooling system and trends in hardware component failures. Field support engineers provide Level 1 support, which consists of hardware problem isolation, diagnosis, resolution and/or workarounds on a 7x24 basis and will escalate problems to Level 2, Cray product experts, when additional technical expertise or urgent problem resolution is required to resolve difficult or persistent problems. Field support engineers open and track software bugs and/or request part failure analysis as required. Other tasks include but are not limited to: Ensure the customer is notified of new hardware releases and patches within one week of release to the general market and provide the customer with the upgrade within two weeks of a particular request. Platinum-Diamond SOHB 2.2-021114 Page 9 of 19 www.cray.com

Advise the customer of necessary updates or additions to their system as appropriate, including software updates and all technical issues that may affect the operation of the customer s system. Use best commercial effort to obtain and provide information, data, decisions, and approvals within a response time as agreed to by the customer and Cray. Update the customer regarding all active maintenance issues as required, including but not limited to detailed descriptions of issues and their resolution. Conduct trend analysis to ensure the on-site hardware inventory and off-site sources remain sufficient to provide the required level of support of the customer s system. 2.2.2 System Support - Software Field support engineers provide on-site software maintenance and support necessary to maximize the utilization of Cray-delivered products. Cray field support engineers provide installation support for software releases, upgrades, and patches, Level 1 software problem isolation, diagnosis, resolution and/or workarounds, analyze problems with system software and/or third-party software to implement workarounds or fixes as needed. When required, field support engineers will escalate problems to Level 2 Cray product experts when additional technical expertise or urgent resolution is needed to resolve difficult or persistent problems. Field support engineers open and track software bugs until final resolution. In addition, field support engineers also review with the customer both software currency and maintenance levels and provide advice and guidance. Cray field support engineers will recommend strategies and procedures for managing software currency and ensure that levels of maintenance are kept up to date. Cray field support engineers will provide assistance with setting up and regularly reviewing the customer s system software strategy. Finally, in collaboration with Cray sales, field support engineers coordinate ordering and delivery of Cray software for the customer as necessary and required to maintain the customer s system. Platinum-Diamond SOHB 2.2-021114 Page 10 of 19 www.cray.com

3. Support Tools Cray provides a complete set of support tools accessible through CrayPort. All customers with current maintenance contracts can request access to CrayPort for as many users as needed. CrayPort is available 24x7 excluding downtimes for routine maintenance or unforeseen outages. 3.1 CrayPort Cray s customer portal, CrayPort, provides access to a variety of information, including bug tracking, Field Notices, technical support cases, documentation, software downloads, and training registration. CrayPort is also the tool that customers use to report and manage system maintenance issues. The Help tab at CrayPort includes a guide that describes the various features of the site. While every effort is made to prepare for all possible contingencies, there may be times where access to CrayPort may not be available due to unforeseen circumstances. If access problems are encountered, customers may contact the CrayPort support organization at csms_support@cray.com. Registering for CrayPort To register for CrayPort, go to http://crayport.cray.com and select the Register for an Account link. Customer may request as many user accounts as needed. The registration form requires the following information: Customer user contact information. Cray system type. Cray system asset or serial number (optional) Company information. Upon receipt of this information, account setup and activation occurs within two business days. Customer login information is sent to the e-mail address provided. 3.2 Defect Management Defects are reported as cases through CrayPort. All pertinent information needs to be captured at the time a case is created. While the issue is still a case in CrayPort, the Field support engineer triages and validates the issue and gather additional documentation if necessary. If a CrayPort case is confirmed to be a product defect, the Field support engineer converts the case to a Bugzilla bug via an automated mechanism that is part of CrayPort's web interface. The automated mechanism will open a bug via Bugzilla's email interface, passing along all required information. 3.2.1 Bugzilla Once a reported case is confirmed as a bug, a bug ID is created in Bugzilla. Data such as product, components, and customer ID are take directly from the CrayPort case fields and used to create the Bugzilla bug. Once a bug ID is created, the bug ID is added to the case and the case is closed. Platinum-Diamond SOHB 2.2-021114 Page 11 of 19 www.cray.com

All bugs, public and private, that are created in Bugzilla are available for review at CrayPort. Additionally, customers can subscribe to a daily email update of their own reported bugs. 3.2.2 Bug Severity The severity of a bug refers to the importance the customer places on the reported bug. In general, these are the basic guidelines for defining the severity of a bug at Cray: Critical: Prevents the customer from meeting their normal production schedules. If an acceptable workaround is available, then the bug can be downgraded to Urgent. Urgent: Limits the customer's ability to meet their normal production schedules. If an acceptable workaround is available, then the bug can be downgraded to Major. Major: Does not interfere with the customer's normal workload or production schedules. Minor: Little or no impact on users and can be bypassed easily. 3.3 Field Notices Field Notices (FNs) are important communications specific to the products a customer owns. These notifications include information updates, product updates, security updates, critical notifications, breaking news, and announcements on the availability of patches or new releases. All customers under a valid Support agreement receive, via email, all the FNs that are related to their Cray products. Customers also have an option to customize their FN subscriptions at CrayPort to receive only those FNs that they consider appropriate for their role. 3.4 Email Notifications All customers under a current, valid Support Agreement are able to subscribe to receive email notifications. These email notifications include all Field Notices, changes, and bug updates related to their products. Customers can set up their accounts for email notifications by changing their CrayPort Profiles, located under the Help drop down menu on the main CrayPort menu bar. In CrayPort Profile, customers can enable notifications for bugs and/or Field Notices under the Preferences heading. 3.5 Spare Parts Under a valid Cray support agreeemnt, Cray provides an on-site inventory of critical spare parts. All spare parts are new or warranted to perform as new. Parts that are installed in the customer system become the property of the customer; parts removed from the customer system become the property of Cray. 3.6 Cray Documentation Cray Documentation includes: Reference information in the form of man pages is provided with Cray software release packages using the man(1) command. Between major releases, software updates, changes/fixes are documented in HTML pages included with the update download package. CrayDoc is Cray s documentation system that contains a collection of open-source software components that allows fast and easy access to Cray documentation from a web browser over Platinum-Diamond SOHB 2.2-021114 Page 12 of 19 www.cray.com

a local path defined by the system administrator. HTML files of most documents and PDF files of all documents are available. CrayDoc runs on any operating system based on UNIX or Linux and includes the following features: Ability to search books, Cray-authored man pages, and the Cray glossary. Accomodates wild card (*) searches. Integrates non-cray documents which support third-party hardware and software. Highlights search matches. Documentation is also available on Cray s public website, docs.cray.com. The documentation that is available on this site is updated with each software release to accurately reflect the behavior/features of the software. Platinum-Diamond SOHB 2.2-021114 Page 13 of 19 www.cray.com

4. Case Lifecycle With Cray 7x24 support, issues can be reported directly by the customer, by Cray field support engineers working with the customer, or by the customer s DSM. Response time will depend on the customer s agreement with Cray, with options such as 4 hours, 2 hours and 1 hour response. Customers can open a case directly by following the steps below. a) Log in to http://crayport.cray.com and select the Case tab. b) Click the Create New Case button and enter the required information. c) Click the Submit button. The system assigns the case to the queue that supports the Cray system. An assigned monitor of the queue responds to the inquiry during the contracted period of service. Note: Refer to the CrayPort Guide on the Help tab for detailed case entry instructions. 4.1 Defining a Case Customers should review the following information prior to entering and reporting a case. This information will be used in resolving the customer issue: Detailed description of the problem. The environment profile associated with the system that is exhibiting the problem (production, test, etc.). What steps were followed that resulted in the problem being reported? What parameters are set for the programs being accessed? Has this problem occurred before? What result was expected? Can the issue be duplicated (reproduced) on demand? Can the issue be described step-by-step to allow it to be reproduced? What troubleshooting steps have been taken to investigate the problem? Have any recent changes been made to the system? Have the processes for using the software been changed? Which programs were involved when the error occurred? If an error message appeared, what is the exact error message and number? What is the severity of the case? 4.2 Case Priority Definition When reporting a case, customers have the opportunity to select the case priority based on the severity of the problem. However, case priority will be reassessed and revised by the support engineer assigned to the case, based on the nature of the issue at each stage of the case. Therefore, the priority of a case will be agreed upon by the customer and the Cray Field support engineer according to the criteria in this document. For example, if a valid workaround is found for a Priority 1 (Critical) issue, the priority of the case may be downgraded to Priority 2 (Urgent). Similarly, if an Urgent case were to deteriorate, it may be upgraded to Critical. It is important to note that frequency and urgency of the response on cases will depend on the severity of each case. The different severity levels are described as follows: Platinum-Diamond SOHB 2.2-021114 Page 14 of 19 www.cray.com

4.2.1 Case Priority 1: Critical Critical cases represent the highest business impact to a customer. A Priority 1 case prevents the customer from meeting their normal production schedules or prevents an acceptance test from proceeding. Product is not operational and all data is inaccessible. Material function of the product is inoperable and cannot be restarted. Intermittent failures occur so frequently that the functionality of the system is rendered inoperable. Data flow is completely stopped resulting in critical impact to the end user's business and no workaround is available. 4.2.2 Case Priority 2: Urgent Urgent cases limit the customer's ability to meet their normal production schedules or will cause significant impact on the acceptance test. Product is operational. Restricted functionality is impacting the end user's business. If not resolved, it could impact long-term operations. If a workaround is available, it is only a short-term solution as it could be labor-intensive or costprohibitive over a long period of time. 4.2.3 Case Priority 3: Major Major cases do not interfere with the customer's normal production schedules or an acceptable workaround exists. Product is operational. Functional limitations or restrictions exist that are not critical to customer operations. 4.2.4 Case Priority 4: Minor Minor cases have little or no impact on users and can be bypassed easily. Questions on product usage, implementation, performance, and general knowledge. Customer enhancement requests are included. 4.3 Definition of Case Status A case will move through various stages during its lifecycle. Below is the list of definitions of case status and the action currently in process: Platinum-Diamond SOHB 2.2-021114 Page 15 of 19 www.cray.com

Table 1. Case Status Case Status Open/Closed Status Description Unassigned Open Waiting to be assigned to an individual owner. Work in Progress Open Waiting for final HW diagnosis, collection of bug information, responses to other queries, and information gathering. Waiting Maintenance Closed Waiting for physical work or SW upgrade/patch installation. Unsuccessful part replacement will cause a case to remain in this state until case can be completed. Waiting Logistics Closed Waiting for any logistics activity, such as part failure analysis or spare parts. Data Gathering / Monitoring Closed Waiting for additional data, or waiting to see if an intermittent issue re-occurs. Completed Closed Case completed. 4.4 Spare Parts Management Under Cray Platinum and Diamond support, a comprehensive set of spare parts is included and delivered on the customer site. The field support engineers assigned to the customer are responsible for the management of the on-site inventory. 4.5 Monitoring Case Progress All customer-submitted cases can be viewed electronically via CrayPort. Customers may monitor their cases by selecting the Case/Bug Entry tab on CrayPort and selecting Search Cases. Cray also provides e-mail notifications when a case is updated or a change in status occurs. 4.6 Escalation Guidelines At any time during the customer s defined local business hours, the customer, the field support engineers or the DSM can escalate a specific support case. Platinum-Diamond SOHB 2.2-021114 Page 16 of 19 www.cray.com

To escalate a case, the customer has two options: 1. Request a case escalation directly to the field support engineer working on the case. 2. Request a case escalation directly to the DSM associated to the customer account. 3. Contact Cray Support via telephone with the specific case number and request to escalate the case. 4.7 Case Stages All issues are initially investigated by the on-site field support engineers associated to the customer. This is also referred to as Level 1 Support. When a field Support engineer is not able to resolve a customer case, the case is escalated to Level 2. Level 2 Support is provided by Cray product experts who serve as the remediation contacts for problems not solved by Level 1 Support. The Cray Level 2 Support product experts may determine that the issue is the result of a product defect, missing functionality, or needs to be addressed by a third-party product vendor. When it is determined that the case is a product defect, the case is escalated to the appropriate development team or Level 3 Support. The Cray development team will determine if the specific issue is going to be included on a fix or a subsequent release of the product. Cray development may also determine that the issue is a feature or function which is not a part of the original product design. The development of an enhancement is up to the discretion of Cray product development; customers are welcome to submit ideas. When it is determined that the issue is related to a third-party vendor, the Level 2 Support product experts engages with the third party technology vendor support organization in the resolution of the issue. Platinum-Diamond SOHB 2.2-021114 Page 17 of 19 www.cray.com

5. Cray Support Contact Information Table 2. Cray Contact Information Cray Support contact information list Toll free support number (Select 1) 1-800-950-2729 Support number (Select 1) (651) 605-8888 24x7 online case reporting tool, CrayPort CrayPort http://crayport.cray.com http://crayport.cray.com Platinum-Diamond SOHB 2.2-021114 Page 18 of 19 www.cray.com

6. Glossary Case (Incident, Call) A case is a single problem, or symptom in the customer s operation of the solution that negatively affects the functionality of the Cray system. A case is made up of all the activities required to resolve the issue once Cray services has been contacted. Every case is assigned a unique number (case ID) that is used to track its progress and referenced when communicating with customers. DSM District Service Manager. Primary service interface between the Cray support team and all customer departments supporting the system. Bug Error, defect, discrepancy. Critical Bug Defect that prevents the customer from running their normal workload or meeting their normal production schedules. Urgent Bug Defect that limits the customer from running a normal workload or from meeting their normal production schedules. Major Bug Defect that does not limit the customer from running their normal production schedules but does not leave the system inoperable. Minor Bug - Minor defect that has little or no impact on users and can be bypassed easily. Bugzila Ticket tracking tool used by Cray to develop or create fixes for a bug reported in a case. Fix (Action, Correction, or Patch) A fix is the solution to a bug or problem. CrayPort CrayPort is Cray s portal to customers. Documentation, case reporting and product information is found in CrayPort. Level 1 Support - Level 1 support means the service provided by Cray Field Engineers, in response to the initial web contact or telephone call by a customer which identifies and documents a problem. This includes problem source identification assistance, problem analysis, problem resolution, installation planning information, knowledge base search, and preventative and corrective service information. Level 2 Support Level 2 support means the service provided by Cray s technical experts after a case is escalated by Level 1 support. Level 3 Support Level 3 support means the service provided for fixes or workarounds. This service involves Cray development team and/or third-party providers. Release A release is the edition of a Cray product after it has been made generally available. Platinum-Diamond SOHB 2.2-021114 Page 19 of 19 www.cray.com