Welcome to Atmos 2.1 System Administration course. Click the Notes tab to view text that corresponds to the audio recording. Click the Supporting Materials tab to download a PDF version of this elearning. Copyright 1996, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC2, EMC, Data Domain, RSA, EMC Centera, EMC ControlCenter, EMC LifeLine, EMC OnCourse, EMC Proven, EMC Snap, EMC SourceOne, EMC Storage Administrator, Acartus, Access Logix, AdvantEdge, AlphaStor, ApplicationXtender, ArchiveXtender, Atmos, Authentica, Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Captiva, Catalog Solution, C-Clip, Celerra, Celerra Replicator, Centera, CenterStage, CentraStar, ClaimPack, ClaimsEditor, CLARiiON, ClientPak, Codebook Correlation Technology, Common Information Model, Configuration Intelligence, Configuresoft, Connectrix, CopyCross, CopyPoint, Dantz, DatabaseXtender, Direct Matrix Architecture, DiskXtender, DiskXtender 2000, Document Sciences, Documentum, elnput, E-Lab, EmailXaminer, EmailXtender, Enginuity, eroom, Event Explorer, FarPoint, FirstPass, FLARE, FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum, HighRoad, HomeBase, InfoMover, Infoscape, Infra, InputAccel, InputAccel Express, Invista, Ionix, ISIS, Max Retriever, MediaStor, MirrorView, Navisphere, NetWorker, nlayers, OnAlert, OpenScale, PixTools, Powerlink, PowerPath, PowerSnap, QuickScan, Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo, SafeLine, SAN Advisor, SAN Copy, SAN Manager, Smarts, SnapImage, SnapSure, SnapView, SRDF, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix DMX, Symmetrix VMAX, TimeFinder, UltraFlex, UltraPoint, UltraScale, Unisphere, VMAX, Vblock, Viewlets, Virtual Matrix, Virtual Matrix Architecture, Virtual Provisioning, VisualSAN, VisualSRM, Voyence, VPLEX, VSAM-Assist, WebXtender, xpression, xpresso, YottaYotta, the EMC logo, and where information lives, are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. Copyright 2012 EMC Corporation. All rights reserved. Published in the USA. 1
This course provides details on the new software administration features of Atmos 2.1. This Atmos software release offers infrastructure changes and numerous user enhancements, including Graphical User Interface (GUI) additions, event management, node configuration and disk management, channel bonding, and Atmos Content Address Storage (CAS). This course also includes a demonstrative video for navigating the new Atmos 2.1 GUI. This course is intended for those already experienced with Atmos system administration and are responsible for installation, configuration, implementation, and/or troubleshooting of Atmos software. 2
This module focuses on the Atmos 2.1 Graphical User Interface. 3
The Atmos 2.1 GUI has many new enhancements that will allow for field personal and customers to navigate faster and gather system information. Operations have been grouped together for quick access. System information is better organized by the use of tabs. When the user logs into the system the most important information will be displayed at the top level screen such as failed disks and Recovery Jobs. Capacity information is available on a per RMG basis and per node basis. Event summaries by time range with drill down to node level event logs are also available. NOTE: IS = Installation Segment - a collection of an even number, 2 to 252, of Atmos Nodes SS = Storage Services RMG = Resource Management Group - a site specific collection of one or more Installation Segments Node = Atmos System, Virtual or Physical MDS = MetaData Services CM = Configuration Manager VE = Virtual Environment GUI = Graphical User Interface API = Application Programming Interface ACDP = Atmos Cloud Delivery Platform CAS = Content Address Storage ISO = International Standards Organization. Also known as International Organization for Standardization. S3 = Amazon Simple Storage Service 4
Once logged in to the Atmos 2.1 system, you will see changes to the system summary page. Event notifications, which are located at the top right corner in which are defined in four categories. The system software version is now reported on this page. Tabs allow users to select areas for system investigation and operational insights. The navigation tree on the far left has been expanded with more detailed configuration capabilities. 5
The Atmos 2.1 software incorporates customer replaceable disk capability. This Customer Replaceable Unit (CRU) feature is built into the Graphical User Interface and is seen from the system dashboard page. If there is a failed disk, it will show up in the failed disks section. There are two actions to choose from if you or the customer encounters a failed disk. The first being Order Disk. By selecting this, a pop up box displays information as seen here. Once the disk is ordered and received, the second link gives instructions on where the failed disk is located and how to replace. 6
Let s now take a look at the Atmos 2.1 Graphical User Interface enhancements. 7
This module focuses on the Atmos 2.1 Event, Disk and Configuration Management. 8
The functionality of event management has changed in Atmos 2.1. Logging of events is now written to the event log database, which replaces the Alert.log. What has been removed in Atmos 2.1? The reporting framework GUI. Customers cannot set configuration of thresholds for event triggers. All events now have preconfigured thresholds. 9
System-level and RMG-level dashboards have Event Summary icons located in upper righthand corner of the GUI. Notification configuration for events is more granular for these options. The navigation menu on the left now has screen options to select specific policies for alert notifications. To configure these policies the system software serial number must be applied first before attempting to configure any of the available notifications. 10
After the user logs into Atmos using a web browser, the system summary page is displayed as shown. Information Events, Error Events, Warning Events, and Critical Events can be viewed for the last 24 hours, 7 days, 30 days, or you can view all events which have occurred in the system. The user can then select the type of event they wish to view. Here we see a critical event chosen. You would select the RMG in question for this critical event and then a second display window will be displayed. By selecting the individual node the user then can view specific information made available for that node. 11
By selecting an individual node in a RMG, the user is able to select the Events Disk & FS or Performance tabs. When choosing the Events tab, a detailed listing is available to view the severity, timestamp, category, source (either a hardware subsystem or a software based service), and the event ID. 12
When the system administrator wants to activate a notification policy, he/she will select the respective category and from there enable or disable: SNMP Trap notifications Email notifications Remote syslog notifications Once selected, the user can test the event configuration by selecting the test button or just submit the settings. The system administrator can also set the notification severity level for SNMP Traps and Email. There are 4 notification levels to choose from. They are: Information Warning Error Critical Email and or SNMP traps will only be sent out if the event s severity is greater than or equal to the configured severity level. 13
By selecting the SNMP Configuration page, the system administrator can choose the Default (system) or specific RMG-level policy to configure. The user can then add a SNMP NMS destination by clicking on the Add link. The system administrator can enable the Atmos SNMP agent and configure the agent to accept SNMP requests from a remote NMS. If the user clicks Add in the SNMP Trap Configuration box, a new input box will appear. Criteria to input would be: The IP address of a NMS (Trap Destination). The remote SNMP port or the remote NMS community name (defaults to public) or the remote SNMP port or the remote NMS community name (defaults to public). More than one NMS can be configured. SNMP traps for events will be sent to all NMSs in the configured list. 14
From the left-hand Notification menu, the system administrator would click on Email On the Email Configuration page, the user then can choose Default (system) or RMG-level policy to configure. You can add a mail sending (SMTP) server and add any number of email recipients. Event messages will be sent to all recipients in this list. 15
To configure remote Syslog Configuration, from the Notification menu, click on Remote Syslog. This will bring up the Remote Syslog Configuration page. The user can choose Default (system) or RMG-level policy to configure. A remote hostname and respective port number can be configured. Remote syslog requires that a user configure syslog parameters on the remote server. This includes enabling the remote server s syslog port to accept messages from the Atmos system. Customers can use the remote syslog to aggregate events from all nodes into one local place. 16
To configure remote SYR, from the left-hand Notification menu, click on System Report On the System report configuration page the system administrator can choose Default (system) or RMG-level policy to configure. You can enable a receipt for SYR notifications, (SMTP must be configured prior to enabling these). The customer name, Site ID, contact email and contact name are required fields. Users can add a Primary Forwarding Server along with the well known port number. This can be an ESRS Gateway or a SMTP server. You can add in any number of Failover Forwarding (ESRS or SMTP) servers. Event XML files will be sent to the Primary Forwarding server (if accessible). If Atmos cannot access the Primary server, it will attempt to send the Event XML file to a failover server. 17
The RMG status screen available in Atmos 2.1 displays various system attributes such as the busiest Atmos disks. It allows the user to drill down and view how full nodes are with data. CPU utilization is available on a per node basis along with how full the metadata disks are. 18
A complete rewrite of Atmos disk management layer has been put into place for Atmos 2.1. Existing python based management scripts have been replaced by a C++ library which is currently executed as part of the cluster management. This new design is more reliable and implements a well understood state machine. The new code handles disk detection, disk formatting, and disk health monitoring. An adjunct component called the Service Configurator also handles disk and file system provisioning to the MDS and SS services. Disk management also interacts with recovery manager to report disk and DAE failures By design, the disk management understands and deals with hierarchical relationship between disks and DAEs and between disks, volumes and file systems. The RMG database is updated for presentation purposes and should be in synch under normal conditions, but Disk Management behavior is no longer dependent on the contents of RMG DB. The default file system for all DAE disks is XFS (ext3 is still used for OS drives). Note: Local Repository in /var/local/maui/atmos-diskman and copied to 2 DAE hosted file systems to support node replacement. Logging provided in /var/local/maui/dm.log Disk related events posted to Event Log (Category = Disk) MDS assigned file systems mounted at /atmos/<fsuuid>. SS assigned file systems mounted at /mauiss-disks/ss-<fsuuid> 19
When a disk is determined to be disconnected or failed, disk management will notify the recovery manager of the failure and if the DAE is healthy. The recovery manager will schedule a Storage Service recovery job when notified. When the DAE is healthy the recovery job will be scheduled to start in one hour. When the DAE is determined to be unhealthy the recovery job will be scheduled to start in one week. Only one recovery job per node will be started by the recovery manager at one time. The user can view scheduled recovery jobs in the System Summary screen of the GUI. The SysAdmin can start pending jobs or stop active jobs and can force more than one job to be run on a node. The GUI will report on completion of the task. All MDS sets must be accessible for a recovery job to complete and have accurate data about objects on the failed disk. 20
Improved disk information details the DAE information along with the slot ID of the failure. Free space and disk status are also displayed. 21
With MDS Management, in the event of a disk failure, MDS evacuation of a disconnected file system will wait for 10 minutes to pass before moving requests to another MDS. Immediate evacuation for file systems with a health status failure will occur. Logging information can be found in the /var/local/maui/dm.log. For Storage Services Management, the system will wait one minute before applying configuration changes. Storage Services file systems will be removed from the configuration upon detection of a disconnect or health status failure. Storage Services file systems are re-added to the configuration upon reconnection of a healthy disk. Logging information can be found in the /var/local/maui/dm.log. 22
Most service and feature configurations are now under the control of the configuration management subsystem created in the Atmos 2.0 software release. This provides the ability to scope configurations to different sets of nodes (Node, IS, RMG, System). Scoping allows a user to have a default system wide configuration for a feature such as SMTP or NTP as well as to create RMG specific configurations as needed. System management automatically ensures a node has the latest configuration information on startup. If the latest configuration information cannot be synched to the node it will remain in maintenance mode (init level 3) until the problem can be resolved, (i.e. the system will not perform any I/O). 23
Configuration information is maintained in the Configuration Management Database and needs to be updated via the GUI, or CLI, or hand-edited changes will be lost. The Configuration Manager ensures that replaced nodes do not lose customizations. Service configuration files should NOT be edited by hand. 24
This module focuses on network management, Channel Bonding and VLAN tagging. 25
Network management in Atmos 2.1 allows for bonding of multiple public Atmos interfaces. VLAN tagging is at Layer 3 of the network stack. This process is now supported via API, no Request for Price Quotation (RPQ) is required. This provides users with additional network flexibility for performance and high availability. The user can now setup data and/or administration path segmentation. The benefits of this is that it integrates the network configuration with the installation process or that VLAN tagging can be implemented at a later time. This provides improved network management visibility. 26
With embedded network management, you now have the ability to channel bond across Ethernet interfaces 1,2,3. Support for VLAN tagging is supported across all external Ethernet interfaces. You can also modify the internal Atmos IP address ranges if so desired. What this gives the customer is complete network management control. It is supported by an embedded CLI. Network configuration information is sent via connecthome to SYR for system records. The network management configurations are preserved with upgrades/node replacements through the use of HAL, the hardware abstraction layer now present across all nodes in the system. 27
The master node can be installed with optional VLAN tagging information for the initial public interface during a new installation. If configured at this time, the VLAN information is propagated automatically for all slave nodes in the segment. A single public IP address will be shared for the management, data and access traffic. All other network management configuration is done after the nodes have been installed, either before or after activation of the installed nodes. 28
You can modify the public (eth1 and above) network by using the network management CLI (netmgr). The network management CLI lets you: Bond physical interfaces for failover and to increase the bandwidth for the public network. You can bond eth1, eth2, and eth3, or any subset of these interfaces. Once the interfaces are bonded, you can create virtual interfaces, assign IP address, and optionally assign VLAN tags. You can also separate the network traffic on eth1, so that access, data, and system management use different IP addresses. Note: Bonding is not supported on Gen1 hardware or when using the virtual edition of Atmos 2.1 29
In this example eth1, eth2, and eth3 can be bonded to form bond0. The bond0 interface is assigned three virtual interfaces bond0, bond0:1, and bond0:2. Each bond0 virtual interface must be assigned its own IP address. bond0 is automatically assigned to the system management network, and the original IP address assigned at installation time will be used (This assignment cannot be modified). You can optionally tag the data, management or access traffic. 30
Here is a list of limitations for each type of network modification. You must document the network changes you make to the system. None of the network changes are shown in the Atmos system management GUI at this time. To start and stop the virtual interfaces, use the ifup/ifdown commands. Do not use the ifconfig command. 31
Ethernet channel bonding and network separation (VLAN tagging) is an industry standard and now a supported feature of Atmos 2.1. It completely integrates the network configuration with the installation procedures or can be modified after installation. Network configuration is automatically reapplied during a node replacement. The CLI can be used for querying and manipulating the Atmos network configuration. 32
Listed here are some basic network management troubleshooting techniques. 33
This module focuses on the S3 native API support 34
Amazon S3 (Simple Storage Service) is an online storage web service offered by Amazon Web Services. Amazon S3 provides storage through web services interfaces (REST, SOAP, and BitTorrent). Amazon launched S3, its first publicly available web service, in the United States in March 2006 and in Europe in November 2007. S3's design aims to provide scalability, high availability, and low latency at commodity costs. Note: REST - REpresentational State Transfer, is a style of software architecture for distributed systems such as the World Wide Web. REST has emerged over the past few years as a predominant Web service design model. REST has increasingly displaced other design models such as SOAP and WSDL due to its simpler style. SOAP - Simple Object Access Protocol, is a protocol specification for exchanging structured information in the implementation of Web Services in computer networks. BitTorrent - peer-to-peer file sharing protocol used for distributing large amounts of data over the Internet. BitTorrent is one of the most common protocols for transferring large files and it has been estimated that peer-to-peer networks collectively have accounted for roughly 43% to 70% of all Internet traffic (depending on geographical location) as of February 2009. 35
Atmos S3 native API support is an Atmos API module similar to the REST API, which supports the majority of the most commonly used S3 calls. This API module enables Atmos compatibility with commercial or open source S3 applications. It also enables increased security with private cloud implementations. The benefits for customers are that the API is fully integrated and a supported feature of Atmos. You can now migrate existing custom S3 applications to the Atmos platform environment. No additional development is needed for application migration. 36
When using Atmos 2.0 software, S3 was supported by Atmos using a S3/Atmos Proxy server, which is a dedicated service that acts like a S3 API gateway and transforms S3 REST API calls into Atmos REST API calls. It is implemented as a web application that can be deployed in any of the servlet containers that conform to the Java Servlet API specification of version 2.5. For better scalability, S3/Atmos Proxy used a distributed centralized storage. This centralized storage is for maintaining configuration information, naming, distributed synchronization, and providing group services. It could be installed in a HA mode, with several coordination instances, which eliminates a single point of failure. 37
EMC source licensed the S3 Proxy code for existing Atmos customers. The licensing is similar to the ACDP portal licensing. The proxy provided a mechanism to migrate applications from S3 to Atmos without requiring application modification or additional development. The client/app just needs to replace the S3 credentials (Access Key ID/Secret Access Key) with the Atmos credentials (userid/shared secret). The proxy service sits between the client/app and an Atmos installation. The proxy intercepts S3 calls, translates them to Atmos requests. On the way back, the Atmos responses are converted to S3 response format. 38
Atmos 2.1 software now has S3 native API support. This native implementation increases performance and decreases latency. There is no need to have external proxy server for S3 translation. With the simple S3 native infrastructure, there are no additional nodes required for support. ESRS and SYR integration is coupled into the management and serviceability aspects of Atmos 2.1. 39
The native implementation of the S3 API is contained in the Atmos 2.1 software. The S3 native API does not require application modification or additional development for existing S3 applications to work with Atmos. The client/app just needs to replace the S3 credentials (Access Key ID/Secret Access Key) with the Atmos credentials (userid/shared secret). The S3 API module runs on the Atmos nodes, just like the REST API module. The S3 module receives the S3 API calls and talks directly to the Atmos Client library. 40
As mentioned, applications written for the S3 API can now communicate directly to an Atmos node. Implementation of this S3 native API does not support the full S3 API call set. Shown here are the supported calls. 41
This module focuses on CAS enhancements to Atmos 42
Atmos 2.1 policies have been enhanced to support specific CAS metadata. There is improved monitoring and support for audit logging. Atmos continues to enhance CAS capabilities in the system architecture. Atmos CAS users can now leverage dynamic object level policy selection. 43
Atmos Storage Administrators can now register tags per CAS application. They can define more granular information for search criteria provided by a combination of subtenantid and UID information. Both System and User metadata tags can now be registered. Once tags are registered with a application, only new objects (CDF and Blobs) will have the tags transmitted to MDS. This transmission of new objects with metadata is a background process. The maximum number of tags that are allowed per application is 100. The removal of tags also supported. With the new capabilities, adding or removing tags will result in audit logging. Centera objects can have tag size of up to 100KB. Atmos MDS supports only up to 1KB values. Data > 1KB is truncated. 44
Here we see a sample Centera CDF object. In this example, we want to register a tag for system type and a tag called state. To do this we need to set up a policy as such. 45
Now there is support in the GUI for customers to define the set of CAS metadata to be transferred with objects. This broadens the available CAS object metadata for Atmos policies. After the tenant and subtenant are created, the user will log into the subtenant where policy specifications, policy transitions and policy selectors are defined. Here we see the UID list where we can setup/view the CAS application configuration. You select the view link which then opens a dialog box to specific application criteria. 46
After setting up application specific information the user will then add a policy specification which defines what to look for. In this slide we see the policy selector with the metadata value which defines the search information. Once saved you will change the default system policy for this given specification. We will create two synchronous copies and asynchronous copy if the state equals Rhode Island. 47
With these CAS enhancements, monitoring has been enhanced to support this feature. There are alerts which leverage the Atmos Event Manager. Call home capabilities can be enabled using SYR. Along with this is audit logging to keep track of who does what in the system. 48
C2A is the migration of objects from a single Centera virtual pool to a single subtenant on the Atmos 2.1 system configured and running CAS. All C2A Migrations requires CCA approval. Consult PS engagement number PS-BAS-CENTRA or via email at this address, c2a_cc@emc.com for more information. 49
In this course we discussed then demonstrated the Graphical User Interface enhancements with a video tour Explained key components of Event, Configuration and Disk management features Defined the Network Management capabilities and showed configuration with a video demonstration Reviewed S3 and the features incorporated into the Atmos 2.1 software Explained the Atmos CAS enhancements This concludes the training. Proceed to the course assessment on the next slide. 50
51