EMC ISILON STORAGE BEST PRACTICES FOR ELECTRONIC DESIGN AUTOMATION

Transcription

1 White Paper EMC ISILON STORAGE BEST PRACTICES FOR ELECTRONIC DESIGN AUTOMATION Abstract This paper describes best practices for setting up and managing an EMC Isilon cluster to store data for electronic design automation. June 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC 2, EMC, the EMC logo, Isilon, FlexProtect, InsightIQ, OneFS, SmartConnect, SmartPools, SmartQuotas, SnapshotIQ, and SyncIQ are registered trademarks or trademarks of EMC Corporation in the United States and other countries. H

3 Table of Contents Introduction... 6 EDA workflows and workloads... 6 EMC Isilon scale-out NAS... 7 Isilon node... 8 Overcome EDA storage challenges with EMC Isilon... 8 Improve runtimes for concurrent jobs... 9 Overview of best practices for EDA... 9 Obtain statistics to tune an EDA workflow... 9 Match workloads with nodes and storage pools... 9 Network connections File system and protocols Limits for pools, directories, files, and names Data protection EMC Isilon SnapshotIQ EMC Isilon SyncIQ EMC Isilon SmartQuotas Identity management, authentication, and access control Permissions Home directories Summary of high-level best practices for EDA Obtain statistics to tune an EDA workflow Optimize performance with storage analytics Test changes before putting them into production Match workloads with nodes and storage pools Analyze connections and access patterns Align workloads with data access patterns Align datasets with storage pools SmartPools for EDA data SSD strategies Global namespace acceleration Write caching with SmartCache Guidelines for file pool management Check the optimization settings of directories and files Plan for network performance and throughput Networking Ethernet speeds Frame sizes Internal InfiniBand network redundancy IP address allocation planning EMC Isilon SmartConnect

4 Connection-balancing strategies High availability with dynamic NFS failover Connection-balancing and failover policies Service subnets and DNS integration SmartConnect and failover IP optimization SmartConnect best practices Size a SmartConnect pool Optimize throughput and performance File systems and data access protocols Structure of the file system Data layout Caching Request block sizes and random read and write performance NFS NFS performance settings Use NFS version 3 or NFS over TCP, not UDP Sync and async options NFS rsize and wsize Hard mounts Multiple clients writing to the same files Enable readdirplus on clients Recommended client mount settings and system controls NFS server threads SMB Change notify SMB signing Limits for pools, directories, files, and names Pool capacity Maximum number of files in a directory Directory structures Maximum depth of a directory Maximum path length of names for nested directories Data protection N+M data protection Data protection best practices Data mirroring Balance data protection with storage utilization Virtual hot spare EMC Isilon SnapshotIQ Snapshot best practices

5 EMC Isilon SyncIQ Business continuance Avoid full dataset replications Select the right source replication dataset Performance tuning guidelines Limitations and restrictions Using SmartConnect with SynclQ Performance and policy job monitoring Target-aware initial synchronization Best practices for SyncIQ jobs EMC Isilon SmartQuotas Include data protection overhead in disk usage calculations Include the space that snapshots consume View reports to manage quotas Additional best practices for setting quotas Permissions for mixed environments On-disk identity Run the repair permissions job after changing the on-disk identity ACL policies for mixed environments Permissions policies Run chmod on a file with an ACL The inheritance of ACLs created on directories by chmod Chown command on files with ACLs Access checks (chmod, chown) Advanced settings Owner and group permissions No deny ACEs Home directories Sizing guidance for storage capacity SnapshotIQ Capacity planning considerations Capacity planning best practices Conclusion

6 Introduction The EMC Isilon scale-out network-attached storage (NAS) platform combines modular hardware with unified software to harness unstructured data. Powered by the distributed EMC Isilon OneFS operating system, an EMC Isilon cluster delivers a scalable pool of storage with a global namespace. The use of distributed software to scale data across commodity hardware sets OneFS apart from other storage systems. Each node in an Isilon cluster controls data requests, boosts performance, and expands the cluster's capacity. For electronic design automation (EDA), the Isilon scale-out distributed architecture minimizes CPU bottlenecks, rapidly serves metadata, and optimizes wall clock performance for concurrent jobs. This paper describes best practices for managing an Isilon cluster to maximize performance for EDA workflows. EDA workflows and workloads An EDA workflow includes a logical and a physical phase. During the logical phase, engineers architect a chip design by compiling source files into a chip model. As engineers create the design, they check out its source code from a software configuration management system, such as Subversion (SVN) or Perforce, to refine specifications. Engineers then simulate the chip design by scheduling and running jobs in a large compute grid. Scheduling the compile and simulation jobs involves using a scheduler, such as the IBM Platform Load Sharing Facility (LSF) or the Oracle Grid Engine. The scheduler distributes the build and simulation jobs to the available slots on the compute resources. Efficiency in creating, scheduling, and executing both build and simulation jobs can reduce the time it takes to bring a chip to market. The logical phase generates an input/output (I/O)-intensive workload: EDA applications read and compile thousands of source files to build and simulate a chip design. The logical phase also invokes other applications, scripts, and verification processes, which vary by environment. A storage system manages the various design projects and files so that different users, scripts, and applications can access the data. Within the storage system, EDA workflows tend to store a large number of files in a single directory amid a deep directory structure on a large cluster, often with more than 20 Isilon nodes. Project directories dominate the file system, and the project directories require performance for the hundreds of projects stored on them. Projects that build source code read and write thousands of small files, while projects that build simulations read and write many large files. The workflow for the projects includes backing up data and taking several snapshots of the directories daily and nightly. Given the large number of files and directories as well as the growing size of design files over time, the storage system must make file management easy and provide seamless access to random file types. EDA uses a large compute grid, or build farm, for high-performance computing. The grid, which can number more than a 1,000 client computers, requires several IP addresses for each Isilon node to distribute client connections effectively and to redistribute connections in case a node fails. A cluster may also include home 6

7 directories for users and the safe archiving of design blocks for reuse in future projects. The clients and applications in the compute grid determine the cluster's workload: Across the EDA industry, I/O profiles vary by tool and design stage, from many small random workloads to large sequential workloads. A common denominator is workloads that run many build jobs concurrently, generating high CPU usage on the storage system as the jobs access directories and files. In many cases, most file system operations get attributes, perform lookups, or retrieve other metadata as the workflow consolidates up to millions of small files, each describing a gate or block, into a file of several terabytes for the physical design phase. Meanwhile, other aspects of the design cycle depend heavily on read and write operations. Hardware-software verification tests, for example, can result in a large single-threaded read operation followed by many slow write operations. For most workflows, the intensity of I/O operations necessitates solid-state drives (SSDs), write coalescing, coherent caching, and clustered RAM to efficiently serve requests for files and metadata. To serve I/O requests with the kind of performance that EDA tools require, the storage system should hold the working set of data in memory. After the logical phase produces a design that works, the physical design phase converts the logical design into a physical chip, verifies it, and prepares it for tape-out to a foundry, which manufactures the chip. The tape-out process combines the many layers of a chip s design into a single model to send to the foundry for manufacturing. The workflows and workloads of the logical phase constitute a requirement to optimize the system that stores the design files. Tuning the storage system to serve data without bottlenecks and to support running concurrent jobs is key to reducing a chip's time to market. As the next sections of this paper demonstrate, the architecture of Isilon scale-out NAS is ideally suited to reduce bottlenecks and optimize concurrency. Later sections consider options for tuning an Isilon cluster to help meet these objectives. EMC Isilon scale-out NAS OneFS combines the three traditional layers of storage architecture file system, volume manager, and data protection into a scale-out NAS cluster. In contrast to a scale-up approach, EMC Isilon takes a scale-out approach by making a cluster of nodes that runs a distributed file system. Each node adds resources to the cluster. Because each node contains globally coherent RAM, as a cluster becomes larger, it becomes faster. Meanwhile, the file system expands dynamically and redistributes content, which eliminates the work of partitioning disks and creating volumes. There is no CPU controller head to cause bottlenecks. Nodes work as peers to spread data across the cluster. Segmenting and distributing data a process known as striping not only protects data, but also enables a client connecting to any node to take advantage of the entire cluster's performance. 7

8 For an EDA workflow, the Isilon scale-out distributed architecture eliminates CPU bottlenecks and optimizes the performance of processing concurrent jobs when you add the right number of nodes to the cluster to serve a compute grid. Isilon node As a rack-mountable appliance, a node includes the following components in a 2U or 4U rack-mountable chassis: memory, CPUs, RAM, NVRAM, network interfaces, InfiniBand adapters, disk controllers, and storage media. Each node runs the Isilon OneFS operating system, the distributed file system software that unites the nodes into a cluster. An Isilon cluster comprises three or more nodes, up to 144. A cluster s storage capacity ranges from a minimum of 18 TB to a maximum of 15.5 PB. When you add a node to a cluster, you increase the cluster's aggregate disk, cache, CPU, RAM, and network capacity. OneFS groups RAM into a single globally coherent cache so that a data request on a node benefits from data that is cached anywhere. NVRAM is grouped to write data with high throughput and to protect write operations from power failures. As the cluster expands, spindles and CPU combine to increase throughput, capacity, and I/O operations per second (IOPS). EMC Isilon makes several types of nodes, all of which can be added to a cluster to balance capacity and performance with throughput or IOPS. Table 1: EMC Isilon nodes Node S-Series X-Series NL-Series Use case IOPS-intensive applications such as EDA High-concurrency and throughput-driven workflows Near-primary accessibility, with near-tape value The EMC Isilon Performance Accelerator extension node provides independent scaling for high performance by adding processing power, memory, bandwidth, and parallel read/write access. Overcome EDA storage challenges with EMC Isilon EMC Isilon S-Series nodes deliver the performance that EDA workflows and workloads demand. The EMC Isilon S200 node delivers 1.1 million network file system (NFS) SPECsfs2008 file operations per second with more than 100 GB/s of aggregate throughput. The S200 combines SSDs with 10,000 RPM 2.5-inch Serial Attached SCSI (SAS) drive technology, two quad-core Intel CPUs, and up to 13.8 TB of globally coherent cache. SSD technology accelerates namespace-intensive metadata operations. You can optimize storage for EDA workflows by creating policies to store metadata and other latency-sensitive data on SSDs. To help eliminate bottlenecks, the OneFS operating system includes several cache types, a prefetching option, and dual 10 gigabit Ethernet (GbE) connections. The globally coherent cache provides rapid access to stored data by connecting to any node. You can also easily scale the globally coherent cache to hold your working data set in memory as the data set grows in size. The local coherent cache delivers rapid access to data stored on a node when a client connects to it. 8

9 By default, OneFS optimizes operations to get attributes (getattr) with a built-in stat() cache. Similarly, OneFS prefetches readdirplus data. You can adjust the number of file nodes that OneFS prefetches for readdirplus an option that you can use to tailor readdirplus to your workflow. OneFS also provides several options for prefetching data to address streaming, random, and concurrent access patterns. The dual 10 GbE connections to each node support the high levels of network utilization that take place during EDA s simulation and verification phrases. Improve runtimes for concurrent jobs A critical question is how many Isilon S-Series nodes should be included in a cluster so that the cluster processes, with the optimal level of performance, concurrent I/O requests from the computer grid. With the right number of nodes, an Isilon cluster can reduce wall clock runtime for concurrent jobs after the time that other storage systems hit a saturation point for the controller. Finding an Isilon cluster s optimal point the point at which it scales in processing concurrent jobs and reduces wall clock runtimes in relation to other systems for the same workload depends on the size of your compute grid, the number of jobs, the working datasets, and other factors. EMC Isilon recommends that you work with your Isilon representative to determine the number of nodes that will best serve your workflow. Overview of best practices for EDA You can tune an EMC Isilon cluster to boost performance and throughput for electronic design automation. But because every EDA workflow, network, and computing environment is different, tailoring a cluster to meet your needs requires research, consultation with your Isilon representative, testing, and a cost-benefit analysis. This document provides guidelines to help you make decisions about how to set up a large cluster that will maximize the storage performance of an EDA workflow. Keep in mind, however, that there are no generic answers or configuration templates for EDA. How you tune your cluster depends on several primary factors: The characteristics of your workflows The characteristics of your working datasets The size of your compute grid Obtain statistics to tune an EDA workflow Before you tune your Isilon cluster or your clients, you should analyze how your EDA workflow interacts with the storage system by gathering statistics about your common file sizes and I/O operations, including CPU usage and latency. Match workloads with nodes and storage pools You should match performance-demanding workflows with high-performance nodes. The Isilon S200 with SSDs performs well for EDA workflows. The SSDs enhance the performance of NFS lookups, getattr operations, and other metadata operations. The 9

10 metadata operations pull data quickly from the disks when the metadata is not found in the cache. An Isilon cluster can also combine nodes of different types into a single file system and then segment nodes into storage pools with different capacity-to-performance ratios. A best practice is to use EMC Isilon SmartPools policies to implement storage pools that maximize performance for working datasets while cost-effectively storing less important data. File pool policies can automate the distribution of different file types to a storage tier that matches their performance requirements. A file pool policy can also change the layout of the data on the underlying disk for optimal read and write performance. Network connections For EDA workflows, you should establish two 10 GbE connections to each node to support the high levels of network utilization that take place during the simulation and verification phrases. Using 10 GbE connections also helps support the workload of EDA tools like Synopsys and Cadence. A best practice is to bind multiple IP addresses to each node interface in an EMC Isilon SmartConnect network pool. Generally, optimal balancing and failover is achieved when the number of addresses allocated to the network pool equals N * (N 1), where N equals the number of node interfaces in the pool. Thus, if a pool is configured with a total of five node interfaces, the optimal IP address allocation would total 20 IP addresses (5 * (5 1) = 20) to allocate four IP addresses to each node interface in the pool. Assigning each workload or data store to a unique IP address enables Isilon SmartConnect to move each workload to one of the other interfaces, minimizing the additional workload that a remaining node in the SmartConnect pool must absorb and ensuring that the workload is evenly distributed across all the other nodes in the pool. File system and protocols You can optimize how the OneFS operating system lays out and prefetches data to match your dominant access pattern concurrent, streaming, or random. For most EDA workflows, the default concurrent access pattern will usually work best. You can also modify the OneFS caches and NFS performance settings to improve performance for EDA. Limits for pools, directories, files, and names A best practice to maintain optimal performance is to fill a cluster's capacity only up to 80 percent per pool, especially for an EDA workflow. In general, it is more efficient to create a directory structure that consolidates files in a single directory than it is to spread files out over many subdirectories because OneFS can more efficiently add a B-tree block to a directory than it can add an inode to a subdirectory. With SSDs, you should limit the number of files in a directory to 100,000. With hard disk drives (HDDs), you should limit the number of files in a directory to 20,

11 You should probably limit the number of subdirectories to 10,000 because exceeding 10,000 subdirectories might affect the performance of command-line tools and other applications. Data protection For most EDA workflows in which optimizing for an aggregation of workloads trumps optimizing for a single job you should consider using +2:1 for clusters with fewer than 18 nodes. For clusters with more than 18 nodes, consider using +2. The protection level that you select, however, depends on various factors, including the number of nodes in the cluster. The right protection level for your cluster might also depend on which version of OneFS you are running. Because so many variables interact to determine the protection level for an environment, a best practice is to consult with your Isilon representative about selecting a protection level. Isilon can analyze the cluster to identify its mean time to data loss and then suggest an optimal policy for you. EMC Isilon SnapshotIQ The best practices for snapshots depend on factors and objectives that vary across environments. For EDA workflows, a key question is whether to take snapshots on a per-project basis or an all-project basis. A per-project approach, for instance, might take a separate snapshot of each directory in which a project resides. An all-project approach, in contrast, might take one snapshot at the high-level directory that contains all the project directories. One strategy is to minimize the number of snapshot policies, thereby reducing the number of snapshots that you need to manage and decreasing the number of snapshot delete jobs. When you delete snapshots, it is preferable to have a small number of snapshots in a snapshot delete job. EMC Isilon SyncIQ For EDA workflows, business continuance is important. EMC Isilon SynclQ maximizes the use of network bandwidth, delivering fast replication times for tight recovery point objectives (RPOs). You can also use SynclQ with EMC Isilon SnapshotIQ for storing as many point-in-time snapshots of your data as needed to support secondary activities like backing up data to tape. The performance tuning guidelines in this document can help you exploit SyncIQ to support business continuance. EMC Isilon SmartQuotas The EMC Isilon SmartQuotas module tracks disk usage with reports and enforces storage limits with alerts. Using the SmartQuotas module is a best practice because it helps optimize storage capacity while minimizing costs. EMC Isilon recommends that you send notifications to users when they approach their storage limits and that you analyze reports to allocate storage more efficiently. Each quota that you set should include the overhead for data protection and the space that snapshots consume. Do not create quotas on the root directory of the default OneFS share (ifs) because the quota might degrade performance. Governing a single directory with overlapping quotas can also degrade performance. 11

12 Identity management, authentication, and access control A best practice is to use Microsoft Active Directory with Windows Services for UNIX and RFC 2307 attributes to manage Linux, UNIX, and Windows systems. Integrating UNIX and Linux systems with Active Directory centralizes identity management and eases interoperability. Make sure that your domain controllers are running Windows Server 2003 or later. If you use Lightweight Directory Access Protocol (LDAP) for Linux systems and Active Directory for Windows systems, the OneFS user mapping service can help manage users across domains. Permissions An Isilon cluster s default settings handle permissions securely and effectively for most networks that mix UNIX and Windows systems. EMC Isilon recommends that you use the option to configure permissions policies manually. Managing permissions policies manually gives you a range of options for responding to the kind of special cases that can surface in a mixed environment. In general, however, EMC Isilon recommends that you avoid changing the advanced settings for permissions. For chmod commands, using the option to merge new permissions with the existing ACL is recommended for mixed networks. Merging the new permissions with the ACL is the best way to balance the preservation of security with the expectations of users. For chown commands, using the option to leave the ACL unmodified is the recommended approach because it preserves the ACEs that explicitly allow or deny access to specific users and groups. Home directories Capacity planning entails scaling an Isilon cluster to accommodate the competing demands of combined workloads. In the case of home directories, workload requirements are driven by several factors: disk capacity to accommodate the combined data storage requirements of all targeted users; sufficient disk throughput to support the combined transactional requirements of all users; and enough network bandwidth to provide adequate throughput. This paper includes guidelines to plan home directories amid competing workflows. Summary of high-level best practices for EDA Analyze storage statistics to identify areas for optimization Maximize the use of EMC Isilon S-Series nodes to deliver high-performance storage for EDA workflows Store metadata and the active working set of file data on solid-state drives Create SmartPools policies to manage how latency-sensitive data is stored Add one or more Isilon Performance Accelerator nodes to help ensure that the active working set stays in memory Work with Isilon sales engineers to establish the right number of nodes for your workflow to improve wall clock performance for concurrent jobs Set up an external network with dual 10 GbE connections to each storage node Bind multiple IP addresses to each node interface in a SmartConnect network pool Use SnapshotIQ and SyncIQ for data protection and business continuance Apply the SmartQuotas module to track disk usage with reports and enforce limits with alerts 12

13 Obtain statistics to tune an EDA workflow Before you tune your Isilon cluster or your clients, you should analyze how your EDA workflow interacts with the storage system by gathering statistics about your common file sizes and I/O operations, including CPU usage and latency. With OneFS 7.0 or later, you can obtain key statistics and timing data for delete, renew lease, create, remove, set userdata, get entry, and other file system operations by connecting to a node with SSH and running the following command as root to turn on the vopstat system control: sysctl efs.util.vopstats.record_timings=1 efs.util.vopstats.record_timings: 0 -> 1 After you turn on vopstats, you can view them by running the following sysctl efs.util.vopstats command as root: sysctl efs.util.vopstats (You can also run the following command on OneFS 6.x, but the results do not include timing data.) Here is an example of the command s output: efs.util.vopstats.ifs_snap_set_userdata.initiated: 26 efs.util.vopstats.ifs_snap_set_userdata.fast_path: 0 efs.util.vopstats.ifs_snap_set_userdata.read_bytes: 0 efs.util.vopstats.ifs_snap_set_userdata.read_ops: 0 efs.util.vopstats.ifs_snap_set_userdata.raw_read_bytes: 0 efs.util.vopstats.ifs_snap_set_userdata.raw_read_ops: 0 efs.util.vopstats.ifs_snap_set_userdata.raw_write_bytes: 0 efs.util.vopstats.ifs_snap_set_userdata.raw_write_ops: 0 efs.util.vopstats.ifs_snap_set_userdata.timed: 0 efs.util.vopstats.ifs_snap_set_userdata.total_time: 0 efs.util.vopstats.ifs_snap_set_userdata.total_sqr_time: 0 efs.util.vopstats.ifs_snap_set_userdata.fast_path_timed: 0 efs.util.vopstats.ifs_snap_set_userdata.fast_path_total_time: 0 efs.util.vopstats.ifs_snap_set_userdata.fast_path_total_sqr_time: 0 The time data captures the number of operations that cross the OneFS clock tick, which is 10 milliseconds. Independent of the number of events, the total_sq_time provides no actionable information because of the granularity of events. To analyze the operations, use the total_time value instead. The following example shows only the total time records in the vopstats: sysctl efs.util.vopstats grep -e "total_time: [^0]" 13

14 efs.util.vopstats.access_rights.total_time: efs.util.vopstats.lookup.total_time: efs.util.vopstats.unlocked_write_mbuf.total_time: efs.util.vopstats.unlocked_write_mbuf.fast_path_total_time: efs.util.vopstats.commit.total_time: efs.util.vopstats.unlocked_getattr.total_time: efs.util.vopstats.unlocked_getattr.fast_path_total_time: efs.util.vopstats.inactive.total_time: efs.util.vopstats.islocked.total_time: efs.util.vopstats.lock1.total_time: efs.util.vopstats.unlocked_read_mbuf.total_time: efs.util.vopstats.readdir.total_time: efs.util.vopstats.setattr.total_time: efs.util.vopstats.unlock.total_time: efs.util.vopstats.ifs_snap_delete_resume.timed: efs.util.vopstats.ifs_snap_delete_resume.total_time: efs.util.vopstats.ifs_snap_delete_resume.total_sqr_time: With OneFS 6.0 or later, you can run the following command to obtain statistics for protocols, client connections, and the file system: isi statistics To view NFS protocol statistics, for example, you can run the isi statistics command like this: isi statistics pstat --protocol=nfs3 Here is an example of the output: NFS3 Operations Per Second null 0.00/s getattr 74.14/s setattr 4.53/s lookup 1.40/s access 9.60/s readlink 0.00/s read /s write /s create 0.20/s mkdir 0.00/s symlink 0.00/s mknod 0.00/s remove 0.00/s rmdir 0.00/s rename 0.20/s link 0.00/s readdir 0.00/s readdirplus 0.00/s statfs 0.18/s fsinfo 0.00/s pathconf 0.00/s commit 10.82/s noop 0.00/s TOTAL /s CPU Utilization OneFS Stats user 1.5% In MB/s system 7.6% Out MB/s idle 91.0% Total MB/s Network Input Network Output Disk I/O MB/s MB/s Disk iops Pkt/s Pkt/s Read 7.52 MB/s Errors/s 0.00 Errors/s 0.00 Write MB/s 14

15 You can also run the isi statistics command with the client argument to view I/O information and timing data by client: isi statistics client And you can run the isi statistics command with the system argument to view CPU utilization by protocol. For example: isi statistics system Node CPU SMB FTP HTTP ISCSI NFS HDFS Total NetIn NetOut DiskIn DiskOut LNN %Used B/s B/s B/s B/s B/s B/s B/s B/s B/s B/s B/s All K K 26K 134M M 37M 101M 178M 289M Other performance statistics appear in the web administration interface. For the various methods of obtaining statistics about the storage system, see the OneFS Administration Guide and the OneFS Command Reference. A key question is, Which types of operations consume the most CPUs or result in the most access latency? Create and remove operations, in particular, can consume more CPUs than other operations, but whether create and remove operations are affecting performance depends on your workflow and working set. For EDA, readdirplus as well as read and write operations can also be costly. If a directory contains a million files, enumeration might cost too many resources in comparison to other operations. Once you can associate a type of operation with a degradation in performance by, for example, analyzing timing data and CPU usage you can tailor the system to process the operation faster. Optimize performance with storage analytics EMC Isilon InsightIQ monitors and analyzes the performance of an Isilon cluster to help you optimize storage resources and forecast capacity. To maximize performance over time, a best practice is to track performance and capacity with InsightIQ, which is an optionally licensed feature. Test changes before putting them into production Before you modify your production cluster, you should, if possible, implement the changes on a test cluster that uses the same version and the same settings as your production cluster. You can validate your key workflows on a test cluster by simulating how your administrators, users, and applications interact with the system. Verifying optimizations on a test cluster can expose issues that could degrade the performance of your production system. But verifying optimizations on a test cluster is unlikely to be feasible for EDA workflows. The test environment will probably not replicate the real workflow, or test results might be inconclusive or, worse, they could be misleading. If you make changes to a production cluster, you must be prepared to revert them immediately if they produce side effects or if they fail to enhance performance. 15

16 Match workloads with nodes and storage pools This section discusses how to match a workflow with the different types of Isilon nodes and how to use storage pools to streamline an EDA workflow. Analyze connections and access patterns A critical factor in building a cluster and tuning its performance is having a clear understanding of the access patterns of users and applications. For example: How many clients does the cluster need to support? What percentage of the client connections do you expect to be active at a given time? What is the dominant file-sharing protocol, Server Language Block (SMB) or NFS? What is the volume of workload that the active connections carry? What types of workloads do the active connections carry? And what are the characteristics of the workloads large files or small files? What are the characteristics of the working datasets? What is the dominant access pattern for each dataset? The number of projected active user connections is a key figure. Storage workloads tend to be driven more by the number of active connections than by the number of overall connections. A thousand users, for example, might not require a high level of sustained throughput if only a hundred of the users maintain active connections at one time. The dominant file-sharing protocol that your clients use can play a role in determining the cluster s requirements for CPUs, memory, and network bandwidth. SMB client connections typically require a higher amount of overhead per client than do comparable workloads from NFS clients, especially those using Network File System version 3 (NFSv3). In contrast, network and disk throughput rates stem more from the workload and use case than from the file-sharing protocol. Align workloads with data access patterns Once you identify your dominant access pattern concurrent, streaming, or random you can optimize how OneFS lays out data to match it. By default, OneFS optimizes data layout for concurrent access. For most EDA workflows, the default concurrent access pattern, with its medium level of prefetching data, will often work best. If your dominant access pattern is streaming that is, it has lower concurrency, higher single-stream workloads you can prefetch data well in advance of data requests to increase sequential read performance. Streaming is most effective on clusters or storage pools serving large files. If your workload is truly random, the random access pattern, which prefetches no data, might increase performance. Table 2: OneFS access pattern options OneFS access pattern Concurrent Streaming Random Description Prefetches data at a medium level Prefetches data far in advance of requests Prefetches no data 16

17 With a SmartPools license, you can apply a different access pattern to each storage pool that you create. This enables you to set the optimal pattern for each dataset. Align datasets with storage pools With a SmartPools license, you can create node pools, file policies, and tiers of storage. Node pools segment nodes into groups so that you can align a dataset with its performance requirements. File polices can isolate and store files by type, path, size, and other attributes. Tiers optimize the storage of data by need, such as a frequently used high-speed tier or a rarely accessed archive. Because you can combine EMC Isilon nodes from the S-Series, the X-Series, and the NL-Series into a cluster, you can set up pools of nodes to accelerate access to important working sets while placing inactive data in more cost-effective storage. For example, you can group S-Series nodes with 900 GB SAS drives and 400 GB SSDs per node in one pool, while you put NL-Series nodes with 3 TB SATA drives in another. As each type of node has a different capacity-to-performance ratio, you should assign different datasets to node pools that meet the dataset s capacity or performance requirements. In this way, the node pools can isolate the datasets of critical applications from other data sets to increase performance for important data and decrease storage costs for immaterial data. After you set up node pools, you can establish file pools and govern them with policies. The policies move files, directories, and file pools among node pools or tiers. Routing files with a certain file extension into a node pool containing S-Series nodes can deliver faster read and write performance. Another policy can evaluate the lastmodified date to archive old files. SmartPools also manages data overflow with a spillover policy. When a pool becomes full, OneFS redirects write operations to another pool. It is a best practice to apply a spillover policy to a pool that handles a critical workflow. SmartPools for EDA data You can build node pools and policies to streamline an EDA workflow. If your EDA workflow includes timely critical data as well as historical, nontimely data, you can set up SmartPools to serve critical data quickly while serving historical data less quickly and less expensively. In addition, you can protect both data types equally. For example, with SmartPools, you can create two pools and one policy. One pool contains higher-performance SAS drives, while the other contains lower-cost SATA drives. Both pools are set at a high protection level. You need to create only one policy to manage the data a policy that keeps current data on the faster drives. To place an active working set on faster drives, the policy can use file attributes to route files to a pool of S-Series nodes with SSDs to increase throughput rates and decrease latency levels. At the same time, the policy can move older files, as identified by the last-modified or last-accessed date, to a data storage target of NL-Series disks to reserve the capacity in the S-Series node pool for newer data. When a policy moves a file or directory from one disk pool to another, the file s location in the file system tree remains the same, and you do not need to reconfigure clients to access the file in the new location. A file pool policy can apply to directories or files, and it can filter files by file name, path, 17

18 type, and size, as well as other file attributes. You can also separate snapshot storage from data storage to route snapshots to a cost-effective target. SSD strategies A SmartPools policy can set an SSD strategy that accelerates access to metadata and files. Moving the file and directory metadata from HDDs to SSDs can yield a noticeable improvement in workflow performance. Table 3 lists the SSD strategies from slowest to fastest. Table 3: OneFS SSD use case options SSD strategy Description Use case Avoid SSDs Writes file data and metadata to HDDs only. Implementing this strategy might degrade performance. The storage pool contains archived data or data without performance requirements. Metadata read acceleration Metadata read and write acceleration Data on SSDs Writes both file data and metadata to HDDs. An extra mirror of the file metadata is written to SSDs. The SSD mirror is in addition to the number required to meet the protection level. Writes file data to HDDs and all metadata mirrors to SSDs. This strategy accelerates metadata writes in addition to reads but requires about four to five times more SSD storage than metadata read acceleration. This strategy is available only with OneFS 7.0 or later. Uses SSD node pools for both data and metadata. This strategy does not result in the creation of additional mirrors beyond the normal protection level but requires significantly increased storage requirements compared with the other SSD strategies. The storage pool contains metadata that is more frequently read than written to, and the workflow s performance depends on rapid metadata read operations. The storage pool contains metadata that is frequently read and written to. The workflow s performance depends on rapid metadata read and write operations. The storage pool contains metadata and files that a critical workflow must access with high performance. Global namespace acceleration When only a few of the nodes in a cluster contain SSDs, you can speed up metadata access an important aspect of an EDA workflow by using global namespace acceleration. If your EDA data consists of many files spread across many project directories, accessing metadata can slow down performance. In such a case, you can forego some faster spinning media for much of the data by speeding up metadata read access with global namespace acceleration. 18

19 With SmartPools, you can keep a copy of all the cluster s metadata on SSDs in a few of the nodes. The result lets you quickly locate any data while reducing the cost of storing many of your files. Global namespace acceleration applies to all the nodes in a cluster; it is not tied to a pool or a policy. Global namespace acceleration, however, requires that a percentage of the cluster s nodes contain SSDs. For more information, see the OneFS Administration Guide. Write caching with SmartCache Write caching improves performance for most workflows. To optimize I/O operations, write caching coalesces data into a write-back cache and writes the data to disk at the best time. EMC Isilon calls write caching SmartCache. In general, EMC Isilon recommends that you leave write caching turned on. OneFS interprets writes as either synchronous or asynchronous, depending on a client's specifications. The impact and risk of write caching depend on the protocols with which your clients write data to the cluster and on whether the writes are interpreted as synchronous or asynchronous. If you disable write caching, OneFS ignores your clients' specifications and writes data synchronously. For more information, see the OneFS Administration Guide. Guidelines for file pool management EMC Isilon recommends the following guidelines to improve overall performance: Plan file pool policies carefully, especially the sequence in which they are applied. Without proper planning and analysis, policies can conflict with or override one another. Note that directories with large amounts of stale data for instance, data that has not been accessed for more than 60 days can be migrated with a file pool policy to archive storage on an NL-Series pool. OneFS supplies a template for this policy. If directories take a long time to load in a file manager like Windows Explorer because there are a large number of objects directories, files, or both enable metadata acceleration for the data to improve load time. For more information on SmartPools file pool policies, see the white paper titled Next Generation Storage Tiering with EMC Isilon SmartPools on the EMC website and the OneFS Administration Guide. Check the optimization settings of directories and files You can check the attributes of directories and files to see your current settings by running the following command and replacing data in the example with the name of a directory or file. The command s output, which shows the properties of a directory named data, is abridged to remove some immaterial data. isi get -D data POLICY W LEVEL PERFORMANCE COAL ENCODING FILE IADDRS default 4x/2 concurrency on N/A./ <1,1, :512>, <1,4, :512>, <1,23, :512>, <2,0, :512>, <3,2, :512> ct: rt: 0 ************************************************* * IFS inode: [ 1,1, :512, 1,4, :512, 1,23, :512, 2,0, :512, 3,2, :512 ] ************************************************* * Inode Version: 6 19

20 * Dir Version: 2 * Inode Revision: 7 * Inode Mirror Count: 5 * Physical Blocks: 0 * LIN: 1:0003:0000 * Last Modified: * Last Inode Change: * Create Time: * Rename Time: 0 * Write Caching: Enabled * Parent Lin 2 * Parent Hash: * Manually Manage: * Access False * Protection False * Protection Policy: Diskpool default * Current Protection: 4x * Future Protection: 0x * Disk pools: policy any pool group ID -> data target s200_13tb_400gbssd_48gb:6(6), metadata target s200_13tb_400gb-ssd_48gb:6(6) * SSD Strategy: metadata * SSD Status: complete * Layout drive count: 0 * Access pattern: 0 * File Data (118 bytes): * Metatree Depth: 1 * Dynamic Attributes (37 bytes): ATTRIBUTE OFFSET SIZE New file attribute 0 23 Disk pool policy ID 23 5 Last snapshot paint time 28 9 ************************************************* * NEW FILE ATTRIBUTES * Access attributes: active * Write Cache: on * Access Pattern: concurrency * At_r: 0 * Protection attributes: active * Protection Policy: Diskpool default * Disk pools: policy any pool group ID * SSD Strategy: metadata ************************************************* Here is what some of these lines mean: This is a OneFS command to display the file system properties of a directory or file. For more information, see the OneFS Command Reference. The directory s data access pattern is set to concurrency. Write caching (SmartCache) is turned on. The SSD strategy is set to metadata, which is the default. Files that are added to the directory are governed by these settings, most of which can be changed by applying a file pool policy to the directory. 20

21 Plan for network performance and throughput As the number of clients that connect to a cluster increases, balancing the connections across network interfaces becomes more important. This section covers best practices for setting up a cluster s network and balancing client connections. Networking A cluster includes two networks: an internal network to exchange data between nodes and an external network to handle client connections. Nodes exchange data through the internal network over InfiniBand. Each node includes redundant InfiniBand ports that enable you to add a second internal network in case the first one fails. Clients reach the cluster with 1 GbE or 10 GbE. Every node includes Ethernet ports. Ethernet speeds Clients connect to a node with either a 1 GbE or 10 GbE connection. The practical throughput of a 1 GbE connection stands at about 100 MB/s per interface. The practical throughput of a 10 GbE Ethernet connection stands at about 1,250 MB/s per interface, but when you use both interfaces at the same time, you are unlikely to see the full combined throughput for a variety of reasons. Here s an example of how to calculate the maximum practical throughput for a cluster. Consider a node that has two 1 GbE connections. The number and size of each node s connections limit the practical throughput for a cluster of nodes of the same type to about 200 MB/s times the number of nodes. For example, a cluster of 10 nodes using both of the 1 GbE external interfaces delivers a total throughput of about 2000 MB/s. For EDA workflows, you should establish dual 10 GbE connections to each node to support the high levels of network utilization that take place during the simulation and verification phrases. Using 10 GbE connections also helps support the workload of EDA tools like Synopsys and Cadence. Frame sizes Jumbo frames where the maximum transmission unit, or MTU, is set to 9000 bytes yield slightly better throughput performance with slightly less CPU usage than standard frames, where the MTU is set to 1500 bytes. For OneFS 6.x, jumbo frames provide about 5 percent better throughput and about 9 percent less CPU usage for 1 GbE connections. For 10 GbE connections, jumbo frames provide about 8 percent better throughput and about 8 percent less CPU usage. Internal InfiniBand network redundancy In a high-performance computing environment, you should use the redundant InfiniBand ports to create a second internal network in case the first network fails. When you build the redundant internal network, be sure to use separate switches. 21

22 IP address allocation planning Internal network As a best practice, you should set up the cluster s internal InfiniBand network for redundancy. Doing so requires two separate subnets, one for the internal-a (int-a) pool and another for the internal-b (int-b) pool. Each internal interface requires a unique IP address. A cluster that might scale to the maximum of 144 nodes requires a larger pool of addresses than a cluster that is not expected to exceed 60 nodes. Once you create an internal subnet, you cannot expand it without taking the entire cluster offline. Thus, before you assign IP addresses to the internal subnet, your plan should include not only your available address ranges but also the number of nodes that you expect the cluster to contain. EMC recommends that you isolate the internal subnets IP address ranges from those of the external network by assigning nonoverlapping private addresses. By isolating the internal IP ranges, you can avoid having routing issues in the cluster and conflicting IP addresses in your enterprise network. External network Both SmartConnect Basic and SmartConnect Advanced static network pools are limited to one IP address per interface. Overprovisioning IP addresses to either type of pool wastes addresses. EMC Isilon SmartConnect By default, the OneFS SmartConnect module balances connections among nodes by using a round-robin policy with static IP addresses and one IP address pool for each subnet. A SmartConnect license adds advanced balancing policies to evenly distribute CPU usage, client connections, or throughput. The licensed mode also lets you define IP address pools to support multiple DNS zones in a subnet. The licensed version supports IP failover, also known as NFS failover, to provide continuous access to data when hardware or a network path fails. Because of the demands that EDA places on CPU usage and throughput, a best practice is to use the licensed version of SmartConnect Advanced. The rest of this section assumes that you have the licensed version in place. SmartConnect pools provide seamless failover for NFS clients. Client connections that use other application-level protocols, such as SMB, do not support the failover mechanism that SmartConnect provides. As a result, SmartConnect Advanced with dynamic pools should be used only for NFS workloads. Static pools are recommended for SMB and all other protocols. SmartConnect requires that you add a new name server (NS) record as a delegated domain to the authoritative DNS zone that contains the cluster. You can improve the performance of both your cluster and your clients by evenly distributing the client connections among the nodes. SmartConnect can remove nodes that have gone offline from the request queue and prevent new clients from mounting a failed node. In addition, you can set SmartConnect to add new nodes to a 22