HP StoreAll Storage and Symantec Enterprise Vault technical reference guide

Technical white paper HP StoreAll Storage and Symantec Enterprise Vault technical reference guide Table of contents Introduction... 2 Purpose... 2 Audience... 2 Terminology and acronyms... 2 Description of capability... 2 Mapping capability to HP Catalog Services... 2 Capability technical design... 3 Overall design approach... 3 Overview of functional component relationships... 5 Mailbox archiving functional component detail... 7 Journal archiving functional component detail... 24 E-Discovery functional component details... 36 Storage functional component detail... 45 Backup and recovery functional component detail... 56 Monitoring functional component detail... 60 Reporting functional component detail... 64 Networking functional component detail... 67 Directory service functional component detail... 68 Naming service functional component detail... 70 Overall process design approach... 70 Business continuity and disaster recovery... 70 Enterprise Vault failover... 70 Performance and event monitoring... 72 Reporting... 72 Appendix Terminology and acronyms... 73

Introduction The goal of this document is to provide solution architects and solution engineers a standard set of guidelines, tools, and methodologies for building a messaging and collaboration (M&C) archiving solution based on Symantec Enterprise Vault 10 and HP StoreAll Storage archiving platform. Note that at the time of writing, Enterprise Vault 10.0.2 was the most recent release and was used in the writing of this document. Purpose This guide provides technical reference information for the HP Messaging and Collaboration Archiving capability. It is created to define the capability functional components, design principles, and to design a Symantec Enterprise Vault 10.0.2 email archiving solution for the HP M&C Archiving capability client. Audience The primary audiences for this capability guide are integration engineers and delivery SMEs who can use the guide as a technical reference along with the Symantec Enterprise Vault Document Set for EV 10.0.2 symantec.com/business/support/index?page=content&id=doc5804. Secondary audiences for this guide are the sales, solution development teams including solution and portfolio architects as well as production change management organizations, or those who need to understand the technical aspects of the capability. Project managers may also use this document to create a work breakdown structure (WBS). Terminology and acronyms This section defines the terminology and common acronyms used in this document. For ease of use, the reference table is placed in Appendix Terminology and acronyms. Description of capability The HP M&C Archiving capability consists of specific hardware, software, and processes that are managed by operational specialists with specific skill sets. It can be deployed within an HP data center or at a customer data center that meets with the HP security, network, and operational standards. The solution is managed by leveraged HP support organizations using their operational tools and processes. HP offers customers consistent, efficient, and low-cost M&C archiving solutions that are fully managed by HP. Such fully managed solution provides an alternative to customers having to build, staff, and administer their own messaging infrastructure. In an M&C archiving solution, the archiving servers and applications are dedicated to a particular customer. Mapping capability to HP Catalog Services Definition of compliance The term compliance as used in this guide relates to the process or processes required to allow an organization to conform to the various rules and regulations that govern the storage and retention of email data. Globally, legislation by regulatory bodies means that many organizations are obliged to retain an ever-increasing amount of electronic correspondence and data. Enterprise Vault enables organizations to capture, store, and retrieve and expire the relevant data. There are many such regulatory requirements covering all kinds of industries in many countries. Compliance requirements can be external, such as the Sarbanes-Oxley Act of 2002 requirement in the US, or the Freedom of Information Act in the UK. Compliance can also refer to internal company policy. Companies may have their own motivation for retaining mail for what we call compliance, such as internal oversight, working with outside counsel, responses to litigation, and so forth. The need for compliance is more frequently associated with the financial services industry, but that is generally because that industry is more highly regulated. Many industries such as government, healthcare, even manufacturing are also subject to stringent regulation and have a need to be compliant in their handling of data. It should also be noted that what can be a legal requirement in one country can actually be illegal in another. For example, in Germany because of strong privacy laws that protect the rights of the individual, it is illegal to collect any personal information without that person s prior consent. 2

By its very nature, compliance generally requires that all email data be stored in a non-alterable format and collected before anyone has had a chance to delete or change it. Within the M&C archive offering, the compliance components of the solution relate to the type of storage used, the process of collecting everything or journaling, and the specialized product called discovery accelerator that provides an integrated application for searching and reviewing data that has been archived using Enterprise Vault. Together, these components provide the framework that allows a company to meet its compliance requirements. These components are: HP StoreAll Storage in Enterprise compliance mode provides the WORM capability that is required to ensure that stored data cannot be modified or destroyed whilst under retention. Data is normally replicated to a separate secondary HP StoreAll Storage in a separate data center to provide the data redundancy required by compliance regulations. Enterprise Vault journal archiving provides the mechanism to collect all mail captured by the Microsoft Exchange journaling process, index the mail, and store it on HP StoreAll Storage device. Discovery accelerator provides the specialized tool to allow compliance officers to perform company-wide searches of the archived data. Within the M&C archive offering, the Enterprise Vault journal archiving components are frequently referred to as compliance archiving. This reflects the close relationship between a business need for compliance and the need to store journaled emails for certain users or groups of users. Note Although HP provides an infrastructure that allows a company to achieve compliance, we do not and cannot provide compliance. HP cannot provide legal advice and will not offer advice on what a company must do to be compliant. The customer must advise us on what their requirements are. We can then implement an infrastructure that will allow them to meet those requirements. Additionally HP has not obtained formal compliance regulatory certifications from any governmental entities (For example: US Federal Drug Administration [FDA]). Capability technical design This section provides a description of the hardware, software, roles, and process components for the solution. It provides information on the technical aspects of the various core components, as well as the optional components. Overall design approach The following points provide a high-level overview of design principles and design approach. Solution selection criteria The solution must provide what the customer requires: mailbox archiving and compliance archiving which includes E-Discovery. It must meet any specific customer business requirements such as message retention, corporate policies, regulatory compliance, and E-Discovery, as well as infrastructure requirements such as the type of storage used for archived data and backup and recovery of the entire email archiving system. System sizing The mail archiving servers and storage must be able to support customers of various sizes. HP M&C Archiving takes a modular approach by devising standard building blocks that are capable of supporting certain level of load, and scale out by adding more building blocks to support higher demand. 3

Networking considerations To support a high archiving ingestion rate and search performance, HP M&C Archiving utilizes Gigabit Ethernet as the standard network type for email archiving servers and archive storage. A Fibre Channel network is used when the application data such as databases and indexes are stored on SAN storage. Large amounts of data may be replicated between primary and secondary HP StoreAll Storage devices via remote replication using the run once feature of the HP StoreAll Storage in conjunction with scripted Enterprise Vault verification. A separate and dedicated network is used for backup and recovery data streams. Disaster recovery considerations In this release, HP M&C Archiving is developing backup and recovery methods for email archiving servers, databases, application, and archived data within a local data center. Site-level failover and disaster recovery methods are also covered. As an email archiving solution is an integration of multiple components or moving parts and is time sensitive between the moving parts, backup and recovery of these components must be accurately coordinated. Software considerations This release of HP M&C Archiving is using Symantec Enterprise Vault 10.0.2 and Symantec Discovery Accelerator 10.0.2 as the email archiving application. Software selections are based on the approved HP M&C Archiving list of materials. Any software not included in the list may not be supported. Server platform virtualization considerations Part of the scope for HP Email Archiving offering is to assess the potential for deploying all or part of an email archiving solution on virtual servers. Our conclusion, for the current capability design, is that there are significant opportunities to use virtual servers to either reduce costs or improve ease of operation. The rational for this conclusion is shown below: The Email Archiving capability recognizes three generic solution sizes small, medium, and large. The medium and large categories are described in detail in the capability documentation Enterprise Vault 10 provides considerable improvements in the area of virtualization over previous versions. Configurations to cope with intense CPU, memory, and I/O workloads are now quite possible on VMware vsphere 4 and above, if the virtual machine (VM) is properly specified. Running Enterprise Vault in a virtual environment has significant advantages in terms of high availability and failover, and as such, EV 10 servers are good candidates for virtualization. The small size category is briefly described as a custom option with all server roles installed on the same server. This is likely to provide a more cost-effective solution than virtualizing any of the components (because virtualization would require the management of more platforms). Additionally, SQL database servers, because of their intensive I/O and CPU requirements, do not make good candidates for virtualization and with the exception of very small implementations (Rule of thumb: less than 500 users) should always be physical within this offering. Supporting servers (Active Directory, internet security and acceleration [ISA], and so on) that might be candidates for virtualization are not delivered by the email archiving service, belonging instead to managed directory services or Managed Messaging. The discovery accelerator server might also be a candidate for virtualization but must be carefully built in a virtual environment to help maximize CPU and disk I/O. 4

Overview of functional component relationships Figure 1 provides a high-level overview of the core functional components that comprise an email archiving solution and the relationships between the components. Figure 1. High-level view of email archiving core functional components Microsoft desktop footprint Outlook Client with EV add-ins, Internet Explorer Intranet access Reverse proxy service Outlook Web access Intranet access Email archiving services Security Billing Reporting Monitoring Backup and recovery Mailbox archiving Journal archiving E-Discovery Exchange messaging services Directory services Naming services Platform Storage services Networking infrastructure Facilities As shown in the Figure 1, the core functional components of HP M&C Archiving, such as mailbox archiving, journal archiving, and E-Discovery depend on multiple layers of other functional components such as storage as well as naming and directory services. The HP M&C Archiving solution also incorporates and leverages some supporting functional components such as backup and recovery, monitoring and reporting, and billing and security. In this release of M&C Archiving, email archiving services are provided to customers connecting from both within the corporate intranet and the public Internet. Table 1. Describes the various components required to make up the HP M&C Archiving solution. Functional components Mailbox archiving Journal archiving E-Discovery Description Provide archiving functions to Exchange user mailboxes. Include the following main subcomponents: Complete audit trail Symantec Enterprise Vault Microsoft SQL Server Managed Messaging (Exchange 2010 SP2) Client-side archiving features PST Migration (Not in scope of this release. High-level guidance only.) Provide journaling and archiving functions to Exchange messaging systems. Include the following main sub-components: Microsoft Exchange Standard or Premium Journaling Exchange Journal Mailbox Server Symantec Enterprise Vault Microsoft SQL Server HP SQL Database Capability Provide E-Discovery functions to archived contents. Include the following main subcomponents: Symantec discovery accelerator Microsoft SQL Server 5

Functional components Storage Backup and recovery Monitoring Reporting Networking Directory service Naming service Remote administration Enterprise Security Event Management (ESEM) Billing Description Provide storage functions to the mailbox archiving, journal archiving, and E-Discovery. Include the following main subcomponents: Application storage for indexes and databases HP StoreAll Storage HP Storage Management Services capability Provide backup and recovery functions to mailbox archiving, journal archiving, and E-Discovery. Include the following main sub-components: Backup and recovery of Enterprise Vault servers, SQL Server, indexes, databases, and vault stores HP StoreAll Remote Replication HP Data Center Backup (DCB) Backup and Restore Services capability Provide monitoring functions to mailbox archiving, journal archiving, and E-Discovery. Include the following main subcomponents: Event Management Solution Engineering capability HP Operations Manager (HPOM) server, agent, and the following monitoring solutions Enterprise Vault Monitoring Microsoft System Center Operations Manager (SCOM) server, agent, and the following management packs: Management pack for Enterprise Vault Note: HPOM is the default-monitoring tool in HP M&C Archiving. SCOM is the alternative monitoring tool when HPOM is not used. Provide reporting functions to mailbox archiving, journal archiving, and E-Discovery. Include the following main subcomponents: Symantec Enterprise Vault Reporting (based on SSRS) Provide networking infrastructure required for mailbox archiving, journal archiving, E-Discovery, and those related to storage, backup, and recovery. Include the following main subcomponents: HP Global Network Services Provide Active Directory functions to email archiving infrastructure. HP Global Active Directory Capability Provide naming functions to email archiving infrastructure. Include the following main subcomponents: HP Network naming and addressing Provide remote terminal functions for administrators to remotely manage the email archiving infrastructure systems securely. Include the following main subcomponents: From SOE Leveraged remote access (LRA) Citrix From OMCnet Citrix/TS Provide ESEM functions to email archiving infrastructure servers. Include the following mail subcomponents: HP Enterprise Security Event Management Capability Provide billing functions to mailbox archiving, journal archiving, and E-Discovery. Include the following main subcomponents: Manual billing process and procedures 6

Mailbox archiving functional component detail Description Over the past years, messaging has become a critical part of every organization. As a result, the number and size of messages sent is growing exponentially and this trend is likely to continue for the near future. Messaging systems like Exchange were never designed or intended to store unlimited amounts of data and may eventually fail if proper control on growth and management of the environment is not exercised. If the size of the mail database is controlled, the total server hardware resource and storage requirements for the messaging environment can be better handled; therefore, assuring a correct messaging service (convenience, performance, and availability) and assuring the possibility to plan the correct capacity is required to support future needs. The ability to archive emails has recently become a major issue in the messaging world. There are two main drivers for this: firstly, storage growth and secondly, compliance or the need to comply with an ever-increasing number of regulatory requirements and laws relating to the storage and availability of data. In the last few years, laws such as Sarbanes-Oxley in the US and the Freedom of Information Act in the UK have made it a legal requirement for all data involved in certain types of transaction to be available for inspection by external auditors and government agencies for extended periods of time. These laws also require that data is stored in such a way that it cannot be altered or deleted until a previously specified retention period has been reached. Full audit facilities must also be available to prove that proper procedure has been followed. These requirements go beyond the technical capabilities of traditional email systems and require the use of specialized hardware and software, or archive solutions. Symantec uses industry-leading technology to deliver a highly scalable, flexible, and reliable solution for today s data archive needs. HP M&C Archiving service is primarily based on HP StoreAll Storage solution designed to store and provide simple, highly scalable, and effective access to fixed content. This offering is targeted at enterprise compliance situations and as such uses the HP StoreAll Storage as the back-end WORM-like storage medium. In non-compliance situations where WORM capability is not required, HP StoreAll Storage devices are also used in non-compliance mode without WORM enabled. Enterprise Vault product architecture Figure 2 provides an overview of the Enterprise Vault product architecture: Figure 2. Enterprise Vault architecture MAPI/ HTTP Microsoft Exchange 2010 SP2 environment Outlook and OWA Application services MAPI Retrieval Archiving Journaling PST Outlook and OWA HTTP Admin Directory Shopping IIS services Indexing Storage Open storage layer Search, Archive Explorer and Accelerator Module Index locations Location 1 Physical storage Store 1 Vault stores Store 2 Store 3 7

Archive software Symantec Enterprise Vault is a leading product suite in the archiving space. It provides the ability for users to search and transparently retrieve archived Microsoft Exchange messages. In addition, it provides journaling facilities for meeting compliance and audit or regulatory requirements around electronic messaging systems (email, instant messaging, and so forth). The product suite is designed to provide lifecycle management for records in Microsoft Exchange message stores, whilst allowing customers to reduce the costs associated with keeping these records for the necessary lifecycle by storing the records on more cost-optimal platforms over time and so is an excellent fit with the solution. Enterprise Vault functionality includes: Applying catalog or index techniques to establish object identification and searchable content metadata Applying retention policy to message objects Creating and maintaining necessary logs and audit trails of object handling events Providing user interfaces supporting access, restoration, and searches for archived objects Safeguarding archived objects from alteration and unauthorized access Archive storage Three distinct storage sets are required for Enterprise Vault. Archive storage Archive storage is used for holding the actual archived content and text or html rendering of those items. This storage is typically very high volume and moderate performance. In compliance situations, this storage must also be capable of providing non-alterable storage, this is known as WORM. Indexing storage Indexing storage is used to hold the indexes for all the archived items. This storage is completely separate from the archive storage and should be located on high speed SAN. For smaller implementations DAS storage is acceptable provided that has sufficient performance. Index storage must not be implemented on NAS devices and must never be implemented on a WORM device. The index storage consumes a volume equivalent to about 13 percent of the archived data (full) or 4 percent (brief). Within this offering, full indexing is the standard configuration and is a prerequisite where discovery accelerator is required. SQL storage SQL is critical to Enterprise Vault. All Enterprise Vault implementations require at least one SQL Server and this is specified with high performance and high redundancy. Normally an SQL cluster is used. This storage for the SQL databases and logs is separate from the archive and index storage and should be located on high speed SAN. For smaller implementations, DAS storage is acceptable provided that has sufficient performance. Compliance vault storage Primary storage for message objects is targeted for magnetic disk; here, two configurations are available. For environments where WORM-like enterprise compliance capability is appropriate or Enterprise Vault s native retention capabilities coupled with non-worm storage is applicable, the HP StoreAll Storage range of NAS devices should be used. These devices offer almost limitless scalability coupled with the flexibility of NAS and the benefits of cross-site replication. HP StoreAll Storage can be configured in WORM-like compliance mode to prevent unauthorized deletion of retained data from applications. For most situations this compliance capability is sufficient to meet regulatory requirements. An additional advantage of using HP StoreAll Storage is its non-proprietary format, all data is stored on a standard NTFS partition and can easily be migrated to different storage in the future. 8

Non-compliance vault storage In the event that compliant WORM or WORM-like storage is not a requirement, many alternate forms of storage can be used for the archive store. However, the recommended storage within this offering remains the HP StoreAll Storage configured in noncompliant (non-worm) mode. For smaller installations where the HP StoreAll Storage is not appropriate, any NTFS volume can be used as an alternative archive store such as NAS, SAN, DAS, or a combination can be used. Enterprise Vault Single Instance In order to benefit from Enterprise Vault Single Instancing, the vault stores are grouped within a vault storage group. The Single Instance boundary is the vault storage group. Within each vault store group, each vault store is assigned a sharing level such as share within group, share within vault, or no sharing. Each item (above 20 KB) is broken into SIS parts. The Enterprise Vault stores a SIS part only once. A vault store has a fingerprint database that maintains each SIS parts fingerprint. The Enterprise Vault checks the vault store groups fingerprint database to identify if any SIS part already exists in the archive. If it does, then Enterprise Vault simply references the existing SIS part. If it does not exist, the SIS part is stored and a new entry is made in the vault store groups fingerprint database. Figure 3 illustrates the relationship between vault stores and the vault store group with respect to sharing boundaries. Figure 3. Sharing boundaries in a vault store group Vault store 1 Sharing level: Share within group Vault store 3 Sharing level: Share within group Vault store 4 Sharing level: Share within group Vault store 2 Sharing level: Share within vault store Vault store 5 Sharing level: No sharing Archive server The Enterprise Vault server is responsible for all major archive activities. It houses the archive software that receives messages, applies retention policies, parses messages for metadata, creates message identities, stores message objects, and manages retrievals, restores, and expires. The earlier sections provided a rough overview of the Enterprise Vault architecture. The main components are described in following sections. 9

Enterprise Vault components Figure 4 provides an overview of the Enterprise Vault components. Figure 4. Enterprise Vault components Users accessing archives Enterprise Vault adminstation console Administrators remotely monitoring Enterprise Vault Enterprise Vault Web Access components Enterprise Vault services and tasks Enterprise Vault operations manager Enterprise Vault reporting Target server SQL Server Enterprise Vault monitoring database Vault store database Enterprise Vault directory database Vault store Partition Enterprise Vault archives Vault store Partition Enterprise Vault archives Enterprise Vault server The Enterprise Vault (EV) server is the core of the email archiving architecture. It provides the link between the Exchange servers, storage devices, and database. Within the archiving process, the EV server connects to the Exchange servers and defines which messages have to be archived. During that archive process, it indexes the data and stores it on a file system connected to the EV server. The vault metadata (SavesetID, vault store partition, and such) are stored into a separate SQL server database. The actual email s metadata is stored within the index structure. The archived messages typically are replaced by a shortcut in the user mailbox. The shortcut can be full item, or an HTML or text preview to the archived message. It can be double clicked which shows the entire contents of the message in its original format. If the customer elects not to leave shortcuts but then decides to use shortcuts at a later date, shortcuts are only created to items after shortcutting is enabled, they are not retrospective, that is shortcuts to previously archived items are not be created. The amount of detail contained in the shortcuts is configurable through the mailbox archiving policy. The more detail stored in a shortcut, the larger the shortcut becomes and the greater its effect on the size of the mailbox, after thousands of shortcuts are created. In addition, since shortcuts exist in the mailbox folders, they add to the count of items in the folders and can affect Outlook performance. Microsoft generally recommends that folder counts stay within a range of 3,500 5,000 items. To help maintain continuity of the vault and folder item count, Enterprise Vault can be configured to delete shortcuts based on several criteria as follows: 1. Shortcut deletion to items that have expired by Enterprise Vault storage expiry 2. Shortcut deletion based on a specified age criteria 3. Shortcut deletion to orphaned items (created by end users deleting archived items via search apps or copying shortcuts and only deleting the original) The shortcut deletion process is scheduled via the Exchange Mailbox Policy properties. The process to delete orphaned shortcuts is a SQL server-intensive process and is estimated to add approximately one third to the shortcut deletion time. If orphaned shortcut deletion is enabled, it is advisable to use the provisioning groups to apply the policy to different groups of users and to activate the orphaned shortcut removal on a policy-by-policy, week-by-week basis. 10

The HP standard EV 10 server configuration requires 16-core CPU with 32 GB of memory. Connecting server IP ports as well as local I/O ports are GigEther IP based. As a general rule of thumb, a server with the specified configuration can handle approximately 7,500 average usage mailboxes (for mailbox archiving). So, one archive server is usually capable of handling up to (related to Managed Messaging): Four messaging servers with 1,875 mailboxes per server Three messaging servers with 2,500 mailboxes per server Two messaging servers with 3,750 mailboxes per server One messaging server with 7,500 mailboxes per server Currently in Managed Messaging, a mailbox server can host up to 16,000 mailboxes. To archive a messaging server with such high number of mailboxes, it is quite likely that you may reduce the number of users per Exchange server. Use the calculation method explained later to determine the actual requirements. In environments with a heavy use of mail, so large individual mail items or a large number of mails per user per day or a combination of both, these numbers could be significantly reduced. It is always important to calculate mail volume accurately before finalizing the EV design as changing things at later days is not a simple matter. Generally, EV servers should be associated with Exchange servers on a one-to-one basis. This provides the best possible ratio to allow for future growth. It is also very important to consider the performance of the Exchange environment. Poor Exchange disk I/O results in poor archive performance. This situation cannot be improved at the Enterprise Vault level. EV can only archive data as fast as it can read it from Exchange. A poorly specified Exchange server has a dramatic impact on perceived EV performance, particularly when daily maintenance of Exchange is in progress. The next table shows the expected ingestion rates for numbers of physical cores where the average message size including attachments is 70 KB and 140 KB. It is assumed that the system is running on VMware and that CPU and memory resources are dedicated (reserved) to the Enterprise Vault server, and not shared with other virtual machines on the host. The rates can vary but typical ingestion rates on a physical server are 10 to 20 percent higher than virtual servers (stated by Symantec). Read the document, Implementing Enterprise Vault on VMware symantec.com/docs/tech180094. Table 2. EV 10 server performance Number of cores Hourly ingestion rate (70 KB items) Hourly ingestion rate (140 KB items) (70 KB items) 60,000 40,000 (140 KB items) 90,000 60,000 Note The following sections provide rule of thumb guidance only. In order to calculate the customer s requirements accurately, Symantec provides a comprehensive sizing tool that should always be used in the later stages of estimation. The latest version of the tool is available from the Symantec partner information website. This tool must be filled in by the HP archiving engineer using input obtained from the customer. For detailed information on EV 10 performance, review the following best practice guide. symantec.com/business/support/index?page=content&id=doc4553 and for VMware implementations, review the guide symantec.com/business/support/index?page=content&id=tech180094 Note In the real world, average message sizes of 140 KB are not uncommon. 11

To accurately calculate the number of Enterprise Vault mailbox archiving servers required for a particular environment, several factors must be considered. They are: The number of users mailboxes that can be archived The number of messages per user to archive each day The average message size The number of hours allocated for archiving each night The number of hours required to perform backup each night Does an email backlog exist, which must be archived within a certain amount of time; if so, what is the size (GB) and number of messages contained in the backlog The estimated annual growth rate of number of mailboxes The estimated annual growth rate of the average number of messages to archive per user It is also assumed that the archiving servers are in close proximity to the source data (target Exchange servers) An EV server scoped for mailbox archiving is also suitable for journaling. Even though there is a lot more to be archived you have a much larger (20+ hours) archive window for journal servers. If the data is available, the number of Enterprise Vault servers can be calculated using the following formula: (Number of mailboxes to archive * number of messages to archive per user per day or number of hours available for archiving) or 90,000 items per hour (based on an average message size of 70 KB) * 75 percent contingency or buffer. The average message size plays a large role in the number of items an Enterprise Vault server can process per hour. It is found that as the average message size doubles from 70 KB, the archiving rate decreases by one third. Therefore, if the average message size is found to be 140 KB, use an archiving rate of 60,000 for the calculations. For your final calculation, always use 75 percent of the final calculated figure. This is in order to provide a contingency or buffer in your calculations. Another common method for calculating server requirement is to use the throughput per hour that a given Enterprise Vault server can process into the archive. For the M&C Standard Enterprise Vault server configuration detailed earlier, a figure of between 5 GB and 9 GB per hour can be expected. Use 5 GB as a conservative value. Symantec s Exchange Store Reporter tool can be used to analyze Exchange mailbox stores and provides estimates of mail volume and sizes. The following examples are based on the fictional company ACME Widgets. In our first consultation, we received limited information from the customer. But we can still provide some rough rule of thumb estimation based on the limited information. Example scenario 1: In our first discussions, ACME Widgets provided the following data to assist with the Enterprise Vault mailbox archiving sizing: There are 8,000 active mailboxes to archive The average mailbox is growing by 20 messages per workday The average message size is 70 KB Four hours per night available as an archiving window The growth rate in number of active mailboxes is expected to average 10 percent for the next three years The average mailbox growth rate (messages per workday) is expected to increase by 20 percent over the next three years A 16-core archiving server is capable of archiving at a rate of 90,000 items per hour; however, we normally fix the size based on 75 percent of this value, which we have often seen as a steady state performance level. Using the known values, we determine that the number of active mailboxes that will require archiving after three years is 10,648 and the number of messages to archive per day per user is 35. The number of Enterprise Vault mailbox archiving servers needed by year three is calculated as follows: (Number of mailboxes to archive * number of messages to archive per user per day or number of hours available for archiving), or 90,000 items per hour (assuming an average message size of 70 KB) * 75 percent. Number of EV servers required to provide mailbox archiving = (10,648 * 35)/4/(90,000*75%) = 1.6 (so 2 EV servers) In this example, no account has been taken of historic mail. Two EV servers will be adequate to cope with existing traffic, but possibly not enough to perform the ingestion. 12

Example scenario 2: ACME Widgets has now provided further data to assist with the Enterprise Vault mailbox archiving sizing. They have made the decision to ingest all their existing mailbox data. There are currently 3,000,000 messages in the mailbox backlog. The backlog must be cleared in one month, which is 20 working days and 3 weekends (one weekend is dedicated to maintenance). There are four hours to archive each weeknight and 10 hours each to archive on Saturday and Sunday. In this scenario, we have been allotted 140 hours to archive the backlog of 3,000,000 messages. We must also consider that during this one-month period, the mailboxes are still accumulating messages at a rate of 20 per day; therefore, we must be able to archive 7,800,000 messages in the allotted time span of 140 hours. To accomplish this, we must be able to sustain an archiving rate of 56,000 messages per hour. Using a 16-core server allows us to achieve up to 90,000 items per hour; therefore, two quad core Enterprise Vault servers should still suffice. Using the two servers, we can easily clear the mailbox backlog within the allotted period of one month and once clear, two servers will be able to process the expected load for the following three years. Table 3. Enterprise Vault mailbox archiving server specifications highlight Hardware Model Processor Memory Internal storage (15k spin speed) SAN Network HP ProLiant DL380 G8 Server 16-core processor 32 GB RAM 2 x 300 GB in RAID 1 for OS, page file, and application files 2 x 300 GB in RAID 1 for Microsoft Message Queuing (MSMQ) 2 x 300 GB in RAID 1 for shopping and exports 2 x HBA Fibre Channel connections (2 x single or dual port HBA cards) Fully redundant Gigabit Ethernet NIC Fully redundant Gigabit backup LAN (Internal Embedded NIC) Software OS Application Microsoft Windows 2008 R2 SP1 64-bit Standard Edition Enterprise Vault v10.0.2 or above Microsoft Outlook 2007 Standard Edition with SP2 Always review the current Enterprise Vault compatibility chart for the most recent requirements. symantec.com/business/support/index?page=content&id=doc5804 Note If you choose to use blade server, HP M&C Archiving recommends the HP ProLiant BL460c Server series. For the blade server configuration, you can reference the above rack server configuration. SQL server The SQL server is used to store the following data: EV server configuration Metadata of archived data Vault store configuration Fingerprint database for single instancing The general rule of thumb is ratio: Provision 4 CPU cores and 8 GB RAM for every four EV servers and for this offering, we do not recommend provisioning more than eight EV servers on a single physical server. 13

Table 4. SQL server specification highlight Hardware Model Processor Memory Internal storage (15k spin speed) SAN Network HP ProLiant DL380 G8 Server 16-core processor 32 GB RAM 2 x 300 GB in RAID 1 for OS, page file and application files 2 x 300 GB in RAID 1 for transactions logs 2 x 300 GB in RAID 1 for TempDB logs 2 x 300 GB in RAID 1 for client DB logs 2 x HBA Fibre Channel connections (2 x single or dual port HBA cards) Fully redundant Gigabit Ethernet NIC Fully redundant Gigabit backup LAN Software OS Application Windows 2008 R2 SP1 64-bit Standard Edition Microsoft SQL Server 2008 R2 64-bit Standard Edition If you choose to use blade server, HP M&C Archiving recommends the HP ProLiant BL460c Server series. For the blade server configuration, you can reference the above rack server configuration. Exchange server The requirements on the Exchange servers are to: Exchange 2010 must be running SP2 with full hotfix rollup or above. This is required to resolve a Microsoft bug that prevents archived or journaled items from being freed up as whitespace. Grant full control to the Enterprise Vault Service Account on each Exchange 2010 SP2 server that will be an archive target. (Run the SetEVExchangePermissions.PS1 PowerShell script. Check the Assigning Exchange Server permissions to the Vault Service account section of the Symantec Enterprise Vault Installing and Configuring document). Remove the throttling restrictions for the Enterprise Vault Service Account by running the SetEVThrottlingPolicy.PS1. Check the Configuring the Exchange 2010 throttling policy on the Vault Service account section of the Symantec Enterprise Vault Installing and Configuring document. If you use a database availability group (DAG) in your Exchange Server 2010 SP2 environment, you must set up archiving for all members of the DAG. You must also target all the DAG member servers within one Enterprise Vault site. (When all DAG member servers are set up for archiving, database and server failovers do not interrupt mailbox archiving.) Create an Enterprise Vault System mailbox on each Exchange 2010 SP2 server that will be an archive target. If you use DAGs in your Exchange Server 2010 SP2 environment, you must create the Enterprise Vault system mailbox in a database that is replicated across the DAG. When there is no database that is replicated across the DAG, you need to create such a database for use by the Enterprise Vault system mailbox. Grant send as permissions to the Enterprise Vault Service account on each Enterprise Vault System mailbox as well. Configure Enterprise Vault to deploy the forms locally in pure Exchange 2010 SP2 environments, even if a public folder store to publish Enterprise Vault Outlook forms via Exchange exists. Install the Enterprise Vault Outlook Web Access (OWA) extensions on all Exchange 2010 SP2 CAS servers. (If there is reverse proxy server such as ISA in place, special rules need to be created to support the EV Web features.) In Exchange 2010, MAPI connections are moved to the Client Access Server role. M&C archive solution does not require dedicated client access server just for the mailbox archiving purpose, because all the mailbox archive activities are scheduled to happen within the archive windows without impacting end users. Detailed information on these requirements can be found in the documents Installing and Configuring Guide for Enterprise Vault at symantec.com/business/support/index?page=content&id=doc4402. For more information on how Enterprise Vault works with Exchange 2010 SP2 servers in a DAG, refer to Setting up Exchange Server Archiving document at symantec.com/business/support/index?page=content&id=doc4410. 14

Outlook Client The retrieval of shortcut archived messages can be made transparent to the Outlook users by using an EV Outlook plug-in, which must be installed on the Outlook Client. The Outlook Client then uses a combination of special Enterprise Vault Outlook forms and the Enterprise Vault Outlook Add-In to interact with the archived items, and retrieve the archived message transparently from the Enterprise Vault server. EV 10.0.2 introduces a new Unified EV Outlook Client that provides the functionality of both the previous DCOM and HTTP-based clients. The new Unified Client will be deployed with this release of the offering. Web clients OWA client To enable users to access archives and manage archived items from within OWA clients, install Enterprise Vault OWA Extensions on Exchange 2010 SP2 CAS servers. Archives can be searched and items can be archived, viewed, restored, and deleted, if permitted. Enterprise Vault buttons and menu items are added for the customer. OWA users do not require the Enterprise Vault Outlook Add-Ins to be installed on their desktop computers. The administrator can configure in the Enterprise Vault Administration Console what functionality is available to OWA users. This release of HP M&C Archiving supports the Search and Archive Explorer functions launched within OWA when accessing the archived items from within the intranet as part of the standard. Internet access via the Search and Archive Explorer functions within OWA requires custom engineering, which involves setting up HTTP or SSL and ISA publishing rules for the Enterprise Vault servers. Outlook Anywhere Client (RPC/HTTP) To enable Outlook 2010 SP2 and Outlook 2010 users to access Enterprise Vault archives using RPC over HTTP, Enterprise Vault Outlook Add-Ins (any type) must be installed on each client desktop computer and the settings configured in the Enterprise Vault Exchange Desktop policy. Read Prerequisites for RPC over HTTP section of the Symantec Enterprise Vault Installing and Configuring document and Chapter 10 Configuring access to Enterprise Vault from Outlook RPC over HTTP clients of the Symantec Enterprise Vault Setting up Exchange Server Archiving document. In this release of M&C Archiving, Outlook Anywhere clients are enabled with Vault Cache where users can access the archived items within the offline vault without the need to connect to the Enterprise Vault servers. If in customer deployment such direct access to Enterprise Vault servers from the public Internet is required, integration engineers conduct custom engineering, which involves setting up HTTP/SSL and ISA publishing rules for the Enterprise Vault servers. Mobile clients The two popular mobile technologies, ActiveSync and BlackBerry, work similarly (server-push synchronization) with a small exception. In the case of archiving, both technologies can exist in the same environment and the archiving process itself is not affected (messages already synchronized to the mobile device can still be archived). Depending on which technology is being used, the end-user experience can vary slightly. The differences between the two are described as follows. ActiveSync With ActiveSync, the shortcuts to the archived messages are actually synchronized to the mobile device; however, the full message does not resolve and the reply or forward actions just send the shortcut. Therefore, if the shortcuts are configured to show the entire message body, an ActiveSync Client is at least able to read the message body after that message has been archived. BlackBerry The synchronized copy of the original email message is retained on the BlackBerry Client. The shortcuts to the archived messages are not transferred to the mobile device. The end user cannot view attachments or use the More feature to get the rest of a message, once the message is archived. Messages can be forwarded and replied from the mobile device after archiving, but any attachments included cannot be sent. Existing shortcuts are not synchronized over to the mobile device. This is only an issue if Enterprise Vault is installed and message is archived prior to implementing the BlackBerry server. The same would apply to a folder if the user chooses to newly synchronize with their BlackBerry shortcuts are not synchronized, just standard Outlook message classes and any new email messages from that point forward are be synchronized by the BlackBerry. Mobile Search an additional Enterprise Vault service can allow archived items to be searchable through a mobile device. Note that Mobile Search is not part of the standard offering and would be considered an uplift. For more information, see Chapter 13 Configuring Mobile Search access to Enterprise Vault of the Setting up Exchange Server Archiving Document at symantec.com/business/support/index?page=content&id=doc5987. 15

Note There are also various partner solutions in this area that allows full archived content and functionality on mobile devices but these are beyond the scope of the standard offering and would be considered custom solutions. Archiving tasks and policies Exchange provisioning tasks Provisioning groups are used to associate Active Directory user groups with different archiving policies. To select the mailboxes to be associated with a provisioning group, any of the following is used: Windows group Windows user Distribution group (the Active Directory Group type, Distribution) Organizational unit LDAP query Whole Exchange Server organization The provisioning groups are then processed by the Exchange provisioning task. This task assigns the correct policy settings to each mailbox. For the HP M&C Archiving solution, it is recommended to use Windows groups. Exchange Mailbox archiving tasks In the administration console, it is necessary to create an Exchange Mailbox Archiving task for each Exchange server with user mailboxes to be archived. These tasks are controlled by the task controller service. To obtain an estimate of the number of items that will be archived, without actually archiving anything, run the task in Report Mode. Exchange mailbox archiving tasks run automatically according to the schedule defined for the Enterprise Vault site. Exchange mailbox policies and archiving strategies An Exchange mailbox policy includes information for the archiving task to use when processing the target mailboxes: Indexing level to use Archiving strategy Archiving actions Whether shortcuts are created and what they contain Shortcut deletion strategy (orphaned and time based) Enterprise Vault Outlook and OWA settings The Enterprise Vault archiving policies options provide the ability to customize the behavior of the archiving task, Enterprise Vault Outlook Add-Ins, and OWA clients. The policy settings can also be locked to prevent end users from changing the administrator-defined settings in Outlook. As with other applications, the more options configured; the more complex the solution becomes. Therefore, general best practice is to roll out a minimum number of policies, provisioning groups, and user features. This generally produces the highest level of user acceptance and experience. In some cases, it is necessary to expand the features but this requirement should be discussed and implemented on a needed basis. Safety copies Safety copies are copies of the archived items. In a mailbox, they are identified by the pending archive icon. The options for removing safety copies (in vault store properties) are as follows: Never After backup After backup (immediately for journaling) Immediately after archive 16

When using HP StoreAll Storage, Enterprise Vault should be configured to wait until a backup of the vault store (or replication) has been completed before removing safety copies (after backup or after backup immediately for journaling). The backup process either changes the archive attribute (non-worm) of the archived items, or in the case of HP StoreAll Storage Remote Replication, it writes a trigger file (subject to custom scripting being deployed). Enterprise Vault recognizes the attribute change or trigger file and removes the safety copies only after a successful backup or replication has taken place. In brief, a script is deployed on the HP StoreAll Storage that continually runs run once replication tasks on the active open partitions. On successful completion of the replication task, the trigger file is written to the root of the partition, thus informing EV that all items older than the trigger file are safe and can be shortcut. Note When using alternate storage medium, a suitable backup strategy must be designed and implemented that is specific to that storage solution. Types of items to archive Enterprise Vault comes with a predefined set of message classes that identify different types of items. Table 5 contains the settings defined in the message classes page of directory properties. Table 5. Default settings Type of item Message class Archived by default Calendar items IPM.Appointment No Contact items IPM.Contact No Documents IPM.Document Yes Electronic sticky notes IPM.Stickynote No Interpersonal messages IPM.Note Yes Journal messages IPM.Activity No Messages posted to a folder IPM.Post Yes Tasks IPM.Task No Note The recommendation is to use the default settings above. Index and databases The following sections provide an overview of the index and databases used for email archiving. Indexing The indexes are organized as follows: There is a separate index for each archive Each index consists of a set of related files Files are held in folders To ensure the most stable indexing consistent environment multiple storage locations are required for all new archive, indexes, and new index volumes. You can choose how much information is indexed for items as they are archived using the following indexing levels: Brief with brief indexing, only information about the item such as the subject and author can be searched. Full indexing searching content for phrases is only available with full indexing. It is recommended to use full indexing, because in HP M&C Archiving we must provide maximum search capability in all configurations deployed. This applies to both mailbox archiving and compliance archiving solutions. 17

Other index location best practices The other index location best practices are: Indexes must always be located on high speed SAN, as NAS storage is not suitable for index data. The indexing service file locations require good random access or a high number of IOPS combined with good transfer rates. Within this offering, RAID 1 arrays must always be used for index data. Provision index locations using mount points not individual Drives. This allows further index locations to be added without running out of disk letters. Exclude all index locations from real time or scheduled antivirus scanning. Disable built-in Windows indexing on the drives that have Enterprise Vault components. Ensure the Windows server page file conforms to the Microsoft best practice of 1.5x physical memory. Set the initial and maximum size of the page file to the same value, for preventing unnecessary fragmentation. In the absence of the above settings, many instances of corrupt indexes have been observed in the field. Databases and SQL server Enterprise Vault has the following core databases: Enterprise Vault directory database Vault store database SIS fingerprint database (if implementing SIS in Enterprise Vault) Enterprise Vault monitoring database SQL Server must be installed and set up before configuring Enterprise Vault. The sort order or collation setting must be case insensitive. (Case sensitive installations are not supported.) The Enterprise Vault Service Account is required to be a member of the local administrator s group (to manage the application, to enumerate the administrative shares, and for specifying placement of SQL Server data files and log files). The EV Service Account must also have database creator rights within SQL Server. The best practice for SQL Server provisioning in a normal load environment is as follows: Up to eight EV servers Provision 4-core and 8 GB RAM for every eight EV servers More than eight EV servers Dedicated server for directory and audit databases with 4-core and 16 GB RAM for every eight EV servers Separate server for other databases. Provision 4-core and 8 GB RAM for every four EV servers DA server Dedicated SQL Server All Enterprise Vault SQL Servers should run a single SQL instance only. 18

Vault store database Each vault store database has an initial storage requirement of 100 MB for the data device and 80 MB for the transaction log device, making a total initial disk space requirement of 180 MB. It is a best practice to locate the SQL Server data files, transaction log files, and the TempDB on separate physical spindles for optimum performance. Each vault store database contains an entry for each item that is archived in the associated vault store, so the vault store databases grow over time. Only when an item is deleted from the archive, its references are deleted from the relevant vault store database. Fingerprint databases When you create a vault store group, a fingerprint database is automatically created for the group. If you are using Enterprise Vault SIS, the fingerprint database may grow rapidly in size. It is important to configure the fingerprint database appropriately for the amount of sharing within the group. Each fingerprint database has an initial storage requirement 244 MB made up as follows: 132 MB for the primary file group 1 MB for each non-primary file group 80 MB for the transaction log device When using SIS within a vault store group, the non-primary file groups can grow rapidly and it is best practice to specify multiple locations on separate devices for scalability and performance. Although there are no specific recommendations on when to create additional file groups. The fingerprint database requires 500 bytes per single instance part (DVSSP). As a general rule of thumb, the fingerprint database size can be calculated using the following formula: 1/r x n x 0.2 x 500 bytes Where, R = average number of identical copies of attachments across user mailboxes (2 if not known) N = number of emails HTTP or SSL The default communication protocol used to access the Enterprise Vault Web Access application is HTTP. If required by the customer, this can be configured to use HTTP or SSL. The IIS server must be correctly configured prior to changing the Enterprise Vault setting. In addition, caution must be taken if messages are already archived in the environment because the shortcuts to existing archived items can fail to work. If this setting is changed when there are existing archived items, only shortcuts created after the change will work properly. It is possible to rebuild shortcuts, but it is extremely CPU intensive and should be avoided. Moving Mailbox Archives Prior to EV10, a feature called Move Archive was introduced and it is continued in this Enterprise Vault version. The Move Mailbox Archive allows movement of the content from an existing Exchange mailbox archives and journal archives to new archives or existing archives in other vault stores. The Move Archive wizard allows us to move archives between vault stores in between sites that are controlled by different Enterprise Vault directories. The Move Archive handles each move operation differently, depending on the archiving status of the source archive and the destination archive. 19

Each archive s status can be active or inactive. Active archives are those that are currently in use by Enterprise Vault as targets for archiving. In the case of mailbox archiving, an active archive is one that is associated with an archiving enabled user, and is the current default archive into which Enterprise Vault archives data. In the case of journal archiving, an active archive is the current default archive for a mailbox journaling task. Inactive archives are those that are not currently in use by Enterprise Vault as targets for archiving. Move Archive supports the move of archives to destination servers that run Enterprise Vault 8.0 SP4 or later. Move Archive does not support move operations for the following archive types: Closed archives File system archiving (FSA) archives Microsoft SharePoint archives Shared archives Exchange public folder archives Move Archive also prevents moves in the following circumstances: If an archive contains items that are under legal hold, such as from discovery accelerator, the archive cannot be moved. Once all legal holds have been removed, the archive can then be moved as desired. The source archive is under retention enforced at the storage level. In this case, the archives are copied but the source archive cannot be removed and must remain in a closed state. The source archive exceeds its archive usage limit and the destination is a new archive. The destination archive exceeds its archive usage limit. In these cases, you should increase the appropriate archive s usage limit on the archive properties: Archive usage limit tab in the administration console. The new archive will be re-indexed. The new archive indexing level inherits the indexing level as set in the Enterprise Vault site properties. For example, if the original archive had a full indexing level, but the new archive resides in an Enterprise Vault site where the default indexing level is brief the new archive will be indexed at a brief level. Note This process is not recommended for moving large numbers of archives. If moving large numbers of archives or if using Enterprise Vault versions prior to EV8 SP4, the following still applies. Pre-EV8.0 SP4 mailbox moves Care should be taken while moving Exchange mailboxes that are already enabled for archiving. A direct relationship exists between the Enterprise Vault server and Exchange server. A mailbox that is moved to a new Exchange server is consequently archived from its original EV server, whereas other users mailboxes on the destination Exchange server are serviced by a different EV server. So, when more and more mailboxes are moved across locations, there will be more and more cross-location traffic. Note One Exchange server can be only serviced by one Enterprise Vault server but one Enterprise Vault Server can archive more than one Exchange server. 20

Figure 5 displays the archiving process of a fictional user Joe s mailbox before the mailbox move. Figure 5. Archive data flow before mailbox move SAN Index MAPI EV Retrieval, Search, and Archive Explorer EV server 1 Mbx archiving Vault metadata Exchange 1 Home forjoe user s mailbox Archive storage Mbx archiving EV server 2 HP StoreAll Storage Exchange 2 Index SAN Figure 6 displays the situation when no action is taken on the EV site after a mailbox move. Figure 6. Archive data flow after mailbox move Job user Index SAN EV Retrieval, Search, and Archive Explorer Exchange 1 Mbx archiving EV server 1 Vault metadata SQL Move Joe user s mailbox Transfer of messages to other Enterprise Vault server Archive storage HP StoreAll Storage MAPI Mbx archiving EV server 2 Exchange 2 New home for Joe user s mailbox Index SAN 21

Therefore, whenever possible avoid mailbox moves between Exchange servers which are archived by different Enterprise Vault servers. In situation in which a mailbox move cannot be avoided, the following table describes some measures to prevent cross-connected archives. Table 6. Preventive measures for cross-connected archives Scenario Mailbox or an Exchange database is moved permanently to a new Exchange Server, which is archived by the same EV server Mailbox or an Exchange database is moved permanently to a new Exchange Server, which is archived by a different EV server Exchange database fails over in a DAG scenario Required steps Create SynchInMigrationMode registry key on the Enterprise Vault server before starting the mailbox migration to the new Exchange server Configure the EV permissions on the new Exchange server Configure an Outlook profile on the EV server that can logon to the new exchange server. Configure a new task under Tasks to connect to the new server name Move the users or database in Exchange Sync the users on the new server A detail description can be found under symantec.com/business/support/index?page=content&id=tech48928. Ensure that the steps 1 4 from scenario 1 (in the above row) have already be done. After this do the following steps: Disable the users via the Vault Administration Console (VAC) Move the user in Exchange Use the move archive wizard to move the users archive to the new server Delete original archives The mailbox archiving task that processed the failed copy continues to process the new active copy of the database until the Enterprise Vault s provisioning task runs. When the provisioning task has run, the new active copy of the database is processed by the mailbox archiving task that is associated with the new host Exchange server. In practice, the failed database might be restored to its initial Exchange host before the provisioning task runs and updates the list of databases that are processed by each mailbox archiving task. In situations where temporary moves are needed the required steps depends on the duration: Short (< one month): Within the same EV site, no action is required. EV continues to operate correctly using DCOM to communicate between EV servers. When the user is moving to a different EV site, disable archiving for the affected users during the time where the mailboxes are stored on the new server. When the users are disabled they can still access the already archived mails but cannot archive new mails. Long (> 1 month): Use the steps described in the table above. 22

Figure 7 depicts the data flow after doing the steps in the Table 6. Figure 7. Archive data flow after archive move Job user SAN EV Retrieval, Search, and Archive Explorer Index EV server 1 Mbx archiving Exchange 1 Vault metadata SQL MAPI Mbx archiving EV server 2 Archive storage Exchange 2 New home for Joe user s mailbox Index HP StoreAll Storage SAN PST Migration Although the PST Migration is not in scope of this release of M&C Archiving, this section provides a short description and general high-level guidelines of the existing PST Migration methods in Enterprise Vault: Scripted migration using Policy Manager: This is ideal for performing bulk migrations of PST files, but it is necessary to collect the PST files in a central location before the migration can start. It offers more flexibility than is available when using the migration wizard (see below). PST Migrator wizard-assisted migration: For a very small number of PST files the PST-Migration Wizard provides a quick and easy way of migrating them to Enterprise Vault. PST Migrator processes PST files that are on a mapped network drive or in a shared network folder. It cannot be used to search across users local hard drives. Server-driven PST Migration: This method locates PST files on users computers and network drives, copies them automatically to a central location, and then automatically migrates them. Unless there are only a few PST files to migrate, Locate and Migrate is likely to require least effort and is the recommended method by M&C Archive. Depending on the selected configuration options, there may be some manual intervention required to approve migration of PST files. Additionally, it is needed to supply the passwords for password-protected PST files. Client-driven PST Migration: This lets configure the users computers to locate PST files and queue them for migration. It is possible to configure the Enterprise Vault Outlook Add-Ins so that users can perform their own PST migrations. The underlying mechanism is Locate and Migrate. This can be useful if, for example: There are users with laptops who are in the office only one or two days a week. Enterprise Vault has no permission to access PST files on the user s computer. Thus making it difficult to obtain their PST files by other methods. 23

It is also possible to configure this facility to migrate PST files stored on network drives, but it is important to know that it cannot migrate files that fall into any of the following categories: Hosted on non-windows file servers, such as UNIX or Linux servers Hosted on distributed file system (DFS) folders Hosted on network shares and not present in the Microsoft Outlook profile Accessed through drive letters or home folders assigned through Active Directory Accessed through network shares to which the EV service account does not have full access Migration recommendations: First migrate a few PST files and then, when all involved people are familiar with the process, increase the number of PST files. Migration is much easier when the PST files located in just a few locations, rather than in many. Sort out the permissions on the PST files before running Policy Manager, otherwise they can fail. There is a Windows 2000 command-line utility, cacls, which you can use to grant the EV Service Account full control access to the PST files. When Enterprise Vault archives items, it also converts the contents to HTML and indexes them. There is a default conversion timeout of 30 minutes for this process. Enterprise Vault makes three attempts to convert an item, and so can take up to 90 minutes before failing an item and moving on to the next one. If there are very large or very complex items in a PST file, it can take a long time to migrate them all. If it is not needed to index the content of the items, then the conversion timeout should be set to just a few minutes. This change to the conversion timeout also affects normal archiving, so remember to reset it to the original value when all the PST files are migrated. It is also possible to improve performance by making Enterprise Vault create text rather than HTML versions of certain document types. For complex PST Migration, it is strongly recommended to use a third-party tool. High volume PST Migration usually requires a separate project to accomplish. Because, the high volume PST migration activities can consume considerable processing power on the Enterprise Vault servers that are already used to provide the daily mailbox archiving and journal archiving services. One must carefully plan the migration hours during the day and ensure sufficient storage is provisioned for the high-volume data. Note For more information about the PST Migration, refer to the chapter 15 PST Migration: Locate and Migrate in the Symantec Enterprise Vault Administrator s Guide. Journal archiving functional component detail Description Journal archiving functional component includes two subcomponents: Microsoft Exchange Journaling and Enterprise Vault Journal archiving. Microsoft Exchange Journaling captures copies of messages for select users (custodians), which are subsequently archived using Enterprise Vault, based on the customer s email retention or archival strategy. In this release of M&C Archiving, Microsoft Exchange 2010 SP2 journaling is used as the journaling tool and Symantec Enterprise Vault 10.0.2 as the archiving tool. Figure 8 depicts the relationship between journaling and archiving when using transport journaling, and the servers used to support such functions. 24

Figure 8. Relationship between journaling and archiving Journaling Archiving Exchange hub transport server (Journal agent) Exchange Journal Mailbox server Enterprise Vault server Journaling features provided by legacy Exchange messaging systems or journaling in a mixed (versions) Exchange environment is not in scope of this release of M&C Archiving. It will be considered part of the custom engineering for any customer that requires support of legacy Exchange messaging system. There are special considerations and complications for Exchange journaling configuration in a mixed Exchange organization. For more information on journaling with legacy Exchange, integration engineers can see the following Microsoft TechNet articles: Journaling with Exchange Server at technet.microsoft.com/en-us/library/aa997525(exchg.65).aspx. Understanding Journaling in a Mixed Exchange 2003 and Exchange 2007 Environment at technet.microsoft.com/enus/library/aa997918(exchg.80).aspx. Note This still applies to Exchange 2010. Exchange Journaling HP M&C Archiving assumes that Managed Messaging or a similar Exchange 2010 SP2 messaging system already exists and is functioning. The following content discusses how HP M&C Archiving uses Exchange 2010 SP2 journaling feature to provide the journaling function. For an overview of Exchange 2010 SP2 journaling, see this Microsoft TechNet article technet.microsoft.com/en-us/library/aa998649(exchg.80).aspx. HP M&C Archiving supports both standard journaling and premium journaling. Standard journaling is configured per mailbox database (the custodians must be grouped per database) and does not require the Exchange Enterprise Client Access License (CAL). Premium journaling can be enabled on a per recipient and per distribution group basis, which is more flexible than the standard journaling, but requires the Exchange Enterprise CAL. Integration engineers have to select the best option that ensures the customer meet the journaling requirements enforced by corporate policies or government regulations. Cost may also be a factor when choosing the journaling options. Exchange 2010 SP2 supports envelope journaling only. With envelope journaling, the original message that matches the journal rule is included as an attachment to the journal report, without any alteration. The body of a journal report contains the sender email address, subject, message ID, and recipient email addresses contained within the original message. This release of HP M&C Archiving only supports situations where the journal mailbox is inside the same Exchange 2010 SP2 organization on a dedicated journal mailbox server. This ensures the journal messages and journal mailbox are secure, reliable, and resilient as explained below. In Exchange 2010 SP2, all communications or sessions between hub transport server and mailbox server are authenticated and encrypted by default. This ensures the integrity of the journal messages when they are delivered to the journal mailbox. Journal mailboxes contain very sensitive information. Some messages can be part of legal proceedings or subject to regulatory requirements, and there are various laws that require messages to remain tamper-free before they are submitted to an investigatory authority. To increase security for journal mailbox, configure it to accept only messages from the Microsoft Exchange sender, and all messages sent to the journal mailbox be sent by authenticated senders. Additionally, take care not to impose mailbox quota limits on a journal mailbox. Note Access to the journal mailbox must be strictly limited and in general, no one except the Enterprise Vault administrators should have access to the journal mailbox. 25

Exchange Server 2010 SP2 ensures that a journal report is never lost due to an unavailable journal mailbox, be it full, not configured properly, or offline. (Lost messages can result in non-compliance to legal and regulatory requirements.) If a journal report cannot be delivered to a journal mailbox, the report remains in the hub transport server s queue until the journal mailbox becomes available. Journal mailbox server When journaling is enabled in Exchange Server 2010 SP2, journal reports are constantly delivered to the journal mailbox. The Enterprise Vault archive application regularly retrieves the journal reports from the journal mailbox, stores them in archive, indexes them, and deletes them from the journal mailbox. Especially when there is a large amount of journal messages generated within the organization, the transactions incurred on the journal mailbox may put a considerable load on the mailbox server hosting the journal mailbox. As a result of this additional load, HP M&C Archiving requires deploying a separate Exchange mailbox server to host the journal mailbox for handling the extra processing and storage load. In some exceptional cases, the existing Exchange mailbox server can also be used to host the journal mailbox, provided it has sufficient processing and storage capacity to handle the added load, and avoid any potential impact on other user mailboxes hosted on the same server. In such case, a separate mailbox database must be created to host the journal mailbox. Separate and dedicated physical storage spindles must be allocated for the journal mailbox database and its transaction logs. The purpose is to separate the journal mailbox from the user mailboxes for avoiding the potential performance impact to the user mailboxes, avoiding the complications between the backup and recovery of user mailboxes, and the backup and recovery of the journal mailbox. Again, deploying journal mailbox on existing production Exchange server that hosts user mailboxes is an exception to HP M&C Archiving standard, and is treated as part of custom engineering for the customer. Table 7 highlights the server specifications HP M&C Archiving recommended for a dedicated journal mailbox server. Integration engineers can also reference the hardware specifications when deploying journal mailbox on an existing mailbox server. Table 7. Exchange Journal Mailbox server specifications highlight Hardware Model Processor Memory Internal storage Network HP ProLiant DL380 G8 Server 1 quad core processor 8 GB RAM 2 x 146 GB in RAID 1 for OS, page file and application files 2 x 300 GB in RAID 1 for journal mailbox database 2 x 300 GB in RAID 1 for transaction logs 2 x 300 GB in RAID 1 for data recovery operations Fully redundant Gigabit Ethernet NIC Fully redundant Gigabit backup LAN (Internal Embedded NIC) Software OS Application Windows 2008 R2 64-bit Standard Edition Exchange 2010 SP2 Standard Edition Forefront Security for Exchange 2010 Note If you choose to use blade server, HP M&C Archiving recommends the HP ProLiant BL460c G8 Server series. For the blade server configuration, you can refer the above rack server configuration. 26

Performance considerations A dedicated journal mailbox server only hosts the journal mailbox. As a significant amount of journal reports are sent to this journal mailbox, and the journal reports (journal messages) are almost immediately read and deleted by Enterprise Vault, it is reasonable to consider this journal mailbox is accessed under a heavy user profile. Based on Microsoft recommendation for mailbox server memory, a mailbox server with one heavy user profile takes 2 GB plus 5 MB per mailbox for memory. So 8 GB of RAM must be sufficient for a journal mailbox server. For processors, Microsoft recommends four processor cores for a mailbox server. HP ProLiant DL380 G8 Server comes with a quad core processor and serves well. To best support the storage I/Os generated by the read and write activities to the journal mailbox, RAID 1 or mirrored configuration is recommended for the internal hard drives. On the Exchange side, especially now that MAPI connections are moved to the Client Access Server role in Exchange 2010, HP M&C Archiving follows the performance guidance Managed Messaging has provided in such as the ratio between the mailbox server role and the client access server role based on the number of processor cores. The journal archive task runs continually, archiving items immediately from the Exchange server journal mailbox. By default, the journal archive task maintains a maximum of five concurrent connections to Exchange server. The impact on Client Access Server, in most cases, can be ignored. So, M&C Archive does not require dedicated Client Access Server for the journal archiving purpose. Capacity considerations Messages stored in the journal mailbox are frequently retrieved and then removed by the Exchange journal task of Enterprise Vault. In case when the Enterprise Vault Exchange Journal task is offline due to server maintenance or failure, messages may accumulate in the journal mailbox server for prolonged time. Thus, sufficient storage space must be provisioned for the journal mailbox. HP M&C Archiving recommends limiting the mailbox database size to 100 GB on a standalone Exchange Journal Mailbox server. This supports the internal storage configurations recommended above. Recovery considerations It is critical to ensure messages stored in the journal mailbox do not get lost even when the journal mailbox database failed. So for recovery purpose, the journal mailbox database and transaction logs are put on separate set of physical spindles. You may also consider precreating a second (but idle) journal mailbox, on a separate, dedicated mailbox database. Options that are more robust are available to enable high availability for the journal mailbox, but these are beyond the scope of this offering. Networking considerations In premium journaling, Exchange 2010 SP2 generates a journal report for every email message that matches the criteria that are configured on a journal rule. Depending on the organization and how the journal rules are configured, Exchange 2010 SP2 may generate a significant number of journal reports. This requires a sufficient network link between the hub transport server and the journal mailbox server. HP M&C Archiving recommends deploying the journal mailbox server within the same LAN as the hub transport servers. Gigabit Ethernet is recommended to guarantee the network throughput. Exchange 2010 SP2 requires at least one hub transport server be deployed within the same Active Directory site where Exchange mailbox servers reside. An Active Directory site is usually formed based on a well-connected network. So it can be safely assumed that a journal mailbox server is always deployed on a well-connected network with at least one hub transport server. The hub transport servers are affected by the additional workload of processing the duplicate mail from the journaling process. Other considerations For other aspects of the journal mailbox server, HP M&C Archiving follows the Exchange mailbox server guidelines developed in Managed Messaging. In case when the existing Exchange environment is not a Managed Messaging solution, some justifications have to be made when deploying a journal mailbox server into that environment. For example, if the customer is not using Forefront Security as the messaging layer antivirus application for Exchange, other messaging antivirus application may be installed on the journal mailbox server. Hub transport server capacity is impacted by the additional workload of processing journaled messages. Considerations must be made for the hub transport design to address the overhead which can be up to 30 percent (as stated by Microsoft). Hub transport journaling agent priority must also be set to 1, so that messages are not dropped by other processes (such as ethical walls) prior to journaling. 27

Journal mailbox placement HP M&C Archiving recommends using a single Exchange Journal Mailbox for journaling in a customer Exchange environment whenever possible. This has several advantages: Reduces processing as the Enterprise Vault Journal archiving server does not need to process duplicate messages from multiple Exchange Journal Mailboxes. When running discovery accelerator, duplicate search results will result when using multiple Exchange Journal Mailboxes. Multiple journal mailboxes One journal mailbox can be used to collect messages for multiple or all journal rules that are configured in the organization. While it is technically feasible to create multiple journal mailboxes for multiple journal rules, it is not recommended unless you are compelled to, due to throughput constraints or unique business drivers that dictate separation of content for different business units. This is because generally all the journal reports from various sources are ingested by Enterprise Vault, stored into the common archive, and then indexed to become searchable or discoverable. It is optimal to have one journal mailbox, since message fan out is reduced and discovery accelerator search results will show duplicate messages in the search and review results when multiple journal mailboxes are used (because of fan out). Archiving server throughput capacity (how much it can index and store in the allotted time) dictates the number of Enterprise Vault journal archiving servers and Exchange Journal Mailboxes required. If the volume calculations show that the sum of journaled mail for two days cannot be ingested over the course of 48 hours, then additional Enterprise Vault journal archiving servers is needed to index and store the content. It is not uncommon to have some excess journaled mail from the workday that is not ingested until overnight. Ensure to account for daily backups and maintenance, weekend backups, and index defrags in your timing calculations. Premium journaling vs. store-based journaling Store-based journaling (similar to legacy Exchange 2003) requires that journaled users are organized on a specially configured mailbox store (or many mailbox stores). If a user is added to the scope of journaled users, their mailbox must be moved from a nonjournaled mailbox store to a journaled mailbox store. This introduces some impacts to the end user as some services can be affected by such a move (such as BlackBerry) and the journaled mail store must have capacity to accommodate the incoming journaled user. Additionally, there is some delay in processing a move request. Management of user placement on special mail stores can be complicated, depending on the characteristics of the customer. This is generally more tolerable, if the journaled user population is fairly static. Premium journaling in Exchange 2010 SP2 is configured to use distribution group members for determining the journaled users. Journaling rules are configured on the hub transport servers and then replicate automatically to all other hub transport servers in the Exchange organization. The hub transport journaling rules setup defines the affected group of users (members of the distribution group) and the place to send the journal reports (journal mailbox). The major advantage of this approach is its simple requirement, to bring users into scope for journaling, add them as members of a distribution group. There is a slight delay of (+ or -) four hours due to membership caching on the hub transport. It is also important to set to transport agent priority to one for the journaling agent. Enterprise Vault journal archiving Description To meet the needs of most compliance situations, it is a fundamental requirement that a copy of all the mails are passed to the archive before it reaches the recipient. This ensures that documents cannot be tampered with or deleted before they have been stored away in the archive. In compliance situations, this is a legal requirement. To provide the journaling facility with Enterprise Vault, journaling must first be enabled within Exchange. For a general description of Exchange journaling, see the Exchange Journaling section. The Enterprise Vault journaling task then works by ingesting items from the Exchange Journal Mailbox into the archive server. When the mail is archived by Enterprise Vault, it is written to WORM storage and cannot be altered in any way. Even if the recipient deletes the email as soon as they receive it, a copy still exists in the archive and can be retrieved for legal investigation, as long as it is retained. 28

Figure 9. Process flow Enterprise Vault services and tasks Enterprise Vault Archives Exchange Journal Mailbox server Journaling is also a prerequisite for discovery accelerator and in any implementation where compliance is a consideration, an Enterprise Vault journal archiving server and associated infrastructure (including Exchange journal mailbox server) must be implemented. For small implementations, where resources permit, the journaling task may be run on an existing Enterprise Vault server, that handles other workload, but this is not recommended. In larger implementations where journaling is required, a dedicated Enterprise Vault journal archiving server will be needed and this will be the standard configuration for the HP M&C Archiving offering. For more information, see the Estimating journaling resource requirements section. Note The HP M&C Archiving solution is designed to be a best fit for most situations. In small implementations, depending on usage and customer requirements, install the Enterprise Vault journal archiving components and the SQL Server all on one physical server. Such solutions may be appropriate on the grounds of cost and must be considered custom. It is a best practice to centralize the Exchange Journal Mailbox servers and the Enterprise Vault journal archiving servers within the Enterprise Vault environment even if there are more than one Enterprise Vault servers and locations. This strategy makes the best use of storage and allows for better deduplication of journaled mails. The HP M&C Archive offering is always predicated on a centralized environment, so only one Enterprise Vault journal archiving server is required. However, when there is heavy load, additional Enterprise Vault journal archiving servers might be needed, and the spreadsheet in the section Estimation tool for journaling can be used as a rule of thumb estimator for the number of Enterprise Vault journal archiving servers that will be required. Within Enterprise Vault, an Exchange Journaling task performs the archiving. One of these tasks can service multiple Exchange Journal Mailboxes, located on multiple Exchange Journal Mailbox servers, although a single Exchange Journal Mailbox is recommended. Exchange Journaling tasks run under the control of the Task Controller Service. When properly configured, the Exchange system journals messages for selected users (custodians) and sends the journal reports to the journal mailbox. The journal report is a combination of the message wrapper (called an envelope) that retains the sender and recipient information, including BCC, in addition to a copy of the original message. Items in journal mailboxes are deleted after they are archived and replicated onto the HP StoreAll Storage, and no shortcuts are created. The archiving of journaled emails takes place when the Enterprise Vault server polls the Exchange Journal Mailbox server. This takes place continually as the journaling task is running on the server. There is a default five minutes delay to help Exchange deduplicate journal reports. Journaling is a real-time activity, so limiting journaling to a few hours during the day is not recommended. The journaling tasks must only be paused for scheduled backups in line with the Enterprise Vault backup procedure and regular maintenance such as index defragmentation. For more information, see Backup and recovery functional component detail section. During this period, journaled emails accumulate on the Exchange Journal Mailbox server and thus the regular maintenance is best configured at a time of low user activity. Note that in situations where journaling is being heavily used in a 24 hours per day environment, it is still a requirement that a regular maintenance window of at least four hours per day is always allowed for backups and similar activities. During this period, the Enterprise Vault servers are in a read-only state so that a consistent backup can be taken. During this maintenance time, journaled email accumulates on the Exchange Journal Mailbox server. This is required for daily maintenance tasks. For more information on backup strategy, see Backup and recovery functional component detail section. Administrators with access permissions to the journal archives can search for messages. Care must be taken to ensure that proper control is placed over on who has access to search the archived data. Built-in auditing enables Enterprise Vault to track all access and changes, so enable this feature. As the HP M&C Archiving offering is predicated on Exchange 2010 SP2 servers, envelope journaling is always enabled. The Enterprise Vault Exchange Journaling task processes journaled emails from Exchange Journal Mailbox servers. 29

Estimating journaling resource requirements Within the HP M&C Archiving offering, the recommendation is to use a single Exchange Journal Mailbox server, with a single Exchange Journal Mailbox. The advantages of this approach are: Processing load is minimized because journaled messages only need to be archived from one location. Discovery accelerator searches do not return duplicates that can occur due to fan out. Spare or additional servers may be considered as part of a more resilient design, but this is beyond the scope of the standard offering and would be considered a custom solution. When a single Exchange Journal Mailbox server, with a single Exchange Journal Mailbox is implemented, the usage calculation is relatively simple, because no account is taken of duplicate mails on multiple journal mailboxes. As a simple rule of thumb, the following calculation can be used: Rule of thumb throughput estimation (based on a dedicated Enterprise Vault journal archiving server) In the absence of actual data on the number and size of messages to process, the following rules of thumb can be used to estimate the Enterprise Vault journal archiving server requirement. The standard Table 3: Enterprise Vault Mailbox Archiving server specifications highlight HP M&C Archiving server specification can process approximately 5 GB per hour into the archive. Number of users * Number of mails per user per day * average mail size * deduplication factor gives a rough estimation of the daily data volume to archive. Convert this figure into gigabytes and divide by three to determine how many hours it will take a single Enterprise Vault journal archiving server to process the data. Remember that four hours are required for backup and maintenance so you only have a maximum of 20 hours per day to process data. Remember to leave some room for growth over the contract period. So, in summary, the amount of data to archive per day = number of journaled users * number of messages per user per day * average message size * deduplication factor (0.67). A standard Enterprise Vault journal archiving server can process a nominal 5 GB per hour. Calculation on throughput of the Enterprise Vault journal archiving server is more accurate than the calculation based on the number of messages because message size changes over time. The amount of data archived per day is the most important variable when attempting to calculate the Enterprise Vault journal archiving server requirement. Example scenario 3: Back at ACME Widgets, the finance department has discovered that they need to comply with Sarbanes Oxley and have made the decision to implement journal archiving and discovery accelerator: They have determined that average message size is 75 KB. Although users keep 20 mails per day, they actually send 20 and receive 60, so a total of 80 mails per day for journaling. Journaling can run for 20 hours per day. In this scenario, we can now calculate that 45 GB of data is archived per day. At a nominal 5 GB per hour, we can easily process the journal load using one dedicated Enterprise Vault journal archive server in 15 hours. Additionally a dedicated discovery accelerator server will be required. So, in order to meet this new compliance requirement, ACME widgets have to purchase the following additional hardware: Dedicated Exchange Journal Mailbox server Dedicated Enterprise Vault journal archiving server Dedicated discovery accelerator server 30

To gain a more accurate estimation of the throughput for journaling, and in circumstances where there are multiple journal mailboxes, the following approach must be used. There are several key pieces of information to consider. These are: The average number of messages to journal archive per workday The average size of messages currently (for example a recent three-day sample) The number of journal mailboxes that will exist in the Exchange environment An estimate of the typical number of internal addresses on a message (this relates to the deduplication factor) The average number of messages to journal archive can be extrapolated by using the Exchange 2010 SP2 cmdlet Get-MessageTrackingLog. An alternative analysis can be obtained by using the results from Exchange Store Reporter (see symantec.com/business/support/index?page=content&id=tech60472). However, it is important to remember that these results are not always accurate as users can move messages into PST files or aggressively delete messages due to low quotas. To accommodate this, 30 percent is usually added to the number obtained from the Exchange Store Reporter results. If Exchange Store Reporter results are not available, then a general rule of thumb of 100 messages sent or received per day per user will be used. The number of journal mailboxes is important because multiple copies of messages generated within Exchange can be sent to multiple journal mailboxes from which Enterprise Vault archives them. Although, Enterprise Vault has the potential to single instance the item in the Enterprise Vault Store, the horsepower required to archive from multiple Journal mailboxes must still be factored. The average number of internal addresses on a message is important because it allows for the calculation of the number of unique messages to be archived by Enterprise Vault. Accounting for multiple journal mailboxes is known as a fan-out factor. The typical number of shared internal addresses on a message is commonly three. Using this value, several fan-out factors have been precalculated based on potential number of journal mailboxes in a customer s environment. Table 8 lists common fan-out factor values found in Exchange environments. Table 8. Common fan-out factor values Number of journal mailboxes Number of journal messages generated per message (fan-out factor) 1 1 2 1.75 3 2.11 4 2.31 By gathering the above statistics, you can determine the number of journal messages to archive per day. With the throughput of 5 GB per hour that an Enterprise Vault journal archiving server can process, we can estimate the number of Enterprise Vault journal archiving servers and storage required for a given environment. Journal archiving runs throughout the day so the archiving rate must keep up with the message flow during the day. Some backlog is acceptable as long as the journal archiving task can catch up during off hours and weekend. The number of Enterprise Vault journal archiving servers required is determined by taking the estimated number of messages sent or received per day divided by the estimated number of internal addresses per message. The resulting number is the unique number of messages to be archived. If there are multiple journal mailboxes, multiply the unique number of messages to be archived by the fan-out factor for the number of journal mailboxes in the environment. This gives you the number of messages to journal archive per day. Best practice shows that 90 percent of these messages fall within a 16-hour window. Multiply the number of messages to journal archive per day by 90 percent then divide by 16 hours. This is the hourly throughput required for this environment. Divide the hourly throughput required by the known throughput of the Enterprise Vault journal archiving servers used for the environment to determine the number of EV journal archiving servers required. The formula written out is: Number of unique messages to archive per day = Number of messages sent or received per day or estimated number of internal sharers per message Number of messages to journal archive per day = Number of unique messages to archive per day * Fan-out factor due to multiple journal mailboxes 31

Throughput required = Number of messages to journal archive per day * 90 percent/16 Number of EV journal archiving servers required = Throughput required or known throughput of single EV server Estimated number of internal sharers per messages is the average number of internal addresses on a message. Best practice in absence of customer data is three. The fan-out factor is based upon the total number of sharers across all the journal mailboxes that participate in sharing within a Vault Store Group. Read the Exchange Journaling section of the Symantec Enterprise Vault 10.0 Performance Guide (symantec.com/docs/doc4553). Example 1: If a customer has, 1,000,000 messages sent or received per day (Assuming standard message size is 70k) and will be implementing one journal mailbox, the number of Enterprise Vault journal archiving servers are calculated as: Number of unique messages to archive per day = 1,000,000/3 = 333,333 Divided by 3 as the number of internal addresses per message in absence of customer estimate Number of journaled messages to archive per day = 333,333 * 1= 333,333 Multiply the number of unique messages to archive per day by a fan-out factor of 1, which correlates to one journal mailbox in the table above. If there are two Exchange Journal Mailboxes, the fan-out factor will be 1.75. Throughput required = 333,333 * 90%/16 = 18,750 You could also add a growth factor to account for increased mail size and increased number of mails over a period of years that corresponds to the life of the hardware (normally three years). Number of Enterprise Vault journal archiving servers required = 18,750/(90,000 *75%) = 0.27 This is based in a 67,500 ingestion rate, which represents 75 percent of 90,000 on an average message size of 70k. If the message size increases, this rate gets lower. Number of Enterprise Vault journal archiving servers required = 1 (rounded up from 0.20) Divided the throughput required by 90,000 items per hour, the benchmark for a 16-core EV server. Example 2: The customer has 5,000 users and has not been able to run Exchange Store Reporter, and does not have any other means to determine the number of messages sent or received per day per user. The rule of thumb value of 100 messages sent or received per day per user is applied. The total number of messages sent or received per day is equal to the number of users multiplied by the rule of thumb value. Total number of messages sent or received per day = 5,000 *100 = 500,000 messages sent or received per day. The number of unique messages is calculated by taking the total number of messages sent or received per day divided by the average number of sharers (rule of thumb is 3). Total number of unique messages to archive per day = 500,000/3 = 166,667 As a rule of thumb, we estimate that 90% of these messages are received within a 16-hour window. Therefore, we can estimate the required throughput of the journal archiving server. Throughput required = 166,667 * 90%/16 = 9,375 items per hour Number of Enterprise Vault journal archiving servers required = 9,375/90,000 = 0.10 90,000 rate is based on the average message size of 70k. If the message size increases, this rate gets lower. Number of Enterprise Vault journal archiving servers required = 1 (rounded up from 0.10) Divided the throughput required by 90,000 items per hour, the benchmark for a 16-core EV server. In addition to the above, national and international laws can have an impact on the location and number of Enterprise Vault journal archiving servers (and Exchange Journal Mailbox servers). For example, data relating to the US military may not leave the US or be processed by non-us personnel. Likewise, in several European countries, data deemed to be personal may not be exported to the US for processing. In Germany, data of a personal nature may not even be collected without the user s permission. When defining the requirements for journaling, it is essential to carefully analyze the customer environment so that a proper design can be proposed. The number of Enterprise Vault servers required is significantly impacted by the profile of the users and the organization. In some circumstances, a single EV server may be able to serve large number of users. In other circumstances, several EV servers could be required for 1,500 users. 32

The HP M&C Archiving recommendation is to always use a dedicated Enterprise Vault Journaling server where journaling is a requirement. For the purposes of this guide, the journaling archiving requirement is treated as a separate entity. In a production environment, Exchange Journaling must be considered in conjunction with other archiving requirements such as Mailbox Archiving and discovery accelerator. Complex environments Example scenario 4: ACME Widgets is expanding, they have just acquired a smaller competitor who makes components for the car industry and the US military; they want to implement Enterprise Vault Mailbox Archiving and journal archiving to meet compliance requirements for the new company. The data from the new company must be kept separate from ACME Widgets, as they are separate legal entities. They have 2,000 mailboxes, 800 in Detroit, 700 in the UK, and 500 in Germany. They have centralized data centers in Detroit and Frankfurt. They want journaling for all UK and US users, and mailbox archiving for all 2,000 users. They want discovery accelerator to monitor users in the UK and the US (not permitted in Germany [Deutchland]). How many EV servers? Two Exchange Journal Mailbox servers, one in US and one in Europe for UK data (US military data cannot be exported from the US, personal data cannot be collected in Germany, and data that can identify an individual cannot be exported from the UK to the US for processing). Two Enterprise Vault journal archiving servers (One for US data and one for UK data) Two discovery accelerator servers (US data must remain in US and German data is excluded) Two Enterprise Vault Mailbox Archiving servers (Detroit and Frankfurt data centers) Two SQL Servers (64-bit) (1 for each data center location) Four HP StoreAll Storage, two in Europe and two in the US (including redundancy) Total=10 Servers Why? Because of US laws on export of military data, separate Exchange Journal Mailbox servers are required in the US and Europe Likewise, two Enterprise Vault journal archiving servers are also required. One for US data that has to remain in the US, and one in the European data center for the UK data that cannot be exported to the US. As already, mentioned, German data cannot be collected because of German privacy laws. For the same reasons as above, two Enterprise Vault Mailbox Archiving servers are also required. One in the US and one in Europe. Two 64-bit SQL Servers are required. In each location, we require a separate dedicated SQL Server. In the above examples, there is no issue with capacity. A single Enterprise Vault server could easily handle the total expected load. However, care must be taken to understand the customer environment before giving estimations on server numbers. The Estimation tool for journaling section can be used to provide an estimation of the likely resources needed for journaling, but legal requirements between different countries are extremely complex and must always be taken into consideration. The customer must advise us on what legal requirements must be met. Rules of thumb (based on Symantec experience) A 16-core processor EV server could in theory support journal archiving for about 60,000 typical mailboxes. In practice, a more realistic rate of 45,000 (60,000 * 75 percent) should be used (based on Managed Messaging usage profile and message numbers). So, in most circumstances, if based on throughput alone, a single Enterprise Vault journal archiving server suffices. A single 64-bit SQL Server instance can support up to eight EV servers. A separate SQL Server should always be used for discovery accelerator. 33

Required metrics The following are the required metrics that need to be considered when conducting a sizing exercise for Exchange Journaling and are discussed in detail below: The number of journaled users The average daily number of emails received per user Average size of email The following information is required to design the journal archiving policies: How long will email be kept? What are the business goals of the retention policy? When will email be removed from Enterprise Vault? Is there a department-level (human resources, legal) retention policy? Will email be retained indefinitely? Are there country-specific data protection and privacy laws? The following information is required to determine audit requirements: Will all Exchange email be journaled for some period of time? Will legal discovery searches be required? In regulated industries such as financial services, retention periods for journaled email can be as short as 30 days or as long as seven to 12 years. In industries such as health care, it can be 100 years or the life of a patient. Answering the archiving policy questions is simpler for customers with a message-journaling focus as their requirements are often driven by specific laws, regulations, or high-level corporate policies. When using discovery accelerator for a large number of legal holds, the legal holds take precedence over retention policy. Legal hold is a discovery accelerator feature that prevents messages from being deleted or expired if they are part of a discovery accelerator case. Once the case is closed in the discovery accelerator application, the legal hold can be removed, and the messages could be deleted or expired normally. Each Enterprise Vault server that runs an Enterprise Vault Storage Service must have a discovery accelerator license to use this feature. Journaling is typically required as part of an overall compliance solution so it can be expected that the Enterprise Vault servers must be capable of performing the type of high performance searches associated with legal discovery. In particular, the indexes must be highly tuned. Estimation tool for journaling Symantec provides a comprehensive sizing tool that is always used for estimations. The tool is available on the Symantec partner website and should be used for providing detailed sizing recommendations. This tool must be filled in by the HP archiving engineer using input obtained from the customer. Storage details The following section provides more detailed storage requirements specific to Enterprise Vault journal archiving. Disk storage detail When implementing journaling, the storage requirement must be carefully estimated. Storage must be allocated for the Vault Store, SQL Server, and indexes. Storage requirements are generally in line with those required for mailbox archiving. For guidance, see the Storage functional component detail section. Note Only storage information specific to journaling is included in this section. Vault store detail When implementing journal archiving, a separate dedicated vault store with its own partition and database is always required. 34

HP StoreAll Storage details The HP StoreAll Storage range of NAS devices can provide archive storage in situations where non-worm or WORM-like storage is a requirement. HP StoreAll Storage is presented as CIFS Share NTFS partitions. As such, it must be backed up to HP StoreOnce or tape on a regular basis, or replicated to a secondary DR HP StoreAll Storage or both, dependent on the customers requirements. Enterprise Vault partition rollover should be used to ensure that archive data is backed up in an efficient manner. Only the most recent data needs to be backed up on a daily basis. Closed partitions can be backed up on a less regular basis. The exact rollover plan depends on the specific customer requirements. As a general rule of thumb, the partitions should be kept to between 200 GB and 400 GB. In order to estimate the amount of storage required for the archive data, you should use the sizing tool from Symantec. As a very rough rule of thumb, you can estimate the amount of storage required for archive data to be approximately half the original volume of data ingested. Index configuration for Enterprise Vault Journaling Archiving When implementing journaling, special consideration must be given to the configuration of the indexes. When running discovery accelerator searches, the indexes are very heavily used and it is important that they are tuned to provide the best possible performance. The indexing service file locations require good random access or a high number of IOPS combined with good transfer rates. Therefore, the index must be hosted on dynamically expandable high-speed SAN partitions and configured to ensure that dedicated LUNs are used and they are able to cope with the high number of IOPS, Symantec Enterprise Vault Performance Guide. Storage team must be consulted on the best configuration options for the SAN. They normally consult the SAN hardware vendor on the implementation who can provide correct configuration details to achieve optimal performance. Typically, LUNs must be created across as many suitable disks as possible, using entire disks rather than partial disks to ensure the best possible performance for the indexes. When implementing journaling, special consideration should be given to additional registry settings that tune the indexes for journaling. See the Symantec Enterprise Vault Performance Guide for detailed instructions on setting up Index Rollover and Index Granularity and other indexing best practices. Technology specifications This section describes the list of materials associated with a component. For Exchange Journal Mailbox spec, see Journal Mailbox Server section. Enterprise Vault server specifications mentioned in Table 9 are required. Table 9. Enterprise Vault Journal Archiving server specifications highlight Hardware Model Processor Memory Internal Storage SAN Network HP ProLiant DL380 G8 Server 16-core processors 32 GB RAM 2 x 300 GB in RAID 1 for OS, page file and application files 2 x 300 GB in RAID 1 for MSMQ 4 x 1 TB in RAID 5 for index data backup (optional, and up to the availability of 1 TB SFF disks) 2 x HBA Fibre Channel connections Fully redundant Gigabit Ethernet NIC Fully redundant Gigabit backup LAN (Internal Embedded NIC) Software OS Application Windows 2008 R2 SP1 64-bit Standard Edition Enterprise Vault v10.0.2 Microsoft Outlook 2007 Standard with SP2 35

Note If you choose to use blade server, HP M&C Archiving recommends the HP ProLiant BL460c G8 Server series. For the blade server configuration, you can refer the above rack server configuration. See the EV compatibility guide for versions of Outlook 2007/2010 and most current supported and required hotfixes for best performance symantec.com/business/support/index?page=content&id=tech38537. List of hardware and software used The hardware specifications for Enterprise Vault server used for journal archiving are the same as those for mailbox archiving, except for the optional internal drives used for index data backup purpose. E-Discovery functional component details This functional component deals with E-Discovery using the Enterprise Vault discovery accelerator. There are two accelerators of which only one, discovery accelerator is made available within this release of the offering. The accelerators are additional Enterprise Vault applications for compliance monitoring and data mining activities such as legal discovery. Legislation by regulatory bodies, such as SEC, NYSE, and FINRA in the US and Freedom of Information laws in Europe, means that organizations are obliged to retain an increasing amount of electronic correspondence and data. The native Enterprise Vault product enables organizations to capture, store, and retrieve relevant data. The Enterprise Vault accelerator products, Compliance Accelerator and discovery accelerator, provide specialized customer applications for capturing, searching, and reviewing data that is archived using Enterprise Vault. Discovery accelerator Discovery accelerator is a fully managed legal discovery and review system that integrates with Enterprise Vault services and archives. Discovery accelerator enables authorized users to retrieve, review, mark, and export emails for lead counsel examination or court-ready production. It is specially tailored for performing detailed searches to find archived items relating to a legal cases or internal enquiries. Using the discovery accelerator smart client application, case administrators can run searches for relevant data. Case reviewers can then review the items returned by the searches and produce the required items in a format suitable for presenting as evidence. Compliance Accelerator Compliance Accelerator provides a capture-and-review system for monitoring employee messages to ensure compliance with industry regulations or company policy. A random sample of journaled messages (inbound and outbound) can be captured daily and reviewed by compliance officers using the Compliance Accelerator Web interface. In addition, Compliance Administrators can run regular searches to find messages that meet certain criteria, for example, messages that contain unacceptable language or particular phrases. To use Compliance or discovery accelerator, the Enterprise Vault journal archiving must be implemented (which also requires the addition of Microsoft Exchange journaling). Note Compliance Accelerator is beyond the scope of release for this offering and is not described in detail in this document. Any reference to Compliance Accelerator is included only to provide a complete picture of the Enterprise Vault compliance capabilities. 36

Discovery accelerator overview Within the HP M&C Archiving offering, only discovery accelerator is provided as an option. Compliance Accelerator is considered to be a custom solution. Discovery accelerator uses the search facility available in Enterprise Vault. It then adds the necessary security that is required in legal discovery to ensure that search criteria and results are secure. In addition, it stores details of all the searches performed, the criteria used, and the items found. These details can be viewed but cannot be changed or deleted from the system. Discovery accelerator is resource intensive and even in small implementations makes significant demands on both the Enterprise Vault Journaling Server and the indexes associated with it. Running discovery accelerator concurrently on the same computer as Enterprise Vault is not generally recommended. As standard, the HP M&C Archiving offering requires separate dedicated Enterprise Vault and discovery accelerator servers and a separate dedicated discovery accelerator SQL database server. A discovery accelerator client database uses four CPUs during a search. As a minimum, a separate dedicated SQL Server instance with four dedicated CPU is required. In medium and large installations, the HP M&C Archiving recommendation is to run the discovery accelerator databases on a dedicated physical SQL Server with a minimum of eight CPU cores. In small installations (less than 100 users with a single reviewer and minimal searching), a second SQL Server instance for discovery accelerator can coexist on the same hardware as the first SQL Server instance which hosts only the EV mailbox and journaling databases but four or more CPUs must be dedicated to that instance. There are also circumstances where it might be appropriate to have completely separate installations of discovery accelerator. For example, where an organization spans international borders and different laws on data privacy and retention apply, or even for separate departments and individual legal discovery users. In such situations, each environment would require its own set of dedicated servers, including Exchange Journal Mailbox server, Enterprise Vault Journal Archiving server, SQL Server database server, and discovery accelerator server. Prior to designing an implementation of the system, a conversation between the compliance officer and discovery accelerator teams must take place to ensure that all necessary options are taken into consideration. It is a good practice to add a second discovery accelerator website (and a separate customer database) in an existing discovery accelerator for troubleshooting and support purposes. Discovery accelerator components Table 10 provides a description of the major components of discovery accelerator. Table 10. Components of discovery accelerator Component Discovery accelerator client Accelerator administrator Web interface Enterprise Vault Accelerator Manager service Client database Configuration database Description Used by discovery accelerator administrators to setup and manage the system, and by reviewers to access the items that they must review. Allows you to set up and manage multiple discovery accelerator databases to store your data. For example, using this facility, you can split your data by date range or organizational unit. Handles requests from the discovery accelerator Web interface and works with the Enterprise Vault components to access archives, perform searches, and so forth. SQL Server database that specifies the location of the client databases, and stores details of the SQL Server, database files, and log files to use. SQL Server database that specifies the location of the client databases, and stores details of the SQL Server, database files, and log files to use. 37

Figure 10 shows how the major components interact. Figure 10. Discovery accelerator process diagram Discovery accelerator requirements Discovery accelerator server and Web service can potentially be resource intensive and for this reason, the discovery accelerator server is normally installed on dedicated hardware. The discovery accelerator service and Web server concurrently coordinate significant activity and therefore require a multiprocessor server. The flow of data through discovery accelerator can result in very high levels of I/O activity on both discovery accelerator server and SQL Server database server. In the HP M&C Archiving offering, it is a requirement that the discovery accelerator Database Server (SQL Server), application server, and EV servers are connected via gigabit network technology. The discovery accelerator server utilizes the standard HP M&C Archiving offering standard build 16-core processors with 32 GB of RAM. It must be installed on the Windows 2008 R2 SP1 (or above) 64-bit Standard Edition operating system. In very complex environments with many reviewers and searches taking place, it is possible that the discovery accelerator services to be split over several physical servers. Such complex environments are beyond the scope of the standard offering and will be considered custom solutions. Within the HP M&C Archiving offering, only two discovery accelerator configurations are considered. A small-sized environment in which the discovery accelerator SQL Server Databases are resident on the same SQL Server instance as the journaling databases, and a large-sized installation in which a separate SQL Server instance is implemented for the discovery accelerator server. The large size implementation is defined as an installation, where there are seven or more Enterprise Vault servers using the first SQL Server instance. A small-sized implementation is defined as an installation, where all the components can be installed on a single physical server. Small-sized implementations are considered custom solutions. 38

The two M&C discovery accelerator configurations are as depicted in Figure 11 and Figure 12. Figure 11. HP M&C Archiving offering small-sized discovery accelerator configuration Enterprise Vault servers SQL Instance 1 SQL Instance 2 Discovery Accelerator Figure 12. HP M&C Archiving offering large-sized discovery accelerator configuration Enterprise Vault servers Dedicated EV SQL Server Dedicated EV SQL Server Discovery Accelerator The discovery accelerator Web server requires memory for previewing, reviewing, and administrative tasks, and demands high CPU requirements for concurrent multi-user reviewing tasks. An HP M&C Archiving offering standard 16-core processor discovery accelerator server running on Windows 2008 R2 SP1 64-bit Standard Edition with 32 GB RAM typically supports up to 300 concurrent reviewers, but this does depend on the working practices of those users. More than 300 concurrent reviewers require multiple Web servers to be deployed in a Web farm arrangement. However, a Web farm is unlikely to be required on a smaller user base. Concurrent Web users also create a high load upon the discovery accelerator service. This load may not coexist with the customer tasks (activities such as searching, exporting, and synchronization) and therefore the discovery accelerator service may need to be scaled out by deploying multiple discovery accelerator servers. Performance considerations The following section provides guidance on performance enhancements required for discovery accelerator and the implications on for the existing Enterprise Vault infrastructure. SQL Server Running multiple SQL Server instances is not recommended because of the load profile of discovery accelerator so the recommendation is to always use a separate dedicated SQL Server. The discovery accelerator SQL database server should be hosted on Microsoft SQL Server 2008 64 bit Enterprise Edition. At least 16 GB of RAM should be installed. If running analytics, you should add a further 4 GB per concurrent analytical collection task. 39

A typical discovery search can be accepted at a rate between 500 2000 items per second. This tends to be very I/O intensive so special attention has to be paid to the database files particularly the logs and their location. During acceptance, the database log file tends to experience sustained periods of 900 IOPS, and the data files experiences peaks of activity between 500 and 3000 IOPS. The discovery accelerator database generally utilizes four processors during discovery search and export activity, and this also depends upon the database server I/O subsystem throughput. An SQL Server instance running on the standard HP M&C Archiving SQL Server specification (64-bit platform) with 32 GB of RAM provides the necessary performance required for the high load of directory assistance. If multiple instances (two) are being hosted on the same physical server, 32 GB of RAM sufficiently supports the added instance. The majority of discovery accelerator actions result in updating the database, and the volume of manipulation means the I/O subsystem has to handle a high level of activity. As previously discussed, the disk sub-system must be configured such that the TempDB and log files are on separate disk spindles. Each customer DB must be on a separate dedicated LUN. The database LUNs must be configured on RAID 10 with high-speed disks (15k) or high-speed SAN and multiple spindles able to provide the high level of IOPS required. Transactions log drives must be configured for RAID 1 15k or high-speed SAN and multiple spindles able to provide the high level of IOPS required. Configure maintenance plans (rebuild indexes, update statistics, and shrink databases). Each database requires the disks to be arranged for two different purposes: the database data and log files. The data files require good random access or high number of IOPS, and therefore a striped array of many disks must be used (using hardware-based RAID and not software). The log files require good sequential write performance. So, each log file must be placed on its own high-speed disk mirror (RAID 10) with good transfer rates. SQL database sizing See the Symantec discovery accelerator 10.0 best practices document for detailed sizing information symantec.com/docs/doc5338. The rule of thumb calculation for database size is as follows: The discovery accelerator configuration database stores details of the following: All registered client databases The Custodian Manager database All client configuration options Errors that the service has logged The database is also used to manage analytics data collection tasks across all clients and cases. In general, the configuration database remains less than 10 MB. However, error conditions can quickly grow the related table. A good rule of thumb is to allow at least 100 MB for the configuration database data file. The details of discovery accelerator Custodian Manager database stores are as follows: All custodians, including their attributes, email addresses, and group membership. Historical record of changes to custodian attributes, addresses, or groups over time. Sizing the Custodian Manager database The Custodian Manager synchronizes custodian details with the corporate directory infrastructure through Active Directory, Domino Directory, or LDAP queries. Use the following rules of thumb to size the Custodian Manager database: Base capacity (MB) = ([3.09e]+[0.23ea]+[2.52g]+[0.23ag]+[0.17en]+[0.05et])/1000 Yearly capacity (MB) = ([0.238ace]+[0.24acg]+[0.05m]+[0.05ect])/1000 40

Where: E = Total employees in directory sources A = Average number of email addresses per employee or group C = Average number of address changes per employee per year G = Total number of groups N = Average group membership per employee M = Typical total number of group movements per year T = Average number of custom attributes (in addition to standard attributes) Note This calculation provides a high-level estimate only. It does not take into account the operating system, paging, and log file devices. It also does not include any additional capacity that may be required. When the Custodian Manager is synchronizing custodian details, the database typically requires 100 to 300 IOPS. Sizing the customer database Use the following rule of thumb to size the core Discovery customer database (excluding analytics tables): Capacity per year (MB) = ([2.53cis]+[0.44acs]+[1.82is]+[11.6ce]+[0.222cism])/1000 Where: C = Average cases per year S = Average number if searches per case I = Average number of items captured per search A = Average number of archives searched per case M = Average number of review marks per captured item C = Average number of items exported or produced per cases year On a server with 8 GB of RAM, a Discovery customer database data file typically requires 250 to 1,800 IOPS and log file around 500 to 1,200 IOPS. However, this varies depending upon concurrent activities and server specification. Analytical tables The analytics tables within the customer database require special consideration. When you enable a case for analytics, discovery accelerator creates a set of new tables specifically for the case. These new tables contain the original item metadata and content, together with the analysis details. The analytics feature collects the original item content and other metadata from Enterprise Vault and inserts it into the tables. Once the data collection is complete, the additional features introduced by analytics are available to end users, and these tables are then subject to a much-reduced load. The analytics tables are broken out into separate file groups, which are created ad hoc when each case is enabled for analytics. Each file is created on a partition selected in a round-robin fashion from the list of available partitions configured for each customer database. In addition, the analytics tables require full-text indexes, which are also created ad hoc when each case is enabled for analytics. Each set of full-text index files are created on a partition selected in a round-robin fashion from the list of available partitions configured for each customer database. The table file groups and full-text indexes should be located on the same partitions, which are then be co-located with the table file group. The discovery accelerator analytics tables grow to a similar size as the total converted-to-html size of all included original items and their attachments. This also affects the discovery accelerator database log file size in a similar manner. Analytics operates on the item converted content provided by Enterprise Vault, so binary data such as image files do not consume space. The converted content is encapsulated in XML that contains other metadata. Many of the analytics table columns are full-text indexed, which adds further overhead. 41

You must size the file group partitions to encapsulate multiple enabled cases of varied size with potentially varied characteristics. So, you must make a high-level estimate that is based on the maximum number of cases and items in analytics at any time, combined with the average size, and overheads accounting for conversations and row sizes. So, the Discovery customer database also requires the following additional storage for the Analytics file groups: Total capacity (MB) = 0.0166097i+0.00025ai Where: I = Total number of items (including attachments) in analytics at any time A = Average original item size Collecting analytics data for a single case is likely to produce sustained periods of very high I/O activity. This varies according to the available physical memory and other concurrent activities. On a database server that provides a throughput of up to 150,000 items per hour, the I/O activity might typically be 1,200 to 2,900 IOPS at the analytics case data file and 100 to 200 IOPS at the customer database log file. The process may take several hours to complete. You can distribute the load from multiple analytics data collections by using multiple file group locations, which are defined when creating the discovery customer database. Divide the estimated storage between at least as many independent partitions as there are expected to be simultaneous cases enabled for analytics. So, if multiple cases are concurrently performing analytics data collection, they should be using different partitions. Once the analytics data collection has completed, the normal I/O activity is much lower. The partition is selected again for subsequent cases, but this should not have any impact on other cases that are still in analytics and located on the partition. Enterprise Vault indexing and discovery accelerator Discovery accelerator search functionality can significantly affect the Enterprise Vault indexing services. If these do not perform effectively, the discovery accelerator searches can be extended. An existing environment can require an upgrade to meet the higher specifications demanded by the discovery accelerator services. The server hosting Enterprise Vault indexing service may be required to handle sustained periods of CPU, memory, and I/O-intensive activity during searches with, by default, 10 concurrent searches running until all indexes are searched. This activity is likely to be repeated throughout the working day as search requests are added by the legal discovery team. This activity has to run alongside any normal journaling, archiving, or retrieval tasks. A search with the default of ten threads typically demands 600 IOPS for 32-bit index volumes or 1,200 IOPS for 64-bit index volumes whilst searching. (However, this varies depending on the hits retrieved per volume.) It should search approximately 5,000 6,000 mailbox index volumes per hour excluding result retrieval, which reduces this rate. Table 11. Estimated index IOPS 32-bit index volumes only 64-bit or mixed 32/64-bit index volumes Small customer stand-alone Enterprise Vault Small customer single Enterprise Vault and discovery accelerator 500 IOPS 1200 IOPS 500 IOPS 1200 IOPS Medium or large customer 1000 IOPS 3200 IOPS A typical search of a single journal index results in around 2,200 IOPS. Once implemented, tune the discovery accelerator and modify the setting in the application Number of Vault Search Threads from 10 to 2. Perform search and monitor IOPS, tune up the number of threads until the IOPS is below the maximum acceptable threshold. This procedure is detailed in the Symantec Enterprise Vault Performance Guide. It is important that index file locations from multiple Enterprise Vault indexing services must never be located on the same storage device (LUN). This can create a significant bottleneck and slow all searches. Index locations The Enterprise Vault indexing service distributes its index files between the file locations specified to the service in a round-robin fashion. This can help to distribute the I/O load during discovery accelerator searches and provide better performance. When an index is created, it is not moved between the locations, so adding locations to an existing system may not affect any improvements. 42

Where it is known that discovery accelerator will be implemented, each file location must be located on a separate LUN to prevent contention between concurrent searches of different indexes. A minimum of five different index LUN locations must be used, ensuring each location is on a different LUN and assigned to a different drive letter or mount point. EV creates eight separate index locations on each of the five separate LUNs. The index files quickly become fragmented on disk (even if there is a large volume of free storage capacity). This file fragmentation can cause a severe performance problem that has to be managed on any index storage device. Either an automated background file defragmentation product or scheduled device defragmentation must be employed. HP M&C Archiving recommends using the built-in defragmentation feature that is included in the Windows operating system. Any scheduled defragmentation should be performed only after the indexing service is stopped to prevent the potential for corruption. Disk defragmentation is covered in detail in the Symantec Enterprise Vault Performance Guide. End-user training Discovery accelerator is a complex product and proper user training is essential. A poorly constructed search can inadvertently cause thousands of indexes to be searched for large volumes of data, perhaps unnecessarily. The following simple practices help improve the overall search performance. The HP M&C Archiving recommends that all the personnel who may configure or conduct searches with discovery accelerator attend the appropriate Symantec training course. The Symantec training website is available at symantec.com/business/training/index.jsp The following points must be considered when planning searches. Do not start a search without criteria. Otherwise, this gets every item from all indexes. Do not use wildcards unless necessary. Otherwise, this can severely impact performance. Make searches specific, and try to include author or recipient details. Specify date ranges. This reduces the number of searched indexes. Avoid overusing search terms. Thousands of terms can cause iterative searches. Select only the vaults relevant to the case in the case properties. Ensure that scheduled searches do not run at the same time as system backups. Quickly accept or reject searches to avoid filling and slowing the database. Test new searches with ad-hoc folders, and then delete the folders as necessary. Technology specifications This section describes the list of materials associated with a component. Keep in mind that the service being supported has specific hardware or software associated with the units specified in the HP Catalog, so substitution of alternate vendor products is not allowed unless approved in advance by the service line, delivery capability organization, and tech policy. List of hardware and software used This section describes the list of materials associated with a component, including a table with vendor part numbers and descriptions. The following highlight the specs of discovery accelerator server used in M&C Archiving. For specs of SQL Server used by discovery accelerator, refer to the Mailbox archiving functional component detail section. The discovery accelerator server directs the flow of data between Enterprise Vault and the discovery accelerator database, and manages the interaction with end users. End-user actions usually result in discovery accelerator database activity, but may also involve Enterprise Vault. The discovery accelerator server can potentially be very resource intensive within this offering; the discovery accelerator server must always be a separate dedicated server. Within this offering, a minimum of eight CPU cores are required. An HP M&C Archiving offers standard 16-core processor discovery accelerator server running on Windows 2008 R2 SP1 64-bit Standard Edition with 32 GB RAM typically supports up to 300 concurrent reviewers. Note Server CPU sizing must not be based on hyper threading. 43

Table 12. Discovery accelerator server specifications highlight Hardware Model Processor Memory Internal Storage Network HP ProLiant DL380 G8 Server 16-core processor 32 GB RAM 2 x 300 GB in RAID 1 for OS, page file and application files 2 x 100 GB SSD in RAID 1 for prefetch cache 4 x 300 GB 15k in RAID 10 for export locations Fully redundant Gigabit Ethernet NIC Fully redundant Gigabit backup LAN (Internal Embedded NIC) Software OS Application Windows 2008 R2 SP1 64-bit Standard Edition Discovery accelerator v10.0.2, Enterprise Vault v10.0.2 Disk considerations for discovery accelerator The discovery accelerator service that runs the customer tasks requires storage to be arranged for two different purposes: the prefetch cache and the export location. By default, the prefetch cache is limited to 1 GB disk capacity, but you may need to increase this to make the best use of it. The export location storage requirements vary depending on the total number of anticipated exports at any one time and the total size of all the original items for each export. Additional space is required for overheads during processing of Exchange message production to PST. The storage needs to be nearly double the total export size during processing. However, once complete, the additional space is released. A single export to native format generally produces 200 300 IOPS. However, exporting to Exchange PST incurs overheads which can produce between 900 and 1,600 IOPS. In addition, concurrent exports can increase this load considerably. Therefore, it may be worth providing multiple export locations to distribute the I/O load. This is obviously heavily dependent on the customer and their needs. The disk storage required for exports and prefetch cache must not become a bottleneck. NAS type storage should not be used. The best devices are local storage, direct attached storage, or partitions on appropriately sized SAN. In this offering, the standard is locally attached DAS disks. For the prefetch cache SSD disks are preferred. This is because of their high performance and for the pre-fetch cache, redundancy is not really an issue. It should, however, be remembered that SSDs are relatively new technology and do have a limited life. An appropriate alternative is to use high-speed SAN, however, there is the additional cost of redundant HBA cards. Discovery accelerator virtualization Within this offering the recommendation is to build discovery accelerator on a physical platform. This is in order to get the optimum performance. However, in smaller implementations or where DA is not intensively used, a virtual platform can be utilized. When using a virtual environment there are important aspects to consider. High speed SAN-based storage on separate dedicated LUNs should be used for the prefetch cache and export locations The LUNs should be mapped directly to dedicated VMFS disks which are presented to the discovery accelerator server Memory and CPU should be exclusively dedicated to the discovery accelerator server If you choose to use blade server, HP M&C Archiving recommends the HP ProLiant BL460c G8 Server series. For the blade server configuration, you can refer the above rack server configuration. 44

Prefetch cache Enabling the prefetch cache can greatly improve performance. The prefetch cache is used by the online preview and reviewing facilities, and the export and production features. Retrieving original items from the cache can perform significantly better than from the Enterprise Vault storage service, and it can prevent the storage service from being heavily utilized during peak times of activity. The recommendation for this offering is to routinely enable the prefetch cache options. Tuning discovery accelerator Discovery accelerator is a very complex product and there are many tuning options that need to be reviewed and modified in order to get the optimum performance. These are highly customer specific and you should always refer to the Discovery accelerator 10.0 Best Practices for Implementation Guide before making changes. In particular, the following items need to be reviewed and if necessary modified: Network sockets Retrieval threads Prefetch cache tuning Search tuning Enterprise Vault indexing search threads Export and production threads Full details can be found in the Symantec document Discovery accelerator 10.0 Best Practices for Implementation Guide. Storage functional component detail Description The storage functional component primarily focuses on the storage required for archive data, index files, and SQL Server databases. HP M&C Archiving supports the following types of storage only: Index files and SQL Server databases located on SAN or external DAS Archive data located on HP StoreAll Storage The design considers storage technologies to support the primary archive repositories and magnetic disk. Major selection criteria are: Operational reliability Access performance adequacy Regulatory guideline compliance where applicable Costs per unit archived Ability to refresh technology over generations Archiving media is an excellent candidate to apply the concepts of information lifecycle management (ILM) against retention periods that could span a hundred years as in the case of health records subject to Health Insurance Portability and Accountability Act (HIPAA) rules. ILM recognizes the need to balance cost against user patterns, data value, and speed of access over time. Where regulatory compliance is required, current guidance support installation of hard disk as the primary repository will achieve necessary access. The HP M&C Archiving solution is built on the HP StoreAll Storage system in both WORM-like or non-worm situations. HP StoreAll Storage The HP StoreAll Storage system provides a single, scalable archive platform to archive content from multiple information sources for longer term data retention. Built on a modular platform, the HP StoreAll Storage provides a modern, scale-out, pay-as-you-grow architecture allowing customers to scale their archives as needed. The WORM feature of the HP StoreAll Storage platform is integrated with Symantec Enterprise Vault application to provide granular retention on a per item basis for content archived from the Symantec Enterprise Vault application. 45

The data mobility features of HP StoreAll Storage such as automated data tiering, rebalancer, and migrator provide rich data management while helping to future-proof the infrastructure from hardware and technology changes. The end-to-end redundant architecture of the HP StoreAll Storage platform provides data high availability, and replication capabilities provide protection against disaster failures. The HP StoreAll Storage configuration necessary to integrate with the Enterprise Vault application will be explored in detail in further sections. For the Enterprise Vault application to archive content to the HP StoreAll Storage system, CIFS shares needs to be configured on either a WORM or non-worm file system on the HP StoreAll Storage. Note that the HP StoreAll Storage system fully supports WORM and non-worm file system within same appliance. The HP StoreAll Storage network configuration is created per the best practices outlined in HP StoreAll Storage best practices guide (see references section). The HP StoreAll Storage system presents the CIFS shares (details outlined in the following sections) to Enterprise Vault via a user or data network. In most environments, the HP StoreAll Storage is a dedicated device and is used exclusively for Enterprise Vault archiving. Leveraging HP StoreAll Storage devices or for that matter any other type of archive storage when used for Enterprise Vault is not recommended. When configuring the HP StoreAll Storage, the following prerequisites must be met before Enterprise Vault can be installed. The following are HP StoreAll Storage configuration task and are performed from the fusion management consol. Normally the HP StoreAll Storage devices must be introduced into the same Active Directory domain as the Enterprise Vault servers and this is the HP recommendation. In situations where this is not possible and the HP StoreAll Storage devices must be installed into a separate domain to the EV servers, an authority s two-way forest trust relationship must be setup. If a firewall exists between the two domains then the appropriate ports must be opened between the HP StoreAll Storage devices, the AD controller of the Enterprise Vault AD domain and the Enterprise Vault servers. These ports and network requirements are documented in detail in the HP StoreAll Storage installation guide. The Enterprise Vault service account must be setup as a local share administrator on the HP StoreAll Storage device. High availability within the HP StoreAll Storage must be configured as required. A method for backup of the HP StoreAll Storage data must be implemented. Either a traditional VTL or tape device or alternatively/additionally, it is possible to use replication to a secondary HP StoreAll Storage in an alternate location. Note that replication-only configurations require custom scripting. Such custom scripting requires a professional services engagement. The script must be developed and tailored to the customer s specific requirements and verified in the customer environment. If only using HP StoreAll Remote Replication to provide the safety copy for Enterprise Vault then remote cluster replication on the HP StoreAll Storage system must be configured as required. If required, WORM options must be set. These are configured at the file-system level within the HP StoreAll Storage device using the Fusion Manager GUI. If both WORM and non-worm options are required, then two separate file systems should be configured. One for WORM and one non-worm. Special notes on backup and replication When using WORM, the method EV uses to determine when a file has been made safe i.e. backed up, changes. Generally, in non-worm situations, Enterprise Vault uses the archive attribute (+A) of the archive files to determine when they have been backed up. All files are initially written with the archive attribute set on (+A). When the backup software backs up a file successfully, it sets the attribute to off (-A). Enterprise Vault uses the change to the archive bit as verification that the file is backed up. It then deletes the original item from Exchange and replaces it with a stub or shortcut. When files are written in WORM, no changes can be made to the file once it s created so neither backup or replication can change the archive attribute to off. This method of verification for a safe copy cannot therefore be used. Instead, Enterprise Vault uses a method called the trigger file. The trigger file method works by writing a small XML file PartitionSecureNotification.XML to the root of each Enterprise Vault partition just after the backup has completed. Any files that are older than the date recorded in this file are considered to have been backed up. In order for this method to work, there must be certainty that all files older than the date stamp are successfully backed up or replicated. When using a single non-worm HP StoreAll Storage and a tape backup solution, it is possible to use the archive attribute method to determine when a file is safe (backed up). Traditionally, backup software normally offers the ability to reset the archive attribute of a file upon successful backup. If using WORM configuration then the trigger file method of verification must always be used. This applies to both tape backup and replication. 46

Setting up CIF shares When using the HP StoreAll Storage, HP best practice is to create a separate CIF share for each Enterprise Vault server that will be running a storage service. The directory structure is then created under that share. The recommendation is to create the shares with a name that reflects the CNAME alias of the EV server they will attach to, for example for EVSERVER1 create \\StoreAll\EVSERVER1. The actual Enterprise Vault archive partitions are created on these shares and the associated directory structure for these should be created on the NAS device prior to installing the EV servers. We would recommend that typically that would take the form of a separate top-level directory for journaling and mailbox archiving and then within those, the rollover structure. Partition rollover best practice It is very difficult to define a single best practice for partition rollover, as all customers are different. In general, you should aim to keep it simple and keep partitions to a manageable number and size. How you configure your partition rollover depends on the individual customer s requirements, but there are some general simple recommendations you can follow, such as: Use a separate partition for every Enterprise Vault server running a storage service Keep it simple Keep partitions to between 200 GB and 400 GB Don t make partitions so big that you would never be able to recover them in the event of a disaster Don t make partitions so small that they become difficult to manage and are rolling over unnecessarily often Enterprise Vault integration Enterprise Vault fully integrates with the WORM features of HP StoreAll Storage. During creation of the vault store partitions, when HP StoreAll Storage is selected as the storage type, WORM or non-worm can be selected. Figure 13. HP StoreAll Storage WORM configuration settings Selecting the option Device Stores data in WORM mode allows content to be stored on the HP StoreAll Storage in immutable form with appropriate retention settings applied by Enterprise Vault. Additionally, the HP StoreAll Storage offers two forms of WORM storage, relaxed where the retention can be reduced or extended and strict mode where it can only be extended. For Enterprise Vault implementations, in most circumstances strict would be the configuration used. The WORM storage options are configured at the file system-level on the HP StoreAll Storage. 47

Architecture diagram (HP StoreAll Storage) WORM-like compliant Figure 14. Architecture of HP StoreAll Storage solution User OWA User outlook (#3) (#1) Existing Exchange environment (#2) (#2) (#1) (#1) Symantec Enterprise Vault Server Existing domain services HP StoreAll Storage secondary (#4) (#5) HP Connections Fiber Attach IP Attach HP StoreAll Storage primary (#6) MS SQL Server 2008 HP 3PAR Storage Remote management Table 13 list the type of connection between the components Table 13. Type connections Ref# on Diagram Component Type Connect (minimum) # Connections (#1) Archive server to Exchange Environment (via IP Switch) GigE 1 (#2) Archive server to HP StoreAll Storage GigE 1 (#3) Remote management 100 MB 1 (#4) SQL Server, remote access GigE 1 (#5) Archive server to HP SAN storage (HP 3PAR Storage) FC/2 Gb 2 (#6) SQL Server to HP SAN storage (HP 3PAR Storage) FC/2 Gb 2 Index files and SQL Server databases Index files and SQL Server databases are located on SAN storage or external DAS storage. Before using partitions on a SAN, the I/O load has to be considered along with any other applications already using the SAN to ensure that performance can be maintained. The implementation must be discussed with the SAN hardware vendor to ensure that optimum performance is achieved. Typically, LUNs must be created across many suitable disks as possible, using entire disks rather than partial disks to prevent multiple I/O-intensive applications from using the same disks. The same performance requirements apply. HP M&C Archiving recommend using RAID 10 configuration across multiple physical disk spindles to meet the high I/O demand. For SAN, the current standard solution is HP 3PAR Enterprise Virtual Array. Other high performance SAN solutions can also be used if they are available. Normally the SAN solution is a completely redundant N+1 hardware device with RAID storage. However, this does not prevent the storage from being rendered inoperative due to a physical environmental issue such as, a complete data center failure. High-availability storage options such as mirroring between data centers are not in scope of this release. 48

In this release, to provide a cost-effective additional measure of protection for the index data on critical systems, local internal hard drives can be used to maintain a second hot copy of indexes, if needed for compliance reasons or as SLA requirements. Vault store configuration For each component (mailbox and journal archiving), a separate dedicated vault store with its own partition and database is always required. The following general rules can be used for estimating the amount of storage needed for the actual vault store. They are based on Symantec experience in the field and represent typical customer values. Take the total size of items to be archived and halve it For email items, divide by the average number of recipients Add 5 KB multiplied by the total number of items Alternatively, 1 TB of Exchange Content =~ 615 GB EV archive storage Index configuration When implementing archiving, special consideration must be given to the configuration of the indexes. When running searches especially with discovery accelerator, the indexes are very heavily used and it is important that they are tuned to provide the best possible performance. The indexing service file locations require good random access or a high number of IOPS combined with good transfer rates. Therefore, the index must be hosted on dynamically expandable high-performance SAN partitions or DAS partition, and configured to ensure the LUNs are able to cope with the high number of IOPS. Although in non-discovery environments, index performance is not so critical, it is recommended to always build servers for high performance and in the HP M&C Archiving offering, indexes must always be designed with discovery in mind. For SAN, the storage team must be consulted on the best configuration options for the SAN. They normally consult the SAN hardware vendor on the implementation, who can give the correct configuration advice to ensure optimal performance is achieved. Typically LUNs must be created across as many suitable disks as possible, using entire disks rather than partial disks. This is to ensure the best possible performance for the indexes and must not be located on the same LUN or spindles as other I/O-intensive applications such as Microsoft Exchange Server or Microsoft SQL Server. Same rules apply when using external DAS storage. Sizing The following sections provides more detailed storage requirements specific to Enterprise Vault Mailbox Archiving. Enterprise Vault has three data sets that must be sized appropriately before beginning email archiving. These are Enterprise Vault Stores, Enterprise Vault Indexes, and Enterprise Vault Databases. Volume considerations When providing storage and volume estimations, the Symantec EV estimation tool must always be used. This tool must be filled in by the HP archiving engineer using input obtained from the customer. Vault Store design Vault Store sizing considerations The vault store is the logical storage unit for storing archived emails. A vault store consists of one or multiple vault store partitions. The partitions define the physical storage location. In this offering HP StoreAll Storage partitions are defined by connection to a NAS share or shares. Within the share, the NTFS directory structure is used to order partitions by year and month. Partitions are closed and rolled over on a month-by-month basis. Only the current month s partition is in the open state and only that partition requires daily backup. The Vault Store estimate is broken down into two parts: the mailbox backlog (amount in Exchange at this time that would be archived with the specified policy) and the steady state archiving (day-to-day archiving). To estimate the size of the vault store partition, you require the following information: Current number of active mailboxes to be archived Average number of messages to be archived from each mailbox 49

Current average size of messages Average number of workdays per user each year Typical single-instance ratio of messages older than the archiving threshold Total size of items to be archived now (backlog) and total count of items to be archived now The estimated annual growth rate of number of mailboxes The estimated annual growth rate of the average number of messages to archive per user The estimated annual growth rate of the average message size Many of these items can be gathered via utilities that gather Exchange metrics. The single instance ratio can be estimated in the absence of customer data. A value of 1.2 or 20 percent will be used. Data Store IOPS requirements The following table provides the IOPS calculations for a typical dual quad core archiving server archiving 4,000 mailboxes at a rate of 50,000 items per hour. These can be used to help determine the proper disk configuration for the Archive Vault Store. The table lists the IOPS for a typical EV server running the storage service. This can be a mailbox archiving server or a journaling server. Remember that each EV server running a storage service will require this level of IOPS. Each EV server has its own share on the NAS device. Table 14. IOPS calculation for a 16-core archiving server Size of data archive (Vault store data written to disk) 7,000,000 KB per hour 6,835 MB per hour 6.6 GB per hour Disk usage 450 IOPS Data profile Sequential writes Storage should be specified for twice this figure. So, the Vault Store must be able to sustain throughput of 900 IOPS per EV server running a storage service. Vault Store sizing detail When providing storage and volume estimations, the Symantec EV estimation tool must always be used. This tool must be filled in by the HP archiving engineer using input obtained from the customer. Enterprise Vault databases The Enterprise Vault SQL Server databases only contain metadata, directory, configuration, audit, and in the case of discovery accelerator, customer database, so the size remain limited. For each Vault Store, there is always an associated Vault Store database within SQL Server. All SQL Server databases are located on the SAN or external DAS Each SQL Server instance has a dedicated LUN for maintaining the SQL Server databases SQL Server backup is accomplished by using the standard SQL Server backup utility; this requires that SQL Server make periodic copies of the databases; as a result, the storage requirement for SQL Server databases must be double than calculated Where possible, SQL Server transaction logs are on high-speed RAID 1 SAN partitions, on discrete disk sets from those used for the Enterprise Vault databases, TempDB, and discovery accelerator customer databases. The HP ProLiant DL380 Gen8 Server has eight storage bays available. In the standard SQL Server configuration, four mirrored pairs (RAID 1) are installed and used for the OS, page file, and temporary databases. 50

Figure 15. Design of SQL Server Storage 146 GB 15k RAID 1 SQL Server 146 GB 15k RAID 1 SAN Database files OS Transcation Logs TempDB Logs CustomerDB Logs SQL Server sizing considerations Directory database The directory database has an initial storage requirement of 10 MB for data and 25 MB for transaction logs making a total of 35 MB. To allow for temporary growth and the transaction logs, at least 5 GB should be made available for the directory database. Vault Store databases For each vault store database, Enterprise Vault initially allocates the following amount of permanent disk space: 100 MB for the data files 80 MB for the transaction log files A basic sizing guide for each vault store database is 250 bytes for each archived item, plus 5 GB for static data, transaction logs, and temporary data fluctuations. So for each vault store database, use the following formula to estimate the size. Number of messages * 250 bytes + 5 GB Fingerprint database The fingerprint database has an initial storage requirement of 244 MB, made up as follows: 132 MB for the primary file group 1 MB for each of the 32 non-primary file groups 80 MB for the transaction log device The non-primary file groups hold the SIS part fingerprints and other information about the SIS parts. If you share items using Enterprise Vault Single Instance storage, the non-primary file groups may grow very rapidly in size. Ensure that there is adequate space for the non-primary file groups to grow as data is added. The New Vault Store Group wizard provides the following options for the initial configuration of the fingerprint database: A default basic configuration, where Enterprise Vault locates the primary file group and all the non-primary file groups on one device. An advanced configuration option, where you can specify separate locations for the 32 non-primary SQL file groups. To ensure acceptable archiving and retrieval performance, it is important to configure the fingerprint database appropriately for the amount of sharing in the vault store group. For optimal performance, do as follows: Use the advanced configuration option to specify as many locations as possible on the SQL Server, maximum of 32. Use a separate device for each location. If you specify more than one location on the same device there is no performance benefit. Note To add or change locations after the fingerprint database is configured is a SQL Server administration task. 51

Limit transaction logs to an appropriate size for your backup and maintenance plan. The following are details of the Vault Store database stores: All sharable item content hash values All sharable item shared link references Typically, all files or mail attachments larger than 20 KB are identified as sharable and a document identity hash value added to the fingerprint database. For message-based archiving this normally represents around 20 percent of the messages. File-based archiving of office type documents is likely to see a similar proportion, but non-office files may vary considerably depending upon the data source. Use the following rule of thumb to size the fingerprint database: Capacity per year (MB) = (pm/r) * (0.5/1024) Where M = Total items archives per year in all vault stores in sharing group R = Expected sharing ratio (assume 2 if unknown) P = Percentage of archived items eligible for sharing (assume 0.2 if unknown) HP M&C Archiving recommendation is to create the 32 non-primary file groups at installation on the same SQL data partition as the other databases. These groups can then be moved to other dedicated LUNs as and when performance warrants it. Note The DVSSP refers to the number of separate parts to the mail such as body and attachments. A general average figure is two. As a rule of thumb, you can also calculate for SQL Server sizing as approximately 1 percent from store size of the Vault Store. For more information on sizing guidelines for discovery accelerator customer databases, see the E-Discovery Functional Component Details section. The sizing value for the databases that is calculated must be doubled to account for temporary backup copies. TempDB database The TempDB database is a shared resource used to store user objects (user-defined objects, temporary tables, and indexes, table variables, tables returned in functions, and system tables), internal objects (work tables for cursors, spool operations, LOB storage, hash join storage, hash aggregate storage, and intermediate sort results), and version stores (row versions from updates). The TempDB database pages quickly move between memory and disk, which become a bottleneck if the disks are inappropriate and the configuration is not tuned. To prevent TempDB file growth causing unnecessary I/O and to ensure that the TempDB files do not become severely fragmented, set the minimum file sizes to 200 MB and grow by 10 percent. Create one data file per CPU (to spread the load and reduce contention), making each file the same size. Enterprise Vault databases Table 15 describes the IOPS calculation for the EV SQL Server databases to determine the proper disk configuration: Table 15. Disk configuration for EV SQL Server database Component Estimated typical sustained IOPS Data Logs Directory 10 to 100 10 to 100 Vault Store DB 300 150 Fingerprint 25 20 52

Discovery accelerator customer databases For optimum performance it is best to size the discovery accelerator customer database so it is not constantly growing. During the creation of the customer database, increase the size to 4 GB and monitor growth rates. Plan for three years of growth and allocate the space to the database ahead of time to avoid automatic growing by SQL Server, which causes performance impacts. It is a best practice for the SQL Server transaction logs to be truncated each day as part of the SQL Server backup. In the event that the SQL Server transaction log backup fails, you must plan for three days of growth. An additional best practice is to perform weekly maintenance that includes rebuilding indexes, updating statistics, and shrinking databases. Table 16 describes the IOPS calculation for the DA SQL Server databases to determine the proper disk configuration: Table 16. IOPS calculation for DA SQL Server databases Component Estimated typical sustained IOPS Data Logs Custodian Manager 100 to 300 150 Customer 250 to 1800 500 to 1200 SQL Server sizing detail To calculate the necessary storage for the vault store databases, use the following formula: Storage required for SQL Server database for Year N (GB) = Total number of messages to archive for Year N * 250 bytes/1024/1024/1024 Note 250 bytes is the estimated size of data added to the Vault Store database for each item archived. It is also important to calculate the size required for the daily SQL Server transaction logs. The formula for this calculation is: Daily SQL Server transaction log requirement (MB) = Number of messages to archive per day * 4 KB/1024 Note 4 KB is an estimate of the size of each transaction for an archived item. Using the data from the example in the Vault Store Sizing Detail section, the year one SQL Server requirements (including the backlog messages = 1,500,000) are: Storage required for SQL Server database (GB) = ([1,500,000 + 6,006,000] * 250 bytes)/1024/1024/1024 = 1.25 GB Daily SQL Server Transaction Log requirement (MB) = (6,006,000/260) * 4 KB/1024 = 90 MB Fingerprint Database = (Number of messages to archive per day * 0.2/2) * (0.5/1024) Directory Database = 5 GB Index storage design Index sizing considerations To enable a user to effectively search Enterprise Vault for archived messages, two levels of indexing are supported. These two levels include basic level, which indexes based on message attributes, and full which indexes words based on their proximity in both message attributes, body text, and any attachments to facilitate the full phrase searching and mining of archive messages. Basic: ~4 percent of equivalent Exchange storage being required Full: ~13 percent of equivalent Exchange storage being required 53

In M&C Archiving, we must be able to search item content for phrases because of using discovery accelerator, so we need to use full indexing. To determine the disk configuration the following calculation can be used: Table 17 describes an EV server archiving 4,000 average user mailboxes at a rate of 50,000 items per hour. Table 17. Calculating index disk configuration Component Estimated typical sustained IOPS Index server 500 to 1000 With heavy DA use Up to 1500 Note The storage should be specified for twice the maximum anticipated load so you should anticipate IOPS of at least 2000 with no DA and 3000 with DA. Index sizing detail The index sizing estimate is based on the total original size of the messages archived and the index overhead factor (13 percent for full indexing). The formula for this calculation is: Storage required for indexes for Year 1 = (Total original size of messages in backlog + total original size of messages archived for year 1) * index overhead Using the numbers again from the example in the Vault Store sizing detail section, the index size estimates for year one (including the backlog) are: Storage required for Indexes for Year 1 = (75+441) * 13 percent = 67.08 GB Enterprise Vault storage calculator Symantec provides a comprehensive sizing tool that should always be used for estimations. This tool must be filled in by the HP archiving engineer using input obtained from the customer. Discovery accelerator storage requirements The following section provides guidance on storage requirements for both discovery accelerator server and the associated SQL Server Database server instance. Discovery accelerator storage The prefetch cache and export location require good random access or high number of IOPS, so a striped array of many disks must be used (using hardware-based RAID and not software). Due to the size and volume of data involved with these activities, the RAID also requires good transfer rates and high-speed disks must be used. These two sets of files have to be located on different arrays to ensure the export throughput and prefetch cache benefits can be maintained. If the customer performs multiple exports at the same time, it is recommended to allocate multiple export locations. Before using partitions on a SAN, the I/O load needs to be considered along with any other applications already using the SAN. This is to ensure that performance can be maintained. The implementation must be discussed with the SAN hardware vendor to ensure that optimum performance is achieved. Typically, LUNs must be created across as many suitable disks as possible, using entire disks rather than partial disks to prevent multiple I/O-intensive applications from using the same disks. Same rules apply when using external DAS storage. 54

A discovery accelerator server should have the partitions described in Table 18. Table 18. Partitions in an accelerator server Partition Configuration Size Comment System drive DAS RAID 1 >70 MB Prefetch cache DAS RAID 1 dedicated spindles on local drives Export location SAN or DAS RAID 5 dedicated spindles on local drives 146 GB Prefetch cache is used to help with review performance >250 GB Export location is generally of large size because the customer is exporting data using discovery accelerator into PST files, to provide to other parties. The customer must be asked how much data they expect to export in one year. If value is unknown, use 100 GB as a starting point. Note The export locations must be excluded from antivirus scanning due to potential file locking issues that prevents completing the export. Discovery accelerator database storage The storage capacity of the discovery accelerator database server must be carefully considered using the appropriate database sizing guide. However, in initial discussions, the following high-level rule of thumb could be used: Capacity per year (GB) = ([S * 2.7] + [E * 2.2])/1024/1024 Where: S = Total number of items captured in all searches performed in all cases per year E = Total number of items exported in all cases per year If the customer cannot supply the inputs, use 100 GB for data files and 50 GB for log files as a minimum starting point. These SAN-based volumes can be increased later. The configuration of SQL Server SAN is critical to the performance of discovery accelerator. The database server storage must be arranged to accommodate the different types of data. Typically, database servers instance must have the following partitions: If there are two instances on a SQL Server, each instance must have its own dedicated LUNs. Table 19. Partitions for database servers Partition Configuration Size Comment System Drive DAS RAID 1 >70 MB Local drives TempDB log partition DAS RAID 1 10k or above >70 GB Local drives TempDB data partition SAN or DAS RAID 10 10k or above >100 GB Separate LUN These partitions must only contain the related database files so that those files do not become fragmented on disk. In large environments with multiple customer data files, you must create multiple customer data files on separate striped disk arrays (RAID 10) and moving tables or indices into separate file groups, which can improve query performance in large databases. Again, this is a decision that must be made by the SQL Server support team based on their standards and procedures. Implementing a discovery accelerator database maintenance plan is absolutely essential to preserve the database performance. Rolling over databases may be required to remain within the available storage capacity. The database storage requirements are high. To reduce storage costs, the database can be periodically rolled over to a new database and the old database archived. Alternatively, the old customer database must be moved to slower storage and the associated discovery accelerator customer tasks disabled. This allows the customer database to be brought online quickly. 55

Backup and recovery functional component detail Description This functional component describes backup retention, backup methodology requirements, and recovery procedures to ensure Enterprise Vault maintained email archive data access is always fully available. WORM storage HP StoreAll Storage supports the WORM file system to retain the data and it is policy based. This helps the customer to comply with their data retention policy by using the WORM-enabled HP StoreAll Storage file system. Non-WORM storage Where WORM storage is not a requirement and HP StoreAll Storage devices are deployed, then the archive data is still subject to the normal requirements for backup. Enterprise Vault requires that archived data be verified as backed up before shortcutting can occur. The archive data must, therefore, be backed up once a day or replicated to a secondary HP StoreAll Storage device (requires custom scripting). It is obviously not practical to back up the entire archive every day, so Enterprise Vault allows you to setup partition rollover. This ensures that only the active partition requires daily backup, closed partitions can then be backed up less frequently. The recommendation is to do partition rollover on a monthly basis. Once a partition is closed, very little changes. The main changes are periodic deletions. Because very little changes and no new data are added, it is quite safe to backup closed partitions on a reduced schedule such as once a month. The exact details of the backup schedule must be tailored to the needs of the individual customer. The active partition must be backed up daily. A full backup of the active partition should be done once a week and incremental backups done at least once a day. When running an Enterprise Vault backup, a script must be run to put the indexes and archives in a read-only state. A sample script is provided in the implementation guide and Enterprise Vault provides a CScript tool to automatically generate a backup script for your environment. It is essential that the backup script is used prior to and after a backup. The act of putting the archives back into read or write mode is what initiates the shortcutting process. Technology specifications To ensure a backup of Enterprise Vault, administrators must backup the original deployment install directories and file sets, as well as other critical components, such as MSMQ, IIS, SQL Server, and Exchange. To backup and restore Enterprise Vault server and data, any backup tool (for example, HP Data Protector) that has the possibility to backup open files can be used. It is set up with multiple jobs in a cyclical schedule to back up parts of the server and data. Microsoft SQL Server 2008 R2 comes with the built-in ability to do an SQL Server online backup of its databases and save a copy of them as flat files to disk (for more information, see msdn.microsoft.com/en-us/library/ms175477(v=sql.105).aspx). A backup solution backs up the copy. A special online database backup utility or module is only required if there is not enough free disk space to use the SQL Server copy option. In this release of M&C Archiving, only the built-in SQL Server backup technology is in scope. Requirements and assumptions The following requirements are needed for the backup and recovery solution: The daily incremental backups for Enterprise Vault (index) data must fit within a maximum of 4-hour backup window to ensure that there is enough time for the archiving process. The daily incremental backups of the active archive partition (in HP StoreAll Storage deployments) must fit within the 4-hour backup window. The daily full backups of the SQL Server databases must also fit within the 4-hour backup window. The weekly full backup for Enterprise Vault (index) data must be finished on one weekend, during a maintenance window. The monthly full backups of the closed archive partitions (in HP StoreAll Storage deployments) must be configured such that it fits within a weekend maintenance window. In large deployments, it may be necessary to stagger these closed partition backups to accommodate large amounts of data. Dedicated Gigabit Network card must be attached to dedicated Gigabit Ethernet backup network. Backup locations and network transmission must meet HP security guidelines (encrypted where necessary). 56

In a customer site deployed solution, the customer is responsible for providing hardware or software backup similar to that provided by HP Backup and Restore Services. Backups are estimated using maximum size and network availability. If a customer exceeds maximum size or network experiences latency, required times cannot be met. This is not expected to occur in normal use. The HP M&C Archiving backup and recovery solution assumes the following: A Gigabit Ethernet NIC cards are included on all Enterprise Vault servers. It is attached to a dedicated backup network. There is no other traffic on the Gigabit Ethernet backup network other than the backup traffic during the Enterprise Server backup. The backup strategies and technology may change to allow faster backups in the future. Required actions for backing up Enterprise Vault application If NetBackup is the backup solution in use, then its agent for Enterprise Vault should be used. NetBackup agent for Enterprise Vault interfaces with EV and automates the read-only state. Due to the inability of the Enterprise Vault index s structure to remain in their normal production (read or write) state during backup operations, it is required that a pre-backup script is run to place all the EV indexes and archive data in a read-only state. If end users have manual archive functionality, and when they run it while the EV Services are in read only mode, the item is placed onto an MSMQ queue and are processed once the EV services are placed back into normal mode. After successful backup, the EV indexes and storage can be placed back into read or write state. To perform these steps, use the appropriate scripts. We recommend using the provided Symantec PowerShell script to generate the backup script. Details of the script and how to use it are found here symantec.com/business/support/index?page=content&id=tech66742. For more details on Enterprise Vault backup, you can also refer the Enterprise Vault 10.0.2 Administrator's Guide at symantec.com/business/support/index?page=content&id=doc5980. Additional index backup In situations where the index data is critical (for example, journal indexes where frequent discovery searches are taking place), it is recommended to maintain a copy of the indexes on local cheap SATA disks. Using the four available drive bays and 1 TB SATA disks, it is possible to provide up to 3 TB for index copy if using RAID 5. The copy must be made with Robocopy and a schedule task. This basic copy of the indexes is in addition to the normal tape backup and provides a simple means of doing a fast restore of a corrupted index. Data type backup usage and retention rules Warning! 1. The vault indexes backup retention period must either be equal or be older than backups of the related EV SQL Server databases. 2. It is critical to have the most up-to-date backup of SQL Server databases. Or, permanent data loss can occur. 3. SQL Server database backups can only be used if they contain all the archive data storage location entries. 4. Using a backup source that does not contain all these entries results in a permanent irretrievable loss of data within the archive. All data that was stored in the archive after the SQL Server database backup was generated is the data that is permanently lost. Restoring aged backups results in permanent archive data loss. To ensure that this does not occur, use only backups that result in restoring all archive storage location entries. Maximum backup retention period Due to above defined functional restrictions on index and SQL Server database backup usability, retention of these backup can be for a maximum of one backup cycle. In this case, a backup cycle is the time between creation of full (complete) backups. When a full backup is made, continuous incremental or complete differential backups must be retained until a new full backup is created. As soon as the new full backup is created, all previous backup sets are no longer useable, since if they were selected for a restore effort, key SQL Server metadata databases entries are not included; these missing entries result in permanent loss of a portion of previously archived data. 57

SQL Server database backup As with any SQL Server database, it is important to have a daily backup plan in place and to monitor the amount of space allocated to the databases. Backups should be done in full recovery mode, with incremental log backups. Enterprise Vault SQL Server databases to be backed up: Enterprise Vault Directory database Enterprise Vault Store databases Enterprise Vault Fingerprint database Monitoring database Auditing database (when used) Discovery accelerator configuration database (when used) Discovery accelerator Custodian Manager database (when used) Discovery accelerator customer databases (when used) The following backup schedules are recommended for SQL Server databases and transaction logs: Back up the Directory database nightly. Additionally, back up the directory database after any configuration changes, to ensure that all services function correctly after recovery. In particular, back up after a new Enterprise Vault service or task is added. Perform a full backup of all vault store databases daily after the main run of the Exchange Mailbox task using a SQL Server maintenance plan. The daily full backup of SQL Server must be coordinated to occur during the time period when the EV services are in read only mode. Backup monitoring database nightly. Backup the transaction logs from journal archiving vault store databases at least every hour to ensure that the SQL Server database backups contain all the archive data storage location entries (For more information, see Data type backup usage and retention rules section). Backup all system databases, especially Master and MSDB daily. Truncate the transaction logs after all the backups. Enterprise Vault SQL Server databases must participate in regular maintenance plans, including re-indexing. For more information on best practices for SQL Server backup, check: HP white paper at download.microsoft.com/download/7/b/1/7b17fc16-444f-4ccd-af90- edb378da8831/hp_whitepaper_sqll.pdf Microsoft guide at msdn.microsoft.com/en-us/library/ms191455(v=sql.105).aspx Data protection backup To safeguard the operational viability of the email archive services, minimally the following solution elements require redundancy or backup. Backup is accomplished employing standard HP data protection services: Operating system on archiving and SQL Servers Perform a full system and file backup, including system state and registry Archiving application software on archiving server SQL Server application software on the SQL Server (For more information on SQL Server databases and transaction logs backup, see SQL Server Database Backup section) Index service file locations Shopping service files Discovery accelerator export directories 58

The following backup schedules are recommended: Backup operating system on Archiving and SQL Servers, archiving application software on archiving server, and SQL Server application software on the SQL Server at least weekly. Backup index service and shopping service file locations daily incremental after the main run of the Exchange Mailbox task and run a full backup on every weekend. All backups must be so timed as to minimize customer impact even though the backup activities can be run without interrupting users access to archived data (read-only access is still possible). Operating systems Standard centralized backup utilities such as HP Backup and Recovery (HP BuR), which is already deployed is used to protect the archive and SQL Server operating systems. Archiving application software Standard centralized backup utilities such as HP BuR, which is already deployed is used to protect the archive server Enterprise Vault application software. SQL Server application software Standard centralized backup utilities, such as HP BuR, which is already deployed is used to protect the MS SQL Server 2008 R2 Standard application software. SQL Server databases Data protection for the Microsoft SQL Server 2008 R2 databases are managed by the HP SQL Server Database Support Group. Their recommended methodology and processes, including the rules from data type backup usage and retention rules section are followed. Indexing and indices Indices are created and maintained to support the search capability within Enterprise Vault. Although these index files can be recreated, it is time consuming, so backups are recommended. It is important to understand that the archive SQL Server databases and the search indices work together and require point-in-time synchronization in the case of backups or restoration. HP StoreAll archived data Standard centralized backup utilities such as HP BuR, which is already deployed is used to protect the archive server and Enterprise Vault application software. Enterprise Vault recovery Restoring an Enterprise Vault environment from backup requires multiple considerations. Recovery scenarios There are several different recovery scenarios depending on which data is lost: User loses a single shortcut Use native deleted item retention in Exchange within a 14-day period. User loses single archived items (only if user can delete items from vault) The Enterprise Vault administrator can recover the deleted item within a 14-day period (for this the dumpster feature in EV has to be enabled and 14 days is the recommended period. Using the Recovered Deleted Items option restores all archived items that have been deleted within the configured time. For more information about this feature, check Recovering deleted items section of the Symantec Enterprise Vault Administrator's Guide. Index data lost or corrupt for one user Recreate the index for the single user by using the standard procedure from Symantec: symantec.com/business/support/index?page=content&id=tech54272. Index data lost for all users Restore from backup Archive data lost Switch to replica storage, or Restore from backup 59

Loss of a single server Depending on the nature of failure and the role that the server performed, it may be possible to utilize the Enterprise Vault Update Service Location (USL) feature to migrate workload to a surviving server. For more information, see Recovery of Enterprise Vault using backups section. Replace the failed hardware component. Restore the OS and all local volumes from backup and verify that the original SAN LUNs are present with their original drive letters. In case when full restore does not bring back the Enterprise Vault server, you may run the Enterprise Vault Configuration wizard to recover the Enterprise Vault services and tasks. For more information, see Recovery of Enterprise Vault using backups section. Loss of all data (except archive data on HP StoreAll Storage) or parts of it Restore all from backup Replace the failed hardware component. Restore the OS and all local volumes from backup and verify that the original SAN LUNs are present with their original drive letters. (Even if server fails) In case when full restore does not bring back the Enterprise Vault server, you may run Enterprise Vault Configuration Wizard to recover the EV services and tasks. For more information, see Environment recovery procedure section. Recovery of Enterprise Vault using backups This recovery procedure assumes that all databases and archived data are still available. To recover Enterprise Vault using existing backups: 1. Restore the backup of the system. 2. If EV services are missing from the service control panel, then use the EV configuration wizard to reconstruct the service information. If the computer is an EV directory service computer, the EV configuration wizard automatically reconstructs the service information. Environment recovery procedure If a disaster occurs in which all or parts of the data are lost except archived data, follow these steps to recover an Enterprise Vault environment: 1. Restore file system backups 2. Restore SQL Server backups (last full backup when all transaction log backups until point of failure) 3. Restore indexing files to their original locations 4. Restore shopping files to their original locations 5. Repeat operations Due to changes made when the last set of backups was done and because certain operations may not have completed before the system failure occurred, the following steps have to be done: A. Repeat archive operations done since the last set of daily backups were made B. Cancel all archive pending items from mailboxes (this can be done for all users by changing a setting in the archiving policy and running the mailbox archiving task in report mode) 6. Repeat all retrieval requests made but not completed because of the system failure For more information on the requirements to properly restore an Enterprise Vault environment, see Symantec Enterprise Vault 10.0 Administrator s Guide at symantec.com/business/support/index?page=content&id=doc5980. Monitoring functional component detail Description This release of HP M&C Archiving supports two monitoring tools: HPOM and Microsoft System Center Operations Manager (SCOM). The HPOM-based monitoring solution is provided by the Enterprise Service Management (ESM) Event Automation Solution Engineering (EASE). SCOM-based monitoring solution is provided by the Workplace Management Framework (WMF). HP M&C Archiving leverages the infrastructure and services from to provide the monitoring and alerting functions. In an HP M&C Archiving solution, HPOM is the default monitoring tool, while SCOM can be used as an alternative monitoring tool when a customer has existing Enterprise Customer Access Licenses (CALs) from Microsoft and chooses to use SCOM for monitoring. 60

In this release of M&C Archiving, the following applications are monitored using HPOM and SCOM: Enterprise Vault 10.0.2 Discovery accelerator 10.0.2 SQL 2008 R2 Exchange Server 2010 SP2 Forefront Protection 2010 for Exchange Technology specifications In the event management architecture, either HPOM or SCOM can be used as monitoring tool that directly monitors and collects event data from the applications such as Exchange, and then feeds the events that meet the predefined criteria to upstream event management components such as HPOM and the event management systems for ticketing and alerting functions. Monitoring with HPOM This section describes the HPOM infrastructure and the standard monitoring solution for the above mentioned applications. HPOM infrastructure The HPOM infrastructure is owned and supported by the Enterprise Instrumentation and Automation Tools (EIAT) capability. HP M&C Archiving subscribes to the service from EIAT capability for monitoring the customer s messaging systems. The following diagram depicts a HPOM monitoring setup where a dedicated HPOM server is located within the same compartment as the HP M&C Archiving servers, and is used to forward the monitored events collected by the HPOM agent to the upstream HPOM server. There may be other HPOM setup where the HPOM agent forwards the monitored events directly to the upstream HPOM server. Figure 16. Monitoring with dedicated HPOM server HP operational support location HP data center or client data center Customer A HPOMw HPOMu HPOM agent HPOM agent HPOM agent Managed Nodes: EV, DA, SQL, and others Discovery accelerator monitoring Discovery accelerator monitoring leverages the same monitoring package used for monitoring Enterprise Vault. Exchange Monitoring FPE Monitoring Monitoring with SCOM This section describes the SCOM infrastructure and the standard monitoring solution for the above mentioned applications. SCOM infrastructure When using SCOM for monitoring, there is a SCOM Operations Console provided to the messaging delivery teams. This operations console provides a rich interface that allows the operations team to interact with the monitored events and alerts. 61

The following diagram depicts a SCOM monitoring setup where a dedicated SCOM server is located within the same compartment as the HP M&C Archiving servers, and is used to forward the monitored events collected by the SCOM agent to the upstream HPOM server. Figure 17. EV monitoring with dedicated SCOM server HP operational support location HP data center or client data center Customer A SCOM Operations console HPOMu SCOM agent SCOM agent SCOM agent Managed Nodes: EV, DA, SQL, and others Discovery accelerator monitoring Discovery accelerator monitoring leverages the same monitoring package used for monitoring Enterprise Vault. Enterprise Vault Operations Manager Enterprise Vault Operations Manager (EVOM) is a Web application that makes remote monitoring of Enterprise Vault possible from any computer with Internet Explorer. EVOM monitors the status of EV services, tasks, and performance counters at predefined intervals and populates this information into a database, for troubleshooting or monitoring the health of the system. Note EVOM is a fairly basic tool and should not be relied upon to provide complete details of the systems health. It just gives a basic overview of the server health and running services. It does not necessarily tell you all is well and does not tell you everything you need to know to ensure the system is operating correctly. EVOM lets administrators monitor the following: The status of Enterprise Vault services. The status of Enterprise Vault archiving tasks. Performance counters for vault stores, disk, memory, and processors. Exchange Server journal mailbox target archiving parameters, including message counts for inbox, archive pending, and failed operations such as Failed DL Expansion. A monitoring agent on each Enterprise Vault server collects data at scheduled intervals, by default every 15 minutes. This data is then stored in the Enterprise Vault monitoring database. The EVOM webpages display the data from when the system was last monitored. Summary pages provide at-a-glance status assessment, while detailed data can help identify problems and bottlenecks. The following components are monitored by EVOM. Enterprise Vault Services (Running or Failed Status Displayed): Enterprise Vault Indexing Service Enterprise Vault Shopping Service Enterprise Vault Storage Service Enterprise Vault Task Controller Service 62

Enterprise Vault tasks (running or failed status displayed): Enterprise Vault Provisioning task Enterprise Vault Mailbox Archiving task Enterprise Vault journal archiving task Table 20. Performance counters Group Processor Memory Logical disk EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores EnterpriseVault :: Vault Stores Counter name % Processor time Available MB % free space Journal archive table size Journal delete table size Journal restore table size Vault store DB backup Vault store DB log % used Vault store DB log backup Vault store DB log size Watch file table size Exchange journal mailbox parameters: Inbox count Archivable Archive pending Failed to copy (Number of items in the failed to copy folder in the journal mailbox. These are normally corrupted messages. You can drag the message onto the desktop and try to open it. If you can, drag it back into the journal mailbox Inbox and let the journal archive task process it. If it cannot be opened, you can leave in the failed to copy folder or delete it.) Failed to store (Number of items in the sailed to store folder in the journal mailbox. These are messages that failed due to a problem with the Enterprise Vault storage service. Drag the items from the failed to store folder to the journal inbox and the journal archive task may try to reprocess.) Failed DL expansion (Number of items in the failed DL expansion folder due to inability to expand one or more distribution lists used in the message.) Failed external filter (Number of items in the failed external filter folder. These are messages that could not be processed by a custom filter.) The EVOM database content is refreshed every 15 minutes (or at the scheduled monitoring interval defined by the administrator). So, the database size is not a concern. Figure 18 depicts the EVOM components and their relationships. Figure 18. EVOM components Monitoring Agent Enterprise Vault Operations Manager Enterprise Vault Server Enterprise Vault Monitoring Database Remote Monitoring Workstation As stated earlier, the EVOM is a complimentary tool that allows administrators to have a quick glance at the system health and status. SCOM is the primary monitoring and alerting tool. All monitoring functionalities provided by the EVOM are also available via SCOM and the management packs. 63

For more details on EVOM, refer to Enterprise Vault 10.0.2 Administrator s Guide at symantec.com/business/support/index?page=content&id=doc5980. Message queue monitoring Enterprise Vault uses Microsoft Message Queue (MSMQ) Server to transfer information between Enterprise Vault components. It is important that MSMQ queues are monitored so that administrators can quickly spot any problems that can occur. Windows Performance Monitor can be used to monitor the performance of the queues. Administrators can find it useful to have the Windows Performance Monitor running continuously, showing the number of messages on all the queues. This way, administrators will quickly become used to the normal behavior of the queues and will notice excessive backlogs. Investigate the cause of any such backlogs promptly. List of hardware and software used Monitoring for HP M&C Archiving is leveraged from the HP capabilities, and as such, there is no need for a separate list of hardware and software for these components in the HP M&C Archiving package. Reporting functional component detail Description HP M&C Archiving Release 4.0 uses the Enterprise Vault reporting feature to provide reporting functions to the email archiving systems. The Enterprise Vault reporting feature uses Microsoft SQL Server Reporting Services as the reporting mechanism. Administrators manage report content and view reports using the SQL Server Reporting Services Report Manager Web application. The Enterprise Vault reporting feature generates some of its Enterprise Vault server reports using data obtained from the Enterprise Vault Directory, Vault Store, and auditing and monitoring databases. Figure 19 depicts the Enterprise Vault reporting components and their relationships. Figure 19. Enterprise Vault reporting components Monitoring agent Enterprise Vault servers Directory database Vault Store database Fingerprint database Monitoring database Enterprise Vault SQL databases Audit database Enterprise Vault Reporting (SSRS) Administrator s workstation Enterprise Vault reporting This release of HP M&C Archiving supports reports that are available from the Enterprise Vault reporting feature, including: Enterprise Vault Operation Reports Archive Quota Usage Archived Items Access 64

Archived Items Access Trends Enterprise Vault Server 24-Hour Health Status Enterprise Vault Server Seven-Day Health Status Exchange Server Journal Mailbox Archiving Health Exchange Server Journal Mailbox Archiving Trends Items Archived Per Hour Mailbox Archiving Status Single Instance Storage Reduction By File Type Single Instance Storage Reduction Per Vault Store Group Single Instance Storage Reduction Summary Vault Store Save sets Vault Store Usage By Archive Vault Store Usage By Billing Account Vault Store Usage Summary Discovery accelerator reports Archive Source Case History Holds Marking History (Item Detail) Production Run Productions Search Criteria Searches Security Subreport Archive Source Discovery accelerator auditing Audited Actions Search Level Review Level Production Level Chain of Custody Production Reports Administrators can do the following: Customize report content using the parameters provided by the report Choose from a number of report export formats, including pdf, xls, HTML, and tiff Schedule reports to be emailed to a configured email address or saved to a shared folder There are several prerequisites for Enterprise Vault reporting. As this feature is dependent on SQL Server Reporting Service (SSRS), the main prerequisite for this feature is SQL Server SSRS must be installed and successfully configured where EV reporting has to be configured. Enterprise Vault reporting has to be installed locally on the SSRS server. SSRS is installed on an Enterprise Vault server. For more details on Enterprise Vault reporting, refer to the Enterprise Vault 10.0 Administrator s Guide at symantec.com/business/support/index?page=content&id=doc4402. 65

Event and diagnostic logging Administrators can also use Enterprise Vault event logs to analyze and diagnose issues. Enterprise Vault uses the following logs: Windows Application Event Log This is used for events that are deemed to be critical. Service start-up and shutdown events are logged here and events arising from the integrated monitoring that is on site properties. These include, for example, warnings if databases have not been backed up recently or if there is a backlog of items in journal mailboxes with a status of archive pending. Enterprise Vault Log This is used for all events that are not deemed to be critical. For example, events relating to the progress of mailbox or public folder archiving. Additionally, events that are placed in the Windows Application Event Log are also placed in the Enterprise Vault log thus ensuring that the Enterprise Vault log contains a complete record of all events. Enterprise Vault Convertors Log This contains events arising from document conversions. For each of the Enterprise Vault services, the level of diagnostics reporting can be selected. The diagnostic reports are logged in the Enterprise Vault Log. You can use the Windows Event Viewer to view these logs. Additionally, you may have your own or various third-party tools that you can use to monitor the Enterprise Vault log entries. Enterprise Vault auditing Enterprise Vault includes flexible auditing that you can enable for individual Enterprise Vault servers. The auditing events are written to a SQL Server database there is a single auditing database per Enterprise Vault site and it must be located on the same SQL Server instance as the Enterprise Vault Directory database. For example, the audit events record the following: The time an event occurred The account that initiated the event The archive in which an item was archived The category of the event, such as view, archive, or delete Auditing can be enabled for a number of different types of event, showing for example, and details of the following: Actions taken using the administration console Searches Viewing an item Deletions For most types of event, detail levels of summary or details can be specified, or both: Summary gives information about the event, such as the date and time, account used, and the vault used. Details list more information, such as extracts from the content of a message, for example subject, mailbox owner, and folder. Note There will be a slight reduction in performance when auditing is enabled. By default, auditing is disabled. Enable auditing when it is required to meet customer requirements. As the auditing database grows, it becomes necessary to either remove data or create a new database. Depending on the requirements for the data (keeping auditing information for a certain time or monitoring the audit events), the following note provides two methods to remove data or create another auditing database. symantec.com/business/support/index?page=content&id=tech35746 66

Networking functional component detail Description This section describes the network topology of application servers, tools, and customers within the HP M&C Archiving solution, and their mode of communication. Network usage In both mailbox archiving and journal archiving, the network is used for the following purposes when Enterprise Vault servers ingesting items from Exchange user mailboxes or journal mailboxes: Communicating with and copying data from the Exchange Servers. Accessing the SQL Server database. Transferring archived data to the storage medium (HP StoreAll Storage). Transferring replica data from HP StoreAll Storage to HP StoreAll Storage. Retrieving archived data from the storage medium for indexing. Reading and writing data to and from the index storage medium. Background activity: communication with the domain controller, user monitoring, and so on. Table 21 shows the expected kilobits per second (Kb/s) when archiving messages of 70 KB from Exchange mailboxes, according to estimation from Symantec based on ingest rate per hour. Table 21. Expected Kb/s Hourly ingestion rate 60,000 90,000 Enterprise Vault server Exchange Server 19,000 28,500 Enterprise Vault server SQL Server (Vault Store) 3,000 4,500 Enterprise Vault server SQL Server (Directory) 1,350 2,000 Enterprise Vault server SQL Server (Fingerprint DB) 170 250 Enterprise Vault server Storage medium 7,200 11,000 Enterprise Vault server Index location 19,200 13,000 The numbers above also apply to journal archiving, except that the expected usage for Enterprise Vault server SQL Server (Directory) will be lower by half. The above ingest rate or hour is based on a less powerful server than the EV server used for the offering. With the higher specifications of the EV servers, the ingestion rate or hour will probably be slightly higher than the rate in the table above. But with a Gigabit Ethernet network as described below, it must still be fine to support the traffics. Increasing the average message size to 140 KB will result in about a 30 percent reduction in throughput. In M&C Archiving, all servers must be connected to a 1000 Mb/s full duplex network using a Gigabit Ethernet Network Interface Card (NIC). The same network connectivity speed must be consistent from the servers NIC to the switches and routers located on the same network when possible. Based on the estimation above, a Gigabit Ethernet network must be more than sufficient to support network usage incurred by mailbox archiving and journal archiving. When services are located within an HP data center, the services of managed networks will be leveraged. Network implementations that involve deploying archiving servers in the data center must be similar to those stipulated by HP Managed Network Services. Remote offices All the application servers are located in the customer s data center. As the archiving solution is suited for centralized deployment, all the archiving servers will be deployed within the same data center as the existing messaging servers. When there are users located at remote offices that are across a WAN link to the data center, it is similar to the Hosted at HP data center scenario described above. Since network, traffics incurred by customer s access to archived emails or during E-Discovery activities are insignificant, existing WAN links supporting the daily messaging traffics will be sufficient. Again, vault cache is useful here, as it will help reduce the amount of requests that goes to the EV servers and provides continuity of service in the event of a network outage. 67

Directory service functional component detail Description Like some applications such as Microsoft Exchange, Symantec Enterprise Vault uses Active Directory to store computer and service account objects, and leverages Active Directory domains as its security and trust boundaries. Unlike Microsoft Exchange which stores application configurations in the Active Directory, Symantec Enterprise Vault stores its configurations in a SQL Server database, named Enterprise Vault Directory database. Note This release of HP M&C Archiving will only support Enterprise Vault v10.0.2 in Windows 2003 and Windows 2008 Active Directory environment. Other versions of Active Directory are not supported in this release. Active Directory organizational unit structures This section will describe the recommended organizational unit (OU) structures for the archiving servers. Server OU structure Enterprise Vault servers will have their own Enterprise Vault Servers OU under the servers OU as shown in Figure 20. Enterprise Vault servers is the exact name for the designated top-level OU for Enterprise Vault servers, as used. Integration engineer must work with Active Directory administrators to get sub-ous defined, based on Enterprise Vault server roles. HP M&C Archiving recommend creating two sub-ous: Enterprise Vault Servers and discovery accelerator Servers. These sub-ous contain server objects based on the corresponding server roles and functions. Figure 20. Server OU structure Archive servers Enterprise Vault servers Discovery accelerator servers Account OU structure Several service accounts are required for Enterprise Vault. These accounts will be placed under the Service Account OU located under the Accounts OU. Figure 21 shows the Account OU structure. Figure 21. Account OU structure Accounts Service accounts 68

Account requirements This section provides some detail about the different accounts required in the Enterprise Vault environment. 1. Enterprise Vault Service Account 2. Enterprise Vault System Mailbox(es) 3. Enterprise Vault OWA Anonymous User 4. Enterprise Vault Operations Manager Monitoring User Account 5. Enterprise Vault Reporting User Active Directory server design As Enterprise Vault uses a SQL Server database to provide directory service functions, it does not have special requirements on Active Directory servers. Email archiving requires an existing messaging system (preferably Managed Messaging environment) in place. It can be safely assumed that the existing Active Directory servers that server the messaging system will be sufficient for Enterprise Vault. Enterprise Vault directory Enterprise Vault uses the Enterprise Vault Directory Service to access this Directory database. Other Enterprise Vault services and tasks use this service to access the configuration information, such as location of the archive and permissions on the archive, in the directory database. Figure 22 depicts how the Enterprise Vault directory service and database works together. Figure 22. Vault service and database Enterprise Vault services and tasks: Storage service Indexing service Shopping service Exchange provisioning task Exchange mailbox archiving task Exchange journaling task Enterprise Vault directory service Enterprise Vault directory database Enterprise Vault provisioning Enterprise Vault uses a GUI-driven provisioning model for Exchange mailbox archiving. This provides for flexibility and control of the archiving settings applied to users or groups of users. The Enterprise Vault administrator configures provisioning groups using the vault administration console. These provisioning groups are associated to specific objects in Active Directory. These can be based on: A specified user(s) A Windows security group or distribution list A Windows OU An LDAP query The Whole Exchange Organization When the Enterprise Vault provisioning groups are created and associated to Active Directory objects, they can be correlated with Enterprise Vault mailbox archiving, PST migration, and retention policies. Different Enterprise Vault settings can be applied to different groups of users. It is likely that the customer will not have pre-existing groups that can be used for Enterprise Vault, thus new Windows security groups will be created. List of hardware and software used Directory service for HP M&C Archiving is leveraged from the HP Global Active Directory capability, and as such, there is no need for a separate list of hardware and software for these components in the HP M&C Archiving package. 69

Naming service functional component detail Description The DNS is a naming service. DNS translates domain names into IP addresses or vice versa, and controls email delivery. Technology specifications DNS must meet the following requirements in the Active Directory environment for archive servers to function properly: DNS servers must be Berkeley Internet Name Domain (BIND) 8.1-compliant or a later version. HP M&C Archiving uses Microsoft Windows 2008 servers to meet this requirement. All the DNS servers must contain a full copy of the DNS zone used by archive servers. Because this information is of a sensitive nature, do not use external or publicly available DNS servers for this purpose. The DNS zone must allow dynamic updates. In HP M&C Archiving deployment, MS DNS will be used. MS DNS is installed on the Windows 2003 or Windows 2008 Domain Controllers. Enterprise Vault DNS aliases When installing Enterprise Vault, DNS alias names (CNAME) must always be used to reference the Enterprise Vault servers, SQL Server, and discovery accelerator. DNS alias are critical for supporting native workload failover (Update Service Location USL ) in addition to permitting more seamless hardware refresh and more importantly consistent end-user experience. For the HP M&C Archiving offering, DNS aliases (CNAME) must always be used for the Enterprise Vault servers, Sites, SQL Server, and discovery accelerator. Using a DNS alias serves the following purposes: If the Enterprise Vault Directory is shared between more than one Enterprise Vault sites, it allows the configuration information for each of the Enterprise Vault sites to be distinguished. It allows future flexibility if you change the computer that is running the Enterprise Vault services. Allows the administrator to easily switch services to another physical computer in the event of a reconfiguration or an emergency. This is achieved by changing the server DNS entries so that a different server is referenced by the alias, it is then possible to run an Enterprise Vault utility called Update Service Locations or USL to quickly switch operation to the new server. The use of USL is covered in detail in the Enterprise Vault Administrator s Guide. List of hardware and software used No special hardware and software needed for the naming service in the HP M&C Archiving package. Overall process design approach This section provides a description of the hardware, software, roles, and process components for design, implementation, and management of the solution. It provides information on the technical aspects of the various core components as the optional components. Business continuity and disaster recovery In most situations and particularly for enterprise-level customers, some form of DR is required. Normally failover to a secondary data center is a requirement. Enterprise Vault failover In order to allow any form of DR to occur, the three main data sets of Enterprise Vault must be replicated to the secondary site. These are: SQL databases Indexes Archive data 70

Replication of these data sets is achieved using HP Continuous Access SAN block-level replication and HP StoreAll Remote Replication. There are two supported solutions for providing failover of the Enterprise Vault servers themselves. Enterprise Vault building blocks failover using USL VMware failover Building blocks failover When using physical Enterprise Vault Servers, the Enterprise Vault building blocks failover process known as update service locations is used. This provides a simple failover mechanism within Symantec Enterprise Vault, which allows the services running on one Symantec Enterprise Vault server to be temporarily hosted on a standby server. This necessitates that for each production EV server, a passive standby EV server is always available in the DR site. Upon a failover, the primary servers are shutdown and the SAN storage containing the indexes and archives is represented to the standby servers. The CNAME aliases of the servers are then re-pointed to the DR servers and the USL process is used to move all the existing services to the DR servers. The disadvantage of this process is that it requires duplicate hardware in the DR site. This in turn requires maintenance and complicates upgrades and patches as the duplicate servers must be kept identical to production. Another disadvantage is that this process requires a manual DNS update, and time for AD replication to complete. Full details of the building blocks failover procedure are documented in the implementation guide. Figure 23. EV DR with building blocks HP Operational Support Exchange Primary DC Exchange CAS Exchange Secondary DR DC Exchange CAS EV Building Blocks HA DA DA EV and servers EV Standby servers SQL HP 3PAR Virtual Copy block level replication (Index + SQL) SQL 3PAR SAN 3PAR HP StoreAll Storage NAS replication HP StoreAll Storage HP StoreAll Storage 71

VMware failover When EV is running on VMware, the full capabilities of VMware can be utilized. Once again, all the data sets must be replicated to the DR site at the SAN level. Each EV server should ideally be configured on its own set of LUNs and VMFS and these LUNs should also be replicated to the DR site. In the event of a DR, the primary servers are shutdown. The replicated LUNs containing the EV server images are re-mounted on VMware at the DR site. The LUNs containing the data sets (indexes and archives) are re-attached and the EV servers are brought back online. The disadvantages of this approach are that it can become more complicated than it first seems. IP address ranges can be different between data centers and sufficient VMware capacity must still be provided to run the VMs in DR. The VMware DR process can be automated using VMware Site Recovery manager and it is highly recommended that this approach be considered. SQL Server failover In a DR scenario, the Enterprise Vault SQL databases must become available in the DR site. The exact method of protection for the SQL databases is a matter for the DBAs and should be deployed as per their standards and procedures. Typically they will use mirroring or log shipping or SAN based replication to provide the SQL failover capability. In general, the SQL Server that hosts the databases in DR will have a different name to the production server. Within the EV configuration we always use CNAME aliases to point to the SQL Server, so in a DR, the CNAME can be re-pointed to a new SQL Server very easily. Write capability in DR It must be remembered that a second copy of the archived data must always be kept and verified before EV will allow shortcuts to be created. When using HP StoreAll Storage, this is not such an issue because traditional backups can be used to maintain the safety copy. In a DR, backups can still be taken and archives may still be written. Once backups are in place and the archive data is secure, the HP StoreAll Storage can be reconfigured to allow write operations and archives may be written. This assumes that updates made in DR and being backed up in DR and that updates to storage in DR can eventually be replicated back to production. Note During normal operation, the secondary (replica) HP StoreAll Storage is configured as read only to prevent any un-intentional write operations on the replica device. See the Backup and recovery functional component detail section. Performance and event monitoring See the Monitoring functional component detail section. Reporting See the Reporting functional component detail section. 72

Appendix Terminology and acronyms Term/Acronym AD API BuR CAS CAS CIRT CMT Compliance archiving CPU DAG DAS DC DCOM DL DNS DOD DOS DPM EOM ESEM EV FSE FTP GC GigE GUI HBA HIPAA HTTPS I/O IdM IIS ILM IOPS IP Definition Windows 2003 or Windows 2008 Active Directory A database that hierarchically stores information about network objects and makes this information available to administrators, users, and applications. Application Programming Interface Backup and Recovery (Exchange) Customer Access Server Content Addressable Storage Computer Incident Response Team Compliance Management Team (Refer to section Definition of compliance archiving for definition and description.) Central Processing Unit Database Availability Group A database availability group (DAG) is the key component of the high availability and site resilience framework built into Microsoft Exchange Server 2010. Direct Attached Storage Domain Controller Domain controllers are servers that store data and manage user and domain interactions, including user logon processes, authentication, and directory searches. Distributed Component Object Model Distribution List Domain Naming Service Department of Defense (US government) Description of Service (Microsoft) Data Protection Manager Enterprise Operations Manager Enterprise Security Event Management Enterprise Vault Forefront Security for Exchange File Transfer Protocol Windows 2003 or Windows 2008 Active Directory Global Catalogs The global catalog is a domain controller that stores all objects of all domains in an Active Directory forest, which makes it possible to search for objects at the forest level rather than at the tree level. Gigabyte Ethernet Graphical User Interface Host Bus Adapter Health Insurance Portability and Accountability Act Hypertext Transfer Protocol Secured Input/output Identity Management Internet Information Server Information Lifecycle Management Input/output Operations Per Second Internet Protocol 73

Term/Acronym IPM KB LAN LDAP LRA LSC LUN MAPI MB MOM MSDB MSMQ MSS NIC NNAM NTFS ODBC OS OU OWA PCM PIM PSS PST RAID RAM RAS RPC Over HTTPS SAN SATA SCDW SDA SDN SEC Service Offering SMC Definition Interpersonal Messages Kilobyte Local Area Network Lightweight Directory Access Protocol Leveraged Remote Access Leverage Services Compartment Logical Unit Number Messaging Application Program Interface Megabyte Microsoft Operations Manager Microsoft Database Microsoft Message Queue (HP) Managed Storage Services Network Interface Card Network Naming and Addressing Management New Technology File System Open Database Connectivity Operating System Organizational Unit Microsoft Outlook Web Access Policy Compliance Management Personal Information Management Microsoft Product Support Services Mail user s Personal Store Redundant Array of Independent Disks Random-Access Memory Microsoft Remote Access Service Microsoft Exchange Server 2010 SP1 and Microsoft Office Outlook 2003 and above, combined with Windows Server 2008, support the use of Remote Procedure Call over HTTP to access Exchange servers. Using the Microsoft Windows RPC over HTTP feature to enable users to connect to their Exchange mailbox eliminates the need for remote office users to use a virtual private network (VPN) to connect to their Exchange servers. Users running Outlook 2003 and above on customer computers can connect directly to an Exchange server within a corporate environment from the Internet. Storage Area Network Serial Advanced Technology Attachment (hard disk interface) System Center Data Warehouse Security Design Assessment The Service Delivery Network (SDN) Framework compartmentalizes Customer HP, Service Delivery, and HP Customer network environments into logical zones. Communication between zones is controlled at distinct control points or security layers. The Service Delivery Network is the central zone that has connectivity through security layers to all other zones. Provides the security mechanisms to protect external customers and internal HP networks. Securities and Exchange Commission Market-facing grouping of related features that addresses a class of business needs that can be industry-independent or industry-specific. Managed by a single service line, however may include multiple service line features. Addresses customer needs recognized in a particular market sub-segment. Service Management Center 74

Term/Acronym SMTP SNMP SOW SP SR TB TDP TSM VCC VSS WBS WINS WMF WORM Definition Simple Mail Transport Protocol Simple Network Management Protocol Statement of Work Microsoft s product Service Pack Microsoft s product Service Release Terabyte Tivoli Data Protection Tivoli Storage Manager Virtual Control Center Volume Shadow Copy Service Work Breakdown Structure Windows Internet Naming Service Workplace Management Framework Write Once Read Many (device) 75

Learn more at hp.com/go/archiving symantec.com/enterprise-vault symantec.com/docs/tech125795 Sign up for updates hp.com/go/getupdated Rate this document Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. UNIX is a registered trademark of The Open Group. 4AA4-5078ENW, March 2013