SOASTA CloudTest Performance Data Retention and Security Policy Whitepaper
Table of Contents Executive Summary: Data Security... 3 1. SOASTA s Data Retention Policy... 3 1.1 Test Development... 3 1. 2 Test Metrics... 3 1.3 Data Encryption... 4 1.4 Test Infrastructure... 4 2. CloudTest Performance Data Q&A... 4 2.1 What is Performance Data?... 4 2.2 What Data is Used During Testing?... 5 2.3 What Happens to the Response Data Received?... 5 2.4 How are Log Files or Other Metrics Handled?... 5 2.5 What Data is Sent Back to the CloudTest Servers?... 6 2.6 What is the Difference Between Scrubbed and Synthetic Data?... 6 2.7 How is Data Handled for Testing Real Transactions?... 6 2.8 How Do We Create Synthetic Data?... 6 2.9 What Happens to the Data Once the Testing is Over?... 6 2.10 How is the Test Data Stored in the Cloud?... 6 2.11 Can Test Result Data be Removed from the Cloud?... 7 2.12 Can Test Result Data be Exported from the Cloud for Offsite Storage?... 7 About SOASTA, Inc... 7
Executive Summary: Data Security Data security is one of the key concerns when it comes to performance testing. Companies do not want their corporate data outside corporate firewalls. SOASTA does not ask customers to export any data or their application outside their firewalls. We can generate synthetic data or use scrubbed data depending on the customer s business and security requirements. We do not save any response data on our CloudTest servers as only key http/https external data is captured for reporting and diagnostic purposes. 1. SOASTA s Data Retention Policy 1.1 Test Development Data that is used for test development by a performance engineer (i.e. HTTP(s) GET/POST requests and responses) is not typically retained by SOASTA. At the customer s request, this data will be deleted immediately after test development. Test definitions (SOASTA test clips) are saved in the SOASTA repository for future testing. These are the steps that comprise the scenario and, while they can be deleted upon request, there is no data or other company private information so they are not typically deleted. 1.2 Test Metrics This includes the performance results of the test(s), such as response times, errors, data transfer rates, etc., as well as infrastructure metrics for any system resources that are monitored as part of the test. Monitored metrics may come from using SOASTA monitoring agents, SSH access to system information or third-party monitoring metrics such as CA s APM (formerly Wily). Test result metrics are kept for a minimum of one year, primarily to allow for comparative reporting over time. At the customer s request, this data (performance metrics and system resource monitors) will be deleted immediately after the customer accepts any test reports that are part of the engagement. PERFORMANCE DATA RETENTION AND SECURITY POLICY WHITEPAPER 3
1.3 Data Encryption Passwords and usernames for the SOASTA CloudTest application and for system resource monitors are encrypted. Monitoring passwords or SSH keys are encrypted and stored in the SOASTA Repository. They are not recoverable or viewable once entered. While it is not implemented as the default, CloudTest supports 3DES encryption of these values in accordance with standards such as the FIPS PUB 46-3 Data Encryption Standard. This can be enabled when compliance is necessary. Synthetic seed data that is fabricated for logging into an application or website to emulate real users can be encrypted as well. All information used to set up montors is retained in the SOASTA repository for future testing. At the customer s request, this data, the usernames and passwords, will be deleted immediately after testing. 1.4 Test Infrastructure CloudTest customers have their own tenant (with associated credentials) and this tenant is maintained for at least one year from date of last test. At the customer request, the tenant will be deleted immediately after all testing and reporting has been deemed completed. The actual load generators and results aggregation servers are temporary cloud-based instances and run only during test execution. This provides an additional level of security as all instances are discarded after each test, with only performance metrics retained in the results database. 2. CloudTest Performance Data Q&A 2.1 What is Performance Data? There are many different types of performance data that are leveraged as part of performance testing. Performance data can be broken up into three major categories: master data, user-generated data and external data. Master data typically exists in the customer s database(s) and is required for conducting business (for example usernames, passwords, social security numbers, etc). User-generated data is anything that the user inputs in editable fields on the application (i.e. new email address, new address, etc). External data is provided upon execution of the application (i.e. confirmation numbers, session ids, etc). Sample of Master Data SOASTA, Inc. PERFORMANCE DATA RETENTION AND SECURITY POLICY WHITEPAPER 4
2.2 What Data is Used During Testing? Data requirements are dependent on the application and the business processes under test. For example, a static site might not require any performance data but just access to the site to make the requests, while a more complex application might require all three types of performance data outlined above. 2.3 What Happens to the Response Data Received? All customer-related response data is discarded from CloudTest servers during load testing. Only performance data related to that response data is retained (response times, errors, etc.). In addition, as noted above, CloudTest servers are temporary instances discarded after each test, with only performance metrics retained in the results database. However, all customer and external data is stored on the CloudTest server during script creation and debugging. The data used to create scripts can be deleted after script creation is completed to ensure no customer data is stored on SOASTA servers. 2.4 How are Log Files or Other Metrics, Which Capture Information About Data Handled? SOASTA does not keep log files or other metrics that capture information about customer data during load tests. But as stated in the previous question we do store this type of data during script creation. Request Response SOASTA, Inc. PERFORMANCE DATA RETENTION AND SECURITY POLICY WHITEPAPER 5
2.5 What Data is Sent Back to the CloudTest Servers? During load testing, only key external data is kept, such as http response codes, cookies, session ids, etc. But all data is sent back to the CloudTest servers to parse, at which point it is fully discarded from any SOASTA servers. 2.6 What is the Difference Between Scrubbed and Synthetic Data? Scrubbed data resides in a customer database and has gone through a process so it no longer includes any real customer data; in essence data that has been transformed and/or created from actual customer data. Synthetic data is generated from scratch and is designed to be an accurate representation of the customer s production database. 2.7 How is data handled for testing real transactions? Real transactions, such as purchases or registrations, have an impact on the database. For testing in the lab, there is generally not a problem as the transaction is only using dummy data and may not actually go through the entire end-to-end process, particularly if the actual order process, for example, is handled by a third party. For testing production systems, more thought needs to go into handling the transactions so that orders aren t accidentally placed or any other real user data isn t impacted. There are a number of approaches that can be taken. Common approaches include: a) If there is an online system that dumps into a queue that an Order management system picks up for fulfillment, stopping the queue and removing items before submitting. b) Some companies have already created many test accounts with addresses and credit cards. Typically there are test credit card numbers and test gift cards. There may be other techniques to identify and filter out test orders, as well. Or, there may be a group of cards that are identified as test cards and a specific list of products from which to purchase. For guest checkouts there may be a specific email format from which to filter out test orders. c) Virtualizing the very end before a transaction gets sent for processing, or hard-coded switches based on customer account or credit card can prevent fulfillment. Aside from creating bad data in the production database, preventing fulfillment is a primary concern. Most of the time fulfillment is processed to a queue or backend system to avoid fraud and most online purchases are not fulfilled in real-time. Thus, you can stop the fulfillment process and remove the test orders prior to allowing fulfillment to process. d) Continue processing all the way up to submitting the order or transaction, and stop. For example, most of the work for an e-commerce site is done prior to the submit. At the same time, you don t want the order taking process to break, so this approach is less desirable. 2.8 How Do We Create Synthetic Data? SOASTA works closely with your application team to ensure a rich spread of representative data is created. Data is also created to target the business process under test and can expand as testing expands. 2.9 What Happens to the Data Once the Testing is Over? When testing is complete, the CloudTest servers only store performance metric data relevant to the test runs. No corporate data is retained on the CloudTest servers. 2.10 How is the Test Data Stored in the Cloud? For tests executed by SOASTA, the test results are stored in the cloud. The results are on Linux servers that are taken down at the conclusion of the testing event. These servers are only available during testing sessions or when results are being analyzed. In most cases, the result data is stored in a relational database in EC2 in an EBS (elastic block store). At the completion of each testing session, a snapshot of customer data is taken. These snapshots are documented and tied to a customer for data cleansing purposes. This data is only available to SOASTA employees and not available to any customers. SOASTA, Inc. PERFORMANCE DATA RETENTION AND SECURITY POLICY WHITEPAPER 6
Sample Performance Metric Data 2.11 Can Test Result Data be Removed from the Cloud? Yes, results can be deleted and removed from the cloud. Results are deleted at the request of the customer. When requested, the results are not deleted until after the result report is completed. Note that once the results are deleted, they are permanently deleted. No backups are made of deleted results. Beyond deleting customer data from the tenant, the hard drive/volume where the data was stored can also be permanently deleted. At the request of the customer SOASTA can facilitate any existing data wiping procedures that are documented and agreed upon prior to an engagement. Typical processes include those that are detailed in documents such as DoD 5220.22-M ( National Industrial Security Program Operating Manual ) or NIST 800-88 ( Guidelines for Media Sanitization ). This includes any operational cleansing that can be performed from a Linux machine on a mounted disk through use of utilities and processes. Since the disks are not owned by SOASTA destruction of the underlying media is not possible. Note that deleting results limits the ability to compare results from prior and future tests. The only way to compare results once a result is deleted is by looking at the result report. 2.12 Can Test Result Data be Exported from the Cloud for Offsite Storage? Yes. All results are exportable in XML format. Results can be exported from the cloud and moved to a customer-defined location for safekeeping. Depending on the length of the test and amount of performance data related to the test, it can take up to 24-48 hours to export the data from the cloud and move to a secure location. Optionally, the results can then be deleted from the cloud as well. About SOASTA, Inc. SOASTA is the leader in cloud and mobile testing. Its web and mobile test automation solution, CloudTest, enables developers, QA professionals and IT operations teams to test with unprecedented speed, scale and precision. The innovative product set streamlines test creation, automates provisioning and execution, and distills analytics to deliver actionable intelligence faster. With SOASTA, companies can have confidence that their applications will perform as designed, even in peak traffic. SOASTA s customers include many of today s most successful brands including American Girl, Chegg, Gilt Group, Hallmark, Intuit, Microsoft and Netflix. SOASTA is privately held and headquartered in Mountain View, California. For more information about SOASTA, please visit www.soasta.com. Headquarters: 650-210-4950 444 Castro St, Fourth Floor Mountain View, CA 94041 FAX: 650-210-4957 EMEA: +44 (0) 1753 752375 Thames Court 1 Victoria Street Windsor, Berkshire SL4 1YB To learn more visit: soasta.com or email us at info@soasta.com 2012 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/or registered trademarks of their respective owners. EST-1008-2-0113 Connect with us: