CLOUD IN MOTION QUESTIONS EVERY LIFE SCIENCES COMPANY SHOULD ASK BEFORE MOVING TO THE CLOUD. FRANK JACQUETTE, JACQUETTE CONSULTING, INC.

Transcription

1 CLOUD IN MOTION QUESTIONS EVERY LIFE SCIENCES COMPANY SHOULD ASK BEFORE MOVING TO THE CLOUD. FRANK JACQUETTE, JACQUETTE CONSULTING, INC.

2 S EVERY LIFE SCIENCES COMPANY SHOULD ASK BEFORE MOVING TO THE CLOUD. To help our clients understand the risks and assess potential cloud vendors I ve put together this list of questions. The questions are based on our own experiences (both good and bad) with helping clients implement cloud-based solutions in validated settings. I ve arranged it so you can directly incorporate the questionnaire as part of a Request for Information (RFI) or Request for Proposal (RFP) document; I ask only that you leave the Jacquette Consulting footer intact. AT ITS BEST... Cloud computing promises many things: lower costs, availability from anywhere, and consistency across all users. Best of all, it enables business users to make the care and feeding of information technology infrastructure Somebody Else s Problem, enabling them to focus on getting their own jobs done rather than engage in endless arm wrestling with computers. AT ITS WORST... However, the cloud presents the possibility of a wide range of disasters. System downtime, slow performance, and lost data are the simple ones, but in the cloud you can also suffer from natural disasters that affect your cloud provider. You can suffer collateral damage if law enforcement seizes computers at the cloud provider s data center. Natural disasters can take out data centers; unethical hackers can break in and steal your data. In the regulated world of life sciences the cloud is an even trickier beast. How do you demonstrate control over a system not directly under your company s fingertips? How do you demonstrate consistency if you re not sure how the cloud vendor manages systems? In addition to the main questionnaire I ve included my own notes about what I would consider good and bad responses to the questions. Your situation may vary, and there may be times when you select a vendor who has less-than-perfect answers because the situation demands it. Even if your vendor is unable to provide great answers to all of the questions you will at least know where you stand and what areas may present additional risk. THE CLOUD IS NOT A TECHNOLOGY TERM, IT S A MARKETING TERM. The cloud presents some unique challenges to validation professionals, but there is nothing about the technology that makes it impossible to validate. If you have additional questions or could use further assistance in assessing or deploying a cloud-based solution, please contact me directly at frank@jacquette.com. 1

3 HARDWARE & FACILITIES Where are the system s servers physically located? Will you enable us to visit the site(s) where our system runs and inspect the system? Does our system reside on a fixed set of servers or is it instantiated across many possible servers? Does our system use directly attached storage such as a hard drive or RAID array attached directly to our server or network attached storage such as a SAN? Is there local hardware redundancy (within the same data center) for hardware failure? If so, does it failover automatically or does it require human intervention? Is there remote hardware redundancy (e.g., at a geographically separate data center)? If so, does it failover automatically or does it require human intervention? If there are multiple systems providing failover capability, how frequently are they synchronized? Does our system s software run natively on its server hardware or is it virtualized? If our system is virtualized, what hypervisor do you use? Is there power redundancy at each of my system s server locations? If so, how long can the system run without externally supplied power? SECURITY Does our system share server hardware with other customers? Does our system share storage hardware with other customers? Does our system share backup hardware with other customers? Does our system share a local area network with other customers? Is there a firewall between our system and systems belonging to other customers? THINGS TO WATCH OUT FOR If the vendor can t (or won t) tell you where your servers are located, it s going to be pretty difficult to demonstrate control and qualification of the environment. You need to be able to verify that the systems are where the vendor says they are. The concept of on demand computing enables a cloud provider to manage hardware resources by adding, removing, or relocating software across many pieces of equipment. While this is good for optimizing resource management for the vendor, it can make validation extremely difficult, and exposes you to additional security risks. If your server is dedicated to you but disk storage is shared, you re still vulnerable to attacks that target shared resources, or to law enforcement seizures that accidentally include your data. Basic hygiene if a server fails or a hard drive dies, will your system go down and for how long? If an entire data center fails because of weather, power loss, fire, etc., how quickly will you be able to get back to work? There may be multiple systems, but they may not all be in sync. If the main system goes down, how much potential lost work is there? The shorter this interval is the better. A technical question that helps you evaluate how hard it would be to move your system to new equipment. Virtualization makes it relatively simple to move entire software environments from one piece of equipment to another. A technical question to help your IT staff understand what they may be working with. Power failures are fairly common even in developed countries, and in developing countries they can be broad in area and long in duration. This help you assess your exposure. Any time that you are sharing hardware with other customers you are potentially vulnerable to attacks launched from other software or users on the same hardware. This could be intentional on the part of the other customer (if, for example, you believe that your competitors might engage in espionage or sabotage) or it could be the result of a security breach of the other customer s software. In addition, if the other customers are subjected to a law enforcement action (such as a seizure of their equipment) you may be caught up in the net. 2

4 SECURITY CONT. Is there a firewall between our system and the Internet? Is our system connected to the outside world via more than one Internet connection? If so, is there a firewall at each connection? How is each data center where my system resides secured physically? Are there other means to access your servers (e.g., dial up modems)? If so, how do you secure these access points? Do you employ wi-fi in your data centers? If so, how is it secured against unauthorized access? Does each of your data centers log physical access? Am I able to request copies of those logs as needed? Do you employ a third party to perform security and penetration testing against your systems? If so, who? Will you provide us with copies of their findings? If we wish to perform our own penetration testing to evaluate your system security will you work with us to do so? Do you train your staff against social engineering-based attacks? Do you have a physical intrusion detection system? Do you have a logical intrusion detection system (to detect attacks via the network)? If you detect a security breach will you inform us? If so, how quickly? THINGS TO WATCH OUT FOR This is basic security; if a cloud vendor has your system wide-open to the entire Internet, there had better be a pretty good reason. Data centers frequently have more than one connection to the outside world; each one needs to be secure. Are the doors to the data center locked? Is there a badge reader, or biometric scanners such as fingerprint scanners or retinal scanners? Are there externally facing glass windows that are easily broken? Older technologies such as modems can make great tools for the vendor to manage the system if the Internet connection fails, but they can also provide another vector for attacks. Wi-fi is an easy target for attackers, and there are several outdated wi-fi standards still in widespread use that are easily compromised. How does the vendor know who has had access to the data center? They need to track this and be willing to share for you to be able to demonstrate control over the environment. Good vendors will constantly search out their own weaknesses and try to improve things. It s a major positive sign when a vendor is proactively asking professionals to try to break in. Likewise a good vendor should be willing to let you test their security. Social engineering is a technique where an attacker gains unauthorized access by convincing the human staff that he or she should have it. Example social engineering techniques include calling in and pretending to be a user who has lost a password, showing up dressed as a Verizon technician and asking for access to the phone closet, and calling a user and pretending to be the help desk in an effort to get the user to reveal access credentials. Defending against social engineering requires training of the people most likely to be targeted, and is essential to create a secure environment. Physical security is much more effective if you know when someone has managed to compromise it. Cameras, motion sensors, etc., are all physical intrusion detection systems. Part of successfully defending against network attacks is being aware of both failed attempts and successful attacks. To do that you must have an intrusion detection system. Many companies prefer not to share information about security breaches because it is embarrassing. However, if you are unaware of successful attacks against your systems you are unable to respond appropriately. You should insist that the vendor inform you promptly of attacks. 3

5 SECURITY CONT. How many security breaches have you suffered in the last three years? How do you dispose of decommissioned media such as hard drives or tapes? Do you have a formal destruction policy for old media? DISASTER RECOVERY Are there local backups? If so, how often is the system backed up? Are there remote backups at a geographically separate location? If so, how often is the system backed up to the remote location? How many generations of backups do you maintain? On what medium do you store backups (hard drive, tape, etc.)? If we request a copy of the most recent system backup will you provide it to us? Is there a cost to do so, and if so what is the cost? Do you perform disaster recovery drills to ensure that your backup and recovery regime is successful? If so, how often, and will you provide us with copies of the results? Do you perform failover drills to ensure that your failover scheme is successful? If so, how often, and will you provide us with copies of the results? How can we access our systems and/or data if your system is down? In the event that your data center(s) is/are down, how long would it take us to access our systems and/or data? If your organization ceases operation or goes out of business, how can we access our systems and data? Are you willing to assume liability for data loss or damage as part of any agreement into which we enter? Are you insured against data loss or system loss? If so, in what amount is the insurance and who is the carrier? THINGS TO WATCH OUT FOR Obviously this number should be as low as possible. Old storage media can be a great way for attackers to acquire your data or examine how your systems work in preparation for an attack. Any destruction policy should include recordkeeping that tracks how and when media has been destroyed. Any vendor that doesn t perform backups should immediately be removed from consideration. Local backups are useless if a facility burns down, so there need to be remote backups as well. Sometimes problems are not detected until after a backup cycle has run, so you may need to go back more than one backup to be able to get back what you need. Having multiple generations of backups makes this possible. This is helpful for your IT staff to know if they need to be able to restore a backup or take over operation of the system. This is important if the vendor is no longer able to meet your needs. Backup regimes are pretty useless if they can t be restored, but that happens a lot more often than you would think. A good vendor will routinely test that both backup and recovery work as expected. Failover is the process by which one computer or set of computers takes over when the first one fails. Like backups, it has to be tested to be sure that it works. If the vendor suffers a complete technical failure, what s your option? Will they put server racks in a truck and drive them to your office? If the vendor has an emergency plan, how long would it take for you to get going again? What happens if the vendor goes out of business? You don t want your systems and data being bought by a competitor at a bankruptch auction, or sitting locked in a warehouse while bankruptcy proceedings play out. This is the acid test of the vendor s confidence in their ability to keep your data safe. You need to make sure that the vendor has the financial resources to cover any data loss that you may suffer. 4

6 OWNERSHIP & CONTROL Who owns the servers and storage upon which our system runs? Who owns the software upon which our system runs? Do you outsource infrastructure or system administration to a third party? If so, who? If we wish to take the system and run it in our own data center with our own staff, will you facilitate the transfer of equipment, data, and software? If so, what are the costs to do so? If we wish to take the system and run it in our own data center, will you license the software to us? If so, what are the costs to do so? If we wish to take the system and run it in our own data center, will you provide support? If so, what are the costs? If your system includes proprietary software are you willing to place a copy of the source code in escrow with a third party in case your business ceases operations? Who owns the data on the system, your organization or ours? OPERATIONS During what hours do you provide live phone-based support? Do you outsource help desk or front-line support to a third party? If so, who? Do you have a response plan in the event that law enforcement arrives at your facility and demands access to or attempts to seize equipment that supports our system? THINGS TO WATCH OUT FOR Basic questions. If the equipment isn t yours, your control will be limited. Similarly, it does little good to own servers but not the software needed to run them. If there s a third party behind your vendor you have to ask them all of these same questions to ensure control. The acid test of the vendor s commitment to make the system work for you. If the vendor runs their own software (e.g., Salesforce.com) this may not be practical, but if you can t take the software with you then you are forever locked in to that provider. Also a good test of the vendor s commitment. If a vendor uses their own software but then goes out of business you have few options. Placing the source code in escrow enables gives you the option to find another vendor to maintain and support the software in the event of a bankruptcy. This is an extremely important question and the answer is not necessarily obvious, so check carefully. If the vendor owns the data, you should probably consider a different vendor, especially if the data is something that gives you a competitive advantage or is subject to privacy regulations. A basic question; the correct answer depends on your business. You want to know whether the vendor will be directly supporting you or whether you have to go through a third party. In the United States there have been several instances where cloud vendors have been shut down by law enforcement (usually over piracy issues.) Law enforcement does not take a nuanced approach to seizure of equipment, so you could be caught up accidentally if another customer s equipment is seized. It is important that the vendor be aware of this possibility and have a plan to minimize the cross-customer impact of any such seizure. 5

7 REGULATORY Our company is regulated by the United States Food and Drug Administration (the FDA.) The FDA will occasionally ban individuals from working in the industry for misconduct or illegal activity. Do you screen employees and subcontractors to ensure that they are not on the FDA s banned list? Do you employ third-party auditors to assess your systems for regulatory compliance? If so, who? Will you provide us with copies of their findings? If we wish to perform our own audit of your system policies and procedures will you work with us to do so? Is each member of your staff specifically trained on how to perform his or her role? Do you maintain training records? If so, may we examine them? Do you maintain standard operating procedures (SOPs) around system administration and maintenance? If so, may we examine them? How do you manage change control around the service? Do you have a quality assurance manager? If so, is that person solely dedicated to a QA role, or do they have other responsibilities within your organization? In our terminology system qualification is the process to ensure that the system will behave consistently on a given set of hardware (e.g., if you move the system from one server to another you have qualified the new server to ensure consistent results.) Do you have a formal process for system qualification? If you perform system qualification, do you track the results of your qualification? Have you ever been audited by an organization that operates under FDA regulation? If so, when? Are you currently involved in any litigation involving past or present customers? THINGS TO WATCH OUT FOR Many cloud vendors are unaware of how the FDA does things, so this is an important question to ask. Like self-imposed security audits, self-imposed compliance audits are a good sign that the vendor takes the issue seriously and is always looking to improve. It s a big red flag if a vendor will not permit you to perform an audit. Training and record management are critical components of compliance but not necessarily well understood by cloud vendors. Change control is essential to be able to perform validation and verification; if the environment or software is constantly changing, you can t demonstrate control and consistency. It is critical that the vendor have a change control process and that they follow it consistently. A dedicate QA manager is a great sign that they understand that compliance is a Big Deal and are willing to dedicate resources to it. Hardware gets old and fails, systems are moved to facilitate data center management, hard drives are upgraded, etc. If the vendor has no way to demonstrate that the system behaves consistently with different hardware you will have a hard time validating anything. And of course qualification is only useful if you can demonstrate that you actually did it. Understanding the FDA and its regulatory environment is always helpful. It s good to know if your vendor is being sued by customers for failing to meet obligations. 6

8 YOUR BLANK GUIDE QUESTIONS EVERY LIFE SCIENCES COMPANY SHOULD ASK BEFORE MOVING TO THE CLOUD. FRANK JACQUETTE, JACQUETTE CONSULTING, INC.

9 VENDOR RESPONSE HARDWARE & FACILITIES Where are the system s servers physically located? Will you enable us to visit the site(s) where our system runs and inspect the system? Does our system reside on a fixed set of servers or is it instantiated across many possible servers? Does our system use directly attached storage such as a hard drive or RAID array attached directly to our server or network attached storage such as a SAN? Is there local hardware redundancy (within the same data center) for hardware failure? If so, does it failover automatically or does it require human intervention? Is there remote hardware redundancy (e.g., at a geographically separate data center)? If so, does it failover automatically or does it require human intervention? If there are multiple systems providing failover capability, how frequently are they synchronized? Does our system s software run natively on its server hardware or is it virtualized? 7

10 VENDOR RESPONSE HARDWARE & FACILITIES CONT. If our system is virtualized, what hypervisor do you use? Is there power redundancy at each of my system s server locations? If so, how long can the system run without externally supplied power? SECURITY Does our system share server hardware with other customers? Does our system share storage hardware with other customers? Does our system share backup hardware with other customers? Does our system share a local area network with other customers? Is there a firewall between our system and systems belonging to other customers? Is there a firewall between our system and the Internet? 8

11 VENDOR RESPONSE SECURITY CONT. Is our system connected to the outside world via more than one Internet connection? If so, is there a firewall at each connection? How is each data center where my system resides secured physically? Are there other means to access your servers (e.g., dial up modems)? If so, how do you secure these access points? Do you employ wi-fi in your data centers? If so, how is it secured against unauthorized access? Does each of your data centers log physical access? Am I able to request copies of those logs as needed? Do you employ a third party to perform security and penetration testing against your systems? If so, who? Will you provide us with copies of their findings? If we wish to perform our own penetration testing to evaluate your system security will you work with us to do so? Do you train your staff against social engineering-based attacks? 9

12 VENDOR RESPONSE SECURITY CONT. Do you have a physical intrusion detection system? Do you have a logical intrusion detection system (to detect attacks via the network)? If you detect a security breach will you inform us? If so, how quickly? How many security breaches have you suffered in the last three years? How do you dispose of decommissioned media such as hard drives or tapes? Do you have a formal destruction policy for old media? DISASTER RECOVERY Are there local backups? If so, how often is the system backed up? Are there remote backups at a geographically separate location? If so, how often is the system backed up to the remote location? 10

13 VENDOR RESPONSE DISASTER RECOVERY CONT. How many generations of backups do you maintain? On what medium do you store backups (hard drive, tape, etc.)? If we request a copy of the most recent system backup will you provide it to us? Is there a cost to do so, and if so what is the cost? Do you perform disaster recovery drills to ensure that your backup and recovery regime is successful? If so, how often, and will you provide us with copies of the results? Do you perform failover drills to ensure that your failover scheme is successful? If so, how often, and will you provide us with copies of the results? How can we access our systems and/or data if your system is down? In the event that your data center(s) is/are down, how long would it take us to access our systems and/or data? If your organization ceases operation or goes out of business, how can we access our systems and data? 11

14 VENDOR RESPONSE DISASTER RECOVERY CONT. Are you willing to assume liability for data loss or damage as part of any agreement into which we enter? Are you insured against data loss or system loss? If so, in what amount is the insurance and who is the carrier? OWNERSHIP & CONTROL Who owns the servers and storage upon which our system runs? Who owns the software upon which our system runs? Do you outsource infrastructure or system administration to a third party? If so, who? If we wish to take the system and run it in our own data center with our own staff, will you facilitate the transfer of equipment, data, and software? If so, what are the costs to do so? If we wish to take the system and run it in our own data center, will you license the software to us? If so, what are the costs to do so? 12

15 VENDOR RESPONSE OWNERSHIP & CONTROL CONT. If we wish to take the system and run it in our own data center, will you provide support? If so, what are the costs? If your system includes proprietary software are you willing to place a copy of the source code in escrow with a third party in case your business ceases operations? Who owns the data on the system, your organization or ours? OPERATIONS During what hours do you provide live phone-based support? Do you outsource help desk or front-line support to a third party? If so, who? Do you have a response plan in the event that law enforcement arrives at your facility and demands access to or attempts to seize equipment that supports our system? REGULATORY Our company is regulated by the United States Food and Drug Administration (the FDA.) The FDA will occasionally ban individuals from working in the industry for misconduct or illegal activity. Do you screen employees and subcontractors to ensure that they are not on the FDA s banned list? 13

16 VENDOR RESPONSE REGULATORY CONT. Do you employ third-party auditors to assess your systems for regulatory compliance? If so, who? Will you provide us with copies of their findings? If we wish to perform our own audit of your system policies and procedures will you work with us to do so? Is each member of your staff specifically trained on how to perform his or her role? Do you maintain training records? If so, may we examine them? Do you maintain standard operating procedures (SOPs) around system administration and maintenance? If so, may we examine them? How do you manage change control around the service? Do you have a quality assurance manager? If so, is that person solely dedicated to a QA role, or do they have other responsibilities within your organization? 14

17 VENDOR RESPONSE REGULATORY CONT. In our terminology system qualification is the process to ensure that the system will behave consistently on a given set of hardware (e.g., if you move the system from one server to another you have qualified the new server to ensure consistent results.) Do you have a formal process for system qualification? If you perform system qualification, do you track the results of your qualification? Have you ever been audited by an organization that operates under FDA regulation? If so, when? Are you currently involved in any litigation involving past or present customers? 15

18 Humanity. Technology. Happiness. 710 PROVIDENCE ROAD, MALVERN, PA / T / F / JACQUETTE.COM