The SaaS LMS and Total Cost of Ownership in FDA-Regulated Companies

The SaaS LMS and Total Cost of Ownership in FDA-Regulated Companies

The SaaS LMS and Total Cost of Ownership in FDA-Regulated Companies By Rob Sims, Director, Life Science, UL EduNeering When a Life Science company is evaluating corporate Learning Management Systems (LMS), calculating a total cost of ownership (TCO) often takes a back seat to gathering the organization s functional and IT requirements. This is understandable, given that Quality Assurance executives must ensure the LMS can satisfy their quality and regulatory goals: qualifications, audit reports, management visibility, etc. However, we have seen that spending some time on TCO has the potential to narrow the focus for an evaluation team that s composed of IT, HR, Procurement and Finance representatives. And when TCO is placed at the forefront of evaluation criteria, more and more organizations will understand why the SaaS-based LMS has grown so popular. In the past five years, the cloud computing adoption has grown steadily. According to a July 2012 CompTIA report, 93% of companies are using some form of cloud solution. And Gartner predicts cloud computing will grow by over 100% to be a $207 billion industry by 2016, while the overall global IT market is forecast to grow at just 3%. TCO is a primary driver for any SaaS (Software as a Service) solution, as a SaaS application typically includes pay-as-you-go pricing, requires less IT resources, no hardware investment, no disaster recovery investment and no hardware maintenance costs. For FDA regulated companies, the most significant cost advantage involves the reduced validation effort, both at start-up and in subsequent years. For many organizations, the validation effort is a time-consuming, resource-intensive process. However, with the SaaS LMS, validation becomes a shared responsibility between the client and the vendor, and this distribution of resources reduces the validation costs significantly, especially when the software is upgraded. That s because with a server-based application, the client s engineering and IT personnel are responsible for designing and maintaining the internal data center, computer hardware, software, security, disaster recovery, etc. And the client s Quality Assurance (QA) team conducts quality and validation activities on the single LMS software code. In the SaaS model, several critical functions of QA, IT and validation are shifted to the SaaS vendor making the SaaS vendor audit a critical activity for the client s validation, auditing, IT and QA team. Page 2

Adhering to FDA Validation & 21 CFR Part 11 Regulations The FDA defines software validation as confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled. (source: www.fda.gov). Even when a software vendor has a well defined process, each business is expected to ensure that the organization s use of the software is validated. The FDA expects developers to follow controls and procedures that are specified in a Software Development Life Cycle (SDLC) to ensure quality. The FDA requires that appropriate testing be performed, and this is achieved by executing test scripts that map back to the software requirements. Testing often consists of Installation Qualification (IQ), in which documentation verifies that the system is installed according to written specifications; Operational Qualification (OQ), in which documentation verifies that the system operates throughout all operating ranges; and Performance Qualification (PQ), in which tests must span the underlying cgmp business process, to ensure that users and administrators trained in the SOPs can accomplish business objectives in the production environment. In addition to validation requirements, the system must meet electronic record and electronic signature requirements as outlined in 21 CFR Part 11. Here are just three critical requirements of Part 11: KEY TOPICS: Adhering to FDA Validation & 21 CFR Part 11 Regulations The LMS Deployment: Auditing the LMS Vendor Reducing Costs Following the Deployment Summary Audit Trails: Use of time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify or delete electronic records. Authority Checks: Use of authority checks to ensure that only authorized individuals can use the system, electronically sign a record, access the operation or computer system input or output device, alter a record or perform the operation at hand. Electronic Records: Signed electronic records need to contain information associated with the signing that clearly indicates these three items: 1) the printed name of the signer; 2) the date and time the signature was executed; and 3) the meaning (such as review, approval, responsibility or authorship) associated with the signature. In a similar way, EU Annex 11 also focuses on the product life cycle, as well as user requirements related to each system. However, Annex 11 places more emphasis on people and management accountability than FDA regulations specifically the individuals with responsibility for the business process and system maintenance. To address these regulatory obligations, clients perform a number of activities: create the Solution Design Document (SDD), develop User and Functional Requirements, develop test scripts and conduct IQ, OQ and PQ testing, among other activities. This effort drives up the IT and validation teams time and effort, and often adds months to any software implementation or enhancement project. Page 3

With a SaaS-based LMS, clients can rely on the vendor to perform and maintain some of these activities. The effort for the client, then, focuses initially on an audit of the data center, as well as the vendor s QA and validation methodology, to ensure these activities are performed at the same standard as would be performed by the client s own QA and validation teams. Typically, the three days or so taken to audit a vendor can dramatically reduce the time spent validating the system. The LMS Deployment: Auditing the LMS Vendor For the SaaS LMS, clients often conduct an audit of the vendor at the vendor s primary location, so the vendor can demonstrate adherence to quality software engineering and testing principles. The client s validation teams will typically review QA documentation and project files that were developed as part of the design, development, testing and implementation of the LMS. The vendor s QA team will need to demonstrate that a valid development methodology is in place and that thorough testing was used to provide confidence and assurance to the client that the LMS is fit for production use. Furthermore, the vendor s QA team must demonstrate that it is involved with these activities, which should occur throughout the entire SDLC stage: Business rules, graphical user interface design elements and interoperability with existing features Functional specification IQ and OQ Process for system releases to clients Test traceability and full product testing before any new LMS code is released to a production environment Typically, when a client audits ComplianceWire, our SaaS-based LMS, these are the documents that are usually requested for review: Business Requirements: This document focuses on the needs of the user and spells out exactly what the system will do Functional Requirements: This document focuses on how the system will do what the user is expecting System Design: This document focuses on capturing the system design based on the functional requirements. Depicting screen layout, system functions and other aspects of the user experience to fulfill the business requirements Requirements Traceability Matrix: This document captures the relationship between the business and functional requirements and the test scripts that satisfy them; this document should reflect the latest enhancements made to the platform Test Scripts: These documents support new enhancements or custom projects. These scripts should cover basic platform functionality, CFR functional and reporting features and high priority areas of the LMS. In addition, custom testing should be executed for any custom programming performed Page 4

Test Plan (TP) and Validation Plan (VP) SOPs: Because ComplianceWire is a single platform, these documents have been adopted as Standard Operating Procedures Validation Summary Report (VSR): This report summarizes our testing activities, discrepancies and other validation activities In many cases, when clients subscribe to ComplianceWire, following an audit of our data center and SDLC methodology, a client s audit and validation team can then focus on creating these documents: Validation Plan: Describes the internal activities that are of part of the overall validation approach to be conducted by the client User Requirements Specification: Provides details on the functionality that the clients demands from ComplianceWire. These required items are often categorized as critical, mandatory or nice to have, and are the basis for the validation/testing effort User Acceptance Test Scripts: Clients should have their own test scripts that either augment our own QA test scripts, or unique scripts that demonstrate that the company has tested specific usage of the system Validation of Configurations and Customizations: Clients often test their specific configurations of ComplianceWire (Security Roles, Custom Fields, Reports, etc.) and any customized solutions The following table summarizes the typical responsibilities as described above. Typical SaaS LMS Validation Responsibilities Vendor Documents Shared During an Audit: Business Requirements Functional Requirements System Design Requirements Traceability Matrix Client Responsibilities: Validation Plan User Requirements Specification User Acceptance Test Scripts Validation of Configurations and Customizations Test Scripts Test Plan and Validation Plan SOPs Validation Summary Report (VSP) Page 5

Reducing Costs Following the Deployment With a server-based LMS, the vendor s release schedule determines ongoing validation/ re-validation activities, and has the potential to drive up TCO, as it impacts the resources of QA, validation and IT. One client noted that IT staff costs associated with periodic review of the server-based LMS, to ensure that validation status has been sustained, is often not noted in the LMS TCO. And for many of our clients who previously relied on a server-based LMS, the costs of re-validating the software prevented them from upgrading the software to a new version, and thus not gaining the benefits of these new features. With a SaaS LMS, the vendor can reduce this effort. First, the vendor can provide clients with a preview period that includes the ability to test the new features on the vendor s test site. Regression test scripts should be provided to aid the client with their own internal validation effort. The vendor should also provide a validation summary report at the end of all release activities. Based on conversations with clients who have actually migrated from a server-based LMS to a SaaS-based LMS, we ve created two scenarios based on a large company (10,000 employees) and a small to mid-size company (500 employees). While the cost assumptions are not based on any single LMS implementation, we do consider actual costs conveyed to us from clients who have migrated their LMS from a server-based LMS, factoring in the cost of IT staff, validation staff and external IT contractors who are involved in the server-based installation and ongoing maintenance. Please note that we do not factor in the migration of historical data, as this cost will vary depending on the amount of historical data to be migrated, regardless of whether the LMS is server-based or SaaS. In both company scenarios, the Year 1 validation cost savings are reduced by the SaaS LMS, as this requires significantly less IT and validation resources. However, cost savings of the SaaS LMS continue during Year 2 through Year 5, because the validation team doesn t have to validate the entire software and IT infrastructure with each software enhancement. This benefit occurs regardless of company size. What we have seen is that in some cases, SaaS provides additional benefits to the small to mid-size company, as it provides built-in scalability to accommodate new users and expand operations without placing a burden on IT infrastructure. Some organizations consider IT and validation resources as a sunk cost because employees are performing the work. As our clients have pointed out, the server-based LMS upgrade still requires a great deal of effort, and to neglect the opportunity costs (the cost of passing by the second-best option when making a decision) associated with employees is an oversight. To paraphrase one client: Other software projects are sacrificed, sometimes for months, when the IT and validation teams have to perform a single server-based upgrade. For that reason, in the scenarios that follow, we have presented costs associated with a server-based implementation that includes IT resources. Page 6

Scenario #1: Large Company 10,000 Global Users Year 1 LMS Cost Comparison Budget: Server Based SaaS Purchase of Licenses $300K - Subscription Fee - $300K 20% Annual Maintenance Fee $60K - Purchase of Hardware/Storage $330K - IT Support $120K - Project Management & Validation Configuration & Interface Dev $500K $50K $350K $75K Total Year 1 Budget: $1.66M $425K Scenario #2: Emerging Company 500 Global Users Year 1 LMS Cost Comparison Budget: Server Based SaaS Purchase of Licenses $25K - Subscription Fee - $25K 20% Annual Maintenance Fee Purchase of Hardware/Storage $5K - $20K - IT Support $10K - Project Management & Validation Configuration & Interface Dev $50K $10K $30K $10K Total Year 1 Budget: $140K $45K Year 2 through Year 5 Cost Comparison Budget: Server Based SaaS Subscription Fee - $1.2M 20% Annual Maintenance Fee Purchase of Hardware/Storage $240K - $120K - IT Support $480K - Validation Related to Upgrades $1M $100K Total Year 2 5 Budget: $1.84M $1.3M Total Year 1 5 Budget: $3.5M $1.725M Total Cost Difference Over Five Years: Companies Save $1.7M with the SaaS LMS Year 2 through Year 5 Cost Comparison Budget: Server Based SaaS Subscription Fee - $100K 20% Annual Maintenance Fee $20K - Purchase of Hardware/Storage 10K - IT Support $40K - Validation Related to Upgrades $120K $10K Total Year 2 5 Budget: $190K $110K Total Year 1 5 Budget: $330K $155K Total Cost Difference Over Five Years: Companies Save $175K with the SaaS LMS Summary: The SaaS LMS has been positioned to allow small and mid-sized companies to gain quality-focused functionality that addresses FDA regulatory needs, while also being scalable to accommodate future growth. From our real-world experience with our clients, we know that a SaaS LMS designed to address FDA validation requirements can lower the total cost of ownership, while also providing predictability of costs over time. Further, SaaS LMSs have been shown to provide reliable access to data anywhere, anytime via a web browser, to accommodate remote employee usage, as well as non-employee training that spans suppliers, clinical sites, vendors, agents and contractors. The critical question that Quality Assurance executives can ask during the evaluation phase is, What will our costs be when the software is upgraded in Years 2 through 5? When the IT team and the validation team estimate the time required of server-based resources, they will be able to calculate the opportunity costs for annual upgrades. Factoring in IT and validation resource costs, as well as the hardware and software maintenance costs related to the server-based LMS, are at the heart of the growth of the SaaS LMS within the FDA-regulated organization. Page 7

About UL EduNeering UL EduNeering is a business line within UL Life & Health s Business Unit. UL is a global independent safety science company offering expertise across five key strategic businesses: Life & Health, Product Safety, Environment, Verification Services and Enterprise Services. UL EduNeering develops technology-driven solutions to help organizations mitigate risks, improve business performance and establish qualification and training programs through a proprietary, cloud-based platform, ComplianceWire. For more than 30 years, UL has served corporate and government customers in the Life Science, Health Care, Energy and Industrial sectors. Our global quality and compliance management approach integrates ComplianceWire, training content and advisory services, enabling clients to align learning strategies with their quality and compliance objectives. Since 1999, under a unique partnership with the FDA s Office of Regulatory Affairs (ORA), UL has provided the online training, documentation tracking and 21 CFR Part 11-validated platform for ORA-U, the FDA s virtual university. Additionally, UL maintains exclusive partnerships with leading regulatory and industry trade organizations, including AdvaMed, the Drug Information Association, the Personal Care Products Council and the Duke Clinical Research Institute. 202 Carnegie Center Suite 301 Princeton, NJ 08540 609.627.5300 UL and the UL logo are trademarks of UL LLC 2013. uleduneering.com WP/13/120513/LS