Five Steps to Evaluate and Select an LMS - Proven Practices (Jun 13) by Steve Foreman June 5, 2013 If an organization is to evaluate learning management system (LMS) products and vendors effectively, what should it do? Finding the LMS that best meets your organization's needs is not easy. There are hundreds of LMS products available. The investment of time and cost that organizations make in learning management systems and related technologies is significant, as are the risks of disruption from selecting the wrong solution. Once the LMS is in place, your organization is likely to be using it for the next three to five years or more. You certainly don t want to pour your efforts into a product that is poorly matched to your needs. A new research report from The elearning Guild Evaluating and Selecting an LMS (Jun 13) gives insight into what organizations that have already been through the process do. In this article I offer the five steps that are proven to lead to the best decisions. Some background: Learning management systems A learning management system (LMS) can help you manage the administration, delivery, tracking and reporting of instructor-led classes and elearning programs. An LMS provides automation that replaces rigorous and expensive manual work, saves time, and enables you to organize your content, data, and learner audiences. It reports on data that can help you measure results and forecast demand. If you already have an LMS and are not satisfied with its features and capabilities, it may be time to consider pursuing a more suitable product. The LMS marketplace is always changing, with new players entering the fray all the time. A product that is well suited to your organization s needs is probably out there. If you don t have an LMS, or you use a custom-developed solution that is more than a few years old, your organization may be missing some beneficial features that are available in commercial products. In The elearning Guild s new research report, survey responses suggest that the process an organization uses to evaluate and select an LMS can have a significant impact on the perceived success of the LMS implementation and the organization s ultimate satisfaction with the solution.
How to evaluate products and select an LMS Generally, the LMS evaluation and selection process should involve five major steps: needs analysis, requirements definition, product vetting, product evaluation, and product selection (Figure 1). Figure 1: The five major steps in LMS evaluation and selection Step 1: Needs analysis A needs analysis is a great way to get started. Interview your leadership and conduct focus groups with stakeholders throughout the organization. If you are planning to replace your current LMS, ask training groups and users what they like most about the current solution, and what they would most like to see improved. If your organization has no LMS, focus your questions on how the organization manages learning currently and what the key problems areas are operationally and financially. Ask leadership broader questions related to the organization s strategic goals and how human performance impacts them. Don t forget to ask IT about any technical standards or constraints you need to consider. Once you have collected sufficient input, you should be able to identify strategic, operational, and technical drivers that you can use as guideposts for defining your LMS requirements Examples of drivers Based on your data gathering, you may identify the following types of drivers: Your leadership states that your organization is expecting to undergo rapid global growth in the coming years a strategic driver. Some of your training managers mention that delays of a day or more in updating learning records for training delivered half-way around the world have been causing problems with report accuracy an operational driver. Your IT department expresses concern about the accessibility and data security regulations it must meet in certain countries where users reside a technical driver. The strategic, operational, and technical drivers you discover during a needs analysis will bring your LMS goals and requirements into focus. This approach will ensure that your LMS selection criteria align with your organization s goals and priorities. Step 2: Requirements definition Requirements form the basis of your LMS selection criteria. The clearer and more complete your LMS requirements, the easier it is to evaluate products. If you are an instructional designer, you may find that writing LMS requirements is somewhat similar to writing instructional objectives. Try to focus on what learners or administrators must be able to do with the LMS. Each requirement should express a need, not a solution. Each requirement should be discrete, without repeating or overlapping other requirements, and you should write it at the same broad-based level of detail. If you end up with somewhere between 30 and 60 requirements, you re probably at the right level of detail. It is sometimes helpful to organize your requirements into three categories: functional, technical, and cost. Functional requirements describe what the system must be able to do from a learning management perspective. Technical requirements describe how the system must fit into the broader
IT enterprise infrastructure. Cost requirements describe how the system matches up with your organization s budget considerations, constraints, and expectations. Examples of requirements Based on the drivers listed in the previous example, you may define the following requirements. Functional requirements: LMS administrators must be able to manage on-boarding curricula with automatic email reminders and progress tracking. Learners must be able to access the LMS and its content in their native language. The system administrator must be able to configure permissions and span of control for various learning administrator groups so that enterprise administrators can manage enterprise-wide content, while local administrators work autonomously on localized content delivered in local time zones to local target audiences. Technical requirements: The LMS must support specific country regulations related to security and accessibility. The LMS must be easily scalable to handle increasing numbers of users and content to keep in step with the growth of the organization. Cost requirements: The LMS must be compatible with a lower-priced database management system. The LMS must provide cost-effective licensing options for the anticipated number of users in your organization. It is important to prioritize your requirements and you can use a number of methods to accomplish this. One effective method is the analytic hierarchy process, because it offers a reliable, fun, and interactive method for achieving rapid group consensus. By prioritizing your requirements, you can use a weighted scorecard to evaluate how well various LMS products meet each requirement and then calculate overall scores that indicate which products align best with your requirements and priorities. Step 3: Product vetting There are hundreds of LMS products available in the marketplace. Your first challenge is to narrow your choices to a short list of products. One way to accomplish this is to identify eight-to-ten distinctive requirements that can help you rule out non-qualifying products. For example, not all LMS products support multiple languages. If this is critical for you, this makes the requirement for multi-language support a good vetting criterion. Similarly, not all products comply with specific accessibility and security regulations; also good vetting criteria. There are many different price ranges for products. Ballpark licensing costs are another good vetting criterion. Once you have defined a set of vetting criteria, you can research product web sites and/or contact vendors to inquire about their product s support for each vetting criterion. Select a short list of products, perhaps eight-to-twelve, which meet the most vetting criteria. You will evaluate these products in more detail.
Step 4: Product evaluation The product evaluation process works like a funnel. With each step in the process, you rule out nonqualifying products and continue evaluating those that best meet your requirements. You will achieve the best results by performing all of the suggested evaluation activities in the order shown in Figure 2. Figure 2: The product evaluation process funnel Request for information (RFI) A good first step is to create a request for information (RFI). An RFI contains a list of all your requirements with from two to six questions per requirement for vendors to answer. Avoid asking yes or no questions. Instead, ask questions that will require vendors to describe how they meet the requirement. When your RFI is ready, contact your short-listed vendors to ask if they are willing to respond. If so, send them the RFI along with a due date for their response. You may need to set some ground rules to keep the process fair and equitable for all participating vendors. For example, you may decide that no vendor will be told who the other vendors are; answers to any questions asked by one vendor will be distributed to all vendors; no RFI responses will be accepted after the due date, etc. Evaluate the vendor responses with a scorecard that allows scorers to rate how well the vendor meets each requirement on a five-point scale (e.g., unsatisfactory, suboptimal, acceptable, strong, and optimal). You may decide to calculate scores for each requirement, each requirement category, and an overall score. Calculate the vendor scores by incorporating your requirement priority weighting so that the overall score reflects how well the LMS meets your requirements, based on what is most important to your organization. After scoring the responses to the RFI, you can rule out some of the lower-scoring vendors and continue evaluating the top contenders.
Use case demonstrations Invite each of the top contenders to visit your organization for a half day to demonstrate their product. A good approach is to send use cases to the vendors in advance and ask them to demonstrate how their product accomplishes each of your use cases. For this purpose, a use case is a scenario or goal and set of parameters that would typically be performed by a learner or administrator. Use cases should reflect the way your organization manages learning. Select some common use cases as well as some that are complex and challenging. This will enable you to compare all vendors based on the same demonstration criteria. Again, you can use a scorecard to evaluate the vendor demonstrations and rule out those vendors whose products did not perform well in the demonstrations. Example: USE CASE #1: CREATE A LEARNING PATH Background: A learning path is provided for each key function in our organization. A learning path consists of a number of learning requirements, each of which is supported by one or more classes, materials, and assessments. An individual requirement within a learning path may have an expiration date, upon which the learner must refresh his or her skills by completing the most recent learning activities to fulfill the learning requirement. Demonstration: Please demonstrate how a learning path can be created in your product. Instructions: Create a learning path mapped to a particular role or function. The learning path should contain at least four learning requirements, which must be taken in an enforced sequence. The first learning requirement is met by completing an instructor-led class. The second learning requirement is met by accessing a document containing policies and performance standards and getting electronic signature sign-off from your mentor that you have reviewed the materials and met with your mentor to discuss them. The third learning requirement is completed by achieving a passing score >= 80% in an online assessment. The fourth learning requirement is completed by fulfilling the criteria of an in situ observation by your mentor. The learner must complete all four learning requirements within a threemonth timeframe to get credit for completing the learning path. The second learning requirement is set to expire and requires retraining annually. Trial version or sandbox As you continue to evaluate the finalists, you may ask for a trial version of the LMS software or access to a sandbox installation where you can explore the finalist products. Hands-on exploration will give you a better sense of the user-interface design, features, and capabilities of the product. Customer references You may ask for customer references from each vendor. By checking with existing customers, you can find out more about the product, the vendor s responsiveness, problems encountered and how they were resolved, lessons learned, and more. Be aware that vendors are likely to give you the names of their most satisfied customers. However, every customer has experienced bumps with their products. So, be inquisitive. Ask how their last upgrade went and what they would most like to see improved in the product. Financial health Finally, you may want to check financial reports (public company financial reports are readily available online, and for private companies you may find information from Dun and Bradstreet. You may also want to check information from industry analysts on the financial health and market share of the products you are evaluating. Some well-regarded learning-technology industry analyst groups include
Gartner Group, Forrester Research, Brandon Hall Research, and Bersin by Deloitte. Learning Solutions Magazine regularly carries industry news and articles. The strength of your vendor s financial position and customer base helps ensure that the vendor will support your product well and will continue to evolve and improve with sufficient funding for product research and development. Step 5: Product selection The final step is to settle on a product. If possible, you can place your organization in a strong position to negotiate the best pricing and terms by selecting two or three finalist products, any of which will meet your needs. LMS vendors are eager to gain your business. To an LMS vendor, each sale represents a multi-year revenue stream. Send a request for proposal (RFP) to each of your finalists asking for pricing quotes, implementation timeframes, and support options. Be sure to ask for hosting options and pricing if you have elected to purchase a cloud solution. Work with your purchasing or procurement departments, if appropriate. Compare the responses, and don t be afraid to go back to each vendor to ask them for a better price or more feature options comparable to incentives the other vendor finalists are offering. For clarity, Table 1 explains the differences between an RFI that you use in the product evaluation phase and an RFP that you use in the product selection phase. Table 1: Differences between an RFI and an RFP Request for Information (RFI) Request for Proposal (RFP) A document that lists all of your LMS requirements along with two to six questions per requirement. You ask a short list of vendors to respond to each question in the RFI. Responses should include thorough answers to all questions. Answers should describe how the vendor s product addresses your requirement. While you may ask for general pricing information, getting a final price for your organization is usually not part of an RFI. You and your colleagues review and score the responses to determine which products best meet your requirements based on the information provided by each vendor. Based on the result of the RFI and other evaluation activities, these finalists are the potential candidates for the RFP. A document that lists all of your requirements, along with the size of your learner audience, systems that need to be integrated with a new LMS, and whether you plan to migrate data from a legacy LMS. You ask a few vendor finalists to submit a proposal to sell you their product. The quote should include licensing fees, hosting fees (for cloud solutions), maintenance and support fees, implementation costs, and the costs of any systems integrations, data migrations, product customizations, or other professional services needed to meet your requirements. Of course, the price will likely be subject to negotiations with the vendor you select. Concluding thoughts Too often, organizations make technology decisions without adequate investigation of the strategic, operational, and technology drivers for their product choices. A thorough needs analysis engages
stakeholders and sponsors throughout your organization to create a big-picture view of your learning management system needs. Clearly defined and documented requirements capture those needs and form the basis of your product selection criteria. A funnel approach narrows hundreds of products down to the handful that are best matched to your requirements. Use-case demonstrations and sandbox exploration provide evidence of a product s capabilities that help you either validate or debunk vendor promises before you make a commitment. Getting bids from two or three vendors whose products are equally good candidates for your organization positions you to get the best terms and/or feature set for your money. Although this process may take more time and effort than you may have anticipated, it helps ensure that you make the optimum decision and end up with an LMS that is best suited to your needs. It has side benefits as well. Stakeholders who are involved in evaluation activities tend to have more ownership of the resulting decision and become acclimated to the product in advance of implementation. Sponsors have increased confidence in the decision because of the due diligence involved in the process. The learning organization builds relationships with its customers, sponsors, stakeholders, and IT, all of which helps set the stage for implementation. Buying an LMS is not easy; just ask all the people and organizations who ve tried (some more than once). It requires due diligence and smart, strategic thinking. Whatever your organization s experience is or has been, you stand a much better chance for success and satisfaction with your product if you follow proven practices in evaluating and selecting an LMS. Learning Solutions Magazine, 2013 The elearning Guild