Fostering Market Efficiency in K-12 Ed-tech Procurement

Transcription

1 Fostering Market Efficiency in K-12 Ed-tech Procurement A Report from Johns Hopkins University to Digital Promise in partnership with the Education Industry Association Jennifer R. Morrison, Ph.D. Steven M. Ross, Ph.D. Roisin P. Corcoran, Ph.D. With Reid, A. J., Ph.D. Center for Research and Reform in Education (CRRE) Johns Hopkins University September 22, 2014

2

3 CONTENTS EXECUTIVE SUMMARY... 4 Method... 5 Results... 5 Action Point I: Allotment of Funding...6 Action Point II: Assessment of Needs...6 Action Point III: Discovery of Ed-Tech Products...7 Action Point IV: Evaluation of Ed-Tech Products...7 Action Point V: Acquisition of Selected Ed-Tech Products...7 FOSTERING MARKET EFFICIENCY IN K 12 ED-TECH PROCUREMENT LITERATURE REVIEW Method Participants Instruments...21 Procedure...21 Results Action Point I: Allotment of Funding Action Point II: Assessment of Needs Action Point III: Discovery of Ed-Tech Products...31 Action Point IV: Evaluation of Ed-Tech Products Action Point V: Acquisition of Selected Ed-Tech Products Ways of Improving Ed-tech Procurement Comparison Between Smaller and Larger Districts Relationship Between Company Size and Provider Responses Cross-Validation Results CONCLUSION Recommended Tools and Supports for Improving Practice What Have We Learned? Nine Notable Takeaways Suggestions for Further Research...61 REFERENCES Appendix A: Core Sample District Characteristics Appendix B: Provider Participant Characteristics Appendix C: Secondary Sample District Characteristics Appendix D: Survey Items Appendix E: Interview Questions...73 Appendix F: Descriptive Statistics and Response Frequencies for Research Question Appendix G: Descriptive Statistics and Response Frequencies for District Size Comparison Appendix H: Descriptive Statistics and Response Frequencies for Provider Size Comparison Appendix I: Descriptive Statistics and Response Frequencies for Cross-Validation Fostering Market Efficiency in K 12 Ed-tech Procurement 3

4 EXECUTIVE SUMMARY In K 12 education, the identification, evaluation, and acquisition of educational technology products (herein referred to as procurement) are essential but can often be a highly difficult process. But is the procurement process as onerous and challenging as certain stakeholders, particularly providers, describe? What are the most prevalent and serious barriers for end-users, such as principals and teachers, to obtain the products they need most? What are the effective practices that make procurement relatively smooth and reportedly successful in some school districts and for some providers? The purpose of this study was to address these issues from the perspectives of diverse groups of educators and providers, using a mixed-method evaluation design. The specific research questions that guided the study were as follows: 1. What does the K 12 ed-tech procurement process for student-facing tools and applications that contribute to personalized learning currently look like for: a. district and provider stakeholders b. different types of stakeholders within school districts c. different sizes of districts (smaller districts, larger districts); and d. different types of providers (smaller firms, larger firms)? 2. What does, or would, a highly efficient K 12 ed-tech procurement process look like across those same dimensions? 3. What are the constraining conditions (i.e., obstacles) that do or could get in the way of an efficient ed-tech procurement process? 4. What are the enabling factors (i.e., best practices) that do or could facilitate an efficient ed-tech procurement process? An Operational Framework Based on our review of the literature and perspectives gained during data collection for this study, we present an operational framework that depicts five key Action Points of typical procurement processes in school districts. These Action Points are interactive and often overlapping rather than an invariant linear sequence. For present purposes, however, they provide an operational framework for relating results to key procurement needs that occur at one time or another along the pathway from the allotment of funding to the acquisition of selected products. Action Point I: Allotment of funding for ed-tech product acquisitions. The amount of funding available to purchase ed-tech products directly influences the degree of participant involvement in subsequent phases. Action Point II: Assessment of needs for ed-tech products. By knowing where and how ed-tech support is needed, school districts aptly put the horse before the cart, so that the search for products (Action Point III) has direction and purpose. Action Point III: Discovery of ed-tech products that address priority needs. This phase exposes school districts to a variety of ed-tech products that perform different educational functions, thus, creating opportunity to further investigate those appearing to offer the best fit. Action Point IV: Evaluation of product quality and effectiveness: Here, by examining evidence about the product, obtaining peer recommendations, observing demonstrations, and conducting pilots (quick-turnaround try-outs), school districts obtain information to guide selection of the product(s) likely to most reliably and effectively support instructional needs and goals. Action Point V: Acquisition of selected products. In this culminating activity, the products selected are acquired through completed purchasing agreements with the vendors. The processes involved may be quite straightforward and rapidly completed, or complicated and slowed by district (e.g., school board) or external (state or municipal) policies. 4 Fostering Market Efficiency in K 12 Ed-tech Procurement

5 EXECUTIVE SUMMARY I. Allotment of Funding II. Assessment of Needs III. Discovery of Ed-Tech Products IV. Evaluation of Products V. Acquisition of Selected Products Method District participants were recruited on a voluntary basis through membership in various professional organizations. The core district participant sample consisted of participants from 54 districts in 31 states. Included were superintendents (n = 43), curriculum directors and related positions (n = 44), business officers and related positions (n = 42), technology directors and related positions (n = 59), and principals (n = 103). Ed-tech providers (n = 47) were also recruited on a voluntary basis to participate in the study. Surveys and interviews were developed by the research team for each of the participant respondent groups (superintendents, curriculum directors, business officers, technology directors, principals, providers). Questions focused on such topics as perceptions of the overall procurement process, sources of information for evaluating products, stakeholder involvement in procurement, financial factors, challenges and enabling factors to procurement, and potential tools and information to improve procurement. Results From Start to Finish: Overall Perspectives Results indicate that while few district stakeholders are satisfied with the efficiency and success of the procurement process, the majority views it as working sufficiently in most respects for acquiring the selected ed-tech products. small percentages of district respondents in any group expressed negative (dissatisfied) views about the procurement processes or their effectiveness in meeting contemporary needs. Districts are grappling with much larger issues such as teacher evaluations, principal recruitment, and implementing Common Core Standards curricula, and thus have less skin in the game than do providers. The latter, in sharp contrast, are extremely dissatisfied, with close to threefourths conveying negative views about these respective questions. Providers also feel dissatisfied with the ease of communicating with districts regarding their products and procurement requirements. These findings foreshadow the differing experiences and reported barriers that these two major stakeholder groups (districts and providers) experience at each of the Action Point phases. Key Findings: Members of all district participant groups have varied impressions of the procurement process. Few view it as working highly effectively or efficiently; however, most feel that it generally operates well enough to acquire the products they want to purchase. Providers are generally dissatisfied, mostly due to challenges with gaining exposure for their products and communicating with district decision-makers. Action Point I: Allotment of Funding The most frequent challenge expressed in open-ended survey responses, and most strongly emphasized by superintendents, related to funding and financial concerns. District participants referenced the cost of items, as well as reductions in the technology budgets for school districts. Fostering Market Efficiency in K 12 Ed-tech Procurement 5

6 EXECUTIVE SUMMARY Key Findings: The apparent tightening of school district technology budgets nationally places increased pressure on providers to market their products in an already highly competitive and stillgrowing industry. For school districts, there is increased pressure to limit purchases to the applications judged more essential overall and less to those that would be more exploratory or supplemental. Although this study did not focus directly on budgeting, an emergent finding was that many districts view ed-tech software products as part of an isolated, supplemental-type budget, rather than intrinsic to broader budget allocations for curriculum and instruction, special education, and so forth. Action Point II: Assessment of Needs Encouragingly, the majority of participants interviewed conveyed that educational goals drive the selection and acquisition of ed-tech products. However, other than determining where student achievement gains were most needed, the use of formal, systematic processes of identifying instructional needs at the school or classroom levels did not appear highly prevalent. District participants rated the technology director as having the greatest involvement in procurement. Notably, teachers were rated as having only a moderate involvement, and principals as slightly more involved than teachers. Key Findings: Nearly all ed-tech products are acquired based on some type of needs assessment. Needs assessments appear to be mostly informal and focused on bolstering student achievement in identified curriculum areas. Formal needs assessments (e.g., surveying teachers and principals, using rubrics or rating scales to determine priorities, or convening review teams to collect data and share findings) seem much more rare. While teachers and principals arguably have the sharpest insights into instructional needs, they appear to be only moderately involved in this capacity. Action Point III: Discovery of Ed-Tech Products Superintendents and other district respondents emphasized the difficulty of sorting through the increasingly large number of products available. This challenge was echoed by providers in terms of gaining district awareness. Key Findings: Discovery is a serious challenge for both school districts and providers, especially for less established providers. Districts in general do not have the capacity (personnel or time) to conduct thorough searches of what is available. Providers in general do not have the capacity or means to broadly expose districts to their products. To the extent that discovery is restricted to a few products that districts happen to identify through searches, peer recommendations, or vendor-driven marketing efforts that reach them, acquiring the most effective ed-tech solutions is largely a hit-or-miss undertaking. The challenges of the discovery process clearly appear to contribute strongly to providers general frustration with contemporary procurement processes. 6 Fostering Market Efficiency in K 12 Ed-tech Procurement

7 EXECUTIVE SUMMARY Action Point IV: Evaluation of Ed-Tech Products District participants overall reported a moderate-to-extensive reliance on pilot tryouts (mostly informal demos and field tests) within the district for ed-tech procurement. Providers, however, had mixed reactions to their opportunities to conduct pilots in district schools. Peer recommendations and rigorous evidence were other key sources of evaluation information. In general, survey responses by district participants were mixed regarding opportunities for products from less established providers or brands to be acquired. In interviews, some participants expressed concerns that the provider may not offer the same level of customer service and support as a more established brand. Further, some expressed concern that emerging providers may have less developed products as compared with established and recognized brands. Participants did acknowledge that some emerging companies have more innovative and engaging products and are willing to adapt product more readily than established brands. Key Findings: There are no readily accessible sources of rigorous evidence on the effectiveness of the vast majority of ed-tech products. As a result, school districts largely depend on recommendations from peers and from their own teachers and principals who have familiarity with the products. Local evidence frequently comes from participation pilot studies of selected products. Because the pilots are informal (e.g., demos, brief try-outs, committee reviews), accuracy of the conclusions reached about product quality may be questionable. Most providers have nonrigorous evidence (from in-house evaluations or data analyses) on product effectiveness. Given providers interest in selling their products, school districts are hesitant to rely heavily on such information. Overall, both providers and school districts are frustrated by what might be described as largely a hit-or-miss approach to the vetting of many ed-tech products. Action Point V: Acquisition of Selected Ed-Tech Products Superintendents, technology directors, and business officers were mostly satisfied with the length of time of procurement. directors, however, were slightly more dissatisfied. Predictably, given costs and the uncertainty involved with waiting for contracts to be approved, providers (with 73.3% negative) were highly dissatisfied. District participants (except principals) generally disagreed that decentralized school procurement processes are desirable. In interviews, district administrators preferred a balanced process of allowing some amount of school-level purchasing with district oversight. Regarding the acquisition modes employed for procurement, participants indicated a moderate reliance on both formal, competitive processes (e.g., RFP) and noncompetitive processes (e.g., sole source or other) in survey responses. Interviews with district participants revealed that the primary factor resulting in a formal process when mandated by state guidelines was the dollar amount of the purchase. Interviews with district participants also revealed mixed views on the use of RFPs. Benefits of using a more formal RFP process included obtaining improved pricing through competition and encouraging districts to clearly establish product requirements prior to evaluating products. District participants also noted, however, the increased workload involved in crafting the RFP, as well as the effort required by providers to respond to the RFP. Fostering Market Efficiency in K 12 Ed-tech Procurement 7

8 EXECUTIVE SUMMARY Key Findings: There was moderate satisfaction with current timeframes by most district participant groups, especially superintendents. District participants are moderately satisfied with or not overly concerned about school board, municipal, or state restrictions. Although RFPs require additional time and work by district personnel, they were viewed by some district interviewees as beneficial for vetting providers and their products. Predictably, providers for whom, more than any other participant group, time is money were much less satisfied than were district participants with procurement timeframes and additional compliance criteria (school board, municipal, state, RFPs) that extend time and costs for them to sell their products. Neither decentralized nor cooperative purchasing received strong support by most district participants as desirable purchasing strategies. Not surprisingly, principals were favorable toward decentralized decisions (i.e., greater school autonomy). Comparisons Between Smaller and Larger Districts Supplementary analyses were conducted to answer research question: What does the ed-tech procurement process look like for district participants within smaller districts as compared to larger districts? In general, and not surprisingly, the procurement process was perceived as smoother and more inclusive by participants in smaller districts than by those in larger districts. Inferably, for the small districts, there appears to be greater opportunity for different stakeholders to communicate about needs and for endusers, such as teachers and principals, to influence discovery and acquisition. As district size increases, reliance on the business office and more formal acquisition processes tends to increase. s in smaller districts, probably due in part to their stronger roles and the quicker turnaround in purchasing desired products, are more likely to regard the procurement process as meeting contemporary needs. Key Findings: Both larger and smaller districts appear to struggle with similar challenges along the procurement pipeline. There were relatively few differences across all the survey item comparisons. Larger districts seem to struggle more, given the more extensive bureaucratic structures and numbers of stakeholders (teachers, principals, administrators), with achieving an inclusive, collaborative process that gives end-users a substantive voice (particularly in needs assessments and discovery). Large districts have potential advantages in their capacity to conduct both pilots and more intensive vetting of providers and products. Relationship Between Company Size and Provider Responses Supplementary analyses were conducted to answer the research question: What does the ed-tech procurement process look like for provider participants within smaller firms as compared to larger firms? Of the 55 comparisons conducted, only three items showed differences between small and large providers. Small providers perceive somewhat more end-user involvement in the procurement process. Both groups are dissatisfied with both the time required for purchasing and with whether the procurement process meets contemporary needs. Small providers are generally more apt than large providers to see the marketplace as tough to penetrate. 8 Fostering Market Efficiency in K 12 Ed-tech Procurement

9 EXECUTIVE SUMMARY Key Findings: Company size did not relate strongly or consistently to perceptions of procurement. Conclusions: Here, we integrate results from the multiple stakeholder surveys and interviews to discuss the main findings. As an organizational scheme, we pose four questions that appear central to school districts needs and activities in procuring ed-tech products. Consistent with the Action Point framework used throughout this report, the questions address, respectively, a focus on needs assessment, discovery, evaluation, and acquisition. 1. What ed-tech product do we need? 2. What ed-tech products are available for our needs? 3. Which available products are the best fit? 4. Can we acquire the products that we select in a timely manner? What do we need? Results indicate fairly consistent reliance by school districts on conducting some type of needs assessment, and moderate satisfaction that the identified needs are ultimately satisfied. Needs, however, were described in interviews in a global rather than specific manner: Raising test scores in a particular subject or facilitating data management, enabling authoring of lessons, and so forth. The specific types of ed-tech support required, such as a tutorial program in math that involves parents in checking students work, or a data management system that includes rubrics for assessing project work were rarely mentioned, although we suspect that some districts do conduct more granular analyses. Conclusions: Some type of needs assessment is frequently conducted at the outset of procurement, but in many cases there is uncertainty about the specific ways that ed-tech products would be used, and what attributes they should have, to address instructional needs. End-users are less involved in the process than providers and district participants (particularly principals) would prefer. Recommendations: Districts would likely benefit from the creation of guidelines and models for structuring instructional needs assessments to ensure that selections and acquisitions are linked directly to priority areas. Districts would likely benefit from guidelines for matching instructional design features of edtech products to needs assessments with regard to learning goals, instructional theory, learner analysis, user interface and support, alignment with curriculum, and so on. Districts should more integrally involve end-users in defining needs more specifically and operationally (e.g., To teach problem-based learning more effectively in STEM classes ). In communicating with district stakeholders, providers should increase awareness of current and future instructional needs so that they can adapt product design and market accordingly. Fostering Market Efficiency in K 12 Ed-tech Procurement 9

10 EXECUTIVE SUMMARY What is Available? The rapidly growing number of products available and lack of a reliable resource to aid in product discovery was a concern expressed by both district and provider participants. Without a central source to identify products, the evaluation of products proves to be a challenge for districts of all sizes. District participants desire product information independent of that offered by providers, and currently lack an efficient means of comparing products and making informed decisions based on objective information. Some districts are aware of various networks or websites that list products, but at this time do not see them as necessarily representing full ranges of products, the best products, or assurance that the products are supported by evidence or peer review. Conclusions: There are likely to be multiple ed-tech products that can potentially support particular instructional needs, but district participants lack an efficient, practical means to learn about what the options are. Because instructional needs are often only generally defined ( Raise fourth-grade math scores ), even within a particular curriculum area, there are a plethora of product genres (e.g., full curricula, tutorials, games, presentational, whole-class vs. personalized, etc.) which complicates discovery even further. Recommendations: Districts should increase use of Requests for Information (RFIs) to alert providers as to product needs and to produce information about potentially relevant ed-tech solutions. Districts should increase use of available dissemination and networking websites, which identify ed-tech products and where they are being implemented for instruction. Districts and providers would likely benefit from an online ed-tech products Ed-tech Product Information Exchange that would (a) list and describe available e-tech products, (b) report formal research studies on products and their results, (c) report pilot studies on products and their results, (d) report consumer reactions to ed-tech products, and (e) facilitate networking and communications between providers, districts, and evaluators. Importantly, this website would serve as a one-stop-shop that combines product information, pricing guidance, and so forth with evaluation findings and customer satisfaction reports. Which Available Products Are the Best Fit? For evaluating available ed-tech products, the present results showed fairly high reliance by school districts on external peers and internal end-users perceptions about the quality of particular ed-tech products. The latter group s recommendations, in turn, appear to be largely based on direct interactions with the products via informal piloting activities. District participants, especially superintendents and principals, also conveyed on the survey and in interviews that rigorous evidence of product effectiveness (where available) was another important source of information for product selections. Conclusions: Some type of evaluation strategy is almost always used by districts in the selection of products. Evaluating potential selections, however, is complicated by several factors: (a) lack of available, credible evidence on product effectiveness, (b) uncertainty about the criteria on which to evaluate products (Student achievement gains? Usability? Professional development support?), and (c) the capacity of districts to conduct their own evaluations (i.e., pilots can be time-consuming and costly). 10 Fostering Market Efficiency in K 12 Ed-tech Procurement

11 EXECUTIVE SUMMARY A valuable source of evaluation evidence for districts is peer recommendations because trusted stakeholders from demographically similar districts can provide candid firsthand impressions of their experiences and satisfaction with a particular ed-tech product. Peers can answer questions about impacts on teachers and students, and discuss the quality and level of the provider s support in offering training and technical assistance where needed. A second valuable source of evaluation evidence comes from conducting structured pilots that collect satisfaction and implementation data from teachers and students, and often, preliminary student achievement outcomes. Pilots provide a district with a firsthand test drive of selected products, so that their potential for wider adoption can be judged. A third valuable, but seemingly more limited, source of evaluation evidence is results from rigorous studies. Having rigorous evidence is certainly an advantage for differentiating an ed-tech product from its competitors, but such studies tend to be costly for providers to commission, and, once performed, may lack relevance to contextual conditions at many school districts or to current educational policies (e.g., Common Core State Standards). Products that facilitate teachers work in planning and delivery instruction, such as managing and interpreting data, evaluating portfolios, and authoring lessons, are not likely to demonstrate measureable effects on student achievement (at least not nearly as quickly and strongly as instructional programs can). Similarly, instructional products used as supplements to regular curricular for relatively small segments of learning time per week may be helpful to students and teachers but produce only small effects on test scores. Recommendations: Districts and providers would be likely to benefit substantially from having guidelines for conducting formal pilots to facilitate discovery and evaluation. The present findings indicate that pilots are highly regarded by all stakeholder groups, but effective strategies are needed for (a) matching products to be piloted to teachers based on interest and instructional needs; (b) funding the pilots; (c) collecting both qualitative and quantitative data on implementation, satisfaction, and educational outcomes; and (d) analyzing, interpreting, and using the data for product evaluation and development. Separate but inter-related guidelines for providers and districts would be invaluable. Districts would be likely to benefit from a national ed-tech product website ( Information Exchange ), as previously proposed for facilitating discovery, to make findings from pilots, rigorous studies, and peer experiences much more accessible. Providers would be likely to benefit from guidelines for how to acquire credible evidence for their products (e.g., engaging third-party evaluators for design reviews, case studies, experimental comparison group studies, etc.). Providers should seek opportunities to collect third-party (independent) evidence from evaluation studies (both treatment-control group comparisons and case studies) to differentiate and support their products. Districts would be likely to benefit from guidelines for accessing and evaluating evidence of effectiveness. Our findings indicate frustration and confusion regarding what constitutes meaningful evidence and how to interpret and weigh evidence from different sources such as rigorous studies, pilots, peer recommendations, and provider studies or data. Can We Acquire the Products that We Want in a Timely Manner? District interviewees conveyed as a general perspective that the procurement of ed-tech entails an increased number of options in the marketplace, less defined criteria for evaluation, and the involvement of more stakeholders than when procuring hardware or textbooks. These components tend to increase the complexity of the process and the timelines involved. A formal, competitive (e.g., RFP) Fostering Market Efficiency in K 12 Ed-tech Procurement 11

12 EXECUTIVE SUMMARY process appears to be employed slightly more often than an informal process, and district participants were moderately satisfied with both forms. Smaller districts, though, indicated less reliance on a formal process compared to larger districts. The use of an RFP, when mandated for ed-tech acquisitions, is triggered by larger dollar purchases (the cut-offs for which greatly vary across districts), but are preferred by some district stakeholders (especially business officers) as ways of more carefully vetting products and comparing costs and services. While varied views were presented by different district respondents within and across groups, there was moderate acceptance (and certainly not strong concerns about) the timeliness and nature of purchasing processes. In general, most district participants believe that they can usually obtain desired products in reasonable time once the necessary prerequisite steps (e.g., for needs assessment, discovery, evaluation) are taken. Little interference was seen from states, municipalities, or school boards. Providers, on the other hand, viewed the purchasing process as requiring a protracted timeline, and if requiring a formal RFP process entailing too much effort and cost for an uncertain result. Conclusions: District purchasing policies do not, in general, extend product acquisition time. RFPs and other competitive processes have value in many cases for districts to more thoroughly vet products and obtain competitive pricing. Cooperative purchasing with other districts is an appealing concept in theory, but in reality, many districts see themselves as having specialized needs, valuing independence, and not wanting to spend extra time working out arrangements with other districts. Decentralized purchasing is appealing for school-based adaptations, but is viewed by superintendents and other central office administrators as generally undesirable (e.g., loss of quality control, fragmenting instructional practices, complicating purchasing district-wide). Governance from state, municipal, or school board policies have potential to interfere with or delay purchasing (and sometimes do), but in general do not appear to have a significant negative impact on ed-tech product acquisition. Recommendations: Providers and districts would likely benefit from district guidelines and other policies that clarify acquisition processes, the use of RFIs, contracting requirements, RFP policies, and expected timeframes for different types of purchases. Providers and districts would likely benefit from policies and specific acquisition strategies that move more directly from successful pilots to timely and broader-based purchasing without the need for new RFPs. Providers and districts would likely benefit from expedited or simplified RFP processes and forms tailored to ed-tech instructional products. Creating templates or checklists of model RFPs, RFIs, and contract terms used by districts will be helpful to buyers and sellers. Providers and districts would likely benefit from educating school boards and states about the unique conditions and needs for acquiring ed-tech software vs. hardware products. 12 Fostering Market Efficiency in K 12 Ed-tech Procurement

13 EXECUTIVE SUMMARY What Have We Learned? Nine Notable Takeaways Although the results reported in the present study coming from six surveys and over 50 interviews intend to provide a comprehensive examination of ed-tech procurement practices, a risk for readers is getting lost in the details and losing perspective of what is most important and impactful. What constitutes importance and impact is, of course, somewhat in the eye of the beholder. Accordingly, we offer with that caveat in mind, what we as the researchers and authors believe emerged from the study as the most significant (impactful, interesting, provocative, supported) takeaways. We present them below in no particular order. Discovering what is out there. The most significant challenge of procuring ed-tech products, as compared to traditional instructional products like textbooks, occurs in the discovery phase. The market is flooded with products across all content areas and many application types. Presently, providers struggle to gain visibility for their products, and school districts struggle to learn what is out there. Involving the end-user. End-users (the practitioners in our classrooms) are often only marginally involved in the identification of instructional needs and selection of products. Unlike textbooks, which universally engage teachers in the same manner (making assignments, explaining content, guiding lesson planning), ed-tech products require much higher and more varied teacher interactivity. Learning how to use an ed-tech product can take substantial time for teachers. Implementing the product in the classroom changes the nature of teaching and other instructional activities. Seemingly, those who are so directly affected by the product should have a more central role in selecting and test-driving it before it is purchased. Knowing what you need. Assessments of instructional needs are most frequently surface in nature, and thus do not identify the specific types of support and product attributes that best support instructional goals. Needs assessments must be more structured and precise. Pilots as a means of discovery and evaluation. Pilots appear to have strong potential for districts and providers to collaborate in field-testing products for broader district adoption. Pilots that are structured and rigorous generate evidence about product efficacy that is not only useful locally but also to other districts considering the same products. National ed-tech product information exchange. A potentially valuable tool to districts and providers would be a national website ( Ed-tech Product Information Exchange ), which extends existing networks and online information sites by providing comprehensive descriptions of products in different areas, evaluation evidence from pilots and rigorous studies, and consumer satisfaction reports. Such a website would greatly facilitate discovery and evaluation, the two Action Point domains where districts and providers struggle most. Similar district viewpoints. For the most part, district participants assuming five different roles (superintendent, curriculum director, business officer, technology director, principals) perceive the procurement process and its strengths, weaknesses, and needs in the same way. While there is less intra-district communication than desired, there are not major disagreements between stakeholders or major dissatisfactions. Improvements in virtually all areas, however, are desired to increase efficiency and success in obtaining and then using the right products. Challenges for providers in a buyers market. Providers, overall, are dissatisfied with many aspects of the procurement processes the time delays, RFPs, communications with district stakeholders, getting products discovered. The root cause, it seems, is that there are so many products competing in a pronounced buyers market, and uncertain pathways for identifying districts needs and becoming noticed and differentiated from the competitors. Completing lengthy RFP applications and waiting for decisions further add to the cost and frustrations. For newer providers, all of these concerns intensify. Fostering Market Efficiency in K 12 Ed-tech Procurement 13

14 EXECUTIVE SUMMARY Only small advantages for small districts. Smaller districts can maneuver more easily through various procurement stages than larger districts due to having less complex bureaucracies and more immediate contact with administrators and end-users. However, for the most part, small and large districts appear to experience procurement very similarly with regard to processes, challenges, and needs. Finding meaningful evidence. Reliance on evidence of product effectiveness in making product selections is highly valued by nearly all district stakeholders. But there are misunderstandings about what constitutes reasonable evidence in the first place and frustrations in finding credible evidence. Few providers (except for the very large companies) can afford, or win large grants to fund a randomized control trial (RCT) to prove product effects on student learning. Products that are used in limited dosages or time periods, as supplementary instruction, or to facilitate teacher grading, data management, or lesson planning may not demonstrate measurable gains in any study, but still have considerable value to students, teachers, administrators, and parents. Therefore, as a practical alternative to complex and expensive RCTs and other highly controlled research studies, credible (third-party) evidence for judging project fit and potential efficacy can come from pilots, case studies, and small comparison-group designs. Yet, few providers, it seems, seek opportunities for their products to be evaluated in the latter types of studies. Suggestions for Further Research While this present study offers a breadth of information on ed-tech procurement gleaned from districts and provider participants, additional research is necessary to further explore this topic and provide recommendations to improve efficiency. We offer the following suggestions for additional research. A quantitative analysis of cost triggers and state policies for smaller and larger districts, resulting in competitive vs. non-competitive contracting. A study of how teachers and principals participate in the procurement process and the implications of greater or lesser involvement on satisfaction with and usage of acquired products. A study of how pilots are conducted by diverse districts, and the degree to which piloted products gain advantages for expedited purchasing and scale-up. A study of how providers determine district needs and the degree to which and how they design products accordingly. 14 Fostering Market Efficiency in K 12 Ed-tech Procurement

15 FOSTERING MARKET EFFICIENCY IN K 12 ED-TECH PROCUREMENT In K 12 education, the identification, evaluation, and acquisition of educational technology products (herein referred to as procurement) is an essential but often highly difficult process. While the venture capital sector of educational technology, or ed-tech, products is flourishing, many providers perceive the procurement process to be a closed system of centralized power that presents many barriers to entry for smaller tech developers and start-ups. Further, procurement policies often vary at the macro (federal and state), meso (district), and micro (individual schools and teachers) levels, creating a Wild West of practices (Younie, 2006). And, although there are exemplary cases of successful procurement processes being implemented, best practices and resources are rarely shared, resulting in a constant reinvention of the procurement wheel and a perpetual cycle of dissatisfaction for many stakeholders: product providers, administrators, educators, and ultimately, students. The procurement of educational technology products in K 12 education can be complicated in some school districts by a labyrinth of administrative and legislative barriers; consequently, in those situations, stakeholders sometimes must overcome numerous obstacles in an effort to promote effective teaching and learning with technology. But is the procurement process as onerous and challenging as certain stakeholders, particularly providers, describe? What are the most prevalent and serious barriers for end-users, such as principals and teachers, to obtain the products they most need? What are the effective practices that make procurement relatively smooth and reportedly successful in some school districts and for some providers? To what degree do different stakeholders within school districts, namely superintendents, business officers, technology directors, curriculum directors, and principals, share similar views about what works well, what doesn t work, and what improvements or types of tools and supports are needed to make procurement efficient and effective? The purpose of this study was to address these issues from the perspectives of diverse groups of educators and providers, using a mixed-method evaluation designs. The specific research questions that guided the study were as follows: 1. What does the K 12 ed-tech procurement process for student-facing tools and applications that contribute to personalized learning currently look like for: a. district and provider stakeholders b. different types of stakeholders within school districts c. different sizes of districts (smaller districts, larger districts); and d. different types of providers (smaller firms, larger firms)? 2. What does, or would, a highly efficient K 12 ed-tech procurement process look like across those same dimensions? 3. What are the constraining conditions (i.e. obstacles) that do or could get in the way of an efficient ed-tech procurement process? 4. What are the enabling factors (i.e., best practices) that do or could facilitate an efficient ed-tech procurement process? In the following sections, we first review relevant literature on the ed-tech procurement process, and then describe the study s methodology, including sampling and instruments. In the largest section, results are reported and interpreted from survey and interview data. We conclude by discussing what was learned with respect to the evaluation questions and offering recommendations for improving future procurement practices for school districts, providers, and the primary consumers teachers and students. In the following section, we review selected findings from the research and practice literature to provide background for the present study. For readers interested in a more extensive coverage, we have prepared as part of this research project a comprehensive literature review in a separate document (Lake, 2014). Fostering Market Efficiency in K 12 Ed-tech Procurement 15

16 LITERATURE REVIEW Background The driving force behind educational technology products is personalized learning. As competencies and fluency with computing technologies increase, so does a demand for flexible learning environments, personalization of curriculum, and just-in-time learning. The 2011 Horizon Report (Johnson, Smith, Willis, Levine, & Haywood, 2011) identifies key trends that suggest a shift toward personalized learning in K 12 schools: a re-examination of the educator s role as a coach or facilitator and the expectation for convenient and adaptive learning (Spector, 2013). Consequently, the traditional one-size-fits-all classroom model no longer seems to fit. Instead, the demand for personalized learning has led to unprecedented advancements in instructional technologies. Educational technology in the K 12 and higher education institutions is big business. Globally, ed-tech product spending reached US$13 billion in 2013, up 11% from 2012 (FutureSource Consulting, 2014). Specifically, PreK 12 spending on educational technology was approximately US$7.97 billion during the school year, an increase of 2.7% from the previous year (Richards & Struminger, 2013). Domestically, the Department of Education implemented a federal budget of $71.2 billion for the 2014 fiscal year, a 4.5% increase from the previous year (Center for Digital Education, 2013). These figures suggest that government is supportive of educational technology, both ideologically and financially; surprisingly, then, it is not simply a lack of funding that complicates the ed-tech procurement process that many schools face. In fact, numerous state and federal government-based initiatives have been implemented in an attempt to better understand and transform existing procurement practices. However, these efforts have in no way addressed the contemporary need to place quality ed-tech courseware products in schools to improve teaching and learning. In 2013, the state of Maine led a collaborative effort alongside Hawaii and Vermont to purchase education technology products jointly. The Multi-State Learning Initiative has developed a standard cooperative state purchasing agreement that helped to leverage better pricing, technology support, and service in exchange for more participating institutions and larger contracts. The initiative provides an alternative to states and districts trying to make technology purchases one at a time (Cavanagh, 2013). Although cooperative purchasing is not a recent development, it has gained popularity recently as school leaders have begun to recognize that a common intersection of frustration and misinformation exists for nearly all school business officers. In 2010, the National Education Plan (NETP) was prepared by the U.S. Department of Education s Office of Educational led by Secretary Arne Duncan. The plan was predicated upon the former NETP produced in 2004 and aimed to implement a new approach to research and development (R&D) in education that focuses on scaling innovative practices in the use of technology in teaching and learning, transferring existing and emerging technology innovations into education, sustaining the R&D for education... and creating a new organization to address major R&D challenges. (U.S. Department of Education, 2010, p. x) Specifically, the NETP suggests the development and adoption of a generally accepted definition of productivity as it relates to education (Goal 5.0 Productivity: Redesign and Transform). NETP strongly recommends improvement for existing policies and technologies for managing costs, including those for procurement (U.S. Department of Education, 2010, p. 73). Further, the plan asserts that the Department of Education can and should encourage common standards that will (a) enable cost-tracking measures for productivity improvement, and (b) develop a platform for collaboration and sharing of policies among federal, state, and local entities in an effort to circumvent roadblocks in procurement (U.S. Department of Education, 2010). The NETP s commitment to reforming procurement strategies highlights the urgency of the issue on a national scale. 16 Fostering Market Efficiency in K 12 Ed-tech Procurement

17 LITERATURE REVIEW Investment in digital technologies is also a popular political platform. The ConnectED initiative (2013) was launched by the Obama administration and charged the Federal Communications Commission to take the steps necessary to build high-speed digital connections to America s schools and libraries, ensuring that 99% of American students can benefit from these advances in teaching and learning. The plan also calls for professional development of teachers, an investment in the ed-tech private sector, and a serious effort to provide rural schools and their students with technological opportunities that are equivalent to those of their urban counterparts. Often, political rhetoric expresses a fervent commitment to connecting every student to the digital age, yet the logistics of procurement are often overlooked. European educational systems have been more proactive in addressing procurement challenges in K 12 education. The Finnish National Plan for Information and Communication (ICT) called for a decentralized procurement of products and services, giving individual schools greater autonomy in their decision-making (Leviäkangas, Hautala, Britschgi, & Öörni, 2013). Consequently, this model of procurement leveled the playing field for smaller companies, which opened the market for more competitive pricing and allowed providers to work more closely with their educational partners in a transparent relationship. Unlike Finland, the United Kingdom reformed its educational policy to broaden procurement practices through government policy. This macro approach yielded many problems with this strategy, including the multi-agency nature of the initiatives and their management; disparities of funding; technology resourcing and procurement; ICT training for teachers and impact on pedagogy (Younie, 2006, p. 387). While little has been done in the United States regarding procurement practices, it would seem beneficial to examine other countries policies because many of the barriers and challenges to effective procurement coincide. Barriers to Procurement For educators, administrators, and vendors, numerous barriers complicate and sometimes impede successful integration of ed-tech products. Among other factors, the following conditions usually lead to a difficult procurement climate: schools lack quality judgment of products, districts are unaware of what is cutting edge, brand is a substitution for quality, no support guide exists, there are no inexpensive alternatives to level the playing field, buying consortiums do not work, lower-level decision making equals higher pricing, quality is uneven, purchasing wrongly means bad tech wins, and finally, big sales forces trump smaller startups (Levy, 2013). Of course, not all of these conditions need to exist for a dysfunctional procurement process to prevail. Often, challenges arise from a breakdown in communication between stakeholders. Although most schools and school districts operate differently, and sometimes mutually exclusively, a chain of command emerges when it comes to procurement leadership and decision-making responsibilities. The majority of K 12 districts (64%) mandate that technology expenditures (regardless of the amount) must be approved at the district-level, and most districts (76%) use a committee-based approach to identifying and selecting which products to request for purchasing (Dyrli, 2007). These purchasing committees usually employ at least three different types of administrators: technology/it directors, superintendents, and business/finance/purchasing directors. Although a cooperative approach to procurement decision-making produces a system of checks and balances and shared responsibility, it also may yield miscommunication. Fluctuating school funding and available resources also influence the procurement process. Because national funding programs like the U.S. Recovery and Reinvestment Act and the Race to the Top and Investing in Innovation programs could dictate the quality of ed-tech procurement in school districts, there is a movement away from federal, state, and district funding, and toward individual pupil-based student funding; however, these programs often do not account for rural or underprivileged students (Bailey, Owens, Schneider, Vander Ark, & Waldron, 2014; Davis, 2013; Rivero, 2009). Broadly speaking, the barriers to a successful procurement process can be categorized as (a) school/ district budget and resources, (b) lack of communication among stakeholders, (c) lack of information Fostering Market Efficiency in K 12 Ed-tech Procurement 17