Lessons Learned from Evaluating Technology-based Products in a Small Business Environment: Balancing Rigor and Resources

Transcription

1 Lessons Learned from Evaluating Technology-based Products in a Small Business Environment: Balancing Rigor and Resources EERS 2014 Stephen James, MPH

2 Overview 3C Institute description/context Product Development model Overview of Projects Approach & Process Lessons Learned Recommendations 2

3 Context 3C Institute For-profit small business (~70 employees) primarily Federal SBIR grants Balance between rigor (need to publish) and what s practical (resources) Multidisciplinary teams: research, software/web development, editorial, graphics, commercialization/branding Company Focus Bridging the research to practice gap 3Cs - Communication, Cooperation, Confidence Create and deliver evidence-based programs and web-based applications to promote health and wellbeing Web-based research support tools, social-emotional interventions for kids, serious games 3

4 Product Development End User Requirements/ System Specification Product Development Product Implementation/ Commercialization 4

5 Project Overview: Assess2Progress (A2P) Decrease effect of disabilities on assessment performance Allows teachers to create/deploy assessments in any subject for K-5 students; integration of Universal Design principles (TTS, visual accommodations) Initial prototyping/mockups, feasibility/usability testing and feedback Team: PI, project manager, assistant project manager, research lead, programmer, graphic artist, video staff, editorial, SME PI SBIR National Institute on Disability and Rehabilitation Research (NIDRR) 5

6 Product Development End User Requirements/ System Specification Product Development Product Implementation/ Commercialization A2P 6

7 Project Overview: Research Implementation and Management System (RIMS) Ease burden on researchers by streamlining recruitment, retention, and data collection and management activities First integrated system of online research tools specifically designed to meet the needs of scientists in the social and behavioral sciences Team: PI, project manager, assistant project manager, research team, web development team, graphic design, video, editorial, SME, commercialization Phase II SBIR National Center for Advancing Translational Science (NCATS) 7

8 Product Development End User Requirements/ System Specification Product Development Product Implementation/ Commercialization RIMS 8

9 Evaluation Overview: Purpose and Approach Purpose: define, improve, and test the products (RIMS & A2P) Secondary: facilitators/barriers to use, usefulness & effectiveness, build interest/buyin, understand targeted end users Utilization-focused evaluation (Patton, 2008) Practical focus (data for CQI), produce usable results Non-prescriptive Involvement of stakeholders in evaluation process 9

10 Evaluation Overview: Purpose and Approach Formative evaluation assess and improve products during development To inform product changes Ongoing assessment of features to determine strengths/weaknesses Documentation of implementation, rapid feedback, flexibility to revise initial plans Logic models and evaluation questions similar between projects 10

11 Evaluation Process During Product Development: Engage Stakeholders Evaluation Planning Evaluation Implementation Analysis & Dissemination Analyze & Disseminate Implement Engage Plan 11

12 Engage Stakeholders Evaluation Process Internal stakeholders: explain evaluation approach/process, obtain buy-in Product end-users: inform product development, obtain feedback, ensure product use Evaluation Planning Product development: Vision document serves as road map for integrating development and evaluation activities Evaluation plan: (outcomes, measures, indicators) and project logic model Project management Timeline Communication plan Decision log Lessons learned document 12

13 Evaluation Process Evaluation Implementation Data collection Online In person (focus groups, questionnaires, product testing) Use of PM/evaluation tools Implementation matrix as formative evaluation tool: tracking evaluation activities as completed, documenting changes/cqi Update vision document, evaluation plan, & timeline as necessary Analyze & Disseminate Mixed methodology in data collection: product evaluation Disseminate results to stakeholders, used to make decisions on product changes 13

14 Lessons Learned 14

15 1) Facilitating evaluation buy-in among internal stakeholders in small business environment Issue: Minimal opportunities to discuss evaluation planning; lack of interest from some team members workload, competing priorities, highly-specialized staff How to address Strategic communication re: evaluation purpose/need (knowing when, how, who) Translate evaluation speak into concrete actions for other team members Tie need for evaluation to common project goal that team needs to achieve Rethink separation between implementation and evaluation (role of PM) 15

16 2) Engaging external stakeholders in the development process Issue: Minimal end user involvement in development due to time & resource constraints How to address System specification: user input to inform product before full development Involve end user through intermittent review at strategic time points (online reviews) 16

17 3) Integrating evaluation with other ongoing project tasks will facilitate evaluation implementation Issue: Limited time and resources to devote to evaluation activities in a small business environment How to address Integrate evaluation with other ongoing project activities PM tools also used as evaluation tools to avoid unnecessary redundancies & save time/energy (e.g., implementation matrix, lessons learned, risk management) 17

18 4) Evaluation team and design needs to be flexible and adaptable, able to quickly respond to project changes Issue: Project scope, timeline, staffing might change rapidly in small business environment due to competing organizational priorities How to address Ensure during evaluation planning that methods/measures to address evaluation questions can be modified along with project without impacting intent of evaluation Use of formative evaluation approach and non-prescriptive framework focusing on CQI and practical use of data allows for greater flexibility in design changes (as these are intended to help shape the process) 18

19 5) Maintaining rigor of evaluation methods with low resources is a fine balance Issue: Sample size constraints often limits analysis methodologies; collection of large amounts of qualitative data is usually not possible How to address Ensure appropriate sample size for proposed analyses Employ mixed methodology in data collection to extent possible to triangulate findings 19

20 6) Role of evaluation team in small business environment is to design strategies to make sure data are practical, useful, and used (to enhance product), while also being publishable (to enhance company reputation) Issue: have to consider what, when, and how to share findings with project team; how to get less rigorous evaluations published/presented How to address Present evaluation data constructively and strategically consider that project staff have various projects, competing priorities, and different subcultures To get evaluation published/presented, be creative in what to focus on and how to approach it 20

21 Recommendations: Balancing Rigor and Resources Balance edu. of internal stakeholders with need to timely execute project Importance of stakeholder participation in product specification and for iterative review for evaluation/feedback Integrate evaluation team and processes into project team and processes Scope alignment with available resources (staff, money, timeline); scope of project may be reduced but evaluation approach can usually be maintained Flexibility/adaptability of project design and team Use of mixed methodology in data collection to extent possible higher quality, more complete data 21

22 Contact: Stephen James 22