A pre-project planning tool for ICT in ITT projects

A pre-project planning tool for ICT in ITT projects

1 Pre-project planning Impact evaluation How can I judge impact? The four dimensions of scale are: This tool allows project leaders to assess the key mediating and moderating factors before implementing an ICT-based project for teaching and learning purposes. It is not intended to be prescriptive but it is evidence-based, being derived from the evaluation of ICT projects funded by the Training and Development Agency for Schools (TDA) in the period 2004-2008. More details of this evaluation can be found at [insert web link]. In addition to the before project planning requirement described here, you must also choose a suitable overarching evaluation framework for your project so that it can be evaluated at each stage of its cycle while it progresses, after it has completed and before it moves into its next phase. Examples of evaluation tools that capture this pre-, while and post-information within such an overarching framework can be found in the evaluation document here [insert web link]. In the development of an evaluation it is important to ensure that we develop research designs that capture what is important rather than what is measurable. (Coburn, 2003, p9). Coburn identified four aspects of scale she considered vital to the success of projects designed to bring about reform in practices. Scale is usually viewed as the increasing take-up of a particular reform, and in her research on teaching and learning reform in schools she suggests that evaluators should be redefining scale in four dimensions as current views are too limiting and take-up does not indicate change. Depth relates to the impact and recognition that the reform has on the individual, ie, changed their behaviour, understand and use the new pedagogy of the reform. Sustainability is the capacity of the organisation increased to enable all staff to maintain these changes? Spread describes the reform in terms of the understanding and acceptance of its principles and norms, not just to schools but also to local authorities and other collaborative groups. Shift in reform ownership no longer an external reform controlled by a reformer but becomes an internal reform, with authority held by the school and teachers within the school who have the capacity to sustain, spread and deepen the reform principles themselves. Key point: Before we start a project we need to determine what success for that project might look like. However, it is always important to be able to recognise unexpected outcomes and changes in direction and modify your plans accordingly. You will need to employ several evaluation tools to monitor your project.

2 An implementation model for ICT in ITT the framework During the ICT in initial teacher training (ITT) evaluation we identified three generic groups of factors whose interaction determined the success of any implementation. These were: the status of the technology being introduced the capacity for innovation within ITT organisations, and the degree of alignment between the innovation and the needs and concerns of individuals and teams within the ITT organisation. In each of the projects evaluated, the relative importance of these groups of factors varied considerably. But each was needed, like a series of gears, to drive forward the innovation. If any of these three gears failed to mesh or was under-geared and could not exert sufficient pressure on the other two areas, then the innovation was likely to stall and not make a substantive impact. Alignment with needs and concerns Capacity for innovation Diagram 1. An implementation model for ICT in ITT Status of technology These are explained in more detail below. 1. The status of the technology being introduced This includes the technical, social and learning status of a technology. The technical status of a technology refers to its ease of use, its degree of compatibility with other forms of technology and existing ICT infrastructure, and its overall reliability and functionality. In the ICT in ITT evaluation, instances of technical issues preventing an effective implementation were relatively few. This reflected the extent to which staff and teams with existing expertise in ICT training and support were involved in leading projects. The social status of a technology relates to its popular image and the extent to which it has been taken up in wider society. This determines the initial acceptability of a technology and the degree to which individuals are familiar with it, factors which were often key in the initiation stages of a project. Certain technologies vary widely in their levels of take-up across different professional and social groups. Different groups can hold very divergent perspectives on whether certain technologies are good or bad, useful or not useful innovations. Such differences in perceived social status are particularly important to the implementation of ubiquitous technologies within education, rather than those with more specialised educational uses.

3 For example, the differentiated engagement of teachers and pupils with the video facilities of their mobile phones and the moral panics that have surrounded the sharing of happy slapping incidents among pupils indicate how the problematic status of a particular technology can limit its uptake in schools. In contrast, a positive perception could make a piece of technology aspirant in the mind of potential users and so encourage uptake. Its legitimisation as a tool for learning is key here, as is an assessment of the digital habits of all stakeholders engaged with an implementation. These may vary widely between trainees and trainers, for example, and even within the group of trainers themselves. Individuals existing levels of familiarity impacts upon the degree of development and training required during any implementation. Although some providers have had virtual learning environments (VLEs) for some time it is hard to imagine, for example, that the current rapid take-up and enhanced use of VLEs by students and staff within ITT would have been possible without the widespread use of instant messaging and social networking sites, making the uploading and downloading of documents and participation of online discussion forums commonplace. The informal learning about ICT that individuals had gone through outside their professional lives changed the ways they used ubiquitous technologies in educational settings. The learning status of a particular technology relates to its perceived utility and applicability to individuals own learning and its potential to support the learning of others. The learning status of a technology is based on an individual s views of what constitutes learning and effective teaching and learning processes. These views have to be attuned to the potential, or affordances, of each specific technology to support learning. For trainers the starting point for this process of attunement was their initial perception of how well a certain technology fitted with their existing pedagogical approach and curricula, before going on to consider how it might support changes and improvements. Successful innovations from past projects were characterised by the following. Movement from consideration of the technical status of the technology in isolation to recognition of the role played by its social and learning status. Selection of technology which had a high social status. Challenging negative perceptions of a specific technology. Recognition of how differential prior engagement with a technology affects initial take-up and overall training and support needs. Utilisation of those with in-depth understanding of the learning potential of technologies to model to others. 2. Capacity for innovation Individual capacity An ITT organisation s capacity for innovation depends on the levels of skills and understandings of individual staff and trainees, the dispositions and norms in teams and groups, and the commitment of leaders across the organisation. During a project it is important that sufficient capacity is either present at each of these levels or can be developed during the length of the project, and that it is effectively coordinated across all three levels. Developing individual skills and understandings within the relatively short time span of projects and within the busy schedules of trainers, trainees and school-based mentors can be problematic. As such, it is this element that requires a high level of planning to ensure success. Three broad strategies can be adopted to overcome this problem. 1. Choose a technology that is relatively familiar to at least some of the key stakeholders involved. This allows them to provide informal learning and support to those who are less familiar with it.

4 2. Focus support on relatively small numbers of individuals and so create a group of internal experts who can then mentor and coach others informally through the use of the technology. 3. Embed the technology within existing practices and make it available to trainees across the range of practice contexts. This allows its use to be modelled and encourage those with limited experience to experiment with it. Team capacity Developing an evaluative culture among the team Organisational capacity Innovations in ICT were often based in specific teams. In higher education institution (HEI) providers, these were often those responsible for ICT across the curriculum, but other subject and non-subject based teams were represented. Similarly, in SCITTs and EBITTs management teams and groups of trainers and mentors were involved. Developing these teams capacity was based on establishing key dispositions and norms that supported innovation. The most significant of these team norms and dispositions were: a) Willingness to experiment and take risks These are important dispositions in terms of allowing individuals to explore and so attune themselves to the potential of the technology to support learning. Modelling risk-taking behaviour and discussing how it impacts upon them and their practice encourages others to take up the technology. b) An openness around sharing practice and insights among the team Developing supportive norms around sharing practice is not just a question of encouraging staff to share but also setting up processes and structures to facilitate it. For example, the lead developer of one VLE adopted a strategy of giving staff enrolment keys that controlled access to the materials they had developed and placed in their areas on the VLE. They could then decide which colleagues they would give these keys to. This gave them a degree of control and security over what they disclosed while at the same time creating an expectation that they would eventually share their emergent practice. A commitment to evaluating among the project team helped them to discipline the implementation and ensure that it achieved its intended impacts. Developing an enquiry-based approach to project development is useful here. Finding out why a project isn t working is almost as important as evaluating whether it is. At the organisational level, sufficient leadership capacity and commitment from leadership at all levels were key in driving forward a successful innovation. Where additional leadership capacity was required it was often provided by those outside formal leadership and management structures. Successful innovations from past projects were characterised by the following. Recognition of individuals existing understanding of the technology and encouraging them to use this to support others. Sequential and focused support for a limited number of individuals who then mentored others. Integration of opportunities to model different uses of the technologies throughout existing provision and across the different contexts in which trainees operate. Encouraging teams to experiment and take risks. Building an ethos of openness and shared learning. Developing structures and processes to support sharing between project participants. Providing discipline through enquiry and evaluation. Offering leadership support at all levels and creating additional leadership capacity.

5 3. Aligning the needs and concerns of individuals and teams The project implementation planner This group of factors has the greatest influence on whether an implementation is successful or not. The factors that are key to the mobilisation of individuals and teams are the most important. Mobilising individuals and teams is more than a question of engineering some form of engagement with a technology. The technology being implemented and how it is designed to be used must meet a significant number of individuals needs and add substantively to the quality of the core activities of key groups and teams. To mobilise individuals and teams the technology being implemented has to: meet individuals key needs add to the core activities of groups and teams be congruent with the overall strategic aims of the organisations involved, and be underpinned by core educational values. If the implementation of the technology is designed to fulfil these requirements, then the innovation should achieve not only a basic level of participation, but should also engage a critical mass of committed individuals and could lead to it becoming an established way of working in the ITT organisation. Major stumbling blocks occurred when projects were designed to meet needs that were already being met quite adequately by non-technological means, or when the benefits of using a technology were so marginal that only those disposed to this type of solution became engaged. Mobilising the efforts of individuals and the resources of teams meant designing an implementation process around clearly defined needs and specific enhancements to key activities and interactions between trainers and trainees. Successful innovations from past projects were characterised by the following. They had to meet a range of individuals specific needs. In the context in which they were operating, they had to meet these needs more easily, or at less cost, than other existing or potential approaches. They needed to add substantively to the quality of the core activities of key teams of trainers and groups of trainees. They had to have a degree of congruence with the overall strategic aims of the ITT organisation. Included below is a project implementation planning tool, which can be used for pre-project planning. The planner covers the issues outlined in the framework. Alignment with needs and concerns Capacity for innovation Status of technology

6 The Pre Project Planner An example of how to complete a line in the planner is included in each section. Hold your mouse over the column headings to see an explanation. Section 1. The Status of the Technology Issue Moderating factors* Mediating factors* Issues to address* Inputs required to address issues Resources/ days costs* Technical status How easy is the technology to use for your various stakeholders? Is it specialised or ubiquitous? What support is available and how accessible is that support? EXAMPLE. Technology is specialist software for science data logging. Trainees will not have encountered this software before. Schools will not have copies. The supplier offers full online support and two members of staff are expert users. Trainees will need support. Schools will need a copy of software and support. Training for trainees and schools. Ongoing mentoring. Budget increase to supply schools. Time to plan project with schools. 20 days? How compatible is it with other forms of technology? Does it require other technology to optimise performance, eg, digital video/editing software, etc? Does its functionality have the potential to add something new to the learning process? Have you assessed the range of functionality and trialled its desired application? Is the technology reliable? If software will it work on a range of hardware? Is it cross platform or open source? How does it fit with your existing ICT infrastructure? Does it operate outside it? Is it supported by existing IT services or by a third party agreement? Social status Have you analysed the digital habits of your key stakeholders? Do those support the use of the proposed technological implementation? How ubiquitous is a particular technology, for example, within particular age groups? How close is the functionality of educational technology to that of more widely available technology, eg, Facebook versus a virtual learning environment? Learning status How do different stakeholders view the learning potential of the proposed technology? Does it fit in with their view of what constitutes learning and effective teaching and learning processes?

7 Section 1. The Status of the Technology (Continued) Issue Moderating factors* Mediating factors* Issues to address* Inputs required to address issues Resources/ days costs* How well does the technology match or fit the existing pedagogical approach of trainers? Is its application immediately obvious to users, for example, the use of digital cameras in early years settings? Section 2. The Capacity for Innovation Issue Moderating factors Mediating factors Issues to address Inputs required Resources/ days costs Individual Capacity Is the technology familiar to a core of the key stakeholders? For example, VLE functionality will be familiar to some trainees due to engagement with social software but this may not be the case with trainers. Example. Some trainers will be unfamiliar with basic operation and in seeing the potential for practice. Most trainees will be able to use the VLE. Attitudes and skills of some trainers. Mentoring of trainers by experienced staff. Days relating to time needed for mentoring and buddy scheme. Time to set up structures to share expertise. Do you have, or can you develop, a small group of external experts who can mentor others in the use of the proposed technology? Do you have a plan to embed the technology wider than the initial group? Will it be available for trainees/trainers in all their professional contexts? Team capacity Is there a willingness to experiment and take risks that is engendered within the team or organisation? Is risk taking and enquiry a key part of your group ethos? Are there structures in place that encourage sharing and collaborative endeavour?

8 Section 2. The Capacity for Innovation (Continued) Does your project have at its core a clear basis of enquiry? Do you know what you want to improve in your practice? Organisational capacity Is there commitment from key leaders at all levels within the formal organisational structures? Is project progress and output part of formal monitoring structures? Are leaders aware of the project and its aims and potential impact? Is there leadership capacity available outside formal structures? Section 3. Aligning the Needs and Concerns of Individuals and Teams Project alignment Moderating factors Mediating factors Issues to address Inputs required Resources/ days costs Do you know which key needs of individuals will be met by the project? No knowledge in this area. Survey constructed to see whether the technology would meet individual needs in terms of teaching and learning. Potential lack of alignment with individuals needs. Survey and research into main individual needs being met by project. Survey design and implementation plus analysis... Does the project add to or enhance the core activities of groups and teams? If so, how? Does the project align with the strategic aims of the organisations involved? If so, how? Do you have an articulated view of your core educational values? How does this project link to the core educational values of groups and teams?

9 Carrying your project forward: impact evaluation and project planning from inception The pre-planning carried out above should prepare you to either go forward with your project or reconsider its viability. Should you wish to go forward, you will need to implement some form of project planning/impact tool within an impact evaluation framework or model. One such framework is the logical model framework. The pre-planning activity will ensure that you have considered all of the necessary inputs required for project success. A logic model presents a picture of how your effort or initiative is supposed to work. It explains why your strategy is a good solution to the problem at hand. Effective logic models make an explicit, often visual, statement of the activities that will bring about change and the results you expect to see for the community and its people. A logic model helps maintain the momentum of the planning and evaluation process and participant involvement by providing a common language and point of reference. A detailed model indicates precisely how each activity will lead to desired changes. In the UK, the logic frame model for evaluation has usually been used for planning and evaluating large-scale projects in developing countries. However, it is now seen as a relevant model for whenever evaluation is considered. (The Kellogg Foundation, 2000). A logic model is a plausible, sensible model of how a programme is supposed to work (Bickman, 1987, p5). It serves as a framework and a process for planning to bridge the gap between where you are and where you want to be. It provides a structure for clearly understanding the situation that drives the need for an initiative, the desired end-state, and how investments are linked to activities for targeted people in order to achieve the desired results. A logic model is the first step in evaluation. The logic model describes the sequence of events thought to bring about benefits or change over time. The elements of the logic frame model are resources, outputs, activities, participation, short-, medium and longer-term outcomes, and the relevant external influences (Wholey, 1983, 1987). Sundra et al (2003) describe the logic model as a visual link of programme inputs and activities to programme outputs and outcomes, and shows the basic (logic) for these expectations. The logic frame model is an iterative tool, providing a framework for programme planning, implementation and evaluation. p6, and was one of the models reflected on by Giaffric Ltd (2007) in constructing its evaluation model for the Joint Information Systems Committee (JISC) (see the complete model in section 5 of this document). At its simplest, the logic model may be illustrated by diagram 2 below. Input Outputs Outcomes Programme investments Activities Participation Short Medium Longterm Diagram 2. A simple logic frame model In practice the diagram is likely to end up being more complex as each of the areas under consideration is set out in more detail. See diagram 3

10 Program Action Logic Model Inputs Activities Outputs Outcomes Impact Participation Short Term Medium Term Long Term Situation Needs and assets Symptoms versus problems Stakeholder engagement Priorities Consider: Mission Vision Values Mandates Resources Local dynamics Collabration Competion Intended Outcomes What we Invest Staff Volunteers Time Money Research base Materials Equipment Technology Partners What we do Conduct workshops, meetings Deliver services Develop products, curriculum, resources Train Provide counseling Assess Facilitiate Partner Work with media What we reach Satisfaction What the short term results are Learning Awareness Knowledge Attitudes Skills Options Aspirations Motivations What the medium term results are Participants Clients Agencies Decisionmakers Customers Action Behavior Practice Decisionmaking Policies Social Action What the ultimate Impact(s) is Conditions Social Economic Civil Enviornmental Assumptions External Factors Evaluation Focus Collect Data Analyzw and Interpret Report Diagram 3. A more complex logic frame model This diagram is taken from the University of Wisconsin Extension Program found at http://www.uwex.edu/ces/pdande Arguments for the use of this model Arguments against the use of this model It integrates planning, performance measurement and evaluation in one model. A logic frame model describes a programme and its theory of change. It is useful in helping to focus an evaluation. Furthermore, suggest Taylor-Powell and Henert (2008), the process can facilitate team-building and stakeholder buy-in, as well as ensuring that implicit programme assumptions are made explicit. Evaluators have found the logic frame model process useful in a wide range of small and complex programmes and interventions in industrial, social and educational contexts. A logic frame model presents a plausible and sensible model of how the programme will work under certain conditions to solve identified problems (Bickman, 1987). Thus, the logic frame model is the basis for a convincing story of the programme s expected performance. A manager has to both explain the elements of the programme and present the logic of how the programme works. Patton (1997) refers to a programme description such as this as an espoused theory of action, that is, stakeholder perceptions of how the programme will work The logical approach does suggest that it is too simple as an evaluation framework as it appears to assume that all projects are linear. It is perceived as rigid and can lead to the simplification of complex social processes. The structure of the logic frame model suggests that everything will go according to plan. Programme activities, outcomes and goals are all laid out in advance, as are indicators with which to monitor these. As such, there is no provision for a change in project direction nor a space for learning to be fed into project implementation (although a logic frame model can be altered during the course of a project, many commentators note that they are rarely revisited) Earle (2003 p2). The most common limitations include a logic frame model represents intention, it is not reality, and it focuses on expected outcomes so people may overlook unintended outcomes (positive and negative).

11 Comments References Evaluators have played a prominent role in using and developing the logic frame model. This may be why it is often called an evaluation framework. Development and use of logic model concepts by evaluators continues to result in a broad array of theoretical and practical applications, say Taylor-Powell and Henert (2008). Other evaluation frameworks, models and tools are presented within the evaluation document at [insert link] Bickman, L, 1987. The Functions of Program Theory. In L Bickman (Ed), Using Program Theory in Evaluation, New Directions for Program Evaluation, 33, 5-18. San Francisco, CA, Jossey-Bass Publishers CeDARE, 2009, ICT in ITT Survey, Final Report, Wolverhampton, University of Wolverhampton Patton, M, 1997, Utilisation-focused Evaluation, 3rd edition, Thousand Oaks, CA, Sage Publications Sundra, D L, Scherer, J, Anderson, L A, 2003. A Guide to Logic Model Development for CDC s Prevention Research Centre, Centre for Disease Control and Prevention http://www.ojp.usdoj.gov/bja/evaluation/guide/documents/ cdc-logic-model-development.pdf accessed 12 March 2009 Taylor-Powell, E, and Henert, E, 2008. Developing a Logic Model: Teaching and Training Guide, Madison, WI, University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation. http://www.uwex.edu/ces/pdande accessed 12 March 2009 UK Evaluation Society http://www.evaluation.org.uk/resources/glossary accessed 5 March 2009 Wholey, J, 1983. Evaluation and Effective Public Management. Boston:Little, Brown. Wholey, J, 1987. Evaluability Assessment: Developing Program Theory. In Bickman, L (Ed), Using Program Theory in Evaluation, New Directions for Program Evaluation, No 33, San Francisco, Jossey-Bass * What negative factors need to be addressed to ensure that implementation is successful? * What factors are in place that can support the implementation? Are they available for use? * After reviewing the moderating and mediating factors are there any deal-breaking issues that need addressing through a commitment of resources? * What inputs do you need to make to ensure that the issues are addressed? * It may be useful to estimate the cost of the resource required in days and or monetary terms to determine a cost benefit analysis in terms of the final project costs and impacts. You will also need to consider how any costs will be met, from project funding or matched by the organisation.