CRITERIA USED IN EVALUATING TECHNOLOGICAL MERIT Curricula, Projects and Organisations
DEPARTMENT OF EDUCATION AND SCIENCE GENERAL MINISTRY OF SCIENCE AND TECHNOLOGY POLICY Deputy General Director of Coordination for the National R&D + Innovation Plan Edited by: SECRETARÍA GENERAL TÉCNICA Deputy General Director of Information and Publications Working group: Miriam Pollastrini, Teresa Riaza, Cristina Costa, Victoria Ley, Pedro Vera, Francisco Lizeaga, Javier Bilbao, Jesús Cuartero, Arturo Soto, José Doncel, Francisco Solís, Javier M. Vassallo, Joan Comella, José Ignacio F. Vera Official Publication Identification Number: 651-06-274-4 Copyright: M-34.984-2006 Printing: OMAGRAF SL http://publicaciones.administracion.es
Presentation A country's competitiveness depends directly on its capacity for invention and innovation and the ability of businesses to exploit the results of high-quality scientific and technological research. For this reason, science and technology policy plays an essential role in the development of today's society, whose economy is based increasingly on knowledge and its effective utilisation. Evaluating the quality of proposals and results from scientific activities in line with internationally verified criteria has served as a useful tool for putting Spain at a level commensurate with its investment in scientific research. However, these criteria are incomplete for assessing activities or results of a technological nature. To remedy this deficiency, the Secretary of State for Universities and Research, the National Evaluation and Foresight Agency (ANEP) and the Spanish Foundation for Science and Technology (FECYT) considered it appropriate to establish a general model of assessment that would enable common criteria to be applied based on verifiable results. These take into account the technological merits of researchers and technologists, the quality of applied research and experimental development projects, and the capacity and quality of the activities conducted by some applied research, technological development and innovation organisations. The development of the proposed technology-evaluation model involved representatives from companies, technology centres, offices that transfer research results, universities and public research bodies as well as managers from general national government and from autonomous regions as well as units such as the Deputy General Directorate for the Coordination of the National R&D + Innovation Plan. In addition to the Department of Education and Science's National Agency for Evaluation and Foresight, the Spanish Foundation for Science and Technology, through its contribution to this initiative, has expressed its interest in facilitating the consideration and use of technological merit within Spain's Science-Technology-Business system. Joan Xavier Comella Carnicé Victoria Ley Vega de Seoane Director General of FECYT Director of ANEP
Curriculum vitae of researchers and technologists A. General Curriculum Vitae Information 1) Staff details. 2) Career path: for each type of professional position held, it will be necessary to describe the type of post and in what type of organisation: 2.1) company, specify type and size (including spin-off); 2.2) universities (public or private); 2.3) public R&D centre (and its relation to the administration); 2.4) private R&D centre; 2.5) bodies of the National Health System, and others. In each case, identify the city/country, how long the position was held, verifiable responsibilities (if this is not specified in the job title itself) and a short description of skills acquired. 3) Current professional situation. 4) Degree/qualifications (for each one, indicate the education centre and year in which the qualification was awarded). 5) Areas of specialisation: brief description using key words of specialisation and the areas of current research, development and innovation. 6) Time spent in prestigious foreign or national institutions: for each stay, describe the institution or organisation, country, start and end dates, type of position, activity and subject, tasks performed and skills acquired (and used). 7) Other positions of responsibility: name, organisation, start and end dates, access system, average annual budget, geographical area of responsibility, size of target group, target institutions, and so on.
B. Research, Development and Innovation Activity B1. Participation in projects 1st. Participation in R&D + Innovation projects, secured through competitive calls for applications. The following information is needed: identification of the project (codes, call for applications, title), time commitment, activity personally carried out (in what capacity: researcher, technologist, university graduate undergoing training, manager and/or coordinator of groups and/or multidisciplinary projects, or other), geographical area, duration and budget, significant contributions made. 2nd. Participation in projects executed under the contract formula without third-party evaluation and in other technology transfer activities. In each of the cases listed below, the activity performed, time commitment and mode of involvement, geographical area of the activity, duration and budget for the activity and any relevant contribution must be provided. a) Results from taking part in contracts/research projects, technological development or innovation, by commission or by agreement. b) Results of technology transfer activities, introducing product improvements into the market or ongoing processes. c) Participation in generating spin-off companies based on technological innovations. d) Development of technology skills/abilities, arising from working on major public facilities and/or joint public/private facilities or own capacity. e) Implementing new techniques or procedures, maintaining large installations or complex equipment. f) Delivering technology services: certification, calibration, analysis or other.
B2. Results of the activity 3rd. Industrial and intellectual property and technology transfer results a) Titles over industrial property and/or applications. The following is to be included: publication number and patent application, reference code and title, geographical area, date of application, where appropriate, processing status, and date granted. The contents of claims are also to be summarised, stating whether it is being, or has been, exploited and, if so, stating the name of the company, the formula chosen and the trade name of the products that have resulted, if applicable. b) Know-how and trade secrets. The object of the know-how or trade secret will be stated. The name of the company, the formula chosen for the exploitation and the trade name of commercial products, if applicable, are to be given. An example might be the software (computer programs and databases) that is being exploited. The object, version, authors, copyright holder and number and income from licenses are to be given, distinguishing between whether they are licenses for use or exploitation. c) Plant varieties. The reference code, name and date of the application are to be given along with a summary of the content of the qualities of the variety. This should include an explanation of whether it is being or has been exploited and, if so, stating the company, the formula chosen, and the trade name. In all cases, it should be stated whether the exploitation is on one's own behalf, or through generating an innovative technology-based company of which the author is a promoter (at least, intellectual promoter). And, in this case, detailing the current situation of the result in relation to the company. 4th. Publications for disseminating results Consideration is to be given to publication in international and national journals, books, papers, etc. Where applicable, the impact factor for journals is to be given, or their dissemination, recognition in the scientific community or in the national or international technological community corresponding to the work's subject area.
5th. Participation in informative or dissemination events Involvement in disclosing the results obtained - through conferences, informative and dissemination events (fairs and exhibitions, informative events, lectures, social media and other appearances). The speaker's status, as willing participant or invited guest, must be certified along with type of involvement: communication, lecture in plenary session, lecture in non-plenary session, poster or other. 6th. Acting as an expert Participation as an expert in supervisory, advisory or evaluation work, exhibition curator, or expert. For each event participated in, describe the type of work and identify the type of committee and parent organisation, start and end dates for the work, the verifiable tasks carried out, skills acquired and results. 7th. Theses directed Identification of each thesis directed: type (Master, PhD, other), the educational centre where it was written and defended (if different), the title of the thesis, the name of the person writing the thesis, the start and end date of production and means of disseminating the results. 8th. Collaborations Collaboration with researchers or technologists from other renowned organisations. The relationship will be described, by identifying the type of collaboration (coordinated projects, co-signed publications, networks, co-management, etc), the researchers, centres, aim of the collaboration, period of time, and results where appropriate.
C. Training activity For each activity, the institution and type of activity (regulated course, other types of course, seminars, and supervised or tutored training projects, or other types) need to be described along with the subject, duration, size and level of the target group, training materials published (type, size, title, co-authors, date). D. Other merits a) R&D or Innovation Management responsibilities exercised, programme management, management of organisation, group management, management of events (courses, seminars, workshops or other). Clarification is needed of the name, organisation, start and end dates, access system, average annual budget, geographical area of responsibility, size of target group, type(s) of target organisation(s), and other. b) Languages: Listed, along with level (low/intermediate/advanced), fluency (reading, writing, speaking, negotiation). c) Grants obtained (institution, funding agency, start and end dates, etc). d) Holding of share in scientific and professional companies; identification, start and end dates, if appropriate, size of company, etc. e) Honorary membership of professional and scientific societies (editorial boards), identification, access date (and final date, if applicable), size of society. f) Participation through selective processes to theme-based networks; identification, access date, selection organisation, participating organisations, size of the society. g) Description of titles and awards obtained that accredit professional quality. Please note: Where people's names are included, the full name (forename and surname) must be given so that the researcher's gender can be deduced.
Applied research and experimental development projects A. Objectives and potential interest of the project A1. Criterion of opportunity. The compatibility of the proposal with the contractor's requirements or the priority areas of the call for applications and/or the programme. 1) The objectives aim to achieve the required results. 2) The instruments and the duration should make the project viable. 3) The members of the team running the project and the other participants must meet the requirements. A2. Criteria for identifying objectives and their degree of novelty. The project must clearly identify the objectives it intends to cover and justify their originality and novelty in relation to the current technical situation. 4) Description of the technical situation. 5) Detailed description (precise) of the planned project objectives. 6) Novelty of the objectives with regard to the area in which it is intended to apply them (determining the scope of the innovation will help determine if it relates to a research, development or innovation project).
A3. Value of the expected results. Relevance and socioeconomic and technological impact of the results that it is hoped to achieve. An assessment will be made of the feasibility from the technological and commercial point of view, and the potential for the results and proposed technologies to break into the market (regional, national and international). 7) Social impact on employment and equality measures. 8) Expected economic impact. 9) Expected technological impact. 10) Impact on the competitiveness of any organisations using the results from the project. 11) Sustainability of the competitive advantages resulting from the project, including ethical, legal, economic, social and environmental.
B. Organisation of material and human resources B1. Methodology and work plan. Adequacy of the experimental methods, the analysis of proposed data and the proposed alternatives in the event of possible problems (Contingency Plan). Coordination and integration of resources used in developing the project. Existence of an organisation chart, schedule and adequate management plan that includes an organisational structure capable of addressing the activities proposed in a coordinated manner. Establishing measures that enable critical decisions to be anticipated and taken and establishing internal evaluation mechanisms for the development of the project and ensuring the quality of results. 1) Compatibility of the proposed material and methods with the project objectives. 2) Innovation of the proposed material and methods. 3) Planning of the project phases, activities and tasks, with those responsible for executing them being identified. 4) Proposal for suitable project management systems, including monitoring and internal evaluation tools and ensuring the quality of the results. 5) Proposed contingency plan for unforeseen or unexpected situations. For example, a protocol for action in the event of possible partial and undesirable results. 6) For coordinated or cooperation projects: the benefits arising from coordination/cooperation, as well as a proposal for adequate coordination mechanisms between the various teams and organisations involved in the project. 7) Specification of the relationships between the participating organisations for the development of the project and exploitation of the results obtained.
B2. Quality and capacity of the R&D team The applicant team's participants (including partners and subcontractors) must have had training in all the areas of knowledge related to the project, work to a suitable and verified quality; and have the right experience to do the work; they must also work to a clear organisational structure. The team should also have the essential material and organisational resources to carry out the work of the project, in line with the proposed methodology and work plan. Members of the applicant group are to indicate any time commitment to other projects over the same period, so that their availability can be measured. An assessment will be made of the involvement of potentially interested third-party organisations as observers and/or recipients of the results. 8) Quality and experience of any team members involved in working on the project. 9) The complementarity of the team members to cover the areas of knowledge required when working on the project. 10) For coordinated or cooperation projects, the complementarity of the teams that will work on the project. 11) Gender parity: no less than 40% of each sex should be viewed as desirable. C. Operating plan and budget. C1. Plan to exploit the project results. Actions planned to exploit the results achieved under the project - whether by companies involved in the project itself or through transfer and/or dissemination activities. 1) Detailed description (precise) of the plan to protect, exploit and disseminate the results from the project. Where appropriate, this will include an agreement governing the rights and responsibilities of team members and legitimate third persons (eg the funding organisation) in regard to the project results. In disseminating the results, the target audience and the media to be used are to be identified. 2) Compatibility of the plan to exploit the project results with their actual potential.
C2. Available resources and budget. The budget must be consistent with the work plan as well as the techniques and equipment to be handled. The sections to be assessed will be those relating to equipment and infrastructure (with a limit on their value), materials, personnel (own and contracted to carry out project-specific work), external collaborations, travel and subsistence (for dissemination of results and coordination between agents-applicants), and other expenses. The budget must be reasonable, consistent with the potential benefits; its implementation must be auditable. (The cofinancing of potential third-party recipients may be positively evaluated). 3) Material resources contributed for developing the project, consistent with the methodology and work plan. 4) Organisational resources provided for developing the project, consistent with the work plan and methodology. 5) Specific details and justification of the budget for the project. 6) Compatibility of the support requested with the project objectives.
R&D + Innovation centres/organisations that conduct applied research, technological development and/or innovation activities. A. The organisation's CAPABILITIES A1. Material resources. This is to contain any details that demonstrate availability within the organisation itself as well as access to infrastructure, equipment and facilities that enable the activities set as objectives to be carried out. 1) Provision of space for the anticipated activities and material resources. 2) Own basic laboratory equipment, analytical, experimentation and testing techniques, pilot plants, demonstration, etc. 3) Availability of other institutions' equipment and facilities, under agreed collaborations. 4) Infrastructure for administration, organisation, management and dissemination of results and technological knowledge. 5) Compatibility of the infrastructure and material resources (infrastructure, buildings, laboratories, equipment, etc) with the activities of the R&D + Innovation organisation. 6) Financial resources (volume, source of income, etc in the last three years). A2. External collaboration activities. Included are all collaboration activities with other institutions that contribute to the organisation's capability - whether for running specific projects, for coordinated projects, or for delivery of courses. 7) Bilateral agreements with any other prestigious organisations working on basic research or R&D that provide additional capabilities in terms of human and technical resources. 8) Bilateral agreements with organisations that have responsibility for running postgraduate and PhD courses as well as courses for their own qualifications. 9) Participation in networks.
A3. Training capability. Consideration will be given to activities related to the provision of teaching activities (whether regulated or not), and researchers/technologists' specialisation in R&D activities and capabilities. 10) Postgraduate and PhD courses. 11) Technology specialisation courses for R&D experts. 12) Courses on research techniques. B. The organisation's ACTIVITIES. B1. Results from the activities. Details are to be given of the tangible scientific/technological results from the organisation' activities. Among the different criteria, an assessment is to be made of the quality indices, degree of innovation achieved, and the impact of the results achieved or expected from transferring technology. 1) Participation and results in competitive projects and for companies. 2) Results from transferring knowledge and technology provided within the economic sectors that are the target of their activities. Degree of implementation. 3) Patents and ownership titles that have been applied for, or are in process, or have been granted and are being exploited. 4) Publications in journals with impact factor, in other journals, in papers and in informative publications. 5) Presentation of papers at international and national conferences. Exhibition of activities at fairs. 6) Quality of results within in the international and national context in the area of activities.
B2. Relationship with the economic and social environment. Activities aimed at customers, internal activities that support these, and other external and internal activities undertaken by the organisation. The organisation and its contribution are to be put in the context of the economic and social environment. 1) Using partnerships with companies, through contracts and agreements, for implementing projects, services and consultancy. Importance of R&D activities under contract (number of customers, associated volume of revenues, etc in the last three years). 2) Importance of technology advisory activities (number of customers, associated volume of income, etc in the last three years). 3) The importance of testing and certification activities (number of customers, associated volume of income, etc in the last three years). 4) Exchange of technologists with companies. 5) Participation in training courses, updating and promotion of R&D. 6) Number of researchers newly recruited and/or undergoing training. 7) Activities as agent in creating companies and driving forward businesses within economic and social environment. 8) Customer training activities (number of customers, associated volume of income, etc in the last three years). 9) Information activities (number of customers, associated volume of income, etc in the last three years). 10) Other services and applications aimed at customers (number of customers, associated volume of income, etc in the last three years). 11) in-house R&D activities itself (number of customers and associated volume of income within the framework of international, national, regional or local R&D support programmes within the last three years). 12) Technical standardisation activities (number of technical standardisation committees participated in and the national or international nature of these within the last three years). 13) Other activities (scientific and technological forecasting and monitoring, communication and dissemination, etc within the last three years).
B3. Effectiveness of funding. To evaluate the effectiveness of public funding received, this section should assess the relationship between the economic resources used and the results obtained. 14) Public and private funding received. 15) Effective management of public subsidies received in accordance with the results of transferring knowledge and developing tools and capabilities for future technology transfer. 16) Effective use of public subsidies in accordance with the impact of the activity on the profit and loss account and any employment generated. C. Objectives and strategic areas. C1. Areas of activity. The importance is to be assessed of the mission, fields of activity, economic and social players involved, and the position occupied by the organisation, based on the general planning of future actions, grouped as a minimum into subject areas. In addition, the relevance of the objectives will also be assessed along with their compatibility both with the priorities of European, national and regional R&D + Innovation programmes and with the priorities of industry), and the relevance of the expected results. 1) Economic, social and scientific/technological importance of the organisation's objectives. 2) Compatibility with the priorities of public programmes and private requirements for R&D + Innovation. 3) Alignment with the requirements and strategic plans of companies in the sector of activity. 4) Potential scope of the organisation's activities. 5) Areas of originality and innovativeness of the subject matters. 6) Degree of specialisation and ability to provide value and differentiation to potential customers. 7) Assessment of the expected results from the perspective of transferring knowledge and technology. 8) Possible synergistic effect and impact of the results on incubation and as a technology company incubator, or on expanding the activities of the organisation itself. 9) In terms of the provision of postgraduate, doctoral or specialist courses, an assessment is to be made of the relevance of the subject matter and the expected number of students.
10) External accreditation of the organisation for carrying out its activities. 11) Contribution from the organisation's affiliations with associations, federations and other entities in terms of achieving its objectives. C2. Feasibility of projects. The feasibility of the planned activities is to be assessed in relation to the organisation's capacity (human resources, physical and organisational infrastructure) and the adequate general planning of work. 12) Level of quality and specialisation of the organisation's R&D + Innovation activities, along with its competitive position in the field of activity. 13) Involvement/collaboration of industry in developing the proposals. 14) Feasibility of the work plan and the general distribution of tasks. 15) Suitability of the human resources and the physical and organisational resources for the objectives and plan of work. 16) Compatibility with agreements with other organisations. 17) Model for monitoring projects. C3. Management and exploitation of results. The necessary details are to be included to enable the management model and the expected effectiveness of public financing to be evaluated. 18) Multi-year draft budget. 19) Proposal for funding. 20) Economic management, internal and external technical audit and quality management model. 21) Model to identify and protect transferable results. 22) Model for disseminating results using mechanisms for transferring knowledge and technology. 23) Model for the economic exploitation of results. 24) Expected efficiency of public financing.
SEPTEMBER 2006 http://www.mec.es DEPARTMENT OF EDUCATION AND SCIENCE SECRETARY OF STATE FOR UNIVERSITIES AND RESEARCH NATIONAL EVALUATION AND FORESIGHT AGENCY (ANEP) FECYT