Evaluation of the European Commission s Traineeship Programme

Transcription

1 Evaluation of the European Commission s Traineeship Programme Final Report - Annexes November

2 Europe Direct is a service to help you find answers to your questions about the European Union. Freephone number (*): (*) Certain mobile telephone operators do not allow access to numbers or these calls may be billed More information on the European Union is available on the Internet ( Cataloguing data can be found at the end of this publication. Luxembourg: Publications Office of the European Union, European Union, 2014 Reproduction is authorised provided the source is acknowledged. Publications office of the European Union pp 21 x 29.7cm ISBN doi: /67135 November

3 Table of Contents Table of Contents... 1 Table of Figures... 2 Table of Tables... 3 Annex 1 Terms of Reference... 4 Annex 2 Benchmarking exercise Annex 3 Interview findings Annex 4 Survey report Annex 5 SWOT analysis November

4 Table of Figures Figure 1: Reasons for applying to the Traineeship Scheme Figure 2: Proportion of respondents that were interviewed in the selection process Figure 3: Satisfaction with the Traineeship Scheme application and selection process.. 59 Figure 4: Former trainees satisfaction with the content of the traineeship Figure 5: Former trainees satisfaction with the Traineeship conditions Figure 6: Perceptions on the ideal length of the traineeship Figure 7: Former trainees satisfaction with social and professional events Figure 8: How Former trainees perceived the utility of the Traineeship for their careers 62 Figure 9: Proportion of survey respondents who were employed after Traineeship/application Figure 10: Survey respondents annual salaries after traineeship/application Figure 11: Frequency with which respondents work with EU institutions and topics related to European affairs Figure 12: Types of organisations that respondents worked for Figure 13: Satisfaction with current employment (1 being dissatisfied and 10 being highly satisfied) Figure 14: Respondents countries of current employment EU Member States Figure 15: Respondents views on their knowledge and understanding of the EU Figure 16: Proportion of respondents that were positive about their general image of the EU Figure 17: Proportion of respondents with a (very) high personal interest in the EU Figure 18: Proportion of respondents interested in a career in one of the EU institutions November

5 Table of Tables Table 1: Overview of interviews conducted Table 2: Summary of key characteristics Table 3: Interviews with EC officials and former trainees and their relevant previous experience Table 4: Variables assessed for the potential use of propensity score matching Table 5: Counterfactual surveys, summary of populations and responses Table 6: SWOT analysis November

6 Annex 1 Terms of Reference 1. Context 1.1 Background Evaluation of the European Commission Traineeship Scheme Contracting Authority: European Commission Most European institutions including the European Parliament, the Council of the European Union, the Court of Justice, the Committee of the Regions, the European Economic and Social Committee, the European Ombudsman, the Court of Auditors, the European Central Bank, and the European Data Protection Supervisor have their own traineeship schemes. The present evaluation only concerns the European Commission Traineeship Programme referred to in the Commission Decision of 2 nd March , commonly known as Blue Book traineeships programme, which is by far the largest of these schemes. It does not concern the traineeships programme for scientific graduates of the Joint Research Centre 2, nor the National Experts in Professional Training (NEPT) scheme 3 run by the Directorate General for Human Resources, the Junior Professionals in Delegation (JPD) programme 4 of the European External Action Service or the so-called atypical trainees to be found in some directorates-general. The European Commission s Traineeship scheme has known a growing success since its launch in Each year, some 1,400 university graduates from all over the world spend five months in the services of the European Commission and of various EU agencies and bodies. Trainees benefit from a unique professional experience in a dynamic international environment, while providing the EU institutions with qualified help and a fresh look on their policies and procedures. Boosted by the growing reputation of the scheme and by the economic downturn, the number of applications has soared to reach more than 14,000 candidates for the October 2013 session corresponding to 20 candidates per each available traineeship. A new, dedicated unit was created at the beginning of 2013 to deal with the increased workload, and rationalisation of selection procedures and other activities is underway to streamline the workflow, improve accountability and transparency and minimise the administrative burden of the scheme. Six years after the last external evaluation, it is time to once again assess the efficiency, effectiveness, utility and relevance of the Traineeship scheme C(2005)458. See See See See November

7 It should also be noted that the Commission has recently launched a broad consultation with social partners on a quality framework for traineeships. The evaluation should take into account the main principles presented in the Commission Communication Towards a Quality Framework on Traineeships Second stage consultation of the social partners at European level under Article 154 TFEU 5 and in the accompanying Commission staff working document, 6 such as the conclusion of a traineeship agreement, transparency of information on the trainee s rights and obligations, duration and remuneration, welldefined objectives, mentoring and high-quality learning content. 1.2 Description of the programme Twice a year, in March and in October, approximately 700 trainees join the services of the European Commission and of the various bodies with which it has signed Service Level Agreements: CLEANSKY (CSJU), EACEA, EACI, EAHC, EDPS, EEAS (both in Brussels and in Delegations), ERCEA, REA, SESAR and TENEA. The vast majority of trainees come from the Member States of the European Union and from candidate countries benefiting from a pre-accession strategy. In addition, a small number of nationals of non-member States are also accepted, within the limits of available resources. There are no age limits for candidates wishing to do a traineeship at the Commission. Candidates must have completed the first cycle of a higher education course and obtained a full degree or its equivalent by the closing date for applications. EU candidates must have a very good knowledge of at least two official languages of the European Union 7, including at least one of the three working languages of the European Commission (English, French or German). Candidates from outside the EU must have a very good knowledge of at least one of the latter three languages. Applications are not accepted from candidates who benefit or have already benefited from any kind of in-service training (formal or informal, paid or unpaid) within a European institution or body, or who have or have had any kind of employment within a European institution or body, for more than six weeks. After a general introductory conference on European matters, trainees are attached to their Directorates General or Services, and are placed under the responsibility of an adviser. The adviser must establish a job description, provide guidance and closely follow the trainee during his/her traineeship, acting as his/her mentor. Trainees are given tasks similar to those of junior administrators at the beginning of their career. In each Directorate General and Service, an official is designated to co-ordinate the management of the trainees attached to that department. This coordinator assists the 5 (COM/2012/0728 final) The Study on a comprehensive overview on traineeship arrangements in Member States (2012) can also provide a useful background on the best traineeship programmes in the EU. See 7 Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovene, Spanish and Swedish November

8 trainees in administrative procedures and acts as contact point/interface between the Directorate General or Service and the Traineeships Office Unit (TOU). Trainees who have completed a period of at least three months receive a certificate specifying the dates of their training period and the department in which they were employed. Internal evaluation procedures, including nominative reports and an online questionnaire, are in place to collect feedback from trainees and advisers. 1.3 Objectives of the programme 8 The main aims of the Commission s traineeships scheme are to improve the trainees employability perspectives by providing them with a unique, hands-on experience of policy making and internal procedures and to help them better understand the objectives and goals of the EU integration processes and policies. The opportunity to put into practice the competences acquired in formal education in a dynamic, multilingual and multicultural environment provides trainees with an advantage in pursuing better professional opportunities. The European Commission, in turn, benefits from the continuously renewed academic knowledge, dynamism and enthusiasm of each successive intake of trainees. These bring their insight and understanding of EU practices and procedures into the businesses and organisations where they are hired, thus improving their ability to operate on an EU level. Trainees finally act as goodwill ambassadors for European ideas and values both within and outside the European Union. 1.4 Description of the selection process The traineeship process The Traineeship Programme is managed by a dedicated unit of the European Commission s Directorate-General for Education and Culture (DG EAC), which is responsible for the overall co-ordination of the programme in all its phases application, pre-selection, final selection, recruitment, welcome to the Commission, evaluation and farewell conference as well as for providing administrative support to the trainees during the stage (assistance with visa, provision of certificates, relations with the DGs in case of problems etc.). The traineeships coordinators in each Directorate General and Service facilitate communications between the TOU, trainees and advisers, providing on-the-spot assistance and explaining procedures. Submission Applications for traineeships are opened 7-8 months before the start of the traineeship period. The online application forms have recently been modernised, in an attempt to streamline the pre-selection procedure. Submission of paper documents has been limited to pre-selected candidates (approximately 20% of initial applications). Rules, eligibility criteria and appropriate instructions are published on the website 9. 8 For a detailed description of the aims of the programme, see Commission Decision C(2005)458 of , November

9 Pre-selection procedure The European Commission Traineeship scheme to be evaluated includes 2 types of traineeships: administrative traineeships and traineeships in the Directorate-General for Translation (DGT, approximately 8% of the total), each of which follows different procedures. For administrative traineeships, committees of officials, organised by nationalities, are convened to examine the online applications and give notes based on three categories: education, motivation and additional merits. Under Education (45% of final marks), the level and quality of diplomas are examined, many candidates having today qualifications that go beyond the minimum, which is a Bachelor degree Masters, PhDs and even post-doctoral studies. Points attributed for Motivation (10% of final marks) refer to the quality of reasoning in the free text accompanying the application, while Additional points (45% of final marks) are attributed for the international profile, relevance and level of language competences, working experiences and particularly interesting specialisations. The main objective of the pre-selection procedure is to offer to all Commission departments the best possible choice of candidates in each field of activity, with a wide range of academic disciplines. The pre-selection procedure also aims to ensure equal opportunities by maintaining, insofar as possible, the same gender balance as the one observed in applications and some proportionality between candidates from the different countries. Based on the points attributed in the pre-selection, a list of candidates containing four times as many applicants as there are training places available is compiled. All candidates are informed of the results of the pre-selection procedure and the database containing the names and curricula of the pre-selected candidates (still known as the Blue Book from its original paper format) is made available to all Commission departments for the final selection process. For translation traineeships, knowledge of source and target languages and the ability to use computer-assisted translation tools are the main criteria. A first selection is done by linguists from the language department of the target language chosen by candidates, producing a shortlist from which the Heads of Unit and Heads of Language Departments make the final choice. Pre-selected candidates are invited to provide paper copies of the documents referred to in their online applications, and a check of eligibility criteria is done by the Traineeships Office Unit and the traineeships correspondents of DGT. Final selection procedure Taking into consideration the size and absorption capacity of each service, the results of the evaluations carried out and the budget availability, the TOU determines the number of trainees to be attributed to each Directorate General, Service, agency or other body for each period. The Commission services search the Blue Book according to specific criteria and an established profile, reserve their trainees from the list of suitable candidates through the online system and prepare a detailed job description for each trainee. Offers are sent by the TOU approximately four months before the start of the traineeship period. November

10 1.4.2 Eligible countries The programme is open to university graduates from all over the world, provided the security and visa procedures are complied with. 1.5 Budget of the programme The budget of the programme amounts to approximately 6,500, per year, for the most part reserved for the monthly allowances, insurance and reimbursement of travel expenses of trainees. Each trainee receives a grant of 1, per month in 2013, corresponding to ¼ of the basic salary of a junior administrator (Grade AD5) working for the EU institutions. Between 10% and 15% of trainees go to agencies and other bodies, which cover the monthly allowances from their own budget. 1.6 Implementation of the Programme Below is a table of the number of applications received and the number of trainees in the last two years: March 2011 October 2011 March 2012 October 2012 March 2013 October 2013 Applications Trainees Evaluation and Monitoring Provisions Monitoring Provisions An online reporting and monitoring system is in place to evaluate the satisfaction of trainees and of the services to which they were assigned. The system consists of a 2- phase evaluation during each traineeships period: Towards the end of the first month, trainees are invited to fill in an online questionnaire focusing on the first phases of the traineeships programme (application, pre-selection, final selection, recruitment) and on practical issues relating to the induction phase (welcome days and conferences, introduction in the Directorates General/services, logistics, infrastructure etc.). This enables the TOU to improve its operations and, if needed, to immediately intervene with the Directorates General and services in order to assist in solving problems or difficulties. Towards the end of the traineeships period both trainee and adviser are asked to fill in an on-line questionnaire. This allows collecting feedback on relations between trainees, advisors, coordinators, the Stage Committee, TOU and all other actors. In addition, the TOU sends separate invitations to write an end-of-stage report. Trainees are invited to present their achievements and to comment on the work content and relevance of their academic background for the activities performed. Advisers are invited to describe the tasks performed and write an appraisal of the trainees contribution. November

11 1.7.2 Previous evaluations An external evaluation was carried out in This resulted in a final report including a list of 12 recommendations. 10 An Action Plan was established in October 2008 for the implementation of these suggestions. Three progress reports followed in February 2009, December 2009 and February 2010, detailing the main changes introduced as a result of the evaluation. Such changes mainly concerned the suggested restructuring of the internal monitoring system and better exploitation of its findings, the introduction of preliminary contacts with candidates through phone interviews and the reconsideration of the system of national quotas. The documents concerning this evaluation will be at the disposal of the contractors. 2. Task specification for the assignment This contract will contribute to the following overall objectives: - a thorough assessment of the Traineeship Programme of the European Commission according to the evaluation questions specified in section 2.1; - the provision of useful recommendations aimed at improving the effectiveness, efficiency, utility and relevance of the Traineeship Scheme. The specific objective of this contract is to procure an external, independent interim evaluation of the Traineeship Scheme covering the period The outputs to be delivered by the contractor are as follows: an evaluation report according to the provisions of section 3 below; in particular, the report must answer to all the evaluation questions specified in section here below. Furthermore, the report must provide viable recommendations grouped under the different evaluation criteria specified in section 2.1. a presentation of the evaluation results to the Steering Group, to the unit concerned and other interested parties (Directorate-general for translation, invited coordinators), to be delivered at the premises of DG Education and Culture when submitting the Draft Final Report. 2.1 Evaluation questions Relevance, consistency, added value 1. To what extent have the programme s aims and objectives proved relevant to the needs of the European Commission and the university graduates interested in pursuing a career in European affairs? 10 November

12 2. To what extent the experience gained during the traineeship is useful to access the labour market? 3. Are the eligibility criteria (academic qualifications, language competence, absence of prior employment in EU institutions or bodies) relevant to the purpose of selecting the best candidates while offering the opportunity of a traineeship to a large number of applicants? 4. To what extent has the programme proved complementary to other EU initiatives aimed at offering university graduates insider knowledge of EU institutions, such as those mentioned in section 1.1? 5. Does participation in the programme appear satisfactory in terms of geographical coverage, the subject areas represented and the types of participants? 6. To what extent does the current programme satisfy the criteria presented in the documents referred to at the end of section 1.1? 7. What are the main added values of the programme? Effectiveness 8. To what extent has the programme been effective in achieving the aims and objectives mentioned in section 1.3? 9. Does participation in the programme appear satisfactory in terms of equality between men and women? To what extent do the programme design and implementation mechanisms promote equal participation of men and women? Could any particular points of improvement be identified? Efficiency 10. Are the size of budget, human and technical resources for the programme appropriate and proportional to what the programme is set out to achieve? Are they sufficient for reaching a critical mass of impact? Could the same results have been achieved with fewer resources? Could the use of other instruments or mechanisms have provided better cost-effectiveness? 11. Could procedures be put in place to improve the benefits of the programme for the trainees, in terms of employability and learning outcomes, and for the host services and departments, in terms of improving the trainees contribution to their activities? 12. Could further synergies with other institutions, agencies and bodies, such as those mentioned in Section 1.1, increase the efficiency of the programme? 13. To what extent are the monitoring mechanisms effective and efficient? What are the areas for improvement? Sustainability 14. Which of the current activities or elements of the programme would be likely to continue and in which form if EU support was withdrawn or substantially decreased? November

13 2.1.5 Utility 15. To what extent have the activities of the programme resulted in any unintended results and impacts (both desirable and undesirable)? 16. To what extent do the results and impacts of the programme correspond to the professional needs of the trainees and the European Commission? 2.2 Other tasks under the assignment The Contractor should: Provide a one-page statement about the validity of the evaluation results, i.e. to what extent it has been possible to provide reliable statements on all essential aspects of the EU intervention examined. Issues to be referred to may include scoping of the evaluation exercise, availability of data, unexpected problems encountered in the evaluation process, proportionality between budget and objectives of the assignment, etc. Make a proposal for the dissemination of the evaluation results, on the basis of the draft Dissemination Plan annexed to these Terms of Reference. 3. Reporting and deliverables General reporting requirements Each report (except the final version of the Final Report) should have an introductory page providing an overview and orientation of the report. It should describe what parts of the document, on the one hand, have been carried over from previous reports or been recycled from other documents, and on the other hand, represent progress of the evaluation work with reference to the work plan. All reports must be drafted in English and submitted according to the timetable below to the responsible body. Electronic files must be provided in Microsoft Word for Windows format. Additionally, besides Word, the Final Report must be delivered in Adobe Acrobat pdf format and in 5 hard copies. The Commission will comment on all reports within maximum 30 calendar days. In the absence of observations from the Commission within the deadline the report will be considered as being approved. Within maximum 14 calendar days of receiving the Commission s observations the Contractor will submit the report in definitive form, taking full account of these observations, either by following them precisely or by explaining clearly why they could not be followed. Should the Commission still not consider the report acceptable, the Contractor will be invited to amend the report insofar as such amendments do not interfere with the independence of the evaluator in respect of their findings, conclusions or recommendations. Inception Report The inception report must describe in detail how the methodology proposed by the Contractor is going to be implemented in the light of an examination of the quality and appropriateness of existing data. It should include the Contractor's understanding of the November

14 intervention logic, as well as the quantitative and qualitative indicators that he/she will use in addressing each of the evaluation questions. A detailed work plan including the allocation of experts per task per number of working-days should also be provided. It shall not exceed 20 pages, annexes excluded. Interim Report The report is to be produced after the desk and field research has been completed, and should, to the extent possible, include some preliminary conclusions. The report must as a minimum provide: An overview of the status of the evaluation project; A description of problems encountered and solutions found; A summary of initial findings and results of the data gathering; An assessment of the data, whether it meets expectations and will provide a sound basis for responding to the evaluation questions; A conclusion whether any changes are required to the work plan, or any other solutions should be sought in order to ensure that the required results of the evaluation are achieved. If any such issues are to be identified, they must be discussed in the meeting with the Steering Group dedicated to this report; A proposal for the final structure of the Final Report, as well as a structure of the Executive Summary. It shall not exceed 30 pages, annexes excluded. Draft Final Report This document should deliver the results of all tasks covered by these Terms of Reference, and must be clear enough for any potential reader to understand. It should contain: Executive summary: It sets out, in no more than 7 pages, a summary of the evaluation s main conclusions, the main evidence supporting them and the recommendations arising from them. Main report: The main report must be limited to 75 pages and present, in full, the results of the analyses, conclusions and recommendations arising from the evaluation. It must also contain a description of the subject evaluated, the context of the evaluation, and the methodology used (including an analysis of its strengths and weaknesses). Annexes: These must collate the technical details of the evaluation, and must include the Terms of Reference, questionnaire templates, interview guides, any additional tables or graphics, and references and sources. November

15 Final Report The Final Report follows in principle the same format as the draft Final Report. In addition to the contents described above (main report and annexes), it should contain a ½ page summary statement of the main evaluation conclusions and recommendations. The Executive Summary should be translated into French and German by a professional translation agency, once it has been approved by the responsible body. The Final Report must take into account the results of the quality assessment of the draft Final Report, and the relevant discussions with the Steering Group insofar as these do not interfere with the independence of the Contractor in respect of the conclusions they have reached and the recommendations made. The contracting authority will publish the Final Report, the Executive Summary and the annexes on the World-Wide Web Organisation, timetable and budget 4.1 Organisation The contract will be managed by Unit D4 of the European Commission Directorate- General for Education and Culture. A Steering Group will be involved in the management of the evaluation. The responsibilities of the Steering Group will include: - providing the external evaluator with access to information; - supporting and monitoring the work of the external evaluator; - assessing the quality of the reports submitted by the external evaluator, while ensuring that the Contractor's independence is not compromised. 4.2 Meetings It is expected that the contractor participate in four meetings in Brussels with the evaluation Steering Group. The evaluation team leader and other relevant experts must participate in these meetings. For these meetings, minutes should be drafted by the contractor within 5 working days, to be agreed among the participants and approved and signed by the chair person, who will be appointed from Unit EAC/R Timetable The indicative starting date is 15 November The implementation of the contract will start after both parties have signed it. The period of execution of the contract is 8 months. The following outline work plan and indicative timetable are foreseen: 11 On the site November

16 Deadline (from starting date) Kick-off [T0)] Inception Report [T0 + 1month] Interim Report [T0 + 4 months] Draft Final Report [T0 + 6 months] Final Report [T0 + 8 months] Task The project is kicked off at a meeting between the Contractor and the Steering Group in Brussels Contractor prepares inception report and presents to Steering Group in Brussels Desk and field research completed. Contractor presents interim report to Steering Group in Brussels. Contractor presents a draft final report, including an executive summary, to Steering Group in Brussels Taking account of the Commission s comments contractor sends final report and summary to Steering Group in Brussels 4.4 Budget The estimated maximum budget for the evaluation of the action, covering all the results to be achieved by the contractor as listed in sections 2 and 3 above, is EUR 80, References 5.1 Basic documents the website of the Traineeships programme: Rules governing the official traineeships scheme of the European Commission (Commission Decision of C(2005)458) 12 the official website of the Stage Committee: Statistics on applications and selected trainees the 2007 external evaluation (executive summary, final report, annexes, quality assessment: Action Plan following the 2007 external evaluation Final Progress Report on the implementation of the 2007 evaluation Action Plan 5.2 Documents and information to be provided after contract signature (not exhaustive) Business Process Management documentation on the pre-selection and selection process 12 November

17 Interim and end-of-stage reports based on the questionnaires filled in by trainees and advisers 6. Requirements 6.1 Methodology The contractor will have a free choice as to the methods used to gather and analyse information and for making the assessment, but must take account of the following: The evaluation must be based on recognised evaluation techniques. The choice and a detailed description of the methodology must form part of the offer submitted. There should be a clear link between the evaluation questions addressed and the corresponding methodology proposed. The evaluation questions can be further elaborated, e.g. by providing operational sub-questions under each question. Considerable emphasis should be placed on the analysis phase of the evaluation. In addressing the evaluation questions, quantitative indicators should be sought and used as far as possible. The contractor must support findings and recommendations by explaining the degree to which these are based on opinion, analysis and objectively verifiable evidence. Where opinion is the main source, the degree of consensus and the steps taken to test the opinion should be given. For the design of the cost-effectiveness assessment, the tenderer should use the results of the "Study on Cost-Effectiveness of Education and Culture Spending Programmes" (Ecorys, 2013) 13 as a basis for their proposal. The tenderer should specify which of the approaches described in the mentioned study they will use or, alternatively, what design they propose for such an assessment. 6.2 Quality assurance The Contractor shall, as a minimum, apply the quality assurance procedures described in the Quality Plan included in their bid for Framework Contract EAC/50/2009. The offer should describe how the Quality Plan will be applied during the implementation of this specific contract. 6.3 Resources The Contractor shall ensure that experts are adequately supported and equipped. In particular, sufficient administrative, secretarial and interpreting resources, as well as junior experts, must be available to enable senior experts to concentrate on their core evaluation tasks November

18 Annex 2 Benchmarking exercise Summary The benchmarking exercise compared the traineeship programme with similar programmes to identify any potential synergies. Desk based research on five traineeship schemes (including three EU schemes) was supplemented by interviews with four of the five institutions and organisations. Key characteristics As a result of an inter-institutional agreement, all paid trainees in the EU s institutions receive compensation amounting to almost 1,100 per month. Arrangements concerning inter alia eligibility and exclusion criteria, working conditions and the duration of traineeships are also aligned across the institutions. Diverging from the Blue Book Traineeship, under the EP and EESC traineeships there is no pre-selection phase or other attempt to ensure the shortlisted candidates are representative of the EU s Member States. Harmonisation between the traineeship schemes The main suggestions interviewees did have for closer alignment centred on the idea of establishing a central portal or single window whereby applicants for traineeships with all EU institutions could apply. Interviewees also highlighted the need for improved job descriptions for trainees across the EU s institutions, with an emphasis of learning and training objectives, coupled with effective monitoring to ensure these objectives are achieved. It was also suggested that regular meetings to share best practice and ways to solve similar challenges could be a way forward to increase harmonisation. Monitoring mechanisms Monitoring and evaluation activities take place to some degree across the EU institutions traineeship schemes. Developing a common approach to monitoring and/ or a common strategy for the dissemination of the feedback collected could help the institutions develop a common strategy to combat shared problems. The benchmarking exercise aimed to compare and assess the complementarity of the traineeship programme with similar programmes and identify any potential synergies. Desk based research into publically available information on five traineeship schemes (three EU traineeships and one each from the Council of Europe and United Nations) was supplemented by semi-structured telephone interviews with representatives of four of the five institutions and organisations The United Nations Internship Programme office did not respond to our requests for information November

19 Interviews conducted Table 1: Overview of interviews conducted European Commission Belen Bernaldo De Quiros Interviewed February, 2014 Head of Unit, DG EAC D4 (Traineeships Office) Luca Tomasi February, 2014 Programme manager, Traineeship Office, DG EAC, Unit D4 Silvia Varela February, 2014 Stage coordinator, DG Translations, Unit R1 Magdalena Blaszkowska February, 2014 Stage coordinator, DG SANCO A5 (Human Resources) Antonio Silva Mendes Director February, 2014 DG EAC Directorate B (Education and vocational training: Coordination of Erasmus +) Maddy Tihon February, 2014 Stage coordinator, DG DEVCO R4 (Human Resources) Silvia Varela February, 2014 Stage coordinator, DG Translation R1 (Traineeships, inter-institutional exchanges, SNEs) Other EU institutions European Parliament (EP) European Economic and Social Committee (EESC) Pierre-Antoine Barthélémy Head of unit for recruitment, DG Personnel Gerardus Nijborg Head of unit for personnel Interviewed July, 2014 July, 2014 Council of the European Tamás Záhony August, 2014 November

20 Union (CoEU) Head of Office, Traineeship Office Other international organisations Council of Europe (CoE) Irena Kazenaite Recruitment and employment management division July, 2014 United Nations (UN) no reply no reply Important observations and analysis from the desk research and interviews are presented below, including a table summarising the key characteristics of the five traineeships we examined. Key characteristics of the traineeship schemes Costs of the traineeship schemes Limited information was available on the budget for the training schemes. The European Parliament (EP) estimated it spends approximately 3,000,000 on an annual basis, amounting to approximately 7,500 for each of its 400 trainees who complete a fivemonth traineeship. The European Economic and Social Committee (EESC) spends a total of 295,000 annually (approximately 5,900 per trainee). While the Council of the European Union spends 583,000 annually (approximately 5,830 per trainee). While exact figures were unavailable, the interviewees from these institutions explained that the vast majority of the budget provided financial compensation to the trainees during their traineeships. As a result of an inter-institutional agreement, all paid trainees in the EU s institutions receive compensation amounting to almost 1,100 per month. Arrangements concerning inter alia eligibility and exclusion criteria, working conditions and the duration of traineeships are also aligned across the institutions. Looking beyond the EU schemes, the Council of Europe s (CoE) scheme appears to be considerably less costly to run. The main reason for this is the lack of financial compensation provided to the vast majority of CoE trainees. While the CoE official we interviewed was unable to provide an exact or even estimated figure, she explained that the main cost of running the scheme would be that of two full time equivalent (FTE) staff who support the recruitment of trainees across the CoE s multiple offices. Application and selection process of the traineeship schemes While the application processes for the other EU institutions traineeship schemes broadly mirror that of the Blue Book traineeship (applicants complete an online application form giving details inter alia of their educational achievements, professional experience and motivation), the main divergence is the absence of a pre-selection procedure in all cases but the Council of the EU. The EP official we interviewed explained that the absence of a pre-selection stage was the result of a lack of financial resources rather than a positive decision against pre-screening candidates to produce a representative sample in the same vein as the EC s Blue Book. November

21 The EESC, on the other hand, felt their services were able to cope with the smaller number of applications (approximately 5,000) they receive. Applications for the EESC scheme are made available to all the institution s units via a digital database which can be searched using specific fields. Criteria for the allocation of trainees to different departments / units In the case of the EP, EESC and the CoE, once the applications received are made available to the respective institutions, the interviewees were unable to explain the basis on which advisors within these units select candidates. It was broadly assumed that the candidates chosen are the best fit for a particular unit s current priorities (perhaps because of specific educational or professional experience or language skills). This approach was not felt to betray a lack of transparency, and rather reflected the way employers select staff to meet their own needs (including the means by which the institutions select their own permanent staff) in the real world. The process at the Council of the EU At the Council of the EU, the online application process is outsourced and managed by EPSO. Once the deadline has passed, the Council is given access to the database of candidates and the Council services work through the applications to find suitable candidates (they are able to search using specific fields). They make a pre-selection of around four times the quantity which will eventually be taken. The traineeship office conducts the final selection. The CoEU official we interviewed emphasised that this process is being reviewed. The idea being that it could be more efficient and effective if the traineeship office undertook the pre-selection and the services carried out the final selection. November

22 Table 2: Summary of key characteristics Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat Concerns Paid Traineeship not traineeships undertaken as a compulsory part of studies/ traineeships for national officials Concerns Long-term traineeship not short-term traineeship or traineeships for holders of grants awarded by outside public bodies Concerns graduate traineeships which are known as Robert Schuman traineeships. Note that there are 2 options, the general option and the journalism option not training placements/conference interpretations and translation traineeships/ad hoc traineeships Concerns unpaid traineeship, minimum of three and maximum of five months. Concerns unpaid traineeship, minimum of three and maximum of six months. Aims of the schemes to give a general idea of the objectives and problems of European integration to provide them with practical knowledge of the working of the departments of the General Secretariat of the Council to offer them personal experience thanks to the contacts made in the course of their everyday work to enable them to further and put into practice the knowledge they have acquired during acquire practical knowledge about how the EESC s various departments operate supplement and apply knowledge and skills acquired during their studies To enable trainees to supplement the knowledge which they acquired during their studies and to familiarize themselves with the activities of the European Union and, in particular, the European Parliament learn about EESC s role and activities at interinstitutional level, as well as relations with EU Member and non-member states to enable trainees to gain valuable experience in a complex multicultural and stimulating work environment for the CoE benefit from the enthusiasm, motivation and skills that new graduates bring to give interns a first-hand impression of the day-to-day working environment of the United Nations To offer interns a real chance to work with our people. As part of our team, working directly with outstanding and inspiring career professionals and senior management, you will be exposed to high-profile November

23 Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat their studies or professional careers and/or work gain experience in a multicultural, multilingual and multi-ethnic professional environment conferences, participate in meetings, and contribute to analytical work as well as organizational policy of the United Nations To offer interns an increasing level of responsibility: Initially interns take on the amount of responsibility they can shoulder; the potential for growth, however, is yours to develop Size of trainee intake/number of applicants Approx. 100 pa / 5,000 applicants Approx. 50 pa / 5,000 applicants (the number of trainees is set by the PER unit in line with both the budgetary resources available and the office space available) Approx. 400 pa / 10,000 applicants Approx. 500 pa (across all CoE offices) / approx. 1,000 applicants centrally (no information available on number of applicants who apply directly to CoE services) No information available Trainee profiles and representativen ess Nationals of Member States of the European Union who have completed at least the first cycle of a course of university studies attested by a degree certificate. Aimed at young university graduates who, in the course of their studies or in some other connection, have already acquired either Trainees must: (a) be nationals of a Member State of the European Union or of an applicant country; (up to 10% of the total Nationality of one of the Council of Europe member states; Have a bachelor's degree or equivalent (a diploma for the first Enrolled in a Master's or in a Ph.D. programme, or in the final year of a Bachelor s programme; or within one year after graduation from a November

24 Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat Thorough knowledge of EN/FR (no application shall be accepted from any person who has already received >8 weeks' in-service training in any department of a European institution or body or an agency or an office of the European Communities, whether paid or otherwise. The same shall apply to any application from a person who is or has been an adviser's assistant or assistant to a Member of the European Parliament, or a consultant, a researcher, a temporary staff member, a seconded national expert, a contract, auxiliary or interim staff member of any institution, agency or representative office of the European Communities. Article 6 (4) of Decision No 118/07) Aim to cover as wide a geographical coverage as possible; as well as a the best possible balance between men and women Preference will be given to a qualification in the areas where the EECS works or a good knowledge of the socio-occupational organisation 15 operating in the EU. University graduates who have completed at least three years of studies Open to candidates who are nationals of Member States and to a certain number of university graduates from non-eu countries Exclusion criteria apply (no application shall be accepted from any person who has already completed >6 weeks' paid traineeship with any European institution, agency or body. The same applies to applications from people who a) are number of traineeships may be offered to third nationals) (b) be aged 18 or over on the traineeship starting date; (c) have a thorough knowledge of one of the official languages of the European Union; (d) not have been awarded any other traineeship or have been in paid employment for more than four consecutive weeks at the expense of the European Union budget; (e) for trainees in the field of childcare, a declaration proving no criminal record, issued in the last year at their most recent place of residence. A declaration specifically issued to individuals having contact with children is required if such a document exists in the cycle of higher education as defined in the Bologna Declaration). Have a desire to acquire practical experience and knowledge of the functioning and activities of the Council of Europe; Have a very good knowledge of one of the two Council of Europe official languages (EN/FR) together with good drafting ability. Good knowledge of the other official language is appreciated Exclusion criteria: no individual may undertake more than one traineeship in the organisation Bachelor s, Master s or Ph.D. programme; Excellent command of English or French; Exclusion criteria: Neither a child or a sibling of a United Nations Secretariat staff member 15 Socio-occupational organisation is a term referred to in the Decision No 115/14 A, governing the EESC traineeship. URL: November

25 Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat those who: have completed or started an academic study of European integration in the framework of a university dissertation or research work or a scientific publication hold or have held a post in the private or public sector dealing with the activities of the European Union. Article 7 (1) serving or have served as an assistant to a member of the EP or b) are working or have worked as a consultant on a research assignment, or as a temporary, contract or agency staff member at an EU institution, agency or representative office) country in question. Exclusion criteria: applicants must not have been awarded any other traineeship or have been in paid employment for more than four consecutive weeks at the expense of the European Union budget Length of the traineeship 5 months (in exceptional circumstance this may be reduced to 3 months; and may be extendable to 6 months although any extension above 5 months will be unpaid) 5 months (can be extendable by a maximum of 1 month subject to budgetary resources and office space) 5 months (no extension possible) 8 weeks to 5 months From 2 to 6 months Remuneration The salary is equal to 25% of the basic salary of an official at grade AD5, step 1 (it is currently set at EUR 1,087 monthly but is reviewed regularly) 25% of the basic salary of an official at grade AD5, step 1 (applying on day 1 of the internship) i.e. EUR monthly A scholarship which is the equivalent of ¼ of the basic salary of an official in grade AD5, step 4 16 which was EUR in For 2013 trainees may receive an allowance of 300 EUR pcm Not remunerated (with the exception of approximately 10% of trainees who receive ad hoc remuneration at the discretion of their unit) Not remunerated 16 Under article 24 in the, Internal Rules Governing Traineeships And Study Visits In The Secretariat Of The European Parliament, it states that remuneration is based on ¼ of the basic salary of an official in grade AD5, step 4. This differs from comparable institutions that provide remuneration in line with ¼ of the basic salary of an official at grade AD5, step 1. Url: November

26 Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat Budget dedicated to the scheme, including funding, wages and social security costs EUR 583,000 pa (for 2014) Includes the salary, health insurance, cost of trips etc. EUR 295,000 pa. Reimbursement of travel expenses at the beginning and end of the traineeships provided that certain conditions apply (page 7 of EESC decision) Approx. EUR 3 million pa. Flat rate payment to towards the cost of travel to the place where the traineeship will take place EUR for each km between 1 and 1000 km; No information on budget available (2 FTE staff administer the scheme) No available information de 0,0884 EUR for each km above 1001 km and less than km EUR for each km above km. (rates for 2012 but these are adjusted on 1 January each year) Health and incident insurance is provided for all trainees (trainees may pay a supplement to insure their spouse or children) Household allowance is available to trainees who are married and/or have dependent children. This is limited to 5% of the reference salary of a grade AD5, step 4 official Reimbursements accommodation for and November

27 Council of the European Union (CoEU) European Economic and Social Committee (EESC) European Parliament (EP) Council of Europe (CoE) United Nations (UN) Secretariat living expenses for missions are available Disabled applicants may receive additional payments of up to 50% of the scholarship Web link(s) eu/contacts/traineeshipsoffice/traineeships?lang=en eu/media/ /web_entraineeships_2013.pdf a.eu/?i=portal.en.train eeships a.eu/resources/docs/de cision en.pdf opa.eu/aboutparliament /en/007cecd1cc/trainee ships.html eb/jobs/traineeships m/watch?v=ah_cpbrte1 8 ob/traineeship_program me bw/home.aspx?viewtype =IP eu/uedocs/cmsupload/decisio n% %20en%20sn re01[1].pdf November

28 Harmonisation between the traineeship schemes The evaluation team examined the scope for harmonisation between the Blue Book traineeship and those of the three EU organisations we studied. The officials responsible for the other EU schemes we interviewed (CoEU, EP and EESC) believed there was limited scope for harmonisation between the traineeship schemes beyond the existing alignment in terms of eligibility, financial compensation, health insurance arrangements and duration. The main suggestions interviewees did have for closer alignment centred on the idea of establishing a central portal or single window whereby applicants for traineeships with all EU institutions could apply. This portal would then funnel candidates through different sections depending on which institution they wished to apply to. The interviewees did not believe a single application form would be helpful as the characteristics sought by the different institutions in their trainees differed. For example, the interviewee for the European Parliament suggested that while the Commission s directorates-general sought candidates who already have some knowledge of the relevant policy area (e.g. competition law, trade), the parliament recruits applicants who are interested in politics and the political process generally as well as other trainees who have an interest in learning more about the administrative functions of the institution (e.g. human resources). Given that the CoEU is already outsourcing its application process to EPSO, it might be worth exploring the benefits of integrating applications to a central portal managed by EPSO. Although one interviewee also voiced the concern that a greater alignment of the traineeships would likely entail a role for EPSO, the EU s personnel recruitment office, which would inevitably slow things down. Potential benefits and drawbacks of further alignment between the Schemes Having highlighted the differences between the Commission s and their own traineeship scheme, the interviewees claimed that the extent to which any further alignment between the schemes would produce significant benefits is limited. Although one interviewee suggested that scheduling regular feedback sessions (for example on an annual basis) would facilitate interesting discussions about problems shared and solutions found. It was felt that whilst there was probably openness to this arrangement, the issue would be finding the time. While not a benefit articulated by any of the interviewees, the possibility of pooling resources to increase the effectiveness and efficiency of traineeships in general could result in the other institutions being able to increase their monitoring efforts. One of the interviewees explained that their monitoring efforts could probably be increased were they to be allocated greater resources. Given that monitoring is already underway within the Commission (and also notably in the CoEU), this suggestion would probably benefit institutions with less monitoring in place more than the Commission traineeship itself. The idea of the institutions working together to consider the ramifications of the recently adopted Quality Framework on Traineeships was also raised as an area where the institutions might benefit from greater dialogue. November

29 Quality Framework on Traineeships Setting the standard for good quality traineeships The Terms of Reference for this assignment stipulate that the evaluation team should take into account the main principles presented in the Commission Communication Towards a Quality Framework on Traineeships and its accompanying working documents. These principles include the conclusion of a traineeship agreement, transparency of information on the trainee s rights and obligations, duration and remuneration, well-defined objectives, mentoring and high-quality learning content. In essence, the Quality Framework is considered to be an important reference point for determining what constitutes a good quality offer of traineeships. Since the evaluation s inception in January 2014, the Council has adopted the Quality Framework in its entirety in the form of a non-binding Recommendation 17. The adoption of the Quality Framework did not go unnoticed by trainees, with events organised in Brussels to debate the impact the Framework could have and how to ensure this hortatory document (urging not mandating) can be used to advance the rights of trainees faced with unscrupulous employers 18. The evaluation team were therefore surprised to learn that some of the interviewees (including those representing EU institutions) were unaware of the Quality Framework and had no plans to review or adapt their traineeships in the light of its recommendations. Learning and training objectives The main divergence between the Framework and the traineeships we examined concerns learning and training objectives. The Recommendation urges Member States to promote best practices as regards learning and training objectives in order to help trainees acquire practical experience and relevant skills; the tasks assigned to the trainee should enable these objectives to be attained. Some of the interviewees agreed that, as things currently stand, it would be difficult for them to confirm that their traineeships always meet this aim of ensuring tasks set enable trainees to achieve relevant learning objectives. Some explained that written agreements between the trainee and the institution are vague on detail regarding a trainee s day to day role, with the task of agreeing objectives falling to individual trainees together with their supervisors (a member of staff assigned responsibility for the trainee throughout the traineeship in a similar manner to the role of advisor under the Blue Book scheme). Compounding this problem, a lack of effective monitoring meant it would be impossible to say whether learning objectives had indeed been attained. It is important to note that the introduction of written agreements and clearly defined tasks in the case of the CoEU was presented in a positive light and that it was actually introduced off the back of feedback received in order to ensure expectations were matched on both sides. 17 Council recommendation on a Quality Framework for Traineeships of 10 March See, for example, European Interns Day ( and #InternsDay on November

30 Linked to the lack of systematically stipulating a trainee s day to day tasks, the most frequent complaint from trainees received by interviewees related to concerns about being assigned sufficiently challenging tasks, being utilised and / or being included as part of the team.. Some thought the best means of avoiding this situation would be to incorporate a clearly defined set of tasks for the trainee in the contract or memorandum of understanding signed with the institution prior to the traineeship. In accordance with the Quality Framework, this would entail, inter alia, the identification of the specific skills to be acquired, supervision and mentoring of the trainee, and monitoring of his/her progress. Monitoring throughout the duration of the traineeship would help to ensure the trainee s actual work corresponded to the job description. Other interviewees, however, took the view that a more fluid, less defined, approach could be mutually beneficial: trainees who proved themselves capable of executing simple tasks proficiently would be rewarded with more complex assignments and increased responsibility. This was how things worked in the real world, they said, and reflected the Framework s stated objective that traineeships should eas[e] the transition from education, unemployment or inactivity to work. Remuneration With regard to remuneration, the Quality Framework recommends that traineeship agreements clarify whether an allowance or compensation is applicable, and if applicable, its amount, a standard which all of the traineeships examined meet. With regard to unpaid traineeships, the Council Recommendation recognises that social costs can arise that may limit the career opportunities of those from disadvantaged backgrounds. The issue of remuneration (or lack thereof) has been at the forefront of the (often negative) attention focused on internships over the years since the financial crisis took its toll on unemployment among the EU s graduates. The EU institutions we looked at all provide the same level of financial compensation to their paid trainees, although most also admit unpaid trainees in parallel to their remunerated schemes. The Council of Europe (which does not pay the vast majority of its trainees) representative acknowledged the apparent contradiction in the failure of a leading human rights organisation to pay its interns but explained that budgetary constraints meant that the organisation was faced with a choice of not paying interns or not having any interns. Monitoring mechanisms While each of the EU institutions we looked at conduct some degree of monitoring activity to assess the experience of trainees within their institutions, the evaluation team were struck by the lack of coordination of these monitoring activities. One of the interviewees explained that while inter-institutional meetings took place on an ad hoc basis there was scope to increase the frequency of these opportunities for dialogue and perhaps develop a common framework for monitoring and evaluation of the surveys carried out by the institutions to gauge the level of satisfaction experienced by EU trainees. Beyond the EU institutions, the Council of Europe explained that dwindling resources had seen that body abandon its previous monitoring activities. The CoE explained that, November

31 while in the past they systematically collected feedback from trainees this was no longer the case. A decision had been taken that given that the resources were unavailable for the analysis of this data it was not worth collecting it. Strategies to follow-up on the monitoring results While the interviewees we spoke with were not short of ideas on what the common problems reported by their trainees were (lack of a sufficiently tailored job description featuring prominently) there seemed to have been little attempt to work with the other institutions to introduce strategies to combat these issues. The EESC s workshop report of June 2014 (see text box below) offers a good example of the kind of solutions trainees themselves would propose to the problems perceived to exist. The report is specifically concerned with the EESC s traineeship but should clearly be of interest to anyone seeking to offer a more user-friendly experience for EU trainees. November

32 European Economic and Social Committee Workshop Report of 20 June 2014 "How to get the most out of my Traineeship" The EESC organised a workshop with trainees of their 2014 Spring session aimed at collecting feedback on their experience and developing suggestions for the scheme s improvement. Based on the workshop s outcome and findings from subsequent surveys and interviews with trainees and staff (supervisors, assistants, heads of unit), the resulting report makes fascinating reading, coming as it does from the perspective of trainees having recently completed the five-month traineeship. Having identified seven areas for improvement it presents recommendations to enhance the EESC s scheme, calling for: Better defined job descriptions for trainees ( All departments could design a specific job description for their trainees, that describes the specific tasks and duties a trainee would be expected to perform during the traineeship in their service ). Increased and on-going monitoring of the traineeship ( Supervisors could provide continuous evaluation to their trainees, or at least both a midterm and an end of traineeship assessment, to ensure trainees are showing the skills [as specified in the job description] ). Standardization of contact with pre-selected candidates, including the provision of a detailed job-description. A uniform and specific briefing for new supervisors (similar to Blue Book scheme advisors) who will manage trainees. Opening the possibility for an enhanced implication of trainees i.e. allowing trainees who have specific skills to move beyond clerical functions and thus increase their value to the hosting unit. Improving/boosting trainees visibility during the traineeship to facilitate employability after the traineeship ( In an institution in which members represent organised civil society from various sectors, networking activities between members and trainees could be encouraged. Members may have advice on how to approach specific sectors, career paths ). The report concludes by offering a model job description for EESC trainees and recommends that a similar workshop take place towards the end of all future sessions to maintain this drive towards improvement of the traineeship. Finally, it seems worth mentioning that the CoEU, EP and EESC interviewees with whom we spoke were very keen to offer their perspective on how a traineeship with the EU could be mutually beneficial to both the hosting institution and the trainee. They also impressed upon the evaluation team their eagerness to increasing dialogue with their counterparts in the other institutions (including the EC) and their interest in receiving copies of this report in order to discuss its findings and recommendations. November

33 Annex 3 Interview findings Background The qualitative interview programme serves to complement the results of the online survey by providing more in-depth and detailed views and opinions on the Traineeship Programme from relevant stakeholders. The sample of interviews includes: Traineeship advisers (15 interviews): This group of interviewees is of particular importance, as they have the most contact with individual stagiaires. The interviews cover advisers views on the selection process, the programme s benefits to trainees and host institutions, the extent to which the traineeship fosters networks of professionals, the programme s efficiency, and monitoring activities are covered. Traineeship coordinators from Commission DGs (10 interviews): Coordinators are closely involved with the programme during the selection process as well as the actual traineeship. Similar to traineeship advisors, they provide feedback in relation to the (pre-) selection process, the benefits of the traineeship, the extent to which the traineeship fosters networks, the programme s efficiency, and monitoring activities. Pre-selection evaluators (10 interviews): Given the considerable resources required for the pre-selection of Blue Book candidates and recent changes to this process, a number of interviews are conducted with pre-selection evaluators. The sample includes evaluators with experience of both the previous and modernised pre-selection processes. The interviews are tailored to collect feedback on the eligibility criteria, the recent changes in relation to the pre-selection process and the overall effectiveness and efficiency of this process. Former trainees (15 interviews): The interviews with former trainees allow us to build on the quantitative, impact-type questions addressed in the survey with qualitative data more conducive to answering questions related to the how and why behind that impact. Since interviewees were selected among willing survey respondents, a sample was chosen that complements the survey in terms of profile and response data. Conducting these interviews after the survey also provides an opportunity investigate particularly interesting or surprising findings in more detail. November

34 Table 3: Interviews with EC officials and former trainees and their relevant previous experience Additional previous experience Conducted interviews Adviser Coordinator Pre-selection evaluator Former trainee Interviewed in capacity as Adviser 15 x 2 2 Coordinator 10 x 2 Pre-selection evaluator x 3 Former trainee 16 x TOTAL Several of the EC officials interviewed had previous experiences of other roles, as shown in the table above. As a result, interviewees often had relevant information on other aspects of the traineeship rather than only in relation to their current capacity. Selection/Application process Recommendation by interviewees Ability to choose whether or not the evaluation was conducted in the office rather than mandate For the final selection process, one way to ensure that the Blue Book is consulted, would be to set a minimum number of candidates to be interviewed by phone to fill a position (3 was the suggested number). EQ3. Pre-selection/Process of selecting candidates for the Virtual Blue Book A series of changes have been made to simplify the pre-selection procedures and move them into the digital age, particularly since October Pre-selection evaluators were asked about the impact of these changes to their ability to carry out their assessments in the most efficient and effective manner. These are explored in turn below. Digitisation Applications are submitted online and supporting documents are provided (also online) only when and if a candidate makes it through the first round of (pre-) selection and into the virtual Blue Book. This change was seen as very positive, resulting in a quicker and more effective assessment process in spite of some initial technical teething problems. The delay in the submission of supporting documents was widely seen as a time-saving, sensible move. As one pre-selection adviser described it: Now that it is online, it is a lot simpler and more efficient. I am happy with the fact that you do not need to check supporting documents. I think it has improved a lot November

35 However, there were some unexpected consequences: because people can submit an application more easily i.e. without submitting all the supporting documents in the first instance they do not necessarily invest as much time/effort in the application resulting in some poorer quality applications (not applicants). In addition, since the assessment of supporting documents takes place later on in the process, applicants can get through the first selection with inflated/inaccurate grades and only be discovered afterwards. This was reported in some cases to increase the workload for the pre-selection evaluators, increasing the amount of time used to filter out candidates that didn t meet the formal requirements. The other implication is that since some applications will not meet the formal requirements but still reach the eligibility phase, these applicants might be displacing eligible candidates. Independent / remote assessment A controversial change which polarised opinion was the capability of the pre-selection to be conducted independently in the evaluators office rather than en masse in a separate designated location. The reported advantages and disadvantages are summarised below: The benefits cited included: Increased convenience, possibility to maintain presence in office; absence of time-waster discussions; and no possibility for peer pressure. The drawbacks included: Inability to focus on evaluation due to competing demands on time and distractions; unable to sense-check scores or discuss/justify divergences; any technical glitches or issues with applications can only be solved with a phone call to the traineeship office cf. quickly asking for help; absence of camaraderie which comes from conducting evaluation en masse; and still time-consuming to print out all evaluations and post them to the traineeship office. Evaluator requirements (nationality, university degree) In order to ensure the process of conducting evaluations is both more fair and free from bias and conducted to the highest possible standard, the requirements for evaluators were adjusted so that: applications now need to be assessed by one official sharing the applicant s nationality and one official from another nationality and all evaluators must be educated to university level. With regards to the former: most interviewees did not have any concerns regarding bias but where people did express an opinion, having different nationalities was seen as a positive step to avoid bias whilst maintaining the insight into the national education system (both for grades and for university calibre). November

36 With regards to the educational requirements of evaluators, it was generally seen as a logical move to ask that evaluators are educated to degree level, not least because this is the minimum requirement for applicants. Requiring evaluators to be AD staff was not always viewed as an entirely positive move. Since the evaluation is a voluntary activity the broad consensus was that it should be open to as wide a group as possible, whilst guaranteeing sufficient knowledge to apply the selection criteria. As one coordinator explained: [it would be] very de-motivating for staff to be told you are not qualified to participate in the evaluation. After all it is a voluntary task. It was also mentioned that any moves to restrict who can participate in the process should be weighed up against the cost of reducing the number of people who are willing to participate. Distribution of points to different selection criteria Changes to the distribution of points across eligibility criteria have been a source of some debate. Feedback from interviewees focused primarily on issues surrounding the awarding of points to academic achievements, which received mixed reviews. For instance, the fact that it was not possible to award points for multiple degrees was seen as a positive shift by some but not by all. The reason for this was described as that the target group for the traineeship, though there is a policy against nondiscrimination based on age, are mainly young university graduates. For example, one evaluator mentioned that: There used to be too much emphasis on work experience but they are supposed to be graduates. Now most of the evaluation is on academic merits which is more appropriate. On balance, there were more positive reviews of the changes overall especially regarding the reduction in points awarded to motivation (which was seen as the most subjective area to evaluate). There were some concerns about the ability to properly reward candidates for their language capabilities, especially the breadth of language spoken. Equal treatment safeguards Since March 2014, at the pre-selection stage evaluators cannot see the age or gender, the anonymity of the applicants being safeguarded. Interviewees seemed to be broadly in agreement that equal treatment was not a cause for concern. Although some respondents mentioned that more context, for example, the age of the candidate, could actually be useful, others took the opposite opinion. One evaluator described how evaluators might discriminate based on age based on conversations with other evaluators. In this case the evaluator mentioned that it was still possible to indirectly determine age through the year of the degree. This would mean that if someone was age biased in their evaluations they could still discriminate, though a majority of evaluators did not believe this was the case. EQ3. Are the eligibility criteria relevant to the purpose of selecting the best candidates while offering the opportunity of traineeship to a large number of applicants? EQ3. Level of satisfaction with eligibility criteria Generally EC official reported high levels of satisfaction with the eligibility criteria but there were concerns raised regarding the difficultly in assessing the academic qualifications of candidates. For instance, the calibre of students from different universities is troublesome when assessors are not familiar with the universities and there is a need to take into account the philosophy of a country to read between the lines of grades achieved. These difficulties are inherent to a process of assessing and hiring candidates from across a wide range of disciplines and countries. It led some November

37 interviewees to stress the importance of providing more and better guidance on how to take into account the context of the individual being assessed. EQ10. Is the size of budget, human and technical resources for the programme appropriate and proportional to what the programme is set out to achieve? EQ10. Level of compliance with selection process deadlines Some interviewees expressed frustration at the short time span between the release of the Blue Book and the scramble to secure the best candidates in a relatively compressed period. This was explained by one coordinator as Once the Blue Book is released we have to make up our minds in a matter of days. There s a deadline to communicate the names to the DG very quickly Another coordinator commented in relation to the deadlines how the previous pre-selection was partly organised over Easter which resulted in little time in assessing and choosing candidates. This was however noted as something out of the ordinary rather than common practice. EQ10. Existence of potential efficiency gains in selection procedure An area which efficiency gains could potentially be made relates to the process of negotiating which unit receives a trainee who is in demand from multiple units/dgs. Currently this process of trading trainees is done on an ad-hoc basis but it would seem from some of the interviewees conducted that this could be managed in a more efficient manner. Final selection EQ3. Eligibility criteria The final selection process was seen as less straight forward by trainees and by EC officials. Through conversations with interviewees one primary recurring concern related to the widespread unofficial process of lobbying. From the perspective of the EC, looking through the Blue Book is time consuming, there are many more qualified candidates than there are positions available, receiving applications directly simply makes their job easier. Having said this, it is important to note that not all officials are receptive to lobbying practices. Indeed, some interviewees informed us they actively ignore direct applications preferring to go through the official process which they believe to be fairer. As explained by one coordinator: We don t take account of lobbying here: we get over 300 s, and I don t use them. [ ] I tell candidates that the Blue Book is used to select trainees. From the perspective of trainees the process is useful if you are in the know, which tends to be correlated with studying at particular universities (The College of Europe and The LSE were mentioned by many interviewees). The asymmetry of the availability of the information is problematic since it offers an unfair advantage to a group of those who are privy to an unofficial practice. One former trainee reasoned: I must say that lobbying to me for a traineeship position appears to be making the fair competition even more haphazard, benefiting those who already have insider knowledge. The lack of transparency involved in the final selection process is not restricted to the issue of lobbying. Interviewed trainees also expressed frustration at the lack of information available regarding who the final selection process works. For instance, one trainee asked whether, having been contacted by a unit, would that mean that November

38 they are still shown as available in the Blue Book or as booked? The process through which trainees are contacted by units is not standardised. Some units will phone a trainee, others will send an . For some it is to see if the candidate is right for the role (language skills and motivation), for others it is only once they have been selected. This causes confusion and in some cases irritation for prospective trainees, who feel that the process is not professional. Also, in a few cases, trainees described how after being interviewed by phone they did not know if they were accepted or not. This made it difficult to answer or accept any subsequent offers from other DGs. One set of unexpected consequences to emerge from discussions with EC staff and trainees related to the length of time between application and final selection which meant some potential trainees had accepted positions elsewhere and issues with resources namely office space which is some cases prevents units from making full use of their quota (see also section on technical resources) EQ5. Level of geographical representativeness of trainees This was not evident in the interviews what level of geographical representativeness there was among trainees. This will however be covered in full through the desk study. In a few cases there were however reports of increased applications from countries with high youth unemployment, such as Italy, Portugal and Spain. EQ9. Perception of gender balance Interviewees did not express any particular concerns regarding gender balance. Advisers tended to reserve judgement regarding gender balance, saying that their experience was too localised to their unit. However the general consensus was that this was not a major problem for anyone. Traineeship design EQ14. Which of the current activities or elements of the programme would be likely to continue and in which form if EU support was withdrawn or substantially decreased? EQ14. Willingness of trainees to participate without a grant For most trainees the grant was not the primary reason for applying though many described it as an important element. There were also in a couple of cases former trainees who described how they would not have been able to complete the traineeship without the grant. For further discussion see EQ10. Level of satisfaction of trainees with grant provided. EQ14. Willingness to of EC Services to engage trainees without a grant or selection process Several interviewees noted that the use of atypical trainees had gone down over time. In these cases members of EC Services expressed hesitation to the practice of employing unpaid trainees. The main reason being that these traineeships not being economically equitable, one evaluator even describing it in terms of slave labour. However, one practice reported was that of units not being allocated a Blue Book trainee being more willing to accept atypical trainees. In a few cases coordinators were willing to take on Blue Book trainees that did not reach final selection as unpaid trainees. The rationale cited was that these trainees November

39 were very qualified, had made it far in the process and by taking on an unpaid traineeship they would at least have something they could fall back on. EQ10. Relative cost and effectiveness of other mechanisms in comparison with the traineeship programme On the cost side of hosting Blue Book trainees most EC officials described it as huge added value with relatively little investment, which was especially true for advisers who saw most of the benefits first hand. Advisers described how the trainees were a great benefit and contributed a lot in terms of workload. Being operationally quickly and in most cases not needing much guidance. Among advisers there was widespread consensus that the trainees represented a cost-effective way of giving work experience to young graduates. As one adviser explained; the cost is low and the trainees are more specialised then most of the people working here. They are also highly motivated. So it is a good contribution. In addition, there were some difficulties in answering this question by EC officials since they at times did not have the ability to readily compare different schemes and alternatives to Blue Book trainees. Also, interviewees noted that that is was difficult to estimate what the Blue Book selection process took in terms of time and resources. In regards to this, one coordinator described how it is often difficult to get volunteers to participate in the pre-selection process. The time limit for pre-selection can be quite short and there were also reports of difficulties in motivating people. Mainly two different categories of alternatives to the Blue Book trainees were however identified by interviewees; other traineeship programmes (such as atypical trainees) and temporary contract staff (intérimaire). Firstly, in comparison to other traineeships such as the atypical trainees the Blue Book trainees were described as having broader responsibility, freer role and more flexible usefulness. A majority of interviewees also described little experience of traineeship schemes other than atypical trainees. In these cases atypical trainees were noted as being used for more specific tasks and were employed during shorter time-periods which reduced their utility. Primarily this was as a result of them not being in same position to develop a deeper understanding of the work in contrast to the Blue Book longer traineeship duration. As explained by one adviser: A-typical trainees are of less quality. This is not because they are bad, but they have much less possibilities: they have much less time, they usually stay less than three months. This means that they cannot do as much as Blue Book candidates who stay five months. Furthermore, each DG appears to have their own procedure in handling atypical trainees. Some DGs for example, only allowing atypical trainees that are undergoing studies to participate, while others were noted to accept atypical trainees that had finished their studies. Blue Book trainees were also described as having a special status since they had undergone a more rigorous selection process, were often highly qualified and had a strong academic background which contributed to the positive view of their capabilities. Conversely, one evaluator argued that atypical trainees had more experience of national administration and work experience, for example already being able to write briefs when starting a traineeship. The predominant view among EC Services was however that the Blue Book trainees contributed a lot more than what it cost them in time and money. Secondly, temporary contract staff or intérimaires, were described as an alternative to trainees. These were often used to bridge vacancies when regular EC employees went on maternity leave or other circumstances that required additional assistance. In fact, November

40 a number of EC officials reported that former Blue Book trainees had actually taken up intérimaire positions after their traineeship. In these cases, EC officials were positive to the idea of contracting temporary workers though sceptical if it was actually cost effective. One coordinator described how most units all want Blue Book candidates, rather than intérimaires. This was a result of trainees being really good and having a strong university background. Other EC officials also argued that it would probably be more difficult to find as good candidates by going through an intérimaire agency, but based this more on what they had heard rather than on first-hand experience. Traineeship period Recommendations from interviewees Improve and structure the exchange with other traineeship schemes was suggested by both advisers and trainees. As of now there was little reported contact between the different schemes. Manage expectations at an early stage, both for the trainee and the unit. In a few cases trainees reported disappointment about the tasks given, not knowing what to expect. Having a clearer picture of what the actual work consists was reported by trainees, advisers and coordinators in order to make it easier for trainees to be confident in their choice of DG. Formalise the Alumni network, give it a clearer identity and make it a useful tool for networking was suggested by a trainee. Having trainees in same locations as other member of the unit was suggested as being the best way of integrating them in the unit and avoid trainees feeling isolated and disconnected. A more systematic approach to coordination between the different traineeship schemes such as joint training sessions was suggested by a coordinator. This was so as to make use of economies of scale and at the same time foster inter-traineeship contact. If the grant would be less that would in some cases exclude qualified applicants that do not have the means to pay for the cost of living out of pocket. A voluntary mentorship by trainees that are ending their stage was suggested by a trainee. The main purpose being to familiarize new trainees, especially non-eu citizens, with what to expect in terms of work, the practical requirements, help finding accommodation etc. Additional help for non-eu citizens when arriving from Traineeships Office and Stage Committee was suggested since they are often unfamiliar with the practical procedures of the new environment. November

41 EQ1. To what extent have the programmes aims and objectives proved relevant to the needs of the EC and the university graduates in pursuing a career in European affairs? Due to the overlapping nature of this evaluation question it is covered under the heading Impact of traineeship. EQ2. (relates to EQ1 as well as EQ8 ) To what extent are the experiences gained during traineeship useful to access labour market? EQ2. (this relates to EQ1 and EQ8). Level of satisfaction of former trainees with skills imparted A majority of former trainees agreed that they had learned a great deal during their traineeship. Often describing how they gained useful insights into the institutional processes and procedures, and the internal workings of the commission as well as more applied and practical skills. Frequently, as a result of unit s specialisation, trainees got highly specific knowledge in relation to certain topics, such as a specific field of law or policy area. At times this was a result of trainees devoting large amounts of the traineeship to certain research projects. Several trainees also noted how they developed language and writing skills during their traineeship, as well as learning how to compose certain documents such as briefs and impact assessments. As one former trainee described it: Most useful, I worked on a very small niche policy, really expanded my knowledge and expertise on this issue, (European film industry and policy), learned in practice how policy was made on that issue, secondly for my language development and working with multi-national colleagues was stimulating. In alignment with the aims of the traineeship, in several cases extrainees reported how they were given the opportunity to applying knowledge from academia in a practical context. Many former trainees pointed out that there was a steep learning curve; trainees were often treated as equals by EC officials with high degrees of responsibility. Being treated as equals also required trainees to work independent which in turn was reported as placing responsibility for learning on the trainees themselves. In several cases both advisers and trainees described the learning during the traineeship in the end being a function of the willingness of the trainee. In addition, many trainees pointed out that learning is not only confined to the specific unit but there are plenty of activities organised by the Stage subcommittees as well as other events hosted by various organisations and DGs that contribute to the acquired knowledge and skills of the trainees. Only a few of the former trainees reported that no specific skills were acquired during their traineeship. In these cases, the skills learnt were often judged by the field the former trainees came from and presently worked in, with a few ex-trainees from more specialized fields reporting disappointment in not learning anything that could be applied in their current work. As one trainee commented: Maybe I expected to learn more about stuff that could be applied to my field. The trainee was not as useful for my background. The few trainees that did not report learning any specific skills did however highlight the increased knowledge and understanding of the institutional framework, the value of a hands on experience in a big institution which relates to the aims and objectives of the scheme itself. November

42 EQ2. Degree that EC Services feel trainees learned useful skills There was broad agreement among EC Services that trainees learnt useful skills. Several EC officials pointed out that many trainees had strong academic backgrounds but often limited work experience. For many trainees the traineeship was the first experience in a professional setting as well as working in an international and multicultural organisation. Trainees are expected to manage their time, meet deadlines and interact with colleagues, in the words of one adviser, trainees were not here to make photocopies and coffee. Another point raised by EC officials was that trainees learn how analyses are applied in real-life and not only for academic purposes. Several EC officials described how, often, this was an eye-opener for trainees who frequently only had a theory based view of how for example a law is applied. That, in the case of applying a law, it is not simply a question of following the stated rules but there are other political considerations to take into account and that needs to be balanced. In essence, trainees get the factoryfloor view of policy making and get an insight into how the institution works in a practical sense. However, the type of work that trainees were tasked with was reported to vary greatly depending on which DG and unit they were posted to. In some cases, this left trainees underutilised. Managing the expectations and monitoring the satisfaction, both from the trainee s point of view as well as the unit was highlighted as an important potential area for improvement. As one coordinator commented: The happiness of trainees also depends on the unit I think, expectations they had, tasks they got to work on EQ2. Level of satisfaction of trainees with networks built Most former trainees recognized that they had greatly improved their contacts and personal network. As one trainee described it: During the work you met interesting people that you make relationships with. I think one of the best parts was the multinational pan-european working environment, that unique EU-experience that you can only get in Brussels. Several former trainees had also managed to leverage the network built during the traineeship to get jobs or were planning to make use of it in the future. What was emphasised was that there were plenty of opportunities to meet interesting people to socialize with and build a useful network. Becoming an insider within the EC environment also makes it easier in terms of knowing who to talk to and where to look for job opportunities. Being outside of the Brussels environment was described as making it more difficult to get access to important information and possible job opportunities. For example, getting the opportunity to do temporary work as an intérimaire. In a few cases there were also reports of trainees being grouped together and given offices in a separate location, away from the units. In some cases this was a space issue but many experienced this as having a negative impact on their ability to build relationships and network with colleagues as well as being part of the day to day working environment of the unit. This was commented on by several former trainees and EC officials, one former trainee suggesting that the outcome of the traineeships to a large extent depends on the social dynamics of the unit and in the way the trainee is integrated in the daily tasks of the unit. November

43 One problem flagged by a respondent was that the traineeship was too focused on lobbying and networking in order to procure a possible future job. In the experience of this trainee, it stole focus from the actual content of the traineeship which should be the main priority. One trainee argued that: Everyone is job hunting; everything is about lobbying and networking. Traineeship is all about networking. Which I think shouldn t be the main focus but rather focus on the content of it. One suggestion from a former trainee was that the present alumni network, AlumNet (administered by the Stage Committee), needs improvement. It does not appear to have a clear identity and there exists a whole eco-system of different Facebook and LinkedIn groups which makes it difficult to identify which are official representations of the Alumni organisation. In addition, the webpage to actually sign up to become an alumnus is currently offline (as of 11/08/2014). EQ2. Level of satisfaction with career advice given It became clear that most former trainees were very satisfied with the support and help their advisers gave for their next step after the traineeship. Advisers performed a range of tasks to help their trainees; they gave recommendations of who to contact, sent s to introduce trainees to employers and suggested jobs that might fit the qualifications of the trainees. In sum, advisers were described as being very helpful in terms of career advice. Or in the words of a former trainee: My stage adviser helped me to look up organisations that were dealing with my unit. Provided me with information, and any time when I needed help or recommendation, they offered to send an . In addition, many former trainees reported how they after finishing the traineeship had a clearer idea of what interested them. In some cases trainees describing how they discovered new areas of interest while others noting that working in the EC was not for them. Job fairs organised by sub-committees were also mentioned as being a good opportunity to explore the different jobs available within the EU context. However, in a few cases, respondents mentioned how the job fairs did not tailor for their specialized interest or field, the job fairs organised being more suited for people with a background in economic, legal or political studies. EQ4. Level of interaction between trainees and participants of other programmes It became clear that the exchange taking part between the different traineeship schemes varied significantly between trainees. Several did however report little sustained contact with members of other traineeship schemes. In these cases, even though trainees reported there were common welcome conferences and other coordinated events, since there was no day-to-day interaction between other traineeship schemes there was little possibility to meaningful interaction. As one trainee explained It s true that some coordinated events between different institutions were held but it was mostly hello/goodbye relationships. Partly this lack of contact with other traineeship schemes can be explained as a result of trainees not sharing the same location, being physically isolated from each other. In one case, an adviser described how trainees were self-contained as a result of the location of the DG, not having the possibility to socialize in the same extent with members of other traineeship schemes. November

44 Conversely, there exists a Stage liaison committee to facilitate exchanges between the different schemes, and many of the Stage sub-committees provide an opportunity to interact with other trainees. An issue recognized by an adviser and also former trainee was the lack of apparent coordination between EU institutions in regards to traineeships. Furthermore, suggesting introducing joint training sessions together, where all schemes could participate. This was argued to increase the contact between different traineeship schemes as well as providing some benefits in terms of economies of scale. Increased contact with other traineeship schemes was also suggested by a couple of trainees who described it as an area of improvement. EQ6 To what extent does the current programme satisfy the criteria presented in the Council Recommendation on a Quality Framework for Traineeships? EQ6. Level of satisfaction of trainees with traineeship conditions The recommendations to member states outlined in the Council recommendation on a Quality Framework for Traineeships are broadly categorised under nine categories and are in large part answered through the desk research. There was however in a few cases themes in the Quality framework which interviewees touched upon. The remuneration and duration of the traineeship will be discussed under separate headings (see EQ10. Level of satisfaction of trainees with grant provided and EQ10. Appropriateness of length of stage). Firstly, in regards to the recommendation Learning and training objectives trainees had differing experiences (see also EQ2. (this relates to EQ1 and EQ8). Level of satisfaction of former trainees with skills imparted). In a few cases former trainees explicitly expressed disappointment in how they experienced the learning and training elements of their traineeship. In these cases, the stagiaires noted that they did not have adequate tasks that would help them in acquiring practical experience and relevant skills (effectively reaching the learning objectives). From the viewpoint of advisers and coordinators, reports differed also in regards to the council s recommendation. It was not evidenced in the interviews that best practice in regards to training and learning objectives was practiced systematically though there were notable exceptions. Some advisers and trainees report how learning objectives were continually monitored and revised during the traineeship, advisers adapting the tasks in line with the trainee s educational background. Secondly, in regards to the recommendation number 11; the conditions under a traineeship may be extended or renewed; several trainees as well as EC officials argued that there should be more flexibility in extending the traineeship period (see also EQ10. Appropriateness of length of stage) Thirdly, in regards to working conditions applicable to the trainee, the rights, coverage in terms of health care and compensation is also mostly covered by desk research. However, most trainees described the traineeship conditions in terms of working hours, as being very good. Work hours often being flexible and other there being good access to office space and technical resources. In some cases trainees even had access to their own private offices. A few former trainees described how they actually stayed longer hours as to get as much as possible out of the traineeship: Well I thought working conditions were really good. Stayed late but was it was voluntarily, I preferred this, I wanted to absorb and get the most of my experience. November

45 There were however second-hand reports of trainees at other DGs that had too much responsibility and were described as over-worked by fellow trainees as well as coordinators. The only first-hand negative experience in terms of working conditions reported was that of working in open-spaced office environment, which was described as being noisy and difficult to concentrate in. One coordinator also raised the issue of trainee s rights to special leave but did not elaborate in detail only describing it as not fair and trainees not enjoying the same rights as regular employees. EQ10. Is the size of budget, human and technical resources for the programme appropriate and proportional to what the programme is set out to achieve? EQ10. Difference between the provision of trainees and demand from EC Services When asked about the amount of trainees the DGs could absorb, not surprisingly, most advisers and coordinators agreed that the demand superseded supply in terms of the provision of trainees. Many qualified applicants with good profiles are consequently not given a stage. A majority of EC interviewees believed that there was a large absorption capacity for trainees; We are willing to have more trainees than we have. And we have pushed to have more. The number we receive is low compared to how many are qualified. The only constraint coordinators and advisers mentioned was budget and in a few cases office space. In a few cases coordinators reported trying to arrange an unpaid internship when possible for applicants not being selected. With regards to the allocation of trainees within the DGs, several coordinators admitted how there often were an element of negotiation to appease the different units who wanted trainees. In order to achieve equitable distribution in relation to trainee allocations, some DGs would allocate trainees on rotating schedule between units. A few interviewees did however comment on this fact as being disproportionate since some units were small and had a more manageable workload. Several EC officials also noted that it was unclear how the Traineeships Office allocated trainees to the different DGs. In a few cases, coordinators argued for allocation more based on workload rather than on, what was assumed, the size of the DG. Similarly, a few coordinators described how the allocation of trainees hadn t reflected the growth of the certain DG in recent years, requesting more transparency in how allocation is decided upon. In addition, several coordinators answered when asked about the amount of trainees, that they thought they should receive more trainees in comparison with other DGs. Noting how the allocation was not entirely equitable and the procedure of allocating trainees lacked some transparency. As a result of having few Blue Book trainees, one coordinator described how they were compensating through more atypical trainees. As one coordinator explained: The problems are more the allocation of trainees to DGs, the quotas! We have a very low quota. Because of that we have a lot more a-typical trainees. This practice was however reported to make them uncomfortable since they were asking the trainees to work for free, since there was no budget to pay for them. EQ10. Level of satisfaction of trainees with grant provided In terms of cost of living, many former trainees described Brussels as an expensive city and the grant barely covering all expenses. However, the grant was described by many interviewees as still very generous compared to most other traineeships in Brussels. The general consensus was that the grant was sufficient to cover expenses November

46 much however depended on what accommodation was secured. For example, one trainee mentioned that I would say it [the grant] was enough, it was fair. [ ] I think it could be helpful to get help with finding accommodation, this is the main problem for trainees in Brussels. In addition, in a few cases trainees cited how the lifestyle trainees were accustomed to affect the perception of the grant, trainees from higher income countries being more dissatisfied with grant than other trainees. In one case a coordinator flagged the difference in cost of living between Luxembourg and Brussels as a problem. Trainees doing their traineeship in Luxembourg were described as having significant difficulties in covering the costs of living with the grant provided. There was however no room for changing the grant since that was explained to be considered as an unfair or discriminatory practice. EQ10. Willingness to accept smaller grant The size of the grant was not cited as the main reason for most trainees in their decision to apply to the traineeship. The learning experience was in most cases the primary reason for applying to the traineeship. However, in a few cases former trainees described how the fact that the traineeship would cover their expenses was a major factor in their decision to apply. In these cases not having a grant would have jeopardised their ability to participate in the traineeship. As explained by one It was an important factor that I knew it was paid, if it wasn t paid I don t sure if I could do the traineeship. This has wider implications on equity of access and what persons would be able to actually participate in a traineeship if a grant was not available. EQ10. Level of satisfaction with support given and received The majority of the advisers and coordinators agreed that trainees received sufficient support. There was however some difference in how this was carried out. Many interviewees noted that it depended on the unit and what work was involved, some trainees not needing much day-to-day support but at other units supervising and support was needed in greater extent: [I]n some units, after one month they are very independent. In other units, not as feasible, policy making for example requires them to be in the job for few months before they are independent. Also, in a few cases advisers admitted that the support given was not sufficient as a result of the traineeship coinciding with a busy period, several missions and holidays, making it difficult to find the time to supervise. One mitigation strategy for this reported by some DGs was to share the responsibility for the trainee and assign more than one adviser. Furthermore, the support given, primarily by advisers, was reported to differ in style and content. Some advisers opting for a more hands-on approach with regular meetings and day to day interactions where as other advisers gave trainees a task and leave the trainees to work more independent, treating them as regular colleagues. One point raised by an adviser is that trainees from other schemes attend a lot more events. It was suggested that there was room for the Traineeships Office to remind relevant institutions of the possibility to organise events for the trainees in a more consistent manner. From the perspective of the trainee, it was clear that most advisers were perceived as very dedicated in supporting their trainees. Most trainees described how they developed both excellent working and personal relationships with their advisers that continued after the traineeship period. As described by one trainee: My stage adviser November

47 was fantastic. I really appreciated to work with her; she loved her work very much. With that kind of enthusiasm it follows into your own work. In a few cases former trainees experienced little support from their advisers, often as a result of advisers being overwhelmed with work or away on mission. In these cases trainees described feeling isolated and unsupported, not having anyone to reach out to for guidance and questions. In regards to the coordinators, most trainees had little sustained contact with them; mostly the contact was focused to the beginning and the end of the traineeship. This contact was in these cases usually manifested through the induction and feedback sessions. Most former trainees were happy with the contact they had with coordinators, however in a few cases the coordinator experience was described as less than satisfactory. The impression given by the trainees was that if there was no issues with the traineeship, usually any additional contact with coordinators was not necessary. In relation to support, one point raised by a trainee was that of voluntary mentorships by previous trainees, especially for non-eu citizens, to get them familiarized to a new city, logistics, what to expect out of the traineeship and other practical advice. Overall the most trainees were satisfied with the work the Stage Committee performed. As described by one trainee: I think it was the best experience in the whole traineeship; it was the existence of these Committee groups. More specifically the sub-committees were people could join. I joined a language sub-committee and took cooking classes, it was really good experience. In only a few cases former trainees described how they were disappointed with the Stage Committee. One interviewee recalled how he perceived members of the Stage Committee supporting each other more than the trainees, not budgeting in an equitable manner which resulted in some sub-committees not receiving any funds at all while others receiving more than they could use. In terms of willingness to receive less support, most interviewees had a difficult time judging this issue, citing how it was difficult to imagine an what impact it would have. EQ10. Level of satisfaction with technical resources Most former trainees did not experience any short comings in relation to technical resources. A minority of EC officials and trainees from different DGS experienced problems in relation to office space, namely the constraints in the amount of office space available and that in some cases trainees were clustered together away from the unit. Especially the latter was both by trainees and advisers noted as being especially detrimental for the social and professional experience of the trainees. EQ10. Appropriateness of length of stage There was close to unanimous agreement among trainees that the length of the stage should be longer, or at least that it should be possible to extend it a couple of months. As explained by one trainee: Biggest complaint is that I would ask for a longer traineeship. Did not have enough time to enjoy it, in the sense that just when you are finished learning all the ins and outs of the unit and the system as whole you leave.. This was reported by many of the former trainees who described how they had just about reached their optimal level of functioning after five months when the traineeship was over. Several trainees argued that leaving at that point when they were just starting to feel confident and integrated in the unit was not beneficial for the unit or themselves. To this end several trainees described how they tried getting temporary November

48 contracts or an extension but often their attempts were not successful. This was also noted by advisers and coordinators who admitted to wanting to keep on trainees for longer or at least to preferring to have flexibility in this respect. EQ8. To what extent has the programme been effective in achieving its main aims and objectives (As per ToR p.4)? EQ8. Satisfaction with tasks provided A number of trainees described the tasks during the traineeship as stimulating and diversified, often applying insights and knowledge from academia in the context of the EC. Trainees describing how progressively the task changed during the traineeship, reflecting the increased competency and knowledge the trainees had acquired. This was also described by both trainees and EC officials as dependent on the trainee, the level of autonomy, motivation and previous experience determining the tasks given. Or as one adviser described it The stage is what you do with it, and I tell everyone that. You could spend boring or brilliant months nearly everywhere in the EC.. However, in a few cases former trainees reported that the work was unstructured and ad-hoc in nature. Often it involved more administrative and operational tasks that did not require integrating and cooperating with members of the unit. A couple of former trainees noted how there was no systematic development plan or programme in place for the traineeship period. One trainee estimated that on personal experience one out of five trainees had faced similar problems during their traineeship. Likewise, as a result of not being assigned enough work one trainee described how they ended up searching for jobs instead; There were days when I didn t have anything to do. There wasn t that much to do so we were all sat in the same room and we just spent the time looking for jobs. This was also pointed out by an adviser, arguing that units who apply for trainees should have a work plan but that it was not always the case. EQ8. Level of agreement of former trainees that tasks were challenging Most former trainees described the tasks performed in terms of challenging and demanding but feasible. Frequently former trainees described how there was a sharp learning curve with was challenging in terms of learning how the unit works, terminology and high degrees of responsibility. One trainee described how the task were not only challenging in terms of difficulty: I was both academically and politically challenged, I learned to be very sensitive to cultural differences between countries. There are 27 different views on subjects, couldn t imagine that the council negotiated all these different views ranging from lobbying, nationalities and personalities. So for me it was both a personal and academic challenge. Most trainees had use of previous academic experience though in some cases the work differed from their area of study. Trainees frequently noted how they learned by working on the job and over time becoming more confident in the tasks. At times former trainees described how it all was a bit overwhelming and scary : In the beginning it was difficult, but the difficulties were that they treated me as a colleague. I wasn t used to that kind of responsibility. A significant smaller number of former trainees were disappointed in what tasks they had been assigned. One trainee noted how another fellow trainee who was highly qualified with several degrees was only used for proof reading. In a similar case, a trainee expressed severe disappointment in not being stimulated or challenged at all during her traineeship, describing how the tasks were so easy that a high school student could have done them. November

49 EQ8. Level of provision of policy-related tasks From the point of view of coordinators and advisers, most describe how the trainees are given a high degree of responsibility from the start and that they are in many cases almost treated as regular colleagues. Many advisers and coordinators were careful to point out that they are not tasked with only clerical or administrative tasks but actually involved in the day-to-day policy related work within the unit. As evidenced by one adviser: We try to involve them into the whole range of work for the unit. Depending on their level of autonomy, we can give them different tasks, but there always needs to be someone from the unit involved. They do important work, compile substantial analysis The amount of policy related tasks given to trainees were often dependent on which unit the trainee ended up in. In some cases trainees played a more supportive role, doing desk research or preparing longer research papers. As one coordinator pointed out, even though the work might be very specific in order to carry it out the need a firm grasp on how the whole system around it works. EQ8. Level of satisfaction of EC officials with work provided by the trainees Blue Book trainees were described by a vast majority of EC officials as having a special status among trainees. EC officials regarded them as highly qualified, motivated and being able to become operational quickly. Not only were trainees noted to help with the workload but often described in terms of bringing new ideas, sharing current academic research and providing a fresh perspective on policy making. A number of EC officials were in agreement that the traineeship was a mutually enriching process, as one adviser explained: It is a very valuable experience. It is a two way thing. You teach but you learn from them as well. [ ] If the opportunity arises [to be an adviser] it gives me great pleasure. In a few cases, trainees were even described as being vital for achieving deadlines and managing the workload of the unit: The trainees are crucial. They are vital! We have tight deadlines and we have to cope with enormous assessment of facts. The trainees are fully integrated in our work. Furthermore, several EC officials described how an important task that trainees contributed with was that of side projects, answering questions that a unit might not have time to answer. Trainees could as result of their freer role in the unit devote time to more strategic and niche projects that were removed from the day-to-day work. Impact of traineeship EQ1. (EQ8, EQ2) To what extent have the programme s aims and objectives proved relevant to the needs of the European Commission and the university graduates interested in pursuing a career in European affairs? EQ1. To improve the trainees employability perspectives Overall, former trainees were in agreement that traineeship definitely had increases their employability, especially in relation to pursuing a career in the EU and the international sphere. A number of trainees noted how their traineeship was a contributing factor to them getting their current job, often being a point commented on during job interviews: I did many interviews, and this [the traineeship] was November

50 something everyone asked me about. It has been the most commented point of my CV As the Blue Book traineeship is acknowledged as being a highly competitive traineeship to be selected to, it was considered by most trainees as very meritorious. Not necessarily for the skills acquired but rather for the competition in getting selected. It does not open all doors however; some former trainees mentioned how the experience of working in Brussels was more of a requirement rather than a distinguishing merit when going through the EPSO process. In addition, trainees from new member states described how the traineeship was regarded with extra importance, it often being the most commented point of their CVs. EQ1. Understanding of the objectives and goals of the EU integration processes and policies Several trainees noted how having an understanding of how EU policies and procedure function had helped them in their professional life. Through the insider knowledge that the traineeship provided, interviewees described how they better knew how to navigate the procedures and bureaucracy as well as where to look for information. Having this point of entry was often described as an asset, several trainees commenting on this fact as being useful in their current job. Trainees also noted how they saw how EU impacted their personal life. One trainee explaining how EU legislation on issues such as roaming charges and cross border mobility was useful in her day to day life. Furthermore, in a few cases former trainees commented how the public s and media s perception of policy making in the EU was a lot different from what they experienced. As one trainee describe it: It [perception of EU integration] only changes if you see it with your own eyes. Most people are too far away from institutions to see how it influences their daily lives. How much actual power it has to change the circumstances of their situation. In contrast to the more negative public view, former trainees experienced the policy making process as being a lot more dynamic and energized with dedicated people trying to make a real difference for member states. EQ1. Put into practice the competences acquired in formal education In several cases trainees reported making use of their academic studies and applying classroom concepts on the tasks given, in the words of a former trainee: University studies were compatible with the work I did. It was interesting to practically apply classroom concepts to the daily workings of an institution. Depending on the DG and the unit the academic experiences were not always applicable, in a few cases former trainees noted that the work was focused on a specific area with its own terminology and technical difficulties that were difficult to be prepared for in terms of previous academic experiences. However, frequently there were parts of their formal education that could be applied in some ways even though the subject they were involved in was foreign. EQ1. Continuously renewed academic knowledge, dynamism and enthusiasm of each successive intake of trainees There was broad agreement in the value the trainees bring in terms of insight into current academic knowledge and the new ideas they bring to the table. Several EC officials cited examples where trainees had brought about changes and had made valuable contributions to the work of the units. At times this involved using novel techniques and approaches that would otherwise not have been available. Frequently November

51 mentioned in relation to knowledge were trainees with unique language skills along with trainees with special skillsets that could be used for more tailored and welldefined projects. In relations to this, some more specialized units noted how having young team members helped them understand the younger demographic better, giving insights into for example how to communicate appropriately with certain target groups and to be more mindful when using jargon. Furthermore, several interviewees described more intangible benefits such as having young and not as institutionalized members of a unit asking naïve questions which in turn forced EC officials to explain their practices and question how they conduct their day-to-day work. As explained by one EC official: A lot of the time the trainees are an extra pair of hands; a bit of new blood into the office environment is also good, younger people with new ideas to shake things up. Related to this, EC officials frequently noted how there were definite social benefits in terms of getting young enthusiastic people with unique experiences and fresh ideas that, as one official put it They are windows to the exterior world. EQ1. Trainees as goodwill ambassadors for European ideas and values Several former trainees described how they were very impressed of the professionalism, dedication and competency of their colleagues in their former units. Similarly, a number of stagiaires also noted how the experience of being in a multicultural environment with people from different backgrounds working together towards a common goal was very positive and made them feel more engaged towards the EU as a whole. For instance, it was noted how even though there was a great deal of diversity within the administration, there was also unity. Even though some trainees described how they were critical of the bureaucracy surrounding the EU they were positive towards the ideals and values of the EU. The fact that former trainees act as ambassadors was also commented on by coordinators and advisers in a few cases, describing how these were intangible benefits when trying to estimate what benefits the traineeship brings. As one coordinator described it: After their traineeship they [trainees] can talk about their experience, hopefully favourably. This is a way for us to make the Commission better known. EQ8 (Q1, Q2). To what extent has the programme been effective in achieving it aims (as per ToR p.4) EQ8. Satisfaction with employment The majority of trainees were satisfied with their current jobs. The range of type of job varied greatly, everything from tourist guide to diplomat in-training. Only in one case did a former trainee express negative views on their current employment. In this case the former trainee did however consider the current job as a stepping-stone to more rewarding future jobs and was content in the fact that it would lead to something better. EQ8. Level of European sentiment among trainees A common experience for many of the trainees was that regardless of being sceptical or positive to European project, they became more aware of the positives and the negatives after having completed the traineeship. Not necessarily reinforcing their prior beliefs but rather giving them a more detailed and nuanced view of the shortcomings and the advantages. As one trainee explained: I m pretty sceptical, I m November

52 feel a bit disappointed of how Europe is ruled. The commission helped me understand things are much more complicated however, and that things take time. The time spent in the stage made the trainees able to understand it more objectively or as one former trainee described it reinforced my realism about the EU. Most did however agree that they were more positive than negative and being more engaged in EU than before the traineeship. Furthermore, several trainees noted how being a part of a unit, getting to know the people who work there and the professionalism and dedication they put into the work had made them proud of having been part of the institution. EQ8. Level of understanding among former trainees of EU integration In most cases former trainees described how they had a positive view of the EU integration project. Especially being exposed to the multi-cultural environment that is the EC and understanding the barriers and limitations that exist was described as being especially useful for the trainees understanding of EU integration. As one trainee described her experience: It influenced my view positively. You get to meet people from different backgrounds/cultures. [ ] It creates opportunities to exchange knowledge and values across cultures. Bringing people from all over Europe together [in the traineeship] helps with European integration. Similarly, one trainee noted how most of the general public is too far removed from the institutions to understand the influence and effect on their daily lives. In addition, several stagiaires noted how the traineeship provided a deeper understanding of the complexities involved and an added insight in the difficulties in aligning different interests and agendas to reach decisions. Even though a couple of former trainees expressed criticisms against how the integration and EU in general was managed, a majority of trainees believed in the integration process. EQ8. Enthusiasm for EU career The majority of former trainees expressed positive opinions in regards to working in the EU. Several already had EU related jobs and in some cases were applying for jobs through the EU selection procedure. In several cases interviewees noted that they already were planning for an EU related career with the traineeship not affecting that decision. In a few cases the traineeship had the opposite effect, convincing trainees that the work in the Commission or more generally in the EU was not for them. As explained by one trainee: After seeing how the commission works, I know how it works, not something that I would like to do, not that I bare any ill will towards it, it s just something I don t want to do. EQ7. What are the main added values of the programme? EQ7. Level of sustained professional contacts between former trainees Most trainees reported having frequent contacts with former stagiaires often developing deep friendships; We are still in touch, it was a small team with trainees from all over the world. This was one of the best things with the stage, meeting all these really amazing people. Similarly, several former trainees noted how the amount of new relationships and friends acquired was one of the major benefits with the traineeship. Though at times not being in the same country, some interviewees described how they would try to meet up when the opportunity arose. Other trainees, described staying in contact through and social media. November

53 EQ7. Level of sustained professional contacts between former trainees and EU officials A majority of former trainees described being in touch with former advisers or other members from the unit they did the traineeship at. The degree of contact varied, mostly the contacts with former colleagues and advisers were career related and involved providing recommendations or other useful contacts for future employment. In a few cases the contact had bridged the normal professional relationship into more of an informal mentorship. The adviser, in these cases, taking on more responsibility for helping the trainee in their professional development than could said to be a part of their job description. To this effect, one trainee described how a former adviser had become not only a mentor but a friend, regularly exchanging s and staying in contact. Especially the adviser were reported by former trainees as being the main point of contact, many trainees making use of this in their career advancement, acquiring references and useful contacts. Frequently coordinators reported the contact with former trainees after the traineeship to be ad-hoc in nature, at times receiving the odd now and then. The only stagiaires that coordinators appear to be in contact with were the ones that return to the DGs either as permanent employees or as temporary contract workers. Advisers, on the other hand, report frequent contacts with former trainees, often describing how they follow their trainee s careers and keep them on their radar. To this point, one adviser noted still staying in contact with trainees who did their stage eight years ago: That is one of the added values of the traineeship. I follow the development of the trainees, and I get use from them. And I am still in touch with former trainees from eight years ago A number of advisers underlined the importance of helping their trainees through sharing their personal network and assisting them in their career. As explained by one adviser: We are still in touch. I consider this part of my role as adviser. To help them connect with people for future employment. I still know all my former trainee, we exchange s, we follow up, I follow their career, they can always ask me to ask specific question, to help them find the right person in the EC. Furthermore, in one case an adviser also described how using his network of former trainees to pass on questions to them as well as consider them when there were suitable openings that fit their backgrounds. Monitoring/Feedback Recommendations by interviewees One adviser asked for more concrete feedback on practical issues such as how much time adviser spent with the trainee, citing how the feedback received from the TO was more generic in nature. EQ13. Timeliness of monitoring activities In regards to the survey that the TO collects from the advisers and trainees at the beginning and the end of the traineeship, most interviewees did not express any particular objection to the timeliness. A few advisers did however not recall actually having taken part in a survey at all. November

54 EQ13. Level of sufficiency of monitoring information collected Several coordinators and advisers noted how the monitoring information collected by TO as being superficial in nature, not always being useful: The information is quite general. I think they should improve the survey. Maybe re-think some of the questions. A more detailed suggestion was given by another coordinator, describing how they wanted feedback on what tasks are perceived as good, whether trainees or units are satisfied and which areas need improvement. On the level of the DG, a number of coordinators described how they tried to conduct midterm and more commonly exit interviews which were more informal discussions about the internship and how they experienced their adviser. This was described as a way of solving the lack of feedback that otherwise characterised the monitoring activities. On the level of the DG, the methods and sufficiency which feedback was collected was noted being done in different ways depending on the DG. In a few cases coordinators and advisers argued that there was no point collecting monitoring information since there is no need, if it is going so well. In these cases, the assumption was mostly based on the spontaneous feedback they received from their trainees. In terms of the feedback advisers received, this was mostly referred to as being in the form one-to-one conversations with their trainees as well as usually having some sort of an exit interview. As previously mentioned, in a few cases coordinators and advisers could not recollect if they had in fact taken part in the evaluation report organised by the TO. EQ13. Cost-effectiveness of monitoring activities The survey by the TO was not perceived by most advisers and coordinators as being especially difficult or time consuming, there was however questions raised in relation to cost-effectiveness since a number of advisers reported they had never accessed the results. The survey was at times described as more of a bureaucratic exercise rather than a valuable tool for feedback. As one adviser explained: I don't know what is the destination of these surveys and if at all they are analysed Particularly advisers were not aware of how and who used the information gathered through the survey, expressing some confusion as to its use and where to access it. EQ13. Use of monitoring information When asked about how the feedback is used several coordinators and advisers remarked how they rarely received any comments from TO. One coordinator described the process of trainees filling in confidential feedback reports which were then sent back to the TO, with little or no feedback ever reaching the coordinators. In several cases, coordinators remarked that there seldom was any negative feedback or feedback in general from the TO which lead them to the conclusion that most trainees were happy. However, the lack of feedback was noted to give coordinators a difficult time in knowing what to improve. As a result of feedback not reaching coordinators, in one DG a coordinator had even arranged their own feedback sessions for the trainees. Furthermore, a couple of coordinators also remarked how they could become more involved in the content of the traineeship if the TO provided more feedback by taking more control over task management, monitoring satisfaction and identifying areas of improvement. As one coordinator described the situation: People think that coordinators are only there to take care of the administrative side of things. But with this [high quality] kind of feedback we could be more involved also in the content [of the traineeship]. November

55 A similar situation was reported among several advisers, describing confusion in regards to who analysed the information collected through the surveys and how it was being used. The feedback to the advisers was mostly in form of exit interviews with trainees were there rarely was any negative feedback. Though, as one adviser commented, it was less likely that trainees actually would give negative feedback face-to-face. In regards to this, one adviser called for more specific advice: Perhaps it would be helpful if they [trainees] were asked concrete feedback on practical issues. Not just how do you get along with your mentor, but more specific things, did your mentor spend enough time with you etcetera. November

56 Pr ofi le Annex 4 Survey report Background/ methodology The online survey allowed us to gather quantitative data and (some) qualitative information on (1) the perceptions and (post-stage) experiences of former trainees, and (2) the experiences of former Blue Book candidates that were not selected for a traineeship. The online survey was used for two purposes: 1. Part A: General assessment of the Scheme: The survey of former trainees gathered feedback on the trainees experiences during and after their traineeships. More specifically, the survey gathered insights into the relevance, effectiveness, utility, and added value of their traineeship experience. 2. Part B: Counter-factual impact evaluation: The counter-factual impact evaluation was based on a survey of both former trainees and unsuccessful Blue Book candidates. By comparing the outcomes in key areas between both groups, we estimated the extent to which any observed effects (e.g. in their careers or engagement with the EU) can be attributed to participation in the Scheme or other (external) factors. In order to do this, we designed two separate surveys, one for former trainees and one for unsuccessful Blue Book candidates. The survey of former trainees assessed their views on and satisfaction with the traineeship. It also gathered information on their subsequent careers as well as interest and engagement with the EU. The survey of unsuccessful candidates elaborated on their views on the application and preselection process. Additionally, in order to provide a frame for comparison with the former trainees, it assessed their post-application career development and interest and engagement with the EU. To select a sample for the counterfactual surveys, we initially intended to employ a technique called propensity score matching. This quasi-experimental technique entails matching former trainees and unsuccessful candidates based on observable characteristics that predict their success in selection for the traineeship. This works to make the two groups as comparable as possible and in turn increase the likelihood that differences in outcomes (career evolution, etc.) are due to the traineeship programme rather than other factors. Matching in this way when selecting the sample can reduce the need for adjustment of differences later on during the analysis. In order to verify the suitability of this method, we examined whether the profile data of former trainees and unsuccessful Blue Book candidates could tell us anything about the likelihood of a given candidate being selected. If so, we could select two comparable groups of respondents that were alike in every respect except for one of them having completed the traineeship. The table below provides an overview of the information used in the analysis. Table 4: Variables assessed for the potential use of propensity score matching Area Variables assessed for the potential use of propensity score matching Age November

57 Languages Previous working experience Education Nationality Country of residence Total number of degrees from tertiary education Length of studies Topic areas of studies Previous work experience (permanent or temporary employment) Previous work experience as a trainee Previous work experience as a trainee in an EU institution Other previous work experience (self-employed or voluntary work) Total number of previous jobs Previous work experience in any EU country Previous work experience in a country other than candidate s country of nationality Total number of languages Native language Number of languages speaking level A, B, and C Number of languages listening level A, B, and C Number of languages writing level A, B, and C Number of languages reading level A, B, and C However, having explored different models and interactions 19, we concluded that the information available did not allow us to narrow the population of former trainees and unsuccessful Blue Book candidates into more similar subgroups. Instead, the analysis showed that, at least in terms of the profile data available, former trainees and unsuccessful candidates were (nearly) as similar as if they had been selected at random By different models and interactions we mean different combinations of variables that (together) could have plausibly predicted the chance of being selected. 20 This means that at individual level the selection of candidates heavily relied on situational criteria and the match between the host institution s interests and the candidate s background. It is also likely that chance played some role. November

58 We cannot exclude the possibility that successful applicants may still have been different from their unsuccessful counterparts in terms of characteristics not observable in the data, such as the propensity to engage in proactive lobbying to secure a place in the traineeship. The survey results should be read with this in mind. However, given the pool of highly qualified candidates it is also very likely that successful candidates had very similar characteristics in aggregate to the unsuccessful candidates. In other words, during the final selection phase host institutions are likely to face situations where they need to choose between multiple suitable candidates. Leading from this analysis, we made two changes to our approach to the survey: Sampling: Given the nature of the selection of Blue Book candidates, in which on observable characteristics both groups were very similar to each other there is no advantage to preselecting matched samples. This eliminated the need for a prior matching exercise, and meant that the best approach was to sample all former trainees and unsuccessful Blue Book candidates. Questionnaire design: Although the profile data indicated that former trainees and unsuccessful Blue Book candidates were nearly identical on known characteristics, it was possible (indeed, likely) that there were differences between them that were invisible to us. We therefore added questions to each survey on topics such as candidates pre-application knowledge of and interest in EU affairs that helped us refine the analysis once all the responses were collected. We then controlled for some of the differences in respondent types in order ensure the comparability between former trainees and unsuccessful candidates and help us assess the traineeship s impact as accurately as possible. Dissemination and responses The survey was circulated among all former trainees and unsuccessful Blue Book candidates from the four sessions for which the Commission had suitable contact data, namely March and October 2012 and March and October The survey was promoted by and accessed via a link to a dedicated survey site. In order to ensure the anonymity of respondents and adhere to the Commission s data protection policy, the TOU was responsible for promoting the survey and collecting responses that were then provided to the evaluation team in an anonymised form. Importantly, the evaluators were able to link responses back to candidate profile data, thereby allowing differences in the samples from the two surveys to be controlled for. The survey was launched on 21 May 2014 and left open until 30 June 2014, with a reminder sent midway through the survey period. The table below summarises population sizes and responses for former trainees and unsuccessful Blue Book candidates. For both groups, response rates were sufficiently high for robust statistical analyses to be conducted. For the survey of former trainees, the response rate allows for a confidence interval of 4.05 with a confidence level of 99%. This means that we can be 99% certain that for a given survey question responses were within 4.05 percentage points of those that would be gathered by getting a response from the whole population of former trainees to whom the survey was sent. For Blue Book candidates the results were even more robust, with a confidence interval of 2.65 and confidence level of 99%. November

59 Table 5: Counterfactual surveys, summary of populations and responses Population Responses Response rate Confidence interval (99%) Former trainees 2, % 4.05 Blue candidates Book 7,567 1,573 21% 2.65 Total 10,283 2,455 24% N/A It is important to note that much of the analysis below relied on comparing the responses of former trainees with those of Blue Book candidates. Where appropriate, for some questions we used statistical controls (such as regression analysis of profile data like ages) to weight responses and maximise comparability and ensure the robustness of the findings presented. Findings This section presents the findings of the survey conducted with former trainees and unsuccessful candidates. The analysis proceeds as follows: we begin with a review of feedback regarding the application and selection process for both former trainees and unsuccessful applicants; followed by experiences of the actual Traineeship experience from former trainees and then findings related to the career outcomes of both former trainees and unsuccessful candidates. Lastly, results among both groups in relation to awareness and interest in the EU are presented. Application and selection process The results of the survey show that most applicants applied to the Traineeship for multiple reasons. Responses indicate that gaining a better understanding of EU policymaking and processes and gaining an understanding of what it would be like to work in an EU institution are of paramount importance. On average nine out of ten candidates confirmed this motivation was either to some extent or to a great extent an important reason for their application. In addition, the vast majority - on average seven in eight would-be trainees - applied either to gain work experience generally or as a step towards a career in the EU institutions. These findings demonstrate the perceived importance of the traineeship in supporting the transition from education to work but especially in providing a unique opportunity to gain insight into working life in an EU institution. The differences between former trainees and unsuccessful candidates were relatively minor in this respect. November

60 To gain a better understandi ng of what it would be like to work in an EU institution As a step towards an eventual career in one of the EU institutions To obtain working experience more generally To gain a better understandi ng of EU policy making and processes Figure 1: Reasons for applying to the Traineeship Scheme Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates % 20% 40% 60% 80% 100% To a great extent To some extent Neutral To a small extent Not at all n = former trainees, n = Blue Book candidates After pre-selection stage and the establishment of the virtual Blue Book, EC host services are responsible for the final selection process. The survey showed that while about half of the former trainees were interviewed during the final selection stage (by telephone or face-to-face). In the case of the shortlisted and eventually unsuccessful Blue Book candidates, around 17% were interviewed. This, of course, relates to former trainees also being more likely to be short-listed than unsuccessful candidates. Figure 2: Proportion of respondents that were interviewed in the selection process Former trainees Blue Book candidates 49% 51% 83% 17% Yes No Yes No n = 876 former trainees, n = 1539 Blue Book candidates In terms of satisfaction with the application process, unsuccessful applicants expressed relatively negative views. These related both to perceptions of the clarity and transparency of the application and selection and the extent to which they felt the process enabled them to show their capabilities. However, it is also worth pointing out that former trainees were also critical of the application process. Less than half of November

61 Clarity of the application process Transparency of the selection criteria Extent to which the process allowed you to show your capabilities them indicated that the selection criteria were transparent to (at least) some extent, and despite their success, only about half felt that the process allowed them to show their capabilities. Figure 3: Satisfaction with the Traineeship Scheme application and selection process Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates n = former trainees, n = Blue Book candidates Satisfaction with the Traineeship 0% 20% 40% 60% 80% 100% To a great extent To some extent Neutral To a small extent Not at all This section relates to the content of the traineeship and thus reports on questions that were only asked of former trainees. The findings indicate that the vast majority of former trainees were satisfied with the content of their traineeship. Almost 70% of the former trainees were either very satisfied or fairly satisfied with the extent to which the traineeship provided them with a hands-on experience, with only 10% being not (very) satisfied. Related to this, the findings also indicate that the vast majority of former trainees (82%) were satisfied with the knowledge and skills that they acquired during the Traineeship. Again, only 10% were not (very) satisfied with the knowledge and skills that they had acquired. November

62 Figure 4: Former trainees satisfaction with the content of the traineeship Satisfaction with knowledge and skills acquired during the traineeship Satisfaction with hands-on experience of EU policy making processes % 20% 40% 60% 80% 100% Very satisfied Fairly satisfied Neither satisfied nor dissatisfied Not very satisfied Not satisfied at all n = Former trainees were also generally positive about the conditions of the Traineeship, especially in relation to the working conditions and guidance provided by the stage advisers. With regard to the level of remuneration, more than 70% were either very satisfied or fairly satisfied. Figure 5: Former trainees satisfaction with the Traineeship conditions The working conditions (such as working times, technical equipment, etc.) The guidance provided by your personal stage adviser The assistance provided by your host institution s stage coordinator The level of remuneration provided % 20% 40% 60% 80% 100% Very satisfied Neither satisfied nor dissatisfied Not satisfied at all Fairly satisfied Not very satisfied n = former trainees What is interesting to note is that, when asked about the ideal length of the Traineeship, 70% of former trainees preferred the Traineeship to last for six months or more (i.e. longer than is currently the case), suggesting that trainees perceive that they would benefit from more time with the Commission. November

63 Figure 6: Perceptions on the ideal length of the traineeship 70% 1% 3% 26% 3 months or less 4 months 5 months 6 months or more n = 860 While about half of former trainees were satisfied with the professional events (e.g. career fairs), a greater percentage (67%) were fairly or very satisfied with the social events. Between 7% and 17% were either not very satisfied or not satisfied at all with the social and professional events, respectively. Figure 7: Former trainees satisfaction with social and professional events Professional events (such as career fairs) Social events and parties % 20% 40% 60% 80% 100% Very satisfied Fairly satisfied Neither satisfied nor dissatisfied Not very satisfied Not satisfied at all n = former trainees Lastly, a majority of trainees perceived the Traineeship as being helpful in the advancement and furthering of job related skills and knowledge. Fully 70% or more (strongly) agreed that they could make better informed decisions about their future career and that the knowledge and skills acquired were useful in applying for and carrying out subsequent jobs. November

64 Figure 8: How former trainees perceive the utility of the Traineeship for their careers Participation in the traineeship enabled me to make better informed decisions about my next career steps The skills and knowledge acquired during the traineeship were useful in carrying out duties in subsequent jobs The skills and knowledge acquired during the traineeship were useful in applying for subsequent jobs % 20% 40% 60% 80% 100% Agree strongly Agree Neither agree nor disagree Disagree Strongly disagree Not applicable n = former trainees Career outcomes A key reason for surveying both former trainees and unsuccessful Blue Book applicants was to observe differences in their subsequent careers. This is explored in the ensuing paragraphs and accompanying charts. Although a large majority of respondents was currently working, there was a significant employment difference between the two groups: 82% of former trainees and 74% of unsuccessful Blue Book candidates reported that they currently were working 21. This implies that those who participated in the traineeship had a better chance of finding a job afterwards than those who had not participated in the Scheme. Of those who were working, 87% of former trainees worked in an EU Member State, compared to 80% of the unsuccessful candidates. These changes carried over time. The graph on the next page shows that after one year, former trainees were six percentage points more likely to be employed (81% of former trainees were employed compared to 75% of unsuccessful applicants). After two years, controlling for differences in the sample sizes and other characteristics, the difference increased to nine percentage points, with 88% of former trainees being employed compared to 79% of unsuccessful applicants. 21 Moreover, of those who were not working at the time of the survey, 36% of former trainees and 63% of Blue Book candidates had worked before. November

65 Figure 9: Proportion of survey respondents who were employed after Traineeship/application 100% 90% 80% 70% Unsuccessful candidates Former trainees 60% n = 674 former trainees, n = 1225 Blue Book candidates The data also showed that, on average, participants in the programme took considerably less time to find a job after completing the Traineeship than their counterparts did after lodging an unsuccessful application. 22 A much larger proportion of former trainees found a subsequent job or traineeship after only one month (59% compared to 41% of unsuccessful candidates). This difference decreased slowly as time elapsed. However, even after one year a much larger proportion of unsuccessful applicants than former trainees were still unemployed (82% of former trainees were employed compared to only 76% of unsuccessful candidates). The survey results suggest that the Traineeship led to a boost in the initial income of participants. While this became gradually less important over time, after two years incomes had become more stable for both groups, meaning that some disparity appeared likely to endure. In other words, the results suggest that the Scheme has given former trainees a head start in terms of their level of income (possibly by enabling them to obtain a better first job). While unsuccessful applicants make up some ground over time, a gap remains, with a considerable income disparity of ten percentage points two years after participating in or applying for a Traineeship. Unfortunately, the nature of the data, which drew on responses from the 2012 and 2013 sessions, did not allow us to examine trends over a longer period. 22 When interpreting the graph, it should be noted that there is a bias in the results in that it compares different points in time (i.e. non-trainees at the time of application and former trainees five months later after having gained relevant working experience with the Commission). Nevertheless, even when controlling for age and time elapsed, the differences remained statistically significant. November

66 EU institutions EU or European affairs EUR per year Figure 10: Survey respondents annual salaries after traineeship/application Months from end of traineeship Unsuccessful candidates Former trainees n = 744 former trainees, n = 1320 Blue Book candidates In terms of respondents types of work, the results show that former trainees deal with EU institutions much more often than those who were not selected for the Traineeship (54% answered always or often compared to only 37% of the unsuccessful candidates). They were also more likely to deal regularly with topics related to EU or European affairs (68% compared to 59%). Figure 11: Frequency with which respondents work with EU institutions and topics related to European affairs Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates % 20% 40% 60% 80% 100% Always Often Sometimes Rarely Never n = 754 former trainees that deal with EU affairs, n = 1373 Blue Book candidates that deal with EU affairs, n = 754 former trainees that deal with EU institution, n = 1342 Blue Book candidates that deal with EU institutions While there was less of a difference in terms of the sector where respondents worked, a substantially larger proportion of former trainees worked for the Commission: 15% of the surveyed former trainees worked for the EC, compared to only 3% of the unsuccessful candidates. This indicates a strong link between completing a Traineeship at the Commission and finding future employment there. November

67 Figure 12: Types of organisations that respondents worked for 40% Former trainees Unsuccessful candidates 35% 30% 25% 20% % % 6 5% % Private organisation European Commission International organisation Other EU institutions n = 705 former trainees, n = 1284 Blue Book candidates In terms of job satisfaction, no strong link emerged to suggest the Traineeship increased contentment with employment. Indeed, the survey results suggest generally high levels of job satisfaction among respondents, with former trainees being slightly more satisfied with their current job than unsuccessful applicants were. On a scale from 1 to 10 (with 10 being the most satisfied), they averaged 0.2 points higher. Figure 13: Satisfaction with current employment (1 being dissatisfied and 10 being highly satisfied) Unsuccessful candidates Former trainees % 20% 40% 60% 80% 100% n = 758 former trainees, n = 1379 Blue Book candidates In terms of respondents countries of residence, large majorities of both groups claimed to work in the EU. While this trend was more pronounced among former trainees (88% of whom worked in the EU) than unsuccessful applicants (79%), the difference is likely to reflect differences in the success rates between EU and non-eu applicants more than the impact of the Programme on participants future country of residence. The results also show that only 22% of former trainees worked in more than one country since finishing the traineeship, compared to 44% of unsuccessful candidates. This is likely to relate to trainees propensity to remain in Brussels after their time at the Commission. November

68 Belgium United Kingdom Germany Italy France Luxembourg The Netherlands Portugal Austria Poland Greece Romania Spain Sweden Czech Republic Denmark Lithuania Latvia Bulgaria Hungary Croatia Estonia Slovakia Slovenia Cyprus Finland Ireland Malta Figure 14: Respondents countries of current employment EU Member States 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Former trainees Unsuccessful candidates n = 672 former trainees, n = 1096 Blue Book candidates Awareness and interest in the EU In order to explore the impact of the Traineeship on participants awareness and interest in the EU, we asked both groups questions on their knowledge and understanding of the EU, as well as their perceptions of the EU, feelings about working for the EU and personal interest in it and how all of these had changed over time. While such self-reported change should be interpreted with caution, the results suggest that the Traineeship had a positive impact. Unsuccessful candidates were considerably more positive about their past knowledge and understanding than former trainees (for example, 46% indicated that their knowledge of the EU was (very) good 3 years ago, compared to only 31% of former trainees). When asked about their current knowledge and understanding, the responses were more closely matched, with both groups considering themselves highly knowledgeable. Over three years, the proportion who indicated that their knowledge of the EU was (very) good increased by 53 percentage points (from 31% to 84%) among former trainees, compared to only 34 percentage points (from 46% to 80%) among unsuccessful candidates. November

69 3 years ago 1 year ago Now Figure 15: Respondents views on their knowledge and understanding of the EU 0% 20% 40% 60% 80% 100% Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates Former trainees Unsuccessful candidates Very good Good Average Not so good Not good at all Don t know n = 839 former trainees, n = 1491 Blue Book candidates The Traineeship also appears to have had a positive impact on participants perceptions of the EU. Again asking respondents to consider their situation 3 years ago and compare it with the present, the proportion of former trainees with a positive image of the EU increased by eight percentage points, while for unsuccessful candidates it decreased by ten percentage points. These results held true regardless of whether respondents currently worked with EU institutions or with EU related topics (as shown in the figure below). Figure 16: Proportion of respondents that were positive about their general image of the EU Former trainees working with EU % 80 Unsuccessful candidates working with EU % 3 years ago one year ago current Former trainees not working with EU % Unsuccessful candidates not working with EU % n = 832 former trainees, n = 1475 Blue Book candidates While personal interest in EU affairs increased for both groups over the past three years, the increase was larger among former trainees. The proportion of former November

70 trainees rating their personal interest in the EU as high or very high three years ago was 64% compared to 97% today, an increase of 33 percentage points. The proportion of unsuccessful candidates rating their interest in the EU as (very) high was 69% three years ago and 96% today, an increase of 27 percentage points. Again, these results were unrelated to whether or not respondents were working with EU institutions or EU related topics. 23 Figure 17: Proportion of respondents with a (very) high personal interest in the EU 3 years ago one year ago current Former trainees working with EU % Unsuccessful candidates working with EU % Former trainees not working with EU % Unsuccessful candidates not working with EU % n = 326 former trainees working with the EU, n = 196 Blue Book candidates working with the EU, n = 1663 former trainees not working with the EU, n = 3740 Blue Book candidates not working with EU Respondents self-reported levels of interest in an EU career were very high and nearly identical, though unsuccessful candidates were slightly more interested. There was also some self-reported change over time. Former trainees level of interest increase by five percentage points (from 73% to 78%) between three years ago and the time of the survey, while for unsuccessful candidates there was a sevenpercentage point decrease, from 87% to 80%. Figure 18: Proportion of respondents interested in a career in one of the EU institutions 100% 90% 80% 70% Former trainees Unsuccessful candidates 3 years ago one year ago current 60% n = 809 former trainees, n = 1446 Blue Book candidates 23 It can be argued that applicants who actively seek out the Traineeship have already a higher interest in and knowledge of the EU. This is also reflected in many applicants backgrounds (which tend to be in international relations, politics, law, and European studies). November

71 Annex 5 SWOT analysis Based on the results of the evaluation and evidence found, we undertook an analysis of Strengths, Weaknesses, Opportunities and Threats (SWOT analysis) related to the Traineeship. This analysis is presented from the perspective of high-level findings and draws on all of the data collection tools in addition to the workshop conducted with interested Commission officials as well as representatives of other EU traineeship schemes. The SWOT analysis is useful for considering and identifying potential and existing gaps as well as to future improvements. We used it principally to inform the overarching conclusions and recommendations for the evaluation. SWOT analysis revolved around the consideration of four areas: Strengths: internal characteristics of the Traineeship that are helpful to achieving the objectives. Weaknesses: internal attributes that are harmful to achieving the objectives: the analysis identifies areas where mitigation strategies will be needed. Opportunities: external conditions that improve the chances of achieving objectives: they should be capitalised upon. Threats: external elements in the environment that could jeopardise the success and damage the chances of achieving the objectives: they should be acknowledged and treated in a planned and organised way. November

72 Table 6: SWOT analysis Helpful to achieving the objective Harmful to achieving the objective External STRENGTHS WEAKNESSES Attributes of the organisation, programme or activity Objectives remain relevant (in fact increasingly so) 24 Managed by Traineeship Office Trainees gain competitive advantage in labour market, expanded network, knowledge of the EC Fosters cross-border mobility and a European labour market Traineeship is cost effective: Commission benefits from productive (cheap) high quality (temporary) labour which outweighs costs of grant and human resources incurred Application process now digitised resulting in significant efficiency gains Objectives not fully coherent Eligibility criteria do not match all objectives to a similar degree Nearly (but not fully) aligned with Quality Framework Technical resources (such as office space on hosting units) not always in place to maximise benefits from Traineeship Legal base gives TOU little influence over final selection procedure Perceptions of opaque final selection processes that are inconsistent and inefficient procedures Monitoring activities currently not purposeful or effective External OPPORTUNITIES THREATS Attributes of the environment Legal base can be revised to reflect need to update objectives and improve final selection procedure EC Traineeship open to young graduates from all socio-economic backgrounds cf. unpaid traineeships Exploitation of social media/other avenues to publicise Traineeship Embrace a leadership role in meeting the Quality Framework Partnership with other EU bodies and institutions Make use of alumni (network) in systematic manner Danger of lock in effect of Traineeships Increasing Euroscepticism and disillusionment with the EU Risk that Traineeship replaces entry level positions in EC Potential for synergies with other EU traineeship programmes might not be exploited Resistance to change among key stakeholders due to longstanding systems and processes 24 Meeting graduate demand for first-hand work experience in EC in context of challenging economic conditions; injection of highly motivated and qualified graduates into EC; creation of goodwill ambassadors for EU in context of widespread and increasing Euroscepticism November

73 This document has been prepared for the European Commission. However, it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

74 HOW TO OBTAIN EU PUBLICATIONS Free publications: via EU bookshop ( at the European Union s representations or delegations. You can obtain their contact details on the Internet ( or by sending us a fax to Priced publications: via EU Bookshop ( Priced subscriptions (e.g. annual series of the Official Journal of the European Union and reports of cases before the Court of Justice of the European Union): via one of the sales agents of the Publications Office of the European Union (

75 NC EN-N