Executive Master s programme in Cyber Security. (wo-master) Leiden University

Transcription

1 Executive Master s programme in Cyber Security (wo-master) Leiden University 22 December 2014 Initial accreditation Panel report

2 Table of contents 1 Executive summary 3 2 Procedure Process Panel Assessment framework 6 3 Programme Administrative data University Programme 7 4 Assessment per standard Intended learning outcomes (standard 1) Teaching-learning environment (standard 2) Assessment (standard 3) Graduation guarantee and financial provisions (standard 4) Final conclusion Recommendations and advice 17 5 Overview of the assessments 19 Annex 1: Composition of the panel 20 Annex 2: Schedule of the site visit 22 Annex 3: Documents reviewed 23 page 2

3 1 Executive summary The executive master s programme in Cyber Security is offered by Leiden University. In the field of Cyber Security there is collaboration with Delft University of Technology and The Hague University of Applied Sciences, working together in the Cyber Security Academy The Hague (CSA) foundation. The programme intends to start in January Standard 1. Intended learning outcomes The executive master s programme in Cyber Security is a multidisciplinary programme for professionals who are working in the area of cyber security for a number of years and who feel the need to supplement their usually monodisciplinary knowledge with a broader view. Cyber security is no longer a merely technical issue, but calls for additional knowledge of non-technical issues, including governmental, behavioural, legal, ethical and economic issues. Cyber security professionals at a strategic and tactical level need knowledge, understanding and skills in line with this new cyber view and must be able to combine them into an integral multidisciplinary approach. On the basis of a common conceptualisation of cyberspace and cyber security, participants have the opportunity depending on their choice and background to broaden or deepen their knowledge in a technical oriented track or a social scientific oriented track (the governance track). The panel has studied and discussed the aim of the programme and notes that the multidisciplinary programme, combining technical and non-technical knowledge and skills, is an innovative and ambitious approach. The panel considers it to be an exciting way to prepare professionals at an academic level to cope with the increasingly complex issues of cyber security. This new programme meets the needs of the work field. The intended learning outcomes of the programme are clearly described, linked to the Dublin descriptors and appropriate for an academic master s programme. Considering the importance of designing security measures that are compatible with the Rule of Law, the panel recommends to explicitly add to the learning objectives that graduates are able to take into account the constraints of positive law. The panel assesses standard 1 Intended learning outcomes as satisfactory. Standard 2. Teaching-learning environment The programme consists of four phases of 15 EC each, spread over a two year s period. Every semester covers a phase: 1. the conceptualisation phase, in which a comprehensive understanding is developed, based on a common conceptualisation; 2. the specialisation phase, in which participants can choose a technical oriented or a social scientific oriented specialisation. The technical track is only accessible for participants with a technical based background or education; 3. the elaboration phase, in which a more detailed understanding of previously discussed subjects and perspectives can be obtained in several in-depth modules; 4. the research phase, in which the participants apply their knowledge and skills in order to compose an MSc thesis based on an individual research project in the area of cyber security. The panel is not convinced that the introduction to the programme s multidisciplinary topics in the first phase will be effective enough without a prior introduction to the vocabulary, theories and methodology of the basic disciplines. The panel, however, respects the programme s choice and advises to monitor closely if this will work out as expected. page 3

4 The panel disagrees with the very limited time devoted to law and legal issues. A multidisciplinary programme on cyber security cannot be complete without a serious introduction to the relevant positive law. It is not sufficient if graduates of this programme know when to involve a lawyer or other legal advisor, they must at least know the basics of the relevant positive law for themselves. A course of at least 4 EC must, therefore, be added to the cross-over knowledge courses and be compulsory for all participants without a background in law. The programme is an executive programme (master after master) that focuses on professionals working in the field of cyber security and is taught in English. The admission criteria and intake procedure are designed to attract this target group. Around 24 participants can be admitted annually. The panel finds the intake procedure well-considered. The programme s educational concept is multidisciplinary, practice-oriented, researchbased, selective and international. The teaching methods are in line with the educational concept and the learning objectives and appropriate for the target group. The programme is taught by lecturers from the universities involved who are active academic researchers. In addition, guest lecturers from relevant industries and organisations are called in. Each course subject has a core lecturer with a scientific background and each track has a track coordinator. The programme has an academic director/programme director, who is responsible for the entire programme, and is supported by a programme manager. With regard to the technical track, only available to technical participants, care should be taken that the quality of the teaching material is effectively guaranteed. The programme is offered at the Campus The Hague (FCDH), partly at the Schouwburgstraat and partly in the new facilities of The Hague Security Delta (HSD Campus). The facilities are excellent. Participants will receive extensive guidance during their study. This is especially relevant for this target group of participants who will have to balance their study with their work and private lives. Summing up, the panel concludes that the didactical approach, staff and facilities of the programme will contribute to a teaching-learning environment that enables the participants to achieve the intended learning outcomes. The programme is ambitious and exciting, but lacks a substantive course in positive law. The panel assesses standard 2 Teaching-learning environment as unsatisfactory. Standard 3. Assessment The panel has studied the assessment system and a sample of exams and has met with the Board of Examiners during the site visit. The panel concludes that the Board of Examiners is aware of its responsibilities and is developing procedures and a year plan for the execution of its duties. The panel advises to use the time before the start of the programme to work this out in more detail. The choice of assessment methods reflects the learning objectives of the courses. Their validity, reliability and transparency cannot yet be fully checked, since no students work is page 4

5 yet available. On the basis of the meeting with the Board of Examiners, the panel is confident that the Board will check these quality requirements. The panel advises to include a check on validity before the exams are taken. The panel assesses standard 3 Assessment as satisfactory. Standard 4. Graduation guarantee and financial provisions Leiden University guarantees that participants who have been admitted to the programme, can complete the entire curriculum. The long-term budget of the programme is solid. Based on the intake for the 2015 start of the programme, the panel concludes that the executive master s programme in Cyber Security is a viable programme. The panel assesses standard 4 Graduation guarantee and financial provisions as satisfactory. Given these considerations, the panel advises NVAO to take a conditional positive decision regarding the quality of the proposed executive master s programme in Cyber Security (womaster) at Leiden University. The panel specifies the condition as follows: within a period of one year, a course in positive law of at least 4 EC must be added to the cross-over knowledge courses and be compulsory for all participants without a background in law. Through meeting this condition, the programme will qualify for having standard 2 assessed as satisfactory. The panel is confident that the programme will manage to remedy this shortcoming within one year. The panel sustains Leiden University s request for the programme to be registered under cross-sectoral as its field of study (CROHO-onderdeel). The Hague, 22 December 2014 On behalf of the Initial Accreditation panel convened to assess the executive master s programme in Cyber Security (wo-master) at Leiden University. Prof. dr. ir. Bart Preneel (chair) Dr. Marianne van der Weiden (secretary) page 5

6 2 Procedure 2.1 Process Application 15 September 2014 (additional material 14 November 2014) Composition panel 29 September 2014 Site visit 21 November 2014 (The Hague) Panel report 10 December Panel Composition of the panel: Prof. dr. ir. Bart Preneel, full professor Computer Security and Industrial Cryptography, KU Leuven, Belgium (chair); Prof. dr. Mireille Hildebrandt, professor Smart Environments, Data Protection and the Rule of Law, Radboud University, The Netherlands; Marc Sel, MSc Information Security, Director at PwC (Enterprise Advisory Services - Information Protection), Antwerp, Belgium; Nienke Bach Kolling BSc, MA student Educational Science & Technology, University of Twente, The Netherlands (student member). Assisting staff: Dr. Marianne van der Weiden, secretary to the panel; Drs. Fred Mulder, policy advisor NVAO and process coordinator. 2.3 Assessment framework The framework for limited assessments of new programmes (Stcrt. 2010, nr 21523) is used for institutions that have obtained a positive judgement following an institutional quality assurance assessment. The assessment is based on a discussion with peers regarding the content and quality of the programme. It focuses on four questions: 1. What is the programme aiming for? 2. How does the programme intend to achieve its objectives? 3. How does the programme intend to assess its performance? 4. Does the programme have sufficient financial resources? These four questions have been translated into four standards. Regarding each of these standards, an assessment panel gives a substantiated judgement on a two-point scale: satisfactory or unsatisfactory. The panel subsequently gives a substantiated final conclusion regarding the quality of the programme, on the same two-point scale. Generic quality The quality that can reasonably be expected in an international perspective from a higher education bachelor s or master s programme. Positive The programme meets the generic quality standards. Negative The programme does not meet the generic quality standards. page 6

7 3 Programme 3.1 Administrative data Country Institution Programme Level Orientation Degree Location(s) Credits Tracks Mode of study Field of study The Netherlands Leiden University Executive master s programme in Cyber Security master academic (wo) Master of Science (MSc) Faculty Campus The Hague 60 EC Track 1. Technical Track 2. Governance part-time executive programme in situ cross-sectoral 3.2 University Leiden University is a research university, founded in 1575, offering approximately fifty academic bachelor s and a hundred academic master s programmes to more than 23,000 students. The university employs over 4,000 staff members (3,404 fte) and is organised in seven faculties, six of which are based in Leiden. The seventh, Campus The Hague, is located in The Hague. 3.3 Programme The executive master s programme in Cyber Security is offered by Leiden University at its Faculty Campus The Hague, in collaboration with Delft University of Technology and The Hague University of Applied Sciences, working together in the Cyber Security Academy (CSA) foundation in The Hague. The programme in Cyber Security takes a multidisciplinary approach and focuses on a comprehensive understanding (technical as well as nontechnical) of interrelated cyber security issues and their impact on society. The programme is aimed at professionals who have already obtained an academic master s degree in either the social sciences, law or technical sciences, supplemented with several years of work experience in the field of (cyber) security and who wish to broaden and deepen their knowledge on issues concerning cyber security. page 7

8 4 Assessment per standard 4.1 Intended learning outcomes (standard 1) The intended learning outcomes of the programme have been concretised with regard to content, level and orientation; they meet international requirements. Findings The information dossier describes how cyberspace has evolved from a mainly technical issue (robust communication services) to a system that also includes a socio-technical layer (the interaction between the people active in cyberspace and the data storage and data processing systems) and a governance layer, consisting of a large variety of human actors and organisations. Similarly, cyber security has developed from a technical focus on information security (confidentiality, integrity and availability of information) to more business-oriented topics, such as cyberspace assets, supplier relationships, business continuity management and compliance. Cyber risk management has to deal with the cyber risks of cyberspace actors (end-users, stand-alone and interdependent organisations, critical infrastructures, states and actors at a global scale) in the cyber context. Risk managers must identify and assess the cyber risks (in terms of frequency and impact), determine the acceptable levels of the assessed risks (involving political decisions) and design a balanced set of preventive and repressive measures to reduce them to acceptable levels. These complex security issues call for cyber security professionals with knowledge, understanding and skills at a strategic and tactical level, in line with the new cyber view. These professionals must be able to combine technical with non-technical issues, including governmental, behavioural, legal, ethical and economic issues, into an integral multidisciplinary approach. The complex relationships between cyber threats, vulnerabilities, cyber security incidents and their probable impacts of all kind require a comprehensive understanding of cyberspace, cyber sub-domains and the use and abuse of the Internet, the actors in the (sub-)domains and their behaviour, principles of cyber risk management, legal and ethical issues, economic impact and the complexity of controlling: governance and management perspectives. The assumption is that nearly all senior cyber security professionals in organisations at this moment are trained in either technical, social or legal disciplines and therefore have insufficient understanding of cyber security in a broader perspective. The executive master s programme in cyber security aims to educate professionals with a master degree and several years of work experience in the field of (cyber) security to become professionals with an integral outlook on cyber security. They will be able to identify a variety of theoretical frameworks and approaches suitable for addressing cyber security issues, and to select the appropriate ones that can contribute to strategic solution for cyber risks (in order to reduce these to acceptable levels) and related issues. They understand which additional expertise should be invoked for identifying, assessing and mitigating relevant cyber security risks. In order to develop this comprehensive understanding a common conceptualisation and basic knowledge is required. On the basis of this common conceptualisation, participants in the programme have the opportunity depending on their choice and background to broaden or deepen their knowledge in a technical oriented track or a social scientific page 8

9 oriented track (the governance track). After this, a more detailed understanding of previously discussed subjects is obtained in several in-depth modules. In the master s thesis participants finally demonstrate the obtained comprehensive understanding of the complexity of today s cyber security challenges. After completing the programme, graduates have understanding, knowledge and skills, at tactical-strategic level, related to (a) the cyber security challenges of today and tomorrow, (b) possible (mono- and multidisciplinary) approaches to take up these challenges, and (c) how their personal knowledge, understanding and skills can contribute to the overall quality of possible solutions. The general learning outcomes of the programme have been formulated in terms of the Dublin descriptors and are listed in the information dossier. In addition to the general learning outcomes, specific learning outcomes have been formulated for the technical track and the governance track. In a spread sheet, the programme has shown which courses contribute to which qualification or learning outcome. On the basis of this information, the panel can confirm that the learning objectives fit the aims of the programme and are of the appropriate master s level. The panel also confirms that the description of the knowledge and skills, with its emphasis on theories and methods, reflects the academic character of the programme, without losing sight of the professional settings in which the knowledge and skills are to be used. The panel appreciates the multidisciplinary approach of the programme and recognises its reflection in the intended learning outcomes. Nevertheless, the panel is of the opinion that more emphasis is needed on the constraints of positive law in cyber security. In the information dossier, law is only explicitly mentioned in the specific learning outcomes of the governance track, but the panel is convinced that all graduates, regardless of their track, should have basic knowledge of relevant positive law. This is of particular importance in a programme that concerns the design, evaluation and/or implementation of security measures in the cyber domain, since infringements of the fundamental rights to privacy and data protection, as well as non-discrimination and the presumption of innocence are at stake, as well as infringements of Intellectual Property Rights and trade secrets. The same goes for cybercrime law, as the detection and the resolution of security threats easily involve actions that by themselves constitute a criminal offence. The panel recommends (1) to explicitly add to the general learning objectives that graduates should be able to take into account constraints of positive law, and (2) to add to the specific learning objectives of the governance track that graduates should have more in-depth knowledge of the relevant positive law. Both from the information dossier, the interviews during the site visit and the panel members knowledge of the field, it is clear that the executive master s programme in cyber security meets the need of employers. Employers in industry and government require their future employees not only to have a command of (basic) technology, but also (basic) knowledge of related non-technical issues and perspectives such as legal, economic and ethical know-how, knowledge of (change) management and governance models and methods and knowledge of characteristics and motives of human behaviour in cyberspace. On the basis of a limited benchmark of comparable programmes, the programme staff concludes that most of the programmes in this field are predominantly technically oriented, some of them are partly multidisciplinary oriented. Considering the rapidly increasing complexity of cyber security issues and the diverse impact of cyber risks, the market will in the near future require a standard of knowledge and skills that is more multidisciplinary. The programme aims to anticipate this development. The panel learned during the site visit that page 9

10 28 applicants have already been accepted for the start of the programme in January 2015, which confirms the market s interest in this programme. Considerations The panel has studied and discussed the aim of the programme and notes that the multidisciplinary programme, combining technical and non-technical knowledge and skills, is an innovative and ambitious approach. The panel considers it to be an exciting way to prepare professionals at an academic level to cope with the increasingly complex issues of cyber security. This new programme meets the needs of the work field. The panel concludes that the intended learning outcomes of the programme are clearly described, linked to the Dublin descriptors and appropriate for an academic master s programme. Considering the importance of legal knowledge, the panel recommends to explicitly add to the learning objectives that graduates are able to take into account the constraints of positive law. Conclusion The panel assesses standard 1 Intended learning outcomes as satisfactory. 4.2 Teaching-learning environment (standard 2) The curriculum, staff and programme-specific services and facilities enable incoming students to achieve the intended learning outcomes Findings Structure and contents of the programme The programme consists of 60 EC, scheduled over four semesters in two years. Participants can enter the programme once a year in January. Each semester comprises 15 EC, with 1 EC being equal to 28 hours of study. Each semester is composed of three consecutive seven-week course modules: six weeks of lectures and individual study and one week of assessment and individual study. The programme consists of four phases. Every semester covers a phase. 1. the conceptualisation phase, in which a comprehensive understanding is developed, based on a common conceptualisation and basic knowledge of cyberspace, cyber subdomains, the use and abuse of the Internet, the actors in the (sub)domains and their behaviour and the principles of cyber risk management. Basic aspects of involved legal and ethical issues, economic principles and their impact and the complexity of governing and managing cyber security are taken into account; 2. the specialisation phase, in which participants can choose a technical oriented or a social scientific oriented specialisation. The technical track is only accessible for participants with a technical based background or education; 3. the elaboration phase, in which a more detailed understanding of previously discussed subjects and perspectives can be obtained in several in-depth modules; 4. the research phase, in which the participants apply their knowledge and skills in order to compose an MSc thesis based on an individual research project in the area of cyber security. page 10

11 The panel has reservations whether it is feasible to develop the intended common conceptualisation, without alignment by first providing an adequate introduction to the vocabulary and methodology of the sciences that are not part of participants background. In the programme design, this is to be achieved in the cross-over knowledge courses in the second phase, when participants with a social science background have a course in Technological perspectives of cyber security, and participants with a technical background are offered a course in Social science perspectives on cyber security. Nevertheless, the panel respects the programme s choice. The unorthodox approach and the possible confusion in the first semester may stimulate the participants to recognise the need to learn more about the background of other disciplines. In this way, the learning effect of the second phase may be larger. The panel advises the programme staff to closely monitor if this order of phases has the desired effect. As mentioned under standard 1, the panel finds it essential that knowledge of positive law is included in the programme as a basic discipline. Currently, for instance, law is part of a session in the course Social science perspectives on cyber security, together with political science and public administration. The committee finds this insufficient and misses serious attention to (the constraints of) positive law on issues of data protection, privacy, cyber crime and intellectual property. Therefore, in addition to the two cross-over knowledge courses Technological perspectives on cyber security and Social science perspectives on cyber security, a third course Legal perspectives of cyber security of at least 4 EC must be added, that must be compulsory for all participants without a legal background. Each participant must be required to follow two of the three courses to complement his or her background knowledge. The panel suggests to reduce the modules to 4 EC each, which will allow for the extra course without disrupting the structure of the programme, but leaves it up to the programme to work out the details. The panel has studied the course catalogue and the Powerpoint slides of the first three modules. The information was not quite sufficient to provide a clear view of the topics and issues that would be addressed and, specifically, to which degree of detail and depth. The slides were still rudimentary and seemed rather basic, but in the absence of the lecture notes of the teaching staff it was impossible for the panel to determine the level of the courses with any certainty. It would have been helpful if the programme had drawn up a clearer mapping of the programme s contents, more detailed than the Excel-sheet which lists the nine key issues without elaboration, and more structured (by discipline and topic) and explanatory than the course descriptions in the course catalogue. For the legal component of the programme, this mapping has already been provided in the additional information on courses and teaching staff. Such a mapping exercise would also help the lecturers to align their courses to each other and to avoid gaps and overlap. Moreover, it would help the programme to position itself. The panel appreciates that scientific writing and research skills are taught in various modules during the first three semesters of the programme. Participants prepare for their thesis in semester 3 by a literature research and undertake their actual research project in semester 4, supervised by experienced staff from academia. Didactical approach The programme is an executive programme (master after master) that focuses on professionals working in the field of cyber security and is taught in English. The admission page 11

12 criteria and intake procedure are designed to attract this target group. Around 24 participants can be admitted annually. Preferably, the participants group is composed of a mix of nationalities, academic backgrounds (technical and social sciences) and employments (diverse organisations and sectors). For the start in January 2015, 28 participants have been admitted. Two of these are non-dutch. For a first batch, the panel finds this a promising start. Admission is open to professionals who have completed a relevant master s programme and have attained a sufficient level of competence in mathematics, methodology and English. Applicants must have several years of work experience in private businesses or public organisations that are involved in organising and handling issues in the area of security and/or cyber security. Professionals with an academic bachelor s degree or a bachelor s or master s degree from a university of applied sciences can be admitted under certain conditions, provided that they have additional relevant certificates and professional experience. Any deficiencies must have been satisfied prior to the start of the programme. Intake interviews are held with each applicant, to find out whether there is a match between the participant and the goals of the programme. Three types of participants can enter the programme: (1) professionals with a technical background, applying for the technical track, (2) professionals with a technical background, applying for the governance track, (3) professionals with a social scientific background, applying for the governance track. The panel supports the decision to make the technical track only accessible for participants with a technical based background or education. An introductory programme prior to the official start offers a general introduction into the programme and provides the participants an opportunity to get to know each other. Lectures on cyber security exemplify the various multidisciplinary perspectives and lectures on literature research and the correct use of sources refresh the participants research skills. The panel appreciates the thorough intake procedure and preparation phase. The programme s educational concept is multidisciplinary, practice-oriented, researchbased, selective and international. Case studies, recent policy papers and guest lectures will contribute to the link with current practice. The programme is academically anchored within relevant disciplines at Leiden University, Delft University of Technology and Erasmus University Rotterdam. By capping the number of admissions, the programme can remain small-scale. The teaching methods are geared to adult professionals. In addition to the standard teaching methods (lectures, seminars and workshops, simulations, presentations, independent study and project work), participants are tutored individually and work together in peer review groups of four to six students. The computer facilities enable interactive, computer-based learning and distance learning, allowing for work and study to be combined. The panel agrees that the didactical approach is well-suited for an executive master s programme and the specific target group. The panel did not yet find strong evidence of the international approach, beyond the claim that the programme intends to attract a substantial number of international students. The composition of the group of lecturers is not very international and, for this first year, only two out of 28 students are non-dutch. The panel suggests to strengthen this aspect by inviting some international guest lecturers. page 12

13 Staff The programme is taught by lecturers from the universities involved and by guest lecturers from relevant industries and organisations. Each course subject has a core lecturer with a scientific background and each track has a track coordinator. The programme has an academic director/programme director, who is responsible for the entire programme, and is supported by a programme manager. All lecturers from involved universities are active academic researchers. The information dossers provided their cv s and an overview of their publications. Most of them have a PhD or professorship and are qualified teachers. At this stage relatively few of them seem to have experience with practical cyber security issues in businesses or large organisations. All lecturers have been trained during Spring 2014 in the design and implementation of multidisciplinary teaching, in aspects of knowledge dissemination and in team-building. Expert meetings are organised regularly for additional training of lecturing and support staff. The panel confirms that the combination of lecturers and their expertise are in line with the demands of the programme. Additional lecturing staff is called in from The Hague University for Applied Sciences as instructors, and from various organisations (e.g. KPMG IT Advisory, Fox IT, KPN ) as guest lecturers. Guest lecturers provide relevant field experience to supplement the theoretical input from the university lecturers. In order to align the contents of both types of lectures, the core lecturer of a module always attends the guest lectures. The panel is of the opinion that the input of guest lecturers is necessary to cover specific topics, especially because of the fast evolution of developments in the real world. The explicit link between the guest lecturers and the core lecturers is well-considered. With regard to the technical track, only available to technical participants, care should be taken that the quality of the teaching material is effectively guaranteed. The plan/do/check/act model as described in ISO 27001, the standard for 'Information Security Management Systems', must be adequately covered, because especially in the planning phase, the multidisciplinary approach is crucial: budget, head count and legal compliance (don't plan for something you cannot realise due to legal constraints) all need to be taken into account. Without adequate knowledge combined with in-depth experience in the field, the quality of what is being taught cannot be overseen. Cyber security is an interesting field from academic perspective but its practical applications in the real world should be taken into account. For example, core lecturers need to have the knowledge to evaluate whether a guest lecturer teaches relevant material with regard to a Distributed Denial of Service (DDOS) attack, to a botnet attack, or to audit processes (ISO versus ISO 27000).The meeting with core teaching staff did not fully convince the panel in this respect. The information dossier mentions that the teacher-student ratio for fifty participants (two cohorts) is 1:22. Lecturer s time includes time for developing and updating courses and setting and marking exams. Core lecturers are all deployed in the programme for a fixed number of hours (4-8) per week during the whole programme, regardless their actual deployment in a certain module. This model makes permanent involvement possible while the actual deployment in the programme can vary. The panel concludes that the quantity of staff is sufficient. page 13

14 The programme committee will consist of two core lecturers and two participants of the programme. It will advise the programme director on the quality of the programme. This is in line with the quality assurance policy of Leiden University. Services and facilities The programme is offered at the Campus The Hague (FCDH), partly at the Schouwburgstraat and partly in the new facilities of The Hague Security Delta (HSD Campus). Both locations have multifunctional spaces and sufficient audio-visual equipment. The Schouwburgstraat offers the Living Lab with several multimedia facilities for students and researchers. The HSD building has additional facilities, such as a Serious Gaming Lab, a Real-time Intelligence Lab and an Experience Lab/Boardroom. Participants have access to all facilities of Leiden University, such as the library, the digital learning environment (Blackboard), sports and other general facilities (student support, language courses, computer work stations, cafeterias, restaurants and printing and copying machines). The modern facilities are an asset for the programme. In addition to the physical facilities, guidance and extra-curricular activities are provided. Each participant will have a progress interview at least once a year, prior to the choice for a track and before writing the thesis. These interviews will be held by the programme director, the programme manager and (one of) the track coordinators. A participant can also ask for an interview with the programme director or a lecturer at any time during the programme. The programme manager is the liaison to the programme director and the lecturers. The peer review groups (mentioned above under Didactical approach) are not only meant for reflection and discussion on the course material, but also for mutual support in the individual studying. When these peer groups are composed, the different sectors in which the participants work, their backgrounds and personal characteristics are taken into account. Extra-curricular activities are the three-day introduction and four to six lectures or seminars on specific cyber security themes and current or upcoming cyber security issues. The panel concludes that participants will receive extensive guidance during their study. This is especially relevant for this target group of participants who will have to balance their study with their work and private lives. Considerations Summing up, the panel concludes that the didactical approach, staff and facilities of the programme will contribute to a teaching-learning environment that enables the participants to achieve the intended learning outcomes. Regarding the structure and contents of the programme, the panel is not sure if the introduction to the programme s multidisciplinary topics in the first phase will be effective enough without a prior introduction to the vocabulary, theories and methodology of the basic disciplines. The panel, however, respects the programme s choice and advises to monitor closely if this will work out as expected. The panel disagrees with the very limited time devoted to law and legal issues. A multidisciplinary programme on cyber security cannot be complete be complete without a serious introduction to the relevant positive law. It is not sufficient if graduates of this programme know when to involve a lawyer or other legal advisor, they must at least know the basics of the relevant positive law for themselves. A course of at least 4 EC must, therefore, be added to the cross-over knowledge courses and be compulsory for all participants without a background in law. page 14

15 Conclusion The panel assesses standard 2 Teaching-learning environment as unsatisfactory. 4.3 Assessment (standard 3) The programme has an adequate assessment system in place. Findings The information dossier describes the various assessment methods and their relationship with the objectives of the courses. The assessment methods are written examinations (open-book, multiple choice and other tests), computer-based or on-line assessments, essays, critical analysis of case studies, presentations, project reports, peer and selfassessment and portfolios. The course catalogue provides information on the assessment method per course. The master s programme will be concluded with a final master project (master s thesis) worth 15 EC. The thesis will be a culmination of the specialised and multidisciplinary knowledge and approaches acquired earlier in the programme and applied to a case study from the field. During the site visit the panel has studied the assessments (written exams) of the courses S1M1 (Introduction to cyberspace), S2G3 (Regulating security in cyberspace), S2T/G1 (Social science perspectives on cyber security) and S3E1 (Actors and behaviour in cyberspace). Unfortunately, the assessment criteria or assessment forms were not included, so that the panel could not determine the validity with certainty. Based on the description in the course catalogue, the panel noted a possible discrepancy between the course objectives and course contents on the one hand and the assessment methods on the other hand in modules S2T2 and S2T3. It seems that the essay (which counts for sixty respectively fifty per cent of the assessment) focuses on a relatively small part of the total course contents. The panel advises the programme to check this and, if necessary, adjust the assessment method. An assessment matrix may be helpful to make sure that all learning objectives per course are covered by the assessment method(s). The Board of Examiners consists of four members and is administratively supported by a secretary. The expertise of the members covers the programme s range of disciplines. The Board is responsible for the quality assurance of the assessments and has drawn up Rules and regulations to this effect. The Board has a regular Board meeting three times per year and, in addition, meets twice per year with the programme committee and the programme management. Two members of the Board have already followed a training provided by ICLON, the educational expertise centre of Leiden University, on the role and responsibilities of a Board of Examiners. The other two members will take part in the next course offered in Spring Since the Board members are also involved in the programme as staff members, they regularly attend the programme meetings which are held every six to eight weeks and where teaching and assessment methods in relation to the multidisciplinary approach are discussed. The panel agrees that these sessions can be useful to work out common criteria for assessment in a newly designed programme, but advises the Board to prioritise its formal role and not to depend too much on informal contacts. page 15

16 In the interview during the site visit, the Board emphasised its role in strengthening the transparency and clarity of the assessments and in explaining the rules and procedures to the lecturers from other universities and organisations. The Board intends to check (samples of) examinations and essays and to draw up a schedule for this, but does not have the intention to check all of the assessments in advance, partly because this is primarily the expertise and responsibility of the lecturers. Based on the panel s observation on the assessment of modules S2T2 and S2T3, the panel advises the Board to take a more proactive stance and, if necessary, to call upon assessment experts, e.g. from ICLON. Considerations The panel has studied the assessment system and a sample of exams and has met with the Board of Examiners during the site visit. The panel concludes that the Board of Examiners is aware of its responsibilities and is developing procedures and a year plan for the execution of its duties. The panel advises to use the time before the start of the programme by working this out in more detail. The choice of assessment methods reflects the learning objectives of the courses. Their validity, reliability and transparency cannot yet be fully checked, since no students work is yet available. On the basis of the meeting with the Board of Examiners, the panel is confident that the Board will check these quality requirements. The panel advises to include a check on validity before the exams are taken. Conclusion The panel assesses standard 3 Assessment as satisfactory. 4.4 Graduation guarantee and financial provisions (standard 4) The institution guarantees students that they can complete the entire curriculum and makes sufficient financial provisions available. Findings The information dossier describes the graduation guarantee and long-term budget of the programme. Leiden University guarantees that participants who begin the programme will be able to complete it. Should the programme be discontinued, participants will be given the opportunity to sit exams up to two years after the nominal duration of the programme. A long-term budget covering multiple years has been drawn up to develop and implement the executive master s programme. The budget is based partially on a one-time incentive grant for developing this new type of programme for the municipality of The Hague. The fee for the programme is in line with market rates so as to guarantee the programme s continuation. The registration and intake procedure has resulted in the admission of 28 participants for the start of the programme in January This is a very promising start, even if a couple of these admissions will not materialise because of a lack of funding. Considerations The panel has ascertained that the long-term budget is solid and that Leiden University will guarantee a sufficient time for participants to complete the entire programme. The number of applications shows that the programme is attractive for the target group. page 16

17 Conclusion The panel assesses standard 4 Graduation guarantee and financial provisions as satisfactory. 4.5 Final conclusion On the basis of the information dossier and the discussions during the site visit, the panel concludes that the executive master s programme in Cyber Security is an ambitious multidisciplinary programme with an innovative combination of technical and non-technical disciplines. The programme is targeted to professionals at master s level who wish to broaden and deepen their knowledge on issues concerning cyber security. The intended learning outcomes reflect the aim of the programme and are in line with the requirements of an academic master s programme, although more explicit attention for the role of law is desirable. The structure and contents of the programme are well-thought out, based on the programme s multidisciplinary approach. The phases of conceptualisation, specialisation, elaboration and research build up towards the attainment of the final qualifications. The minimal attention for (positive) law is a severe shortcoming and must be remedied before the programme can be unconditionally approved. The didactical approach, the quality of the academic staff, supplemented with the up to date field knowledge of guest lecturers, and the services and facilities will contribute to a coherent teaching-learning environment. The Board of Examiners consists of members from all relevant disciplines, is aware of its responsibilities and actively prepares to implement its role after the start of the programme. The assessment methods are in line with the learning objectives. The panel advises the Board of Examiners to check the validity of the exams before they are set. Leiden University guarantees that participants who have been admitted to the programme, can complete the entire curriculum. The long-term budget is solid and the number of applications for the first year is promising. The panel concludes that the executive master s programme in Cyber Security is a viable programme. 4.6 Recommendations and advice Standard 1: The panel recommends (1) to explicitly add to the general learning objectives that graduates should be able to take into account constraints of positive law, and (2) to add to the specific learning objectives of the governance track that graduates should have more in-depth knowledge of the relevant positive law. Standard 2: The panel advises the programme staff to closely monitor if the order of phases has the desired learning effect. page 17

18 The panel advises to create a more detailed educational reference framework ( onderwijskundig referentiekader ) and to map the contents of the programme to this framework in order to help the lecturers to align their courses to each other and to avoid gaps and overlap. The panel disagrees with the very limited time devoted to law and legal issues. A basic course of at least 4 EC must be added to the programme and be compulsory for all participants without a background in law. Standard 3: The panel advises the Board of Examiners to take a more pro-active stance in checking assessments in advance and, if necessary, to call upon assessment experts, e.g. from ICLON. An assessment matrix may be helpful to make sure that all learning objectives per course are covered by the assessment method(s). page 18

19 5 Overview of the assessments The panel presents its assessments per standard, as outlined in chapter 4, in the following table. Standard Assessment 1 Intended learning outcomes Satisfactory 2 Teaching-learning environment Unsatisfactory 3 Assessment Satisfactory 4 Graduation guarantee and financial provisions Conclusion Satisfactory Conditionally positive page 19

20 Annex 1: Composition of the panel Prof. dr. ir. Bart Preneel (chair) Prof. Bart Preneel is full professor at the KU Leuven where he heads the COSIC research group. His main research interests are cryptography, information security and privacy. He has served as panel member and chair for the European Research Council, as panel member of the visitation committee Informatics in The Netherlands (2013), and as president of the IACR (International Association for Cryptologic Research). He is a member of the Permanent Stakeholders group of ENISA, of the Academia Europaea, and of the Belgian Privacy Commission (subcommittee use of national number). He has served on the advisory board of several companies and regularly consults for government and industry. He has been invited speaker at more than 100 conferences in 40 countries. In 2002 he received the title of Honorary Certified Information Security Manager (CISM) from ISACA and in 2014 he received the RSA Award for Excellence in the Field of Mathematics. Prof. dr. Mireille Hildebrandt Mireille Hildebrandt is Professor of Smart Environment, Data Protection and the Rule of Law at the institute of Computing and Information Sciences (icis), Faculty of Science, Radboud University Nijmegen; Professor of Technology Law and Law in Technology at the research group on Law, Science, Technology and Society studies (LSTS), Faculty of Law and Criminology, Vrije Universiteit Brussel; Associate Professor of Jurisprudence, Erasmus School of Law (ELS), Erasmus University Rotterdam. The research of Hildebrandt is focused on the implications of increased implementation of artificial intelligence on democracy and the Rule of Law, she publishes widely on the nexus of philosophy or law and that of technology and teaches Law in Cyberspace to master students of the Kerckhoff Master at Radboud University Nijmegen. Marc Sel MSc Marc Sel Is working for PricewaterhouseCoopers 'Advisory Services' in Belgium as a Director, specialised in IT Security. He joined the firm in Jjanuary 1989 as a Consultant. Over time, he specialised in the field of security, both from the technical and from the organisational/management perspective. He performed specialised in-depth reviews, assisted clients with the selection of solutions, and performed implementations. Areas I worked in include authorisations and access control, network security, PKI, smartcards, biometrics, as well as information security organisation and policies, standards and guidelines. He obtained degrees from Technicum North Antwerp (Antwerp, Belgium ), Brussels Free University (VUB, Brussels, Belgium, 1984) and Royal Holloway University Of London, UK, He holds CISA (1993), CISM (2004) and CGEIT (2008) certifications from ISACA. Since 2006 he is also accredited "Lead Auditor" for ISO/IEC assignments. Nienke Bach Kolling BSc (student member) Nienke Bach Kolling is a Master student Educational Science & Technology at the University of Twente and bachelor student in the teacher training programme at Saxion University of Applied Sciences. She was chairman and commissioner educational affairs at the study association for Psychology and Educational Science, and member of the programme committee at the University of Twente. She has several years of experience in testing programs and institutions as a student member for the NVAO. page 20