LTAS 22.10a-b Massive Open Online Course (MOOC): Goldsmiths Introduction to Meteor.js Development ( MOOC 3 ) Web Application Development with JavaScript and MongoDB ( MOOC 4 ) Responsive Website Tutorial and Examples ( MOOC 5 ) Responsive Website Development and design Capstone Project ( Capstone Project ) Part of a MOOCs specialisation: Responsive website development and design Purpose Papers LTAS22.9a-b are submitted to the LTAS to support the launch of the following MOOCs which form part of the MOOC specialisation: Responsive website development and design: LTAS 22.9a MOOC Panel Report: Introduction to Meteor.js Development ( MOOC 3 ) LTAS 22.9b MOOC Panel Report: o Web Application Development with JavaScript and MongoDB ( MOOC 4 ) o Responsive Website Tutorial and Examples ( MOOC 5 ) o Responsive Website Development and design Capstone Project ( Capstone Project ) Background The Creative Programming MOOC team from Goldsmiths was selected by Coursera to develop a specialisation in Full Stack Web Development following a competitive interview round. The title of the specialisation was changed from Full Stack Web Development to Responsive website development and design following extensive Search Engine Optimisation analysis. This is the first specialisation launched on Coursera by a UK university. The first course of this specialisation was launched on the 15 September 2015 with the full specialisation available in December 2015. A MOOC specialisation is a multi-mooc sequence completed by a capstone project. To date, Coursera has launched 34 specialisations. In April 2015 Coursera announced a request for proposals in a number of high-demand topic area specialisations for launch in September 2015. University of London Lead College MOOC course teams were invited to express interest in these developments. Recommended Action Members of the Learning, Teaching and Assessment Sub-committee are invited to note that, on the basis of the recommendations made by the Panels considering the MOOCs: Introduction to Meteor.js Development ( MOOC 3 ) Web Application Development with JavaScript and MongoDB ( MOOC 4 ) Responsive Website Tutorial and Examples ( MOOC 5 )
Responsive Website Development and design Capstone Project ( Capstone Project ) Chair s Action has been taken to support the launch of these MOOCs as was agreed at the LTAS meeting on the 17 th September 2015 (refer to LTAS Minute 1182). [Secretary s note: Given the number of recommendations for the Introduction to Meteor.js Development MOOC, the Panel report suggested that that the efficacy of the course's delivery and the quality of the student experience would be enhanced significantly if these recommendations were implemented before launch. The MOOC team confirmed to the Panel that they had reviewed all of the recommendations and would complete the proposed action as detailed in their panel response before the launch of the course.] Quality, Standards and Governance University of London International Academy December 2015 2
LTAS22.9a LEARNING, TEACHING AND ASSESSMENT SUB-COMMITTEE Report of a panel convened on behalf of the Learning, Teaching and Assessment Subcommittee to consider the approval of a Massive Open Online Course (MOOC): Introduction to Meteor.js Development As part of a MOOCs specialisation: Responsive website development and design Date: The Panel considered the MOOC in the time period 5 11 October 2015. PART A: THE APPROVAL PROCESS AND FORMAT OF THE EVENT Introduction and Background 1. MOOCs allow participants to undertake free, bitesize units of learning. The University of London International Academy became a partner with Coursera 1 in September 2012 and is the chosen platform for the International Programmes MOOCs. To date, the University of London has launched 13 MOOCs with a further 5 MOOCs planned for development n 2015-16 in addition to the development of the first UK MOOC specialisation. The MOOC Introduction to Meteor.js Development forms part of the Responsive website development and design specialisation. 2. A MOOC specialisation is a multi-mooc sequence completed by a capstone project. To date, Coursera has launched 34 specialisations. In April 2015 Coursera announced a request for proposals in a number of high-demand topic area specialisations for launch in September 2015. University of London Lead College MOOC course teams were invited to express interest in these developments. 3. The Creative Programming MOOC team from Goldsmiths was selected by Coursera to develop a specialisation in Full Stack Web Development following a competitive interview round. The title of the specialisation was changed from Full Stack Web Development to Responsive website development and design following extensive Search Engine Optimisation analysis. This is the first specialisation launched on Coursera by a UK university. The first course of this specialisation launched on 15 September 2015 with the full specialisation available in December 2015. Structure 4. The specialisation will contain five MOOCs and a capstone project and are listed below along with the launch date: MOOC MOOC title Launch date 1. Responsive Website Basics: Code 15 September 2015 with HTML, CSS, and JavaScript 2. Responsive Web Design 15 October 2015 1 https://www.coursera.org/ 3
3. Introduction to Meteor.js Development 4. Web Application Development with JavaScript and MongoDB 5. Responsive website tutorial and examples Capstone Responsive website development and project design capstone project 16 November 2015 15 December 2015 15 December 2015 15 December 2015 5. All MOOCs within the specialisation will be created on Coursera s new platform called Ondemand. Previously, Coursera used a session-based platform with set start and end dates for each course (along with set assessment deadlines); on the Ondemand platform, courses will always be open. This provides learners with more flexibility by allowing them to fit the courses around their own lifestyle and work commitments and has proved very popular as an alternative to the session based courses. Certificates 6. In order to obtain a Course Certificate (this is available for each MOOC within the specialisation), a learner must verify each of their assessment attempts through recording their unique typing patterns and submitting photo identification, via a webcam. The learner will receive a certificate if they have passed the course and have paid the requisite fee ($79). Upon successful completion of the capstone project, learners will be eligible for the Specialisation Certificate. 7. The Course Certificate is designed to provide participants with feedback to indicate that they have met the intended learning outcomes. MOOCs are not credit bearing. Therefore the Course Certificate would not be eligible for Accreditation of Prior Learning for a University of London award. Content 8. A full stack web developer is the industry term for someone who builds and designs responsive websites. Essentially, this is a programmer who has capabilities in all the technical layers of a typical website. They should understand how underneath the familiar interface of their favourite web sites there are several layers of technology, leading from the text layout in the user interface all the way back to the database structure. 9. This specialisation takes a novice programmer through all of these layers; it starts at the front end with user experience design using HTML and CSS then moves into client side scripting with JavaScript and JQuery. Following that, the student will learn the meteor.js framework, which makes it possible to implement a reactive, multi-user web application using JavaScript and MongoDB. Along the way, the learner will be exposed to supporting knowledge such as the use of git repositories and unit tests, JavaScript libraries and HTML/ CSS templating with Bootstrap. Throughout the first four courses, the student will see an image sharing application built from scratch, eventually using all of the techniques taught. In the fifth course, four new meteor.js applications will be built from scratch. Finally, the learner will carry out a structured project from a set of options or of their own design, which will challenge and expand their knowledge of all the 4
layers. 10. As students progress through the MOOCs, they will begin to understand how the web pages they look at every day are generated from dealing with basic content layout through to how that content is stored and accessed from a database, and everything in between. This learning will be reinforced by threading a single project throughout the specialisation, where the students gradually build a complete website as they proceed. Each MOOC will add parts to the website, until by the end of the last MOOC they have seen the top to bottom construction of a fully functional, non-trivial, full stack website application. MOOC structure 11. Each MOOC within the specialisation will follow the same structure to provide consistency and familiarity across the courses. Each MOOC will contain four modules of work which is roughly equated to about two to three hours of learning and therefore we would expect students to complete a module within a week. Each module is broken down into lessons which will either be a single video lecture, a reading or a form of assessment. Assessment 12. Formative: Each lesson will have a short pop quiz (no more than 3 or 4 questions in a multiple choice questions quiz style) to test the learner s knowledge. All MOOCs in the specialisation will incorporate formative programming assignment exercises for students to test their code writing skills. 13. Summative: Each module within each MOOC will have a summary quiz of ten questions, which will each be weighted to 10% of the overall grade for the course certificate. Each MOOC will also have two peer assessments where students will be asked to practically apply their learning from that module; this piece of work will be assessed by 5 peers. Both peer assessments will be weighted at 30% of the overall grade for the certificate. In order to pass the course a student must obtain an average mark of 65% across all summative assessments. Capstone project 14. The capstone project is started once a learner has completed and passed all five MOOCs. In the capstone project, learners will be asked to create a complete, responsive, multi- user, data-driven website which aims to solve a particular problem. Learners will be encouraged to create a website addressing a problem that they are interested in, however students will also be given project ideas suggested by industry partners. 15. The capstone project will be a slightly longer course, currently structured at 8 weeks long and, during these weeks, learners will consolidate learning from across the five previous MOOCs. 16. Only learners who have passed the five MOOCs within the specialisation (and have paid for the certificate) will be eligible to graduate to the capstone project. 5
International Programmes Approval 17. The University of London International Academy has procedures in place to ensure that University of London International Programmes MOOCs are informed by good practice and provide a coherent learning experience. A Panel is convened on behalf of the Learning, Teaching and Assessment Sub-committee (LTAS) to review each proposed MOOC. 18. Members of the Panel were invited to consider the coherency of the curriculum, the appropriateness of the teaching, learning and assessment activities, and the overall experience of the MOOC participants. 19. The MOOC review process was conducted online, with discussion carried out via email. Panel members were provided with a pro-forma to help focus their considerations. The Panel Chair maintained oversight of the Panel s comments and confirmed the outcomes of the review. Panel Constitution 20. Membership of the MOOC Review Panel was as follows: Dr Roger Mills (Chair) Miss Rachael Trubshaw Mr Dele Ogunjumelo Ms Sarah Sherman Mrs Sharon Simpson (Secretary) Member of the Learning, Teaching and Assessment Sub- Committee Distance Education Consultant Peer Reviewer Lecturer and Award Leader for BSc Computing Science Staffordshire University Student member of the University of London Systems and Technologies Sub-Committee E-Learning Manager, Bloomsbury Learning Environment Member of the Systems and Technologies Sub-Committee Quality Manager University of London International Programmes PART B: CONSIDERATION OF THE MOOC Introduction 21. Panel members were provided with access to Coursera s Ondemand platform (learner view) where the materials to be considered were available. In line with feedback from MOOCs 1 and 2, formative programming assignment exercises for students to test their code writing skills are also to be incorporated. These exercises were not available at the time of the review as they were still being tested by Goldsmiths software engineer. The course team had been focusing on ensuring that the video lectures would be in place for the peer review event; all quizzes were available and 6
one summative assessment (peer assessment) was available. The Panel considered all four modules of MOOC 3 of the specialisation. 22. Materials for consideration by the Panel included: Videos/video lectures Peer assessments/quizzes Overview information provided to participants Summary of Observations Made by the Panel 23. A synopsis of Panel members observations concerning the MOOC is provided in the following paragraphs. The Appropriateness of the Title of the MOOC in Relation to its Content 24. The Panel acknowledged that the choice of title for the MOOCs was appropriate for the content of the course, and felt that it would be understood by students who had knowledge of meteor.js, or those who had completed MOOCS 1 and 2. 25. However, there was a technical leap in understanding from the first two MOOCs and a brief explanation of Meteor.js would be helpful. 26. It was also suggested that, if the purpose is to up-skill and introduce web development to a new audience, the course could be more usefully named Introduction to developing interactive web applications using meteor.js. 27. The Panel further commented that initial expectations of the course were confused as to whether the intention of the MOOC was to learn how to build an interactive website or an app ( presumably a web app and not a native app so that it can be accessed on both a mobile device s browser and a computer browser) or both. The Panel would welcome greater clarity. Coherency and Relevance of the Curriculum 28. As part three of a six course specialisation, the Panel found that it seemed to develop very steadily and clearly. The modules were deemed to be well-structured and contain topics relevant to achieving the learning outcomes of the MOOC. 29. Given the practical nature of the subject, the simple and straightforward module structure of a video followed by a short quiz was considered to be sufficient. However, it would be useful to include some examples of existing websites that would allow the participant to consider what type of website they will be creating, e.g. YouTube or Facebook. A critique of these commonly used websites would be helpful, either from the instructor or as a collaborative exercise in the discussion board. 7
The accuracy of information provided to prospective participants 30. It was not clear from the information on https://www.cousera.org/learn/meteordevelopment how long each week was supposed to take to complete in total. Nor was it clear, from the introduction, what Meteor.js is. It would therefore be helpful to have an example of a website that uses it, for example Amazon. 31. The description about the level of preparedness expresses that the course is aimed at beginners who have not done any programming before, albeit with a decent level of general ICT skills or that if students have programmed before, the course contains information of interest to those at intermediate and advanced levels. The Panel felt that this may not be an entirely fair statement. As this is a very specialist topic, it was assumed that potential students would understand the terminology and nomenclature used, such as Mongo DB. Accordingly, the Panel suggested (noting that it is stated that this is a part of a 6-course series ) that a statement be made to indicate to those enrolling for this MOOC that they should either have completed MOOCs 1 and 2 or be involved in web development in some capacity. 32. The fee structure is clear. 33. The Panel noted that the at the end of this course expectation was understandable and clear to students. Teaching, Learning and Assessment Methods 34. The Panel found the learning objectives to be very clear. There is a clear statement about what activities students should do together with the time taken over them. 35. The teaching, learning and assessment methods clearly help in confirming that the student has understood the learning content of the modules. The use of short quizzes is very beneficial to the learning process and the instructions for the assessment methods were found to be clear and complete. 36. It is not clear what percentage of each assessment point is worth. For example, whether the 1-hour website design would be worth more than 4 quizzes. A clear indication of weighting would be helpful. 37. The collections of modules build up to help the participants in meeting the progression requirements. However, some modules contain much new material and would benefit from being broken down a little more. 38. With regard to the Discussions, the expectations of use were not clear. The introduction directed students to forums to discuss the course material with fellow students and advised them to introduce themselves in the Meet and Greet forum, but this forum could not be located. 8
The suitability of the assessment activities to test the learning outcomes described in the MOOC outline 39. The assessment activities were thought to be sufficient for testing the learning outcomes of the MOOC. The Quizzes keep the student engaged and motivated and the content of the assessment activities are directly relevant to the learning outcomes. The final assignment is very useful for demonstrating knowledge of the concepts learnt. 40. The Panel noted that Quiz feedback is much better here than in the previous MOOCs for both correct and incorrect answers. Spelling and consistency in the use of capitals needs to be checked. 41. In general, the course author should review the use of capitals and punctuation in the quizzes to ensure consistency, for example, start all optional answers with a capital or start all optional answers in lower case. 42. With regard to the end-of-lesson quiz questions, there were only two, but a further question would be beneficial to provide a further opportunity to test for understanding. Students are advised that to complete they must answer at least 2 correctly. In the absence of a third question, this should be revised to read To complete, answer all/both questions correctly. The viability of the technical delivery of the MOOC 43. The Panel encountered various technical issues during the course of the review: In order to see week 5, it was necessary to scroll down the Course Content menu on the left hand side, so there was a concern that students may miss out on week 5. Once a video has been viewed, it does not start at the beginning when revisited. An explanation is given for those working on a PC or Linux, but the main teaching is conducted via MAC. Navigation can be tricky on an ipad; it was not possible to navigate back from the Coursera site describing the specialisation to the individual courses without using the back button. A link directly to the course is recommended. There were some broken links and some typos. The alignment of the MOOC with the University of London s brand and associated reputation 44. The MOOC positions the University of London as being relevant and current in the use and teaching of web development and technology. This can be seen from the use of modern web development technology and tools. 45. It was noted that placing the University of London logo icon on the left hand side of the page or within the header of the page may reinforce the brand. 9
46. Use of the word confer; in the statement Participation in or completion of this online course will not confer academic credit for University of London programmes could be replaced with entitle for clarity to users whose first language is not English. 47. There are many inconsistencies with the use of capital letters. Such a course should model correct spelling and formatting of terminology used in the technical world and as a reflection on the reputation of the University as the course provider. The overall participant experience of the MOOCs 48. Overall the Panel felt that the videos in the MOOC were engaging and that the introduction to each module was motivating and an encouragement to watch the video. Their short length was appreciated. However, the videos where the lecturer walks around the grounds were somewhat distracting although it was acknowledged that this introduced a little variety. 49. The videos and teachings are clear and provide an opportunity to check for understanding of the concepts via the quiz at the end of the video. However, as noted earlier, attention should be given to formatting, for example, the pop up text boxes do not need full stops and a spell check should be undertaken; Meteor and Mungo should always be capitalised (there are many examples in the quizzes where this is not the case). 50. The Panel noted that a MAC was used and although the lecturer was good at explaining alternative platforms, they questioned whether there had been any research into the most commonly used system beforehand, noting that students may have a block to using a new system. 51. The numbering of the sessions was confusing; week 1 should be the first module, whereas it is currently the Course Overview. It is then confusing that Week 2 is module 1. The introduction should be in week 0 or designated as Introduction. The various lessons within each module are not numbered (although the videos are); the video for MOOC 3 week 1, for example, is numbered 3.1.1, but this may not be clear to users, especially users who are only taking this MOOC. 52. The Coursera statement on financial assistance includes a request to Demonstrate values of academic integrity and contribute positively to the course s community, but it was not apparent how this could be evidenced before the course is studied. 53. The first 2 Discussion forums have titles in common and with the same purpose. The Panel would expect this to be corrected. 54. An interest was also expressed as to how many requests had been made for financial assistance and how many had been accepted. PART C: CONCLUSIONS Outcome 10
55. The Panel gives its support to the introduction of the MOOC Introduction to Meteor.js Development 56. The Panel suggested some enhancements for MOOC 3 a. The inclusion of some working examples of interactive websites with the tutor s comments, for example a screencast with observations/comments. b. The use of more animations in the teaching. In some modules, the majority of the content was just the source code displayed on the computer of the tutor. Include some diagrams and animation which highlight the concepts being taught. c. The use of additional media files other than image media files, for example audio or video files. d. Removal of the advertising on the learner pages 11
LTAS22.9b LEARNING, TEACHING AND ASSESSMENT SUB-COMMITTEE Report of a panel convened on behalf of the Learning, Teaching and Assessment Subcommittee to consider the approval of two Massive Open Online Courses (MOOCs) and a Capstone Project: Web Application Development with JavaScript and MongoDB ( MOOC 4 ) Responsive Website Tutorial and Examples ( MOOC 5 ) Responsive Website Development and design Capstone Project ( Capstone Project ) As part of a MOOCs specialisation: Responsive website development and design Date: The Panel considered the MOOC in the time period 1 8 November 2015. PART A: THE APPROVAL PROCESS AND FORMAT OF THE EVENT Introduction and Background 57. MOOCs allow participants to undertake free, bitesize units of learning. The University of London International Academy became a partner with Coursera 2 in September 2012 and is the chosen platform for the International Programmes MOOCs. To date, the University of London has launched 13 MOOCs with a further 5 MOOCs planned for development n 2015-16 in addition to the development of the first UK MOOC specialisation. The MOOCs and Capstone Project Web Application Development with JavaScript and MongoDB, Responsive Website Tutorial and Examples and Responsive Website Development and Design Capstone Project form part of the Responsive website development and design specialisation. 58. A MOOC specialisation is a multi-mooc sequence completed by a capstone project. To date, Coursera has launched 34 specialisations. In April 2015 Coursera announced a request for proposals in a number of high-demand topic area specialisations for launch in September 2015. University of London Lead College MOOC course teams were invited to express interest in these developments. 59. The Creative Programming MOOC team from Goldsmiths was selected by Coursera to develop a specialisation in Full Stack Web Development following a competitive interview round. The title of the specialisation was changed from Full Stack Web Development to Responsive website development and design following extensive Search Engine Optimisation analysis. This is the first specialisation launched on Coursera by a UK university. The first course of this specialisation launched on 15 September 2015 with the full specialisation available in December 2015. Structure 60. The specialisation will contain five MOOCs and a capstone project and are listed below along with the launch date: MOOC MOOC title Launch date 1. Responsive Website Basics: Code 15 September 2015 with HTML, CSS, and JavaScript 2. Responsive Web Design 15 October 2015 2 https://www.coursera.org/ 12
3. Introduction to Meteor.js Development 4. Web Application Development with JavaScript and MongoDB 5. Responsive website tutorial and examples Capstone Responsive website development and project design capstone project 16 November 2015 15 December 2015 15 December 2015 15 December 2015 61. All MOOCs within the specialisation will be created on Coursera s new platform called Ondemand. Previously, Coursera used a session-based platform with set start and end dates for each course (along with set assessment deadlines); on the Ondemand platform, courses will always be open. This provides learners with more flexibility by allowing them to fit the courses around their own lifestyle and work commitments and has proved very popular as an alternative to the session based courses. Certificates 62. In order to obtain a Course Certificate (this is available for each MOOC within the specialisation), a learner must verify each of their assessment attempts through recording their unique typing patterns and submitting photo identification, via a webcam. The learner will receive a certificate if they have passed the course and have paid the requisite fee ($79). Upon successful completion of the capstone project, learners will be eligible for the Specialisation Certificate. 63. The Course Certificate is designed to provide participants with feedback to indicate that they have met the intended learning outcomes. MOOCs are not credit bearing. Therefore the Course Certificate would not be eligible for Accreditation of Prior Learning for a University of London award. Content 64. A full stack web developer is the industry term for someone who builds and designs responsive websites. Essentially, this is a programmer who has capabilities in all the technical layers of a typical website. They should understand how underneath the familiar interface of their favourite web sites there are several layers of technology, leading from the text layout in the user interface all the way back to the database structure. 65. This specialisation takes a novice programmer through all of these layers; it starts at the front end with user experience design using HTML and CSS then moves into client side scripting with JavaScript and JQuery. Following that, the student will learn the meteor.js framework, which makes it possible to implement a reactive, multi-user web application using JavaScript and MongoDB. Along the way, the learner will be exposed to supporting knowledge such as the use of git repositories and unit tests, JavaScript libraries and HTML/ CSS templating with Bootstrap. Throughout the first four courses, the student will see an image sharing application built from scratch, eventually using all of the techniques taught. In the fifth course, four new meteor.js applications will be built from scratch. Finally, the learner will carry out a structured project from a set of options or of their own design, which will challenge and expand their knowledge of all the layers. 13
66. As students progress through the MOOCs, they will begin to understand how the web pages they look at every day are generated from dealing with basic content layout through to how that content is stored and accessed from a database, and everything in between. This learning will be reinforced by threading a single project throughout the specialisation, where the students gradually build a complete website as they proceed. Each MOOC will add parts to the website, until by the end of the last MOOC they have seen the top to bottom construction of a fully functional, non-trivial, full stack website application. MOOC structure 67. Each MOOC within the specialisation will follow the same structure to provide consistency and familiarity across the courses. Each MOOC will contain four modules of work which is roughly equated to about two to three hours of learning and therefore we would expect students to complete a module within a week. Each module is broken down into lessons which will either be a single video lecture, a reading or a form of assessment. Assessment 68. Formative: Each lesson will have a short pop quiz (no more than 3 or 4 questions in a multiple choice questions quiz style) to test the learner s knowledge. All MOOCs in the specialisation will incorporate formative programming assignment exercises for students to test their code writing skills. 69. Summative: Each module within each MOOC will have a summary quiz of ten questions, which will each be weighted to 10% of the overall grade for the course certificate. Each MOOC will also have two peer assessments where students will be asked to practically apply their learning from that module; this piece of work will be assessed by 5 peers. Both peer assessments will be weighted at 30% of the overall grade for the certificate. In order to pass the course a student must obtain an average mark of 65% across all summative assessments. Capstone project 70. The capstone project is started once a learner has completed and passed all five MOOCs. In the capstone project, learners will be asked to create a complete, responsive, multi- user, data-driven website which aims to solve a particular problem. Learners will be encouraged to create a website addressing a problem that they are interested in, however students will also be given project ideas suggested by industry partners. 71. The capstone project will be a slightly longer course, currently structured at 8 weeks long and, during these weeks, learners will consolidate learning from across the five previous MOOCs. 72. Only learners who have passed the five MOOCs within the specialisation (and have paid for the certificate) will be eligible to graduate to the capstone project. International Programmes Approval 73. The University of London International Academy has procedures in place to ensure that University of London International Programmes MOOCs are informed by good practice and provide a coherent learning experience. A Panel is convened on behalf of 14
the Learning, Teaching and Assessment Sub-committee (LTAS) to review each proposed MOOC. 74. Members of the Panel were invited to consider the coherency of the curriculum, the appropriateness of the teaching, learning and assessment activities, and the overall experience of the MOOC participants. The Terms of Reference for the MOOCs Review Panel are attached in Appendix 1. 75. The MOOC review process was conducted online, with discussion carried out via email. Panel members were provided with a pro-forma to help focus their considerations. The Panel Chair maintained oversight of the Panel s comments and confirmed the outcomes of the review. Panel Constitution 76. Membership of the MOOC Review Panel was as follows: Dr Roger Mills (Chair) Mr Gavin Allanwood Mr Dele Ogunjumelo Ms Sarah Sherman Mrs Sharon Simpson (Secretary) Member of the Learning, Teaching and Assessment Sub- Committee Distance Education Consultant Peer Reviewer Course Leader for Web Design & Development School of Film, Media & Performance at University of Central Lancashire. Student member of the University of London Systems and Technologies Sub-Committee E-Learning Manager, Bloomsbury Learning Environment Member of the Systems and Technologies Sub-Committee Quality Manager University of London International Programmes PART B: CONSIDERATION OF THE MOOC Introduction 21. Panel members were provided with access to Coursera s Ondemand platform (learner view) where the materials to be considered were available. The Panel considered all four modules of the remaining MOOCs of the specialisation. 22. Materials for consideration by the Panel included: Videos/video lectures Peer assessments/quizzes Overview information provided to participants Summary of Observations Made by the Panel 15
23. A synopsis of Panel members observations concerning the MOOCs is provided in the following paragraphs. The Appropriateness of the Title of the MOOC in Relation to its Content 24. The Panel agreed that the choice of title for MOOC 4 was appropriate for the content of the course. It was commented that it would be possible for a non-subject specialist who had only learned about Mongo during the previous MOOC review, to still understand what this course will offer. However, it was also noted that the title now included DB (database), but had been written in MOOC 3 without DB, for example Mongo filters, Mongo Collections. 25. The inclusion of the word tutorial in the MOOC 5 title was questioned as this is implicit. The title was deemed to undersell the course and gives the impression that the course would not be very dynamic. The responsive element was not clear and the course clearly needs to differentiate from the previous Responsive Web Design (MOOC 2). The title Exploring, Creating and Evaluating Responsive Websites was therefore suggested. 26. The content of the Capstone Project focuses on the steps required in develop a web application and also on the business aspect of app development, for example, identifying target market, pricing etc. The title could be updated to reflect this content. It was also suggested that capstone was a term that may not be recognised by the diverse audience for MOOCs and, noting that the presenter used the label Project in the introduction, that Portfolio or Integrated Project might carry more meaning. Coherency and Relevance of the Curriculum 27. The panel acknowledged that the modules of MOOC 4 are well structured and contain topics relevant to achieving the goal of the MOOC. Also, the course seems to fit in logically to the programme of the specialisation. 28. The simple structure of MOOC 4 was welcomed as the content is quite heavy. The modules are broken down into bite-sized video chunks that are no longer than 20 minutes. As the approach used by the course is to model hands-on processes followed by a quiz for understanding, videos under 10 minutes were considered to be preferable. It was not clear to the panel, without actually completing the course whether the set of learning outcomes for each module is achieved by completing the activities since the language used it different. For example, in the first Module, the learning outcome Use iframes to create separate DOMs is presumably covered in the video Bootstrap it! In order to meet the MOOC review objective to ensure the collection of modules allow participants to achieve the stated learning outcomes, the module topics should be titled more clearly and obviously. This suggestion would have been a helpful aide for the MOOC review panel. 29. It was noted that the word mobile appears only in the About this Course information and course outcome 1, yet it was not thought that a truly native mobile app can be produced using JavaScript. (MOOC 4) 30. In general, MOOC 5 was recognised as being heavier than the previous course. Each module focuses on the development of a different type of responsive website yet the previous MOOC was dedicated to creating one website (the collaborative text editor). There is quite an increase in the volume of course content and a higher-level of workload if the students are following the tutor s work step-by-step as they did 16
previously. The emphasis should therefore be on how a dynamic portfolio website is built rather on a how-to guide. 31. Overall, the content of MOOC 5 was considered to fit together well and the structure would appear to provide logical progression from topic to topic towards achieving the goal of the MOOC. Clearly students will have had to do previous MOOCs to follow this, or be quite competent professionals. 32. However, the panel highlighted the importance of keeping the language used consistent to prevent confusion. It was proposed that an explanation of the course taxonomy (or reference to the use of lesson ) could be included on the Course outline/info pages (MOOC 5). 33. MOOC 5 identifies Learning Outcomes for each topic ( lesson ) within each module. This is inconsistent with the previous MOOCs in the specialisation. It was surmised that the Learning Outcomes in the Introduction section of a module refer to the whole module and the proceeding learning outcomes are for each topic/lesson in the module. If so, occurrences of the use of the word lesson need to be double-checked, for example, the following page, which lacks some detail: https://www.coursera.org/learn/responsive-websiteexamples/supplement/neuh1/learning-outcomes. It would be useful for each Learning Outcomes page to have a title so it s clear which lesson is being covered. 34. The modules of the Capstone Project were considered to be well structured and contain topics relevant to achieving the goal of the MOOC, although it was commented that there was no reference to briefs from industry in the modules. 35. The Capstone Project Learning Outcomes 2 and 3: Work to a brief supplied to one of our industry partners or to your own brief Document and receive feedback on the development of your website from an idea to a fully functional piece of software in a series of short, peer assessed screencast videos read like tasks rather than outcomes. 36. The Capstone activity plan could not be downloaded and it would be very useful if this was made possible. (Secretary s note: the panel had queried why there were more weeks on the plan than modules. It has since been confirmed that before the courses launch, Coursera will reformat the structure to show 4 weeks of content. The Introduction module and the first module of content will be combined into the first week) The accuracy of information provided to prospective participants 37. As referred to (in para 29) above, the words native mobile (MOOC 4) may mislead. The panel agreed that there should be an explanation of the word native to explain that it is a dedicated application that runs on a mobile device rather than via the device s browser. 38. The panel suggested some clarifications to the course information/course outline page for MOOC 4: The pre-requisites for MOOC 4 (mentioned in the introduction) were clear, but could be made more explicit in the course description. There are course learning outcomes and module/week learning outcomes for MOOC 4. These should all be included in the course information. 17
Students about to take this course may not know what unit tests are. The use of confer in the note Participation in or completion of this online course will not confer academic credit for University of London programmes and change (e.g. grant or entitle ) requires amendment( as previously suggested for the review of MOOC 3) Some typographical errors were also identified (see Appendix 2) 39. As referred to (at 25) above, the word tutorial in the MOOC 5 title did not convey the right impression of the specific content to students. 40. It was noted that the grading and assessment information for MOOC 5 could be clearer avoiding relative terms like both and all and that it would be helpful to have a table listing each assessment element with the corresponding weighting. 41. There was a lack of clarity in the MOOC 5 About this Course section in relation to the reference to different needs and whether this referred to accessibility. 42. The information provided in the Capstone Project outline was clear and contains relevant details. Teaching, Learning and Assessment Methods 43. As with the previous MOOCs in the specialisation, the format of MOOC 4 is clear and consistent with a video tutorial followed by a quiz containing two questions. The development is very logical and systematic but moves quickly. The coding steps in the presentation could be slowed down as it was sometimes too fast (although the user can pause or rewind the presentation). Feedback from students on the pace will be interesting. 44. The collection of modules in MOOC 5, progressively help the participant in meeting the progression requirements. However, the speed of the progression must be quite difficult even for experienced programmers. 45. MOOC 4 takes the student through developing an example app and in this case it is a collaborative text editing tool. It also demonstrates how to learn from existing applications (e.g. CodeMirror) to develop applications, which is an approach that participants can continue to use on their own following the course. However, the panel would be interested to know whether students would actually be able to develop their own apps, as a result of undertaking this MOOC, without the step-by-step guidance this course offers. Although the MOOC provides excellent teaching on exactly how to create a collaborative text editor and other tools that commonly exist (e.g. IM client), it was not clear to a non-subject specialist whether the course provides enough detail around the generic skills required for development and whether these examples are broad enough to demonstrate more generally how to develop. 46. In Recap (MOOC 5) learning outcomes 1-3 are about context, communication and process. Learning outcome 4 seen is rather weak and refers to site not website. It was suggested that as the first video is called Recap the first quiz in the MOOC introduction could be formulated around what you ve learned in the previous MOOCs that you ll need for this one. This would overcome the issue of some of the introduction quizzes containing just the one question. 47. The instructions for the assessment methods of MOOCS 4 AND 5 were clear and complete. 18
48. The collection of modules for the Capstone Project, progressively help the participant in meeting the progression requirements. Specific examples would be helpful (see Appendix 2) 49. The instructions for the assessment methods of the Capstone Project were clear and complete. However, it would be desirable to provide some reference material to assist those making peer assessments. This could be done by presenting an industry brief as a case study, including aspects that went well as those that did not go so well. The suitability of the assessment activities to test the learning outcomes described in the MOOC outline 50. Overall, the assessment activities were thought to be sufficient for testing the learning outcomes of MOOC 4. There was some clarification required on some of the assignments as follows: The objective or purpose of the Working in the console assignment other than trying out a few activities within Meteor and proving they were achieved by taking screenshots. It was considered that using various console tools to dig around in your meteor applications is somewhat vague This assignment does not appear to require screenshots to be taken demonstrating the objectives have been achieved. Given this assumption, clarification is sought as to whether reviewers are simply required to try out the app? Do participants get credit for their reviews of each other s assessments? Is this moderated? The value of peer assessment is that students understanding of a topic is strengthened via the process of evaluating each other s work. Is there a marking crib-sheet or are the instructions used? Is this process explained adequately? 51. The MOOC 4 & 5 quizzes were considered to be not too difficult to complete and to contain a sufficient amount of questions (although two rather than three would be a more usual convention to check for understanding) which should both keep the student engaged and motivated. Also, the content of the assessments activities are relevant to the learning outcomes. However, most of the questions are related to the programming language syntax and statements and it would enhance the MOOC if these questions were mixed with questions on web development concepts. 52. The panel also felt that it would be helpful if there were some indication of which module learning outcomes are intended to help fulfil course learning outcomes. A matrix to illustrate this would be useful. 53. The instructions for the assessment activity at the end of each week of MOOC 5 could be more explicit. In some earlier examples there has been a breakdown of marks for each section of the assignment and this reflects good practice. 54. The MOOC 5 assignment Responsive Website Tutorial and Examples: Peer review 1 provides a very good opportunity for the student to demonstrate what they have learned so far in the course. There is sufficient guidance as to what is expected. The panel queried whether there would be full instructions as to how to evaluate a peer s assignment. 55. The role of the capstone in enabling students to obtain a certificate is clear, but 19
more detail should be provided about how assessment will work: You will be assessed on the key issues of your presentation needs to be broken down into elements with marks. 56. The panel had the following observations on the quizzes in the Capstone Project: The first quiz of the Capstone Project did not deliver the humour intended and should either be removed or the questions should be changed to make it more serious. The second quiz in Module 1 is a To Do list of the things you plan to do for your project next week. The purpose of writing a To Do list as an open question was not apparent. The panel queried whether this was because there is no other tool in Coursera and questioned whether the student would be able to access it again to return to it. If this was not the case, it was proposed that it would be more useful to post a message on a Forum. Check The answers to the Module 2 quiz re elevator pitch The Module 2 elevator pitch assignment as a recording was well received; this should be enjoyable for the student to both make and assess The second To Do list (in Module 2) assumes a copy of the first To Do list was retained. There was concern that this may not be the case, therefore the instructions would need to be reviewed or an alternative way of submitting the list be found. It would be useful to provide an example for the screencast assignment. Some typographical errors (see Appendix 2) The viability of the technical delivery of the MOOC 57. The panel commented that some of the MOOC 4 & 5 videos start just before the introductory screen with the University of London and music. The music was not well received by the panel as it prefaced each video and became irritating after a few times, especially as it was not possible to turn it off and given that the MOOC would be studied over several hours. The suggestion was that it just be used at the beginning of each week. 58. The panel felt that the technical delivery of MOOC 4 was good, but made the following comments about specific elements: The boxes in the video (8.5.1 video) are not referred to in the oral presentation and the relevancy was not clear. The video sticks in places in 5.1.3 and ends rather abruptly. There is a screen shot of code before the video starts. It was possible to get question 1 of Reactive Data both correct and incorrect at the same time The introduction indicates that there would be a dedicated Meet & Greet forum added to the Discussions, but this was not yet apparent. 59. There were some technical issues to be addressed before launch of the MOOCs: MOOC 5 Sub-titles come on before video starts sometimes. See e.g. 5.1.4 The Apple remains visible at times The link at 5.2.5 link on screen does not work. It is not clear how it can be accessed (see minute 3.53 on video) The link to the transcript in Project Structure actually links to subtitles (and in other modules too) The titles could better describe the place in the course, for example, Week 5>1. Recap of weeks 1-4; Week 5>2. Introduction 20
In the peer assessments, it would be helpful to be able to download the preview grading criteria as it is difficult to read. 60. Overall the technical delivery of the Capstone Project was good although there were some areas to be addressed: The presenter s name should come up on the screen There was a technical issue at the beginning of the Week 5 video Presenting your project In Module 5, the text box to add the end of project git log says What do you think? Is this hard-coded in Coursera and not editable? There were also some questions with wrong answers, for example, question number 3, the following seems to be taken as the correct answer: An elevator that is slightly damaged and therefore leans slightly to the side The link to the transcripts actually links to subtitles. The alignment of the MOOC with the University of London s brand and associated reputation 61. The panel asserted that MOOC 4 is clearly associated with the University of London s brand and associated reputation; All videos are adequately branded; the Home and Course info pages state that the course is co-authored by the UoL and Goldsmiths. 62. Furthermore, the MOOC s under review position the University of London as being relevant and current in the use and teaching of web development and technology. This can be seen from the use of modern web development technology, tools and open source code. 63. The panel also commented that the consistency of language and spelling is paramount if the University is proffering its web design/development expertise to a global audience. The course editors/proofreaders need to ensure that throughout the course, terminology is consistently (and correctly) spelled, capitalised, hyphenated, etc etc. Also, the course taxonomy should be consistent (i.e. lesson/topic/module/session). 64. As commented in para 57 above, the music was not well received by the panel. This continued to irritate and could not be turned off. The overall participant experience of the MOOCs 65. The videos are clear. The demonstration with the specialisation leader was very good, making it collaborative. There was also a very good introduction to the Recap section. This included good use of text in presentation to emphasise some points. In general, the MOOC was very engaging and the presenter is an increasingly engaging and effective teacher.. As the specialisation has developed, the presenter has become more confident and natural in front of the camera. 66. The sub-titles are a real help even to native English speakers. The colloquialisms in speech and subtitles. e.g. gonna, coz/cuz, kinda help to create an informal atmosphere. The participant experience is anticipated to be good. 67. The pink/orange background in the videos, with the presenter in front of the computer is much better than a white background 21
68. The introduction to each module of MOOC 4 is good and motivating. The introduction to the development of collaborative editor was very good. There was also a nice demo for reactive data using various browsers. The videos and teachings are also clear. 69. It would be good if the new tutor for MOOC 5 introduced himself at the beginning. The outgoing tutor says in the previous MOOC that a new tutor will be taking over but it would be helpful to have this repeated either on the screen by the new tutor. 70. The course is quite tough and there should be clear statements about the previous knowledge /previous MOOCs required. It is clearly not self-standing now because there are references back to previous MOOCs. Summaries are good and helpful. Presumably clear information will be provided to help students gain access to all the promised resources. 71. To avoid interrupting the flow of the presentation, the black screen slides for MOOC 5 could be replaced with an overlay, whiteboard or text bubble. 72. CLI commands appear out of context in bulleted lists and may be better seen only in the context of a CLI. 73. The Administration module could be better titled, such as Dynamic and Responsive. 74. The audio levels on some videos are low. 75. The use of PowerPoint in MOOC 5 to break up the video frame between the segment where the presenter speaks to camera and the Meteor application demonstration was commended as was the way in which course authors are encouraging the students to use the frameworks described in the MOOC in the real world, enforcing the link between theory and practice. 76. The panel felt that the videos in the Capstone Project where the specialisation leader is talking with the instructor were awkwardly filmed. Learning from video is quite difficult but presumably the students who are studying this programme will be comfortable with the approach. James was really engaging, but the interview was not well-staged and was unnaturally scripted (for example, in the Module 1 video, the specialisation leader claimed not to have heard of SWOT analysis. It would have been better filmed either as an interview or as an informal chat. Student views on the programme will be critical. It will also be interesting to see the comments of the student who complete or don t complete the MOOC. PART C: CONCLUSIONS Outcome 77. The Panel gives its support to the introduction of the MOOCS Web Application Development with JavaScript and MongoDB Responsive Website Tutorial and Examples Responsive Website Development and design Capstone Project 78. The Panel suggested some enhancements for the MOOCs as follows: (MOOC 4:) 22
a. The pre-requisites for MOOC 4 (mentioned in the introduction) could be made more explicit in the course description. b. It is difficult to learn such complex programming through video and it would have been beneficial to be able to print-out the screens shown. It is however, very helpful to see the code built up step by step. c. A glossary of all the terms and technical vocabulary (spelled correctly) used across the specialisation and within each MOOC would be excellent. If there is not such a feature built into Coursera, perhaps the authors can create a webpage, discussion thread or a downloadable PDF. d. Mix up the MOOC 4 quiz questions with questions on web development concepts and programming language syntax to achieve a better balance of assessment relevant to the learning outcomes. e. Use different font for code snippets in quiz questions. See Appendix 2 for suggestion. f. Slow down coding steps and also show users what you have done before progressing. g. Some consistency should be in place for popups; some commands or statements are highlighted in the video while others are not highlighted. h. Give thought to showing the code developed during the summary at the end of each module. i. Consider organising code from the start instead of at the end of the MOOC as there was a module on organising code. j. In the discussion of scope, relate some of the details in scope using template helper functions to Object Oriented concepts, for example, encapsulation. (MOOC 5:) a. Provide an explanation of the course taxonomy (or reference to the use of lesson ) could be included on the Course outline/info pages. b. Clarify the grading and assessment information for MOOC 5 by avoiding relative terms like both and all and by providing a table listing each assessment element with the corresponding weighting. c. Revisit the learning outcomes 1-3 in Recap (MOOC 5) to strengthen learning outcome 4 and to correct typographical errors d. Revisit the quiz in the MOOC 5 introduction to address what you ve learned in the previous MOOCs that you ll need for this one to replace one-question quizzes. e. Review the number of questions in the MOOC 4 & 5 quizzes and consider mixing the questions relating to the programming language syntax and statements with questions on web development concepts. f. Produce a matrix to illustrate which module learning outcomes are intended to help fulfil course learning outcomes. g. The new tutor for MOOC 5 should introduce himself following the handover between tutors, either on screen or in person. h. Provide a clear statement, for MOOC 5, about the previous knowledge/previous MOOCs required and clear information to help students gain access to all the promised resources. i. Consider replacing the black screen slides for MOOC 5 with an overlay, whiteboard or text bubble to avoid interrupting the flow of the presentation. (Capstone Project) a. Provide examples of the techniques discussed, for example, a SWOT analysis of a web app, an example elevator pitch. There could be links to videos or other documents containing examples. b. After all these weeks of study, the ending is sudden; it would be a nice touch to have a final, farewell video from the Couse Director. 23
c. The instructor could have been introduced. He has an interesting and relevant background in web development and it would be interesting to include, a short interview on his start-ups. d. Some of the videos used could have been introduced at the end of the modules in the previous MOOCs to bring about some inspiration and to reduce the number of videos that are coding related. 24