NMLS Course Evaluation Form The following form is to be used in the evaluation process for determining whether or not to approve a request to have a course NMLS approved. Part One of the form focuses on content and provides the overall course status. Part Two focuses on compliance with NMLS functional specifications. After completing this form, a copy of it is kept as part of the course provider s permanent file. App ID: Provider: Course Title: Review Completion Date: Ranking: Y (Yes) = Acceptable; includes minimum explanation. N (No) = Needs work; may be missing critical elements and/or provides weak explanation. Part One Overall Instructional Design Category Y N Category Y N Syllabus: - Identifies the course and success requirements. - Lists the key objectives of the course and identifies specific points or concepts that the students need to focus on. - Identifies the instructor and contact information Student Support and Resources: - Course provides necessary course-specific resources, materials, contact information, and technical support. Course Objectives: - Identify what the student will learn by taking the course. - Identifies what success factors are for completing the course. - Aligns with classroom format and activities. Course Organization and Design: - Course is well organized and easy to navigate. - Aesthetic design presents and communicates course information clearly throughout the course. - All pages/modules are visually and functionally consistent throughout the course. Materials/Text: - Materials are current (timely) and appropriate for the course topic. - Materials are formatted in such a way as to be useful and support effective instruction. - Materials contribute to ensuring course quality and rigor. Instructional Design/Delivery: - Course offers ample opportunities for interaction and communication student-t-student, student to instructor, and student to content. - Course goals are clearly defined and aligned to learning objectives. - Course provides multiple activities that help student to develop learn the material and reinforces learning (online only). - Activities are design to promote instructor to student interaction (online only). Instructional Methods/Strategies: - Instructional methodology provides for instructor/student interaction. - Activities support learning objectives. - Instructional methodology is appropriate for delivery format. Use of Technology (online only): - Course uses a variety of technology tools to appropriately facilitate communication and learning. - Course optimizes features of the Learning Management System (LMS) to delivery an engaging course. - Information is chunked appropriately. - Ample evidence of instructor/student interaction. Course Completion Requirement: - Requirements for course completion are articulated. - Measures for course completion are evident. Student Feedback: - Instructor offers multiple opportunities for students to give feedback on course content. OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 1
Approval Criteria (expected response is yes; any no response requires clarification) Y N Criteria 1 Possess Learning Objectives: does the course contain learning objectives that describe what the student should know or be able to accomplish upon completion of the course? Criteria 2 Course Must Have Sufficient Material: does the course contain sufficient course materials to support quality instruction relevant to the subject of the course. Criteria 3 Sufficient Difficulty: is the course of sufficient rigor as to prepare the student to work within the industry? If this is a continuing education course, does the course possess a degree of difficulty and it designed to enhance the professional competence and/or professional responsibility of the mortgage loan originator? Criteria 4 Be of Sufficient Length: does the course meet the minimum number of hours as required by the SAFE Act and/or the length being requested for approval? - For PE Comprehensive 3/3/2/+12 electives (20) - For PE State Comprehensive 3/3/2/+State hours + Remaining Hours (20+) - For 8HR Core 3/3/2 (8) - For CE Comprehensive 3/2/2/+1 elective (8) - For CE State Comprehensive 3/2/2/+State hour(s) (8+) - For CE Core 3/2/2 (7) - Ethics shall include instruction on fraud, consumer protection, and fair lending issues. Criteria 5 Be Delivered in an environment Conductive to Learning: is the course being administered in a physical environment that is appropriate for adult education? If this is an online course, is a Learning Management System (LMS) being properly employed? Criteria 6 Have a Defined Start and End time: for instructor-led online pre-licensure courses, is there evidence that the course is designed with a fixed start and end time (course must not be self-paced)? Final Recommendation: Approved (course meets all criteria) Approved with Condition(s) (minor content changes or recommendations should be addressed but are not severe enough to warrant deficiency. Post comments below.) Deficient (course design or content requires modification and/or updates/changes before approval can be granted. Post comments below.) Not Approved (course does not meet one or more of criteria established. Post comments below.) OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 2
Part Two The following form is used as a supplement to Part One in the evaluation process for determining whether or not to approve a request to have a course NMLS approved. The following evaluation checklist references the 2012 Functional Specifications for All NMLS Approved Courses. Ranking: Y (Yes) = Acceptable; includes minimum explanation. N (No) = Needs work; may be missing critical elements and/or provides weak explanation. Category Y N Course Syllabus 1.6.1 The course syllabus is and/or communicates the following: - Course Provider Contact Information - Instructor(s) Contact Information (If Applicable) - Course Description - Required Course Material, Textbook, etc - Course Completion Requirements (including passing scores, retake parameters, etc) - Policies Regarding Attendance and Behavior - Course Outline with time breakdown - Learning Objectives - Ability (including time-frame) to retake a test, quizzes or re-access content - Any Technology Requirements, including need for specific software, web browser compatibility, camera, etc and Technical Support contact info, hours available, etc. Course Content Organization 1.9/ Course Design 5.2 The course content is and/or contains: - Organized in a linear fashion to present information in a logical manner - Broken into specific modules, and contains multiple modules (a module is a defined single topic to learning within the content of a larger course subject) SEE 5.2.1 - Modules are broken into specific units of instruction, and each modules contains multiple units of instruction (a unit of instruction is a smaller topic of instruction, that, when combined with other units of instruction, support the learning objectives of a larger course module) SEE 5.2.2 - Information within each module is properly chunked SEE 5.2.3, 4, 5, 6 Requirement to Cite References 1.6.2 The course contains: - Appropriate citations provided for federal and state laws - Appropriate citations and acknowledgements provided to preserve copyright and intellectual properties OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 3
Interaction 1.10 The course has and/or contains: - Interactivity that is appropriate for this class format - Activities incorporated at multiple points throughout the whole course - Obvious and consistent opportunities for student-to-content interaction throughout the course - A level of difficulty that necessitates the need for the adult learner to listen and/or read and engage the content for comprehension in order meet course completion requirements Inclusion of Case Studies 1.10.4 - Course contains Case Studies - Case Studies reinforce learning objectives -Case Studies provide context for how federal law and/or state law applies in real world scenarios -Case Studies are of sufficient detail and complexity to ensure the student grasps the importance and applicability of the issue being discussed -STATE SPECIFIC CONTENT Case Studies are relevant to the specific state law, regulation, statute, and/or rule being covered Quizzes 1.10.3 & 5.2.7 Quizzes must: - Each module contains a quiz - inform the student how much time is allowed to complete the quiz - inform the student of the passing score - the passing score is high enough to give it legitimacy - passing score is appropriate for the number of questions (5.2.7.3) - support learning objects - posses rigor - have at least 3 questions for a module that is 10 minutes or less (5.2.7.1) - have between 5-10 questions for a module that is greater than 10 minutes (5.2.7.1) - be only multiple choice and not contain true/false questions (5.2.7.1) - drawn from a question bank with a ratio of 2:1 (5.2.7.2) - be passed before student can move on to the next module or unit or instruction (5.2.7.3) - present a high percentage of different questions in a randomized order upon retake (5.2.7.3) - inform the student if a wrong answer was selected (5.2.7.4) - does not allow blow-through of the quiz (i.e. selecting wrong answer in order to retrieve the correct answer and then use that to pass upon a retake (5.2.7.4) OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 4
End of Course Assessment 1.13.1 The final exam is and/or contains: - Course contains a final exam - Supports learning objectives - Designed to assess learning - Promotes critical thinking and requires student to reflect on the course material - Final exam has a passing score of minimum 70% - Requirement to pass final exam for course credit - Contains only multiple choice questions and does not contain any true/false questions - Constructed so as to prevent pattern guessing (ie there are not a significant number of answers that are B) - Does not account for more than 20 minutes of accredited course time for a course 3 hours or more - Does not account for more than 10 minutes of accredited course time for a course 2 hours or less - does not allow a student to game the test (1.13.1.f) - present a high percentage of different questions in a randomized order upon retake (1.13.1.g) Final Exam Question Bank: - For 1-2 hour course, final exam is at least 15 questions - For 3 or more hour course, final exam is at least 25 questions - Final exam is drawn from a question bank with a 2:1question ratio (ex. If a 2 hour course has a final exam of 20 questions, the question bank it pulls from must contain 40 questions.) Time Requirements 1.5 & 5.3 - Course contains enough material and activities to satisfy the hours requested for approval based on a 50 minute clock hour (1.5 & 5.3.4) - Course contains enough material to meet the minimum time required in the SAFE Act for the specific education topics (PE courses 3:3:2:12 / CE courses 3:2:2:1) (1.5 & 5.3.4) - Course provider has a means by which to monitor and ensure seat time. Seat time is the time in which the student is engaged in the course and engaged with the instructor, other students, course content, and activities (1.5 & 5.3.4) - Has a Mandatory Logout after 6 minutes of inactivity (5.3.1) - Timeout occurs without warning or visual countdown (5.3.1) - When logged out for inactivity, course bookmarks at the beginning of the until of instruction and no time awarded for time spent in the unit (5.3.1) - Course is constructed to prevent blow through (5.3.2) - Course modules are locked to create linear movement through the course (5.3.3) - Course is constructed to prevent parking or other means by which seat time is by-passed (5.3.4) OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 5
Learning Management System (LMS) 5.1 - Provides access to course material via the Internet 24 hours a day/7 days a week - Allows student to logout and login and automatically bookmarks progression - Provides a report of student progression and course completion - Authenticates that the student matches the registration - Supports and/or administers quizzes, exams, other activities, and content formats such as text, video and/or audio - Contains verifiable means to measure a student s time in the course - Provides student a visual indictor representing progression in course - Allows student to re-access completed modules for a specified amount of time determined by course provider - Allows for quizzes and final exam to be retaken for a specified amount of time determined by course provider - Technology requirements, including need for specific software, web browser compatibility, continuous interaction and awareness of course timer, ability to reenter the course, etc. agree with those disclosed Track all Changes- Please call-out all revisions and deletions made to the course material upon resubmission with highlight, redline, etc. This applies to changes made in response to the Evaluation Comments below and to any other changes made since the previous review. Failure to track changes will delay the re-review. The following contains an explanation for an N mark in any one of the above categories and/or additional evaluator comments, questions, or explanations. Evaluation Comments: OSS NMLS Course Approval Evaluation Form 1.8.2013 Page 6