Interior Design Outcomes Assessment Report Sophomore Portfolio Review Spring 2006

Transcription

1 Interior Design Outcomes Assessment Report Sophomore Portfolio Review Spring Contents: I. Introduction II. Goals and Outcomes III. Assessment Procedures and Methods IV. Findings V. Discussion and Next Steps VI. Program Reflection and Goal Setting VII. Appendices

2 Interior Design Report Sophomore Portfolio Assessment Spring 2006 I. Introduction The Interior Design department assessed sophomore portfolios in spring 2006 to help determine which students would be certified into the major. This assessment also supports work towards accreditation from Interior Design s professional accrediting body (FIDER) and WSU s regional accrediting body. II. Goals and Outcomes The Interior Design faculty aspires to be one of the top 3 programs in the United States. Through the department s recognition of the significance of WSU s strategic plan, and accreditation requirements from its professional accrediting body (FIDER) and WSU s regional accrediting body, the faculty has identified a single, comprehensive outcome. Graduates of this program will possess the knowledge and skills necessary to successfully enter and engage the profession of interior design as entry-level designers. The faculty has identified the following 6 critical outcomes essential for students to achieve the aforementioned goal. Performance Criteria. Define problems as it relates to the field of Interior Design 2. Understand the meaning and impact of design as it relates to human interaction, technology, theoretical frameworks, and interdisciplinary efforts 3. Analyze information, evaluate issues, and set priorities 4. Explore and generate creative solutions via a systematic and coordinated design process that integrates functional and aesthetic concerns. Justify and defend design solutions as it relates to socio-economic and global issues 6. Convey intent in a professional manner appropriate to the audience 2

3 III. Assessment Procedures and Methods Overview As outlined in the 2006 Interior Design Assessment Plan, the portfolio review process is the central method for assessing student achievement in Interior Design. In particular, the sophomore portfolio is used as a mechanism for identifying students who meet departmental standards and can therefore be certified into the major. In Spring 2006, 30 sophomore portfolios were evaluated using the instrument developed by Interior Design and attached as Appendix A. The 6 point scale was an absolute scale, with high end scores equal to the expected competence of students at graduation and success in the profession. The threshold for passing sophomore portfolios was 2, with scores given in.2 increments. Thus, a rater who scored a portfolio with a 2 had determined that the work demonstrated competency necessary for the student to enter the Interior Design major as a junior. Portfolios were independently evaluated by two faculty members from Interior Design from the Spokane and Pullman campuses. Students set up their portfolio in a classroom, and reviewers moved through the room assessing student work; individually, reviewers engaged students in a discussion regarding the portfolio as an additional means of gaining data for assessment, particularly the student s ability to communicate design intent with the appropriate design terminology. Afterwards, reviewers met to reach consensus on the overall determination for each portfolio, Pass or Fail. When there was disagreement, sections of that student s portfolio were evaluated by a third rater. (Two portfolios were evaluated by only one rater due to lack of time.) Reliability Faculty agreed that the double review process similar to that used by the Educational Testing Services [ would be appropriate for establishing and monitoring inter-rater reliability. Further, by adopting the rigor embodied in the double review approach faculty establish expert validity in the assessment process. To establish reliability validity, raters participated in a norming session, rating a portfolio based on the scoring criteria and then negotiating a consensus on how the criteria would be applied. IV. Findings. Overall performance. The overall pass rate was 30%. (See Figure ) o 9 out of 30 sophomore portfolios passed, reaching the 2 level and giving those students admission into the interior design major. o 2 out of 30 sophomores did not reach the 2 level, meaning those students were not eligible to continue in interior design. 3

4 Figure Overall performance Figure 6 Inter-rater reliability 4

5 2. Inter-rater reliability. Reviewers were 76% consistent in their overall ratings of Pass or Fail, according to the ETS method of calculating the percentage of overall agreement which establishes reliability. In general, ETS scores of 80% or higher are considered reliable. (See Figure 6) 3. Student performance by dimension. Student performance was low in all dimensions; the range of average scores is limited, and not statistically significant. (See figures 2- below) o For each of the three constructs, Design Fundamentals, Interior Design, and Communication, average scores were below the pass rate of 2. o Highest average scores (.9) were found in 2-D design elements and in verbal communication. o Lowest average scores (.84.86) were found in color, function and aesthetics, presentation methods Figure 2 Student performance in 3 constructs Overall Construct Averages Average of Student Scores Construct Construct 2 Construct 3 Construct Dimensions

6 Figure 3 Student performance in construct, Design Fundamentals Construct (Design Fundmentals) Average Scores Average of Student Scores Design Elements & Principles Human Factors Color Construct Construct Dimensions Figure 4 Student performance in construct 2, Interior Design Construct 2 (Interior Design) Average Scores Average of Student Scores D design elements 3D design elements Space Function & Aesthtics Construct Dimensions Creativity Construct 2 6

7 Figure Student performance in construct 3, Communication Construct 3 (Communication) Average Scores Average of Student Scores Illustration Presentation Methods Verbal Communication Construct Dimensions Written Construct 3 V. Discussion and Next Steps Based on this data, the overall performance of sophomores seeking admission to the Interior Design major remains below the competency level established by the program. Preparing students to succeed in their sophomore portfolio has consequences for design of individual courses and course sequences and scaffolding within the program (syllabus, assignments, and activities), particularly in the 00 and 200 level courses. In that light, and to further strengthen the Interior Design program, CTLT advises the following steps:. As a department, answer key questions about the evaluation process and the implications of the current results on course design, assignment design, formative feedback, and instructional practices in 00 and 200 level courses, including: a. How does the program currently prepare students for their sophomore portfolio and evaluation? How could the program better prepare them, providing formative feedback prior to this high stakes review? b. Should the portfolio assessment results serve a purely summative purpose of admitting or denying students entry into the major, or should it formalize the formative aspect with additional opportunities for review or a formal appeals process? c. Should students who do not pass the portfolio review be allowed to use the feedback formatively, and resubmit their portfolios? 7

8 d. Should faculty rate the sophomore portfolios of their own students? e. The discrepancies between the written comments made on sophomore portfolios often offering encouragement and the low scores on criteria suggest additional calibration of program expectations will be useful. 2. Adopt the revised 7-point rubric ( Guide for Rating Program Outcomes INTERIOR DESIGN ), attached as Appendix B, and formalize weighting of the various criteria used in the portfolio review, including the Process evaluation done prior to the formal portfolio presentations. a. Provide faculty with the opportunity to norm using the rubric prior to assessing student portfolios, to increase reliability. b. Clarify the conditions under which a portfolio will be rated by a third person. c. Invite outside professionals to give input on criteria. 3. Using the rubric, integrate assessment criteria into the design and evaluation of assignments at all levels in this program, including 00 and 200 level courses. For example: a. Use the Interior Design rubric more overtly in instruction and as a guide for grading. Specifically, include the program goals and the rubric in course syllabi and link it to assignments. b. Reserve some class time, preferably at the beginning and middle of the semester, for norming sessions with students to maximize students ability to understand and apply the rubric. (CTLT designers are available to provide support for this activity as well.) c. Create assignments in 00 and 200 level Interior Design courses which require students to give one another feedback using the rubric, and offer students opportunities to revise work after receiving feedback from peers and/or faculty; this should begin with the earliest courses taken to help students develop familiarity and move toward competency in the key areas for the sophomore portfolio. d. Require that students attach a rubric-referenced self-assessment to key assignments in 00 and 200 level courses. e. Monitor the use of rubric in Interior Design classes to guide ongoing refinement of rubric and assignments. f. Review and refine first and second year assignments using the rubric. Solicit faculty reflections on strategies for revising assignments and refining their sequence and scaffolding. (These reflections are invaluable evidence of quality enhancement essential for accreditation as well as program improvement.) g. Make other modifications to the design and sequence of key assignments based on department answers and policy decisions in connection with questions posed in Recommendation above. h. Continue to strengthen the alignment between assignments and the outcomes/criteria. (CTLT designers are available to provide support in syllabus and assignment design.) 4. Using the new rubric, identify patterns of weakness in sophomore portfolios and concrete methods to address them; for example, if writing is a constraint, a plan for additional systematic work can be developed in conjunction with the Writing Program or CTLT. 8

9 . Move toward use of Sharepoint e-portfolios to support the portfolio process throughout a student s four years in interior design, especially to provide essential and timely formative feedback prior to sophomore portfolio review and to document the evaluation of portfolios and the outcomes assessment process. 6. Work with CTLT to guide mapping of existing program criteria to the Six Learning Goals of the Baccalaureate. VII. Appendices A. Evaluation tool used through Spring 2006 B. New rubric: Guide for Rating Program Outcomes, Interior Design 9

10 Appendix A: Evaluation tool used through Spring 2006 Interview Review Sheet (Scale from 0 6, 0 no evidence of indicator, 6 Evidence of mastery of the indicator) Design Fundamentals Demonstrated understanding of design elements and principles Human factors (i.e. ergonomics, anthropometrics, ADA) - Projects may reveal accurate furniture relationships and scales and space adjacencies meet client needs Color selection and application is based on a strategy (via principles, theories, and systems) Interior Design Application of 2D design elements and principles are easily recognized in simple linework compositions or floor plans Application of 3D design elements and principles are recognized in the of the spatial envelope, though may be compromised by function development Space planning (adjacencies, circulation; and, articulation and shaping of space) allows for ease of movement Selection and application of materials and products meets user s needs for both function and aesthetics Creativity is demonstrated by applying design elements in unique approach to the problem Communication Manual drafting and lettering demonstrate care in execution, with a hierarchy of lineweights and legible lettering Illustrative sketching / rendering / perspectives / model building provide insight into the design solution Presentation Board methods visually communicate in a manner that support and convey design intent Verbal Communication (organization and clarity)- Describes strategy for exploration in terms of design elements Written Communication- Written concept statements, keying of materials and furnishings, and other basic cross referencing of drawings (section & elevation symbols) are evident 0

11 Total Score Average Comments: Professionalism Timeliness and organization of work Positive attitude toward the discipline Ability to accept and process critique information Comments:

12 Appendix B. New rubric: Guide for Rating Program Outcomes, Interior Design ID Guide for Rating Program Outcomes INTERIOR DESIGN Define problems as it relates to the field of Interior Design. Emerging Identifies the problem as unidimensional. Oversimplifies or overlooks the complexity of the problem Developing Clearly identifies and summarizes the issue or problem at hand. Recognizes complexity, but has difficulty relating them. 7 6 Mastering Identifies the complexity and embedded or implicit aspects of the problem and assesses their relationships. Identifies a different and inappropriate issue or problem. Understand the meaning and impact of design as it relates to human interaction, technology, theoretical frameworks, and interdisciplinary efforts. Emerging Identifies the meaning and impact of human interaction, technology, and interdisciplinary efforts. Describes theoretical frameworks in terms of basic design theory (elements and principles of design) Developing Recognize and interprets theories of human interaction. Incorporates technological issues into design solution. Applies basic theoretical frameworks to design solutions. Discusses the interdisciplinary relationships of the project. 7 6 Mastering Expands understanding of theoretical frameworks. Integrates and combines multidisciplinary theories into unique, effective design solutions. Assesses technological options and applies based on an understanding of performance criteria. 2

13 Applies interior design skills Effectively analyze information, evaluate issues, and set priorities Emerging Describes information provided or personal observation. Confuses observation with interpretation. Examines issues, but may have difficulty classifying them. Lists priorities as stated by the problem or may focus on less relevant concerns Developing Discovers, examines and interprets information (mainly qualitative) and relates its relevance to the design problem. Compares issues and classifies as a set of priorities. Applies results to design problem. Discovers personal value judgments. 6 7 Mastering Evaluates and assesses qualitative and quantitative information. Challenges existing knowledge. Clearly articulates between fact and opinion; and acknowledges value judgments. Combines and classifies the issues, perhaps in new ways. Assesses prioritization as it relates to the multi-faceted goals of the design problem. Explore and generate creative solutions via a systematic and coordinated design process that integrates functional and aesthetic concerns. Emerging Interprets programmatic requirements as the guiding force to define strategy for exploration. Explores a minimal number of solutions. Abandons options quickly preventing a coherent progression of design development. (Student looks for the immediate right answer.) Solves the problem by meeting programmatic requirements and applying basic design theory Developing Describes strategy for exploration as being a two-pronged approach considering aesthetics and human interaction. Analyzes critically throughout the process (programming, schematic design and design development). Explores options in greater detail making informed modifications to improve solution. Applies process of exploration at different scales. Demonstrates a move toward creativity in the interpretation of final solutions. 6 7 Mastering Demonstrates that the design process is a complex exploration of a design problem with numerous issues interdisciplinary in nature. Identifies and analyzes embedded issues not previously specified as needing attention. Assesses theoretical frameworks for appropriate and effective application. Explores problems at the microand macro-levels, demonstrating an understanding of the intricacies and subtleties of interior space as well as its 3

14 Demonstrates difficulty in balancing programmatic requirements and aesthetics. For example, design solutions may sacrifice aesthetics due to an over-emphasis on function. relation to the larger urban/global context. Reflection is evident throughout the process, demonstrating critical, analytical and strategic thinking. Designs a pragmatic, insightful, sensitive and creative solution. Justify and defend design solutions as it relates to socio-economic and global issues. Emerging Describes programmatic requirements as main source of justification. Acknowledgement of socioeconomic and global issues is not expected; they are only defined conceptually. Does not provide indication of reflection on own assertions and/or responses to the design solution Developing Applies an understanding of the users and their circumstances in a discreet experiential space as the basis of justifying design solutions. Identifies and describes the urban context, but is unable to discuss it. Interprets universal and green design as examples of global consideration. Discusses economics relative to final cost, not a budget with limitations. 7 6 Mastering Identifies and clearly states conclusions, implications, and consequences. Compares design solution to a larger context beyond the immediacy of site and location. Expands justification to larger social, economic and global issues. Integrates and explains the interrelationship between design, the built environment, and the global society. Assess issues of economics as not only a budget, but a larger social phenomenon and its impact on the design process/solution. Objectively reflects upon own assertions and/or responses to the design solution. 4

15 Convey intent in a professional manner appropriate to the audience. Emerging Identifies content based on prescribed requirements Developing Distinguishes a hierarchy of visual relevance. 6 7 Mastering Assesses and prioritizes the information to be delivered. Demonstrates knowledge of various forms of visual communication. Visual and verbal communication is basic and often lacks detail and clarity, distracting from design intent. Explores various forms of communication. May, at times, select presentation that is most comfortable, not most effective. Apply visual and verbal communication methods in a manner that support and convey intent. Assesses various methods of communication and the intended audience. Integrates media for maximum impact and clarity.