STIMS the Sustainability, Technology and Innovation Concurrent Master's Degree: Assessments and Outcomes

Transcription

1 STIMS the Sustainability, Technology and Innovation Concurrent Master's Degree: Assessments and Outcomes James L. Barnes, Ed.D., Susan K. Barnes, Ph.D., Michael J. Dyrenfurth, Ph.D., Gareth O Donnell, Ph.D., Kathryne A. Newton, Ph.D., Robert J. Herrick, Frank Palisi, M.S. Barnes Technologies International, LLC/ Barnes Technologies International, LLC/Purdue University/Dublin Institute of Technology/ Purdue University/ Purdue University/ Purdue University Abstract This paper describes the assessment model used to document the impact of a European-United States Atlantis grant project awarded by the Fund for the Improvement of Postsecondary Education (FIPSE, USA) and the Education, Audiovisual and Culture Executive Agency (EACEA, EU). The results reported in the paper reflect progress through Year III of a four year project. The partner institutions included Dublin Institute of Technology (DIT), Universitat Politècnica de Catalunya (UPC), and Purdue University (PU). Two goals of the project were to advance sustainable full-semester student exchange between the European project participants and their US counterparts and to accelerate the development and support of collaborative cross cultural, multi-disciplinary learning environments focused on innovative Engineering, Design and Technology. The result was a highly successful concurrent (dual) master s degree where, upon successful program completion, each student earns two masters degrees, one from Purdue University, and the other from the European university of their choice. To measure the extent to which the project goals were met, a process-outcome evaluation design was used. Both qualitative and quantitative methods and measures were used to evaluate the degree to which the team accomplished the proposed goals. Qualitative measures were used to report outcomes to the project team as to contextual/environmental, input/communication, process and outcome variables that were assessed, and used to strengthen the program on an ongoing basis. In addition, quantitative measures were used to conduct performance outcomes evaluations of the program s goals and objectives that are required by the US Department of Education s FIPSE Program. The findings provided information to the principal investigators on how well the program s goals and objectives had been met and what adjustments must be made to fill gaps in the program. The evaluator also evaluated the project in light of two of the Government Performance and Results Act of 1993 (GPRA) measures and found strong support

2 that the program is making more than adequate progress on the two GPRA performance measures. Introduction Globalization, technological innovation and sustainability are critical issues that continue to impact the critical infrastructure of most nations in the world. Significant to this concern is the lack of preparation of leaders with significant capabilities in these areas. In the Americas and in Europe, advanced programs in leadership development are often configured as master s degree programs both conventional and professional [1]. There exists a wide range of masters programs that include professional masters, MBAs, online degrees, and conventional campusbased experiences. But, few if any focus specifically on the intersection of Technology Globalization Innovation and Sustainability! The Sustainability, Technology and Innovation Concurrent (Dual) Master s Degree (STIMS) Program incorporates the best of what is known about technological innovation, research and development, and sustainability [2], [3], [4], [5], [6], [7], [ [8]). The program was also designed around the best practices of joint, dual, concurrent degree international programs [9], [10], [11], [12], [13], [14], [15], [16]. To address this critical need Purdue University (USA), the Dublin Institute of Technology (Ireland), and the Universitat Politècnica de Catalunya (Spain) proposed and won a joint Education, Audiovisual and Culture Executive Agency (EACEA, EU) and the US Fund for the Improvement of Postsecondary Education (FIPSE, USA) Atlantis project. This grant provided funding for the development and implementation of a transatlantic four-semester dual master s degree initiative across the three-institution consortium. To highlight the success of this FIPSE Atlantis Program, key background information, outcomes, and results, and the assessment model are presented in this paper through Year III of this four year project. Project Objectives & Outcomes This project s chief objective and outcome is the exchange of master s level graduate students among its three partners. These students will benefit from an innovative concurrent master s degree program that develops leading edge understandings and skills with technology, innovation and sustainability and that also promotes transatlantic mobility. By design the outcome will include global perspectives, multiple culture awareness, and sensitivities, as well as dual professional level language capabilities. Important additional outcomes include: Scholarship pertaining how to do effective exchanges and promote international collaboration Collaborative research and teaching based on increased mutual understanding and faculty contact due to mobility

3 Better administrator, faculty and student understandings of cultures and global perspectives Enhanced procedures in place for collaboration and exchange due to increased transparency, and reciprocal recognition of credits Increased transatlantic faculty and administrator mobility MS Degree Outcomes Upon completion of the Masters of Science in Technology Degree focused on Technology Innovation and Sustainability, students will demonstrate: 1. Enhanced capability with research and development. They will: become familiar with research and experimental design sufficient to apply it to real world problems be able to apply design and development procedures to real world problems demonstrate the necessary professional, research and development skills common to engineering and industrial technology disciplines, as evidenced by successful completion of either a thesis or a directed project that are required for successful life-long learning and professional contribution. 2. Global perspectives on technology, technology management and sustainability. They will be able to: employ project management, technology assessment, and sustainability analyses applicable both to their native cultures as well as other cultural settings establish the appropriateness of technology for specific cultural settings use knowledge of how industrial technology impacts society and organizations, from both a technical as well as a leadership and management perspective demonstrate critical thinking in these arenas. 3. Innovation and related process skills. They will be able to: employ key creative and innovation generation procedures sophistically retrieve information from databases and global sources advance product realization and commercialization skills 4. Awareness of and capability with entrepreneurship procedures. They will be: capable with entrepreneurial procedures and skills able to secure business related information from sources around the world 5. Enhanced cross cultural communication and professional effectiveness. They will able to: become proficient in professional communication (reading, writing and speaking/presenting) in at least two languages demonstrate effectiveness in operating in a culture/country other than their native one. demonstrate ethical leadership and a commitment to their personal professional development and life-long learning. 6. A graduate level of technological expertise in one or more of the technology fields.

4 Approach to Evaluating Program Impact on Student Learning Barnes Technologies International LLC (BTILLC) evaluated the STIMS Program based on how the Project Director and STIMS Program leadership team developed their implementation plan. BTILLC evaluated how the project s management team planned the system of input, communications channels, and processes used to establish a baseline. BTILLC combined Stufflebeam s Context, Input, Process and Product Evaluation Model (CIPP) [17], [18], Kirkpatrick s Four Levels of Evaluation Model (reactions, learning, transfer, and results) [19], and Wilder s Collaboration Inventory Factors (environment, membership characteristics, process and structure, communication, purpose, and resources) (Mattessich, Murray-Close, & Monsey [20]; Meyerson [21] to conduct the third party program evaluation of the impact of FIPSE Atlantis Concurrent MS Degree Project. The CIPP Model provides comprehensive approach to evaluating programs, projects, personnel, products, institutions and systems. This evaluation approach is built on the assumption that anything that can be evaluated can be successfully evaluated at various stages of its development, context, input, process, and product. Simply put, the CIPP Model focuses on What needs to be done?, How should it be done?, Is it being done?, and Did it succeed? The Kirkpatrick model is based on four levels of evaluation. Each level represents a different measure of the effectiveness of the program. BTILLC has successfully used this approach on numerous third party program evaluations of company programs, educational projects, and agency impact. The data collection methods to be used by BTILLC during this third party evaluation have been used in similar evaluations. The Wilder model used in this evaluation is based on five measures. Level I Reactions evaluation measured how participants in the STIMS Program reacted to it. It attempted to answer questions regarding the participants perceptions. Did the team members feel they accomplished the STIMS Program goals and objectives? Is the STIMS Program relevant to their proposed goals and objectives? How did participants react to their concurrent MS Degree experiences? Level II Learning evaluation assessed the extent to which teachers have advanced their skills, knowledge, or attitude. Methods of evaluation for Level II included both formal and informal evaluations, as well as team assessment, and self-assessment. Level III Transfer evaluation assessed the change in behavior that has occurred during the development and implementation processes due to the STIMS Program. Level IV Results evaluation measured the final results that occurred because of processes used by the STIMS Program leadership team [19]. Each of the evaluation measures were analyzed based on short-term, intermediate, and longerterm outcomes. BTILLC has used this model to identify changes that are logically expected as

5 result of the project s activities, are within the sphere of influence of the project, and generally accepted as valid by the various stakeholders of the STIMS Program. This integrated program evaluation model assessed to what degree the STIMS Program provides a value added experience for the participants due to a comprehensive, coherent and sustainable plan. At the heart of every phase of this external evaluation was assessing the degree to which the STIMS Program team supported their participants through proven strategies and programs. Key variables, based on the goals and objectives of the project for context, input, process and product, were evaluated to assess to what degree the STIMS Program was meeting the goals and objectives. BTILLC supported Kirkpatrick s four levels of evaluation with Wilder s six factors of evaluation that were broken down as short-term, intermediate, and longer-term outcomes. This integration provided a more robust evaluation technique to assess to what degree the STIMS Program is meeting its goals and objectives. Short-term outcomes refers to change in knowledge or awareness that typically precedes change in behavior or practice. Intermediate outcomes refers to behavioral change in knowledge and awareness. Longer-term outcomes refers to more global change, either affecting more people than those directly served or they involve more profound and lasting changes that the program directly influences [21]. The findings and results of the qualitative and quantitative assessment have provided recommendations to the principal investigators for how well the program s goals and objectives have been met and what adjustments must be made to fill in gaps. Stock and Flow Model The stock and flow model is used to convey the relationships and dynamic interactions of components of a system. It is a modeling technique for designing, interpreting, and discussing complex problems and the behavior change of a system. BTILLC used stock and flow modeling to study the relationships and dynamic interactions between Stufflebeam s CIPP and Kirkpatrick s Four Levels models of program evaluation with the program outcomes and instruments used to assess them, which is presented in Figure 1. Arrows indicate the direction of the relationships. The stock and flow model highlights the efficiency of using one instrument to measure more than one outcome. Faculty interviews, for example, provided data for examining four of the program outcomes. Note that the Stufflebeam/Kirkpatrick evaluation models are housed in the same block and that the relationship with outcomes is bidirectional, indicating that all levels of evaluation are used to

6 inform the outcomes over the period of the activity. Also, all levels are used to interpret the meaning of the results. The Results and Findings Administration and partnership development The FIPSE Atlantis Concurrent Degree Program evolved out of the relationship and trust built by Dublin Institute of Technology (DIT) and Purdue University (PU) through their work on the previous FIPSE-funded Design, Entrepreneurship, Technology, Engineering, Collaboration, Transatlantic (DETECT) Exchange Mobility project. Universitat Politècnica de Catalunya (UPC) was asked to join DIT and PU in this concurrent degree effort because of their strong background in sustainability. DIT already had an established relationship with UPC that made establishing them as a partner an easy process. From the onset, the three institutions involved their executive administration and that led to a strong foundation and commitment being established for STIMS program. Instruments Outcomes Stufflebeam/Kirkpatrick Pre- Departure Evaluation of Exchange Faculty Interviews Leadership Interviews Learning Gain Innovation Gain Cultural Sensitivity Knowledge Transfer Context Input Process Reactions Evaluation Learning Evaluation Transfer Evaluation Results/Accomplishments European Qualifications F k Sustainability Product Results Evaluation Figure 1. Stock and flow model indicating the relationships between the instruments, outcomes, and evaluation models.

7 Collaboration Evaluators used the Wilder Collaborative Factors Inventory Areas to establish baseline data for the level of collaboration with the project leaders. Areas of concern in the first two years of the project were the scores for collaboration factors measuring pace of the project and the sufficiency of funds, staff, materials, and time. This year very few project leaders participated in the Wilder Collaborative Factors Inventory. However, BTILLC did investigate those collaboration components through other methods, including conducting interviews, making direct observations, and analyzing other anecdotal data. The results of this investigation provided evidence that these concerns have been adequately addressed and that the administration of the STIMS Program continues to work well. The leadership teams from PU, UPC, and DIT built a strong working relationship through continuous and open communication, numerous face-to-face and electronic meetings, faculty exchanges, and a high level of senior administrative involvement. The strengths and expertise of each university complement each other, resulting in a strong leadership team that continues to sustain the project. Communication One of the key success elements of projects such as the STIMS Program is a strong communication network among the partners that allows for leaders of each institution to have an opportunity for input into the planning process so that they develop ownership in the project. Thus, a strong trust relationship is built to assist the leadership team to resolve issues when they occur. The team established a variety of mechanisms to ensure an open dialogue amongst the partners to discuss and develop the key components of the STIMS Program. Each institution established a prime point of contact and they involved university personnel at all necessary levels of each university in the communication network. A critical element was the establishment of significant communications at the beginning of the project and during the first few years to help establish these three graduate level programs in their respective institutions and work together to solve the various complex issues that arose during the project s implementation. Key participants met regularly, either in face-to-face-meetings, , telephone, or via video conferences using such technologies as oovoo video call. PU established a SharePoint TM account to store and share STIMS Program documents. Each institution created a website, linking universities and to other key links necessary to administer the project. BTILLC analyzed communications factors using the interviews with the leadership team to gain an understanding of how well the STIMS Program communication process was working. The leadership from each institution indicated that academic officers, faculty, and the International Programs Office have continued to be key university personnel involved in the communication process. In addition to the academic officers, faculty, and the International Programs Office being the key university personnel involved in the communication process, the leaders, as in 2011, also indicated more involvement from the president s/rector s office and the registrar s and

8 record s office. The project s leadership felt that the students also were a major part of the communication process. In the Year III interviews, team members concurred that the openness of the communication process amongst the STIMS Program partners remained a strength of the program. Overall, members who participated in the interviews felt they were as informed as often as they should be about the project and that information was exchanged in both formal and informal ways. Planning vision and process Implementing an effective and efficient planning process is another key element to the success of any project of the nature of the STIMS Program. The planning process must follow a logical pattern towards implementation and must be designed so that all partners, at all necessary levels, have ample opportunity for input. The process must assure that the project is adequately staffed and funded and must also include measures for sustainability beyond the life of the grant period. Finally, the planning process must include the development and signing of all necessary governing documents by each institution. When STIMS Program leaders were asked how well the planning process was organized and if it followed a logical pattern, they agreed the process was working well. They explained that the planning meetings were scheduled to allow the partnership goals and objectives to be accomplished on time and that responsible parties were able to keep up with the work necessary to coordinate all the people, institutions, and activities related to the project. The leaders responded that the planning process had equal participation from all its partners and that the project was supported by an adequate number of people with the requisite skills and expertise. Most of the program leaders responded that the program had adequate funding. The STIMS Program leadership continues to give credit for the well-articulated design and implementation plan to the strength of their partnership. The evaluator feels these opinions can be attributed to the fact that PU and DIT already had a strong working relationship and that they looked carefully to find an appropriate partner with comparable academic rigor and the ability to contribute the missing component of a campus that operates totally on sustainability principles. The evaluator, through his interaction with the partners, has seen significant evidence that the partnership is stronger in Year III than it was in previous years. Because of the strength of the partnership and the trust established by the key people from each institution, the leaders overwhelmingly felt that the design and implementation plan has clear long-term return on investment and tangible outcomes. These responses are consistent with the leader s responses from Years I, II, and III. Curriculum alignment and approval Over the past year the three institutions have met repeatedly either face-to-face or through videoconferencing to develop, gain approval, and implement the STIMS Program curriculum. In Year

9 I, PU and UPC already had curriculum in place, but DIT needed to develop new courses and have them approved, which was accomplished. The result of DIT s work was the development of six programs of study. The six different programs of study were developed to address the variations as to when students started the STIMS Program, fall or spring, and on which of the three campuses they began the program. Key to the success of implementing new programs, especially programs that have never been implemented at an institution, can be the considerable task of having the new program approved. None of the three institutions had previously implemented a concurrent international master s degree program, therefore, each institution was embarking on new territory. All three institutions successfully had their programs approved. This success was due to three key factors: 1) ongoing communications amongst the institutional Atlantis leadership teams and with their respective administrations; 2) having all the appropriate university administrators and faculty involved from the program s onset; and 3) having the Memorandum of Understanding (MOU) and other governing documents in place. Each year, new students began their program of study at their home institution. Based on the reports of the project leaders and students the curriculum implementation in Year III went very well. Students were aware of when and where they were to exchange and the courses they were to take when and where. They were also aware of the language fluency requirements and of the preparation needed to be successful in a country where their native language is not the predominate language of the community. Marketing recruitment The leadership team was asked what they thought about using a variety of the recruitment techniques to make students aware of and interested in the STIMS Program. The leadership identified five key recruitment techniques as being the most important for the STIMS Program: Graduate Program Office announcements, college announcements, student interviews, in-class announcements, and departmental websites. Admissions Agreements have been made on the admissions standards and processes to follow so that students in the program are admitted to all three partner universities. In addition, the three partner institutions have finalized the admission criteria to be used between each university for the admission of students into all three programs. When thinking about the program outcomes and the selection of students to participate in the STIMS Program, the leaders identified four key areas for the selection of students: undergraduate GPA, grades in specific courses, field-specific experience, and student attitude. In the fall semester 2011, the beginning of Year II, ten students began the STIMS Program five from PU, five from UPC and none from DIT. In the spring semester 2012, DIT enrolled an

10 additional four students. By the end of Year III (Table 1) 45 students were participating in the Atlantis Concurrent MS degree program. The leadership team needed to keep the student enrollment manageable and work hard to recruit students. A breakdown of student mobility by institution is detailed in the table below. One PU student dropped out of the Atlantis program during her final semester. BTILLC interviewed the student and concluded it was based on two factors: 1) giving up family for one year and 2) equipment necessary for her to complete research was not the same at each institution. Table 1. Atlantic Concurrent Degree Program Student Enrollment Years II and III Institution Completion Fall Semester Spring Semester Fall Semester Spring Semester Fall Semester 2012 Spring Semester 2012 PU/DIT PU/UPC PU/DIT PU/UPC PU DIT UPC Language and cultural development The language component, and eventually language gain of the students, is a crucial matter to address when establishing an international concurrent degree program, especially when one partner language is different from the other institutions. PU requires a Test of English as Foreign Language (TOEFL) score for all international students. DIT and UPC have similar requirements for their international students. As of Year III, the three institutions are currently exploring language development and support options to help students during their studies. The students were asked three questions about their foreign language capability upon entering the STIMS Program and two questions towards the completion of their first year in the program. Upon entering the program, BTILLC had students report their language fluency and their ability to read and explain scientific and technical information in a foreign language. As the students completed their first year of study, BTILLC asked the students about their level of confidence while living and studying in a foreign country to see if there was any difference in their confidence level since entering the program. Most students, when entering the program, indicated they had studied a foreign language, but did not speak it well. They responded neutral to being able to read and explain scientific and technical information in a foreign language. When surveyed at the end of their first year of study, four students agreed, four students were neutral, and one student disagreed that they felt confident with their foreign language fluency while living in their exchange country. They also indicated similarly about their confidence level with their foreign language fluency for studying in the exchange country. In Year III, language fluency continued to be a concern. While graduating students indicated they managed to survive language barriers, they, as well as other students participating in the program,

11 indicated some problems studying, reading and explaining scientific and technical literature in their exchange language. Faculty development Most of the STIMS Program leadership work has focused on building a strong foundation through a strong planning process that has included some effort towards faculty development. The roots of a strong faculty development component need to be established during the initial start-up year, and they have been established. Faculty development increased as the program matured and faculties were provided opportunities to collaborate on course development, research, conference papers, and other scholarly activity. The leadership of all institutions visited each other s campuses to explore potential faculty development opportunities. Faculty development and mobility continues to be an ongoing discussion during the leadership planning meetings. When the faculty leaders were asked about faculty mobility activities, the majority of the leaders responded that the partnership had made very good progress toward establishing faculty mobility activities. Last year, the leadership team developed a Sustainability Technology and Innovation (STI) Masters Faculty Mobility Proposal, a STI Masters Summary for Faculty, and an Atlantis Faculty Mobility Application. This year, the team implemented this mobility program. Evidence that the program is working well includes extended faculty exchanges and collaborative research papers and presentations. Sustainability Sustaining an initiative beyond the grant funding period is an important consideration that needs to be built into the initial planning process. The STIMS Program leadership team has successfully established a strong foundation that should lead to the program being sustainable. During the first two years of the project, the STIMS Program leadership team identified four key areas where there was opportunity for continuing growth and development. These areas include: Learning outcomes assessment Faculty mobility Directed project/thesis Language preparation In the evaluator s review of project documents, evidence was found to support the conclusion that the leaders have made adequate progress in addressing these four areas. In addition to the documents developed in the first years, the evaluator worked with some faculty to develop strategies to measure student learning through coursework. Last year, project leaders began to investigate the potential for collaboration with institutions and organizations beyond higher education. The project leaders continue to expand their list of United States companies with facilities or financial interests in Ireland and Spain. Using this list, project leaders will generate a plan to network with those companies to gain additional support for the Atlantis project.

12 Institutional change and support Creating institutional changes goes hand-in-hand with developing sustainable programs. This initiative matches the strategic plans of the three institutions. In addition, this program fits well within the direction of accreditation. This year, BTILLC talked to project leaders about their reflections regarding institutional support and institutional change. The leaders of the STIMS Program agreed that they were receiving institutional support. When asked to think about the program outcomes for learning gains that lead to institutional change and to identify factors that would mostly likely lead to institutional change, the teams expressed the importance of expanding alliances with other institutions and expanding institutional visibility. When discussing the program outcomes relative to learning gains for faculty and administrators, the leadership team members agreed that this component could be accomplished best by sharing successful ways of implementing international student exchange programs, sharing innovative pedagogical strategies among partner institutions, developing new courses, and expanding collaborative conference presentations among partner institutions. Academic and intellectual achievement All three institutions have agreed on the academic standards acceptable for the STIMS Program. They have also agreed to a three-person oversight committee with representatives from each institution. This three-person committee also has served as the directed project or thesis committee. BTILLC asked the leadership to think about the program outcomes relative to learning gains for graduate students. The leadership team confirmed that the outcomes established early in the program were still appropriate. These items are: Developing an ability to speak another language well enough to function in another country Developing an ability to read and present in another language Expanding cross-cultural and global awareness and experiences Interacting with faculty Independence/self-reliance Continuous improvement in coursework Learning under a different educational system Developing new logic and problem solving strategies Expanding ability to design/apply sophisticated methodological techniques Expanding ability to think critically Developing stronger oral communication Developing stronger written communncation Increasing accountability

13 Increasing educational aspirations The leadership team was also asked to think about the program outcomes relative to the assessment of learning gains and how that assessment should be accomplished. The STIMS Program leadership felt most strongly that course-embedded assessments, such as facultydeveloped test/exams, host faculty assessment, student presentations, and faculty assessment, in addition to language proficiency tests, were the most important factors to the assessment of learning gains. BTILLC asked the students about key factors that impacted student learning and academic achievement. BTILLC used five questions for entering students to gain an understanding the confidence of being prepared and succeeding in the STIMS Program. BTILLC followed up with the students toward the end of their first year with nine questions about academic achievement. Upon entering the STIMS Program, the students were asked about how confident they were about their STIMS Program. All students either felt very confident or confident about the educational experience they were beginning. The students were also asked about their knowledge of global issues, processes, trends, and systems. Seven out of ten students responding to the survey indicated they were confident with their knowledge of global issues, processes, trends, and systems. Six out of ten students responded that they were confident with their knowledge of other cultures. However, when asked about their knowledge about world geography and conditions, the students were almost evenly divided between being very confident, confident, or neutral. Finally, students entering the program were asked about interacting with professors who will teach them during their FIPSE STIMS Program. Eight out of ten students indicated they felt confident interacting with professors who would teach them during their STIMS Program. At the end of their first year in the STIMS Program, the students were asked if the expectations they had when entering the program were being met. Five out of nine students responding to the survey indicated they agreed that their expectations were being met, while three students strongly agreed and one student was neutral as to whether their expectations were being met by the Atlantis Concurrent MS Degree program. The students were also asked if the courses they had taken were meeting their educational needs. All students either agreed or strongly agreed that the courses that they had taken met educational needs. The students were questioned about their interactions with faculty. They also were asked about faculty being available for help with coursework. The students responded identically to these two related questions, four strongly agreeing, four agreeing and one neutral. BTILLC was also curious if the students understood the requirements for earning a concurrent degree in the Atlantis Program. The responses puzzled BTILLC, because this material was given to all students, covered in orientation, and published on the STIMS Program website. Three students strongly agreed, four agreed, one was neutral and one disagreed that they understood the

14 requirements for earning a concurrent degree in the Atlantis Program. BTILLC was interested in how curious the students were about their exchange city and country. Four students indicated they strongly agreed that they had been studying about the exchange city and country to better prepare themselves for the exchange experience. Three students agreed and two students were neutral about studying about the exchange city and country to better prepare themselves for the exchange experience (Tables 2, 3, and 4). Table 2 Student learning outcomes N=16 Question SD D N A SA The expectations that I set for myself before entering the Atlantis Concurrent Degree Program are being met. The courses that I have taken are meeting my educational needs The professors are effective teachers My research activities have been rewarding My interaction with faculty has met my expectation Faculty have been available for help with coursework I understand the requirements for earning a concurrent degree in the Atlantis Program I have been studying about my exchange city and country to better prepare me for my exchange experience. I feel confident with my foreign language fluency for living in my exchange country I feel confident with my foreign language fluency for studying in my exchange country Table 3 Continuing Student Survey results, Fall 2011 N=16, Spring 2012 N=23 Items Fall Semester 2011 Spring Semester 2012 SD D N A SA SD D N A SA I have changed my expectations that I set for myself before entering the Atlantis Concurrent Degree Program. I feel that the concurrent masters degree is my best option for a master degree. The orientation information provided me by my home institution about what I should expect during my exchange was helpful. My research activities have been rewarding I have maintained contact with my advisor during the summer Faculty have been available for help with coursework during the summer I have been studying about my exchange city and country to better prepare me for my exchange experience. I feel confident with my foreign language fluency for living in my exchange country. I feel confident with my foreign language fluency for studying in my exchange country. I feel confident of being able to read technical and scientific literature in the language of my exchange country. I feel confident of being able to explain technical and scientific literature in the language of my exchange country. I feel confident about my knowledge of global issues, processes, trends, and systems. I feel confident about my knowledge of other cultures I am acceptant of other cultures I feel confident about interacting with local citizens in another country I feel confident about my knowledge of world geography and conditions I feel confident about using local transportation systems in another country. I feel confident about taking excursions in another country

15 I feel safe living in another country I know the key historical events of my exchange countries I know the key cultural events celebrated annually in my exchange countries. I know the types of government in my exchange countries and how they function. I know the major products manufactured in my exchange countries I know the major multi-national corporations that are based both in my home and exchange countries. I know the key technology or engineering careers in my exchange countries. I know the key occupations in my exchange countries I know the key tourist attractions in my exchange countries Table 4 Post Assessment of Completers N=5 Question SD D N A SA The expectations that I set for myself before entering the Atlantis Concurrent Degree Program were met. The courses that I took in my masters program met my educational needs The professors were effective teachers My research was rewarding My interaction with faculty has met my expectation Faculty were available for help with coursework I understood the requirements for earning a concurrent degree in the Atlantis Program I began my thesis/capstone research at the appropriate point in my studies My thesis/capstone advisor was very helpful to me throughout my thesis/capstone process My thesis/capstone research was well coordinated by my thesis/capstone committee The differences in laboratory equipment at the different universities hindered my ability to complete my thesis/capstone research in a timely fashion. I feel more confident studying in a foreign country I feel more confident with my foreign language fluency for studying in my exchange country than I did when I began studying in my exchange country. I feel more confident of being able to read technical and scientific literature in the language of my exchange country than when I began studying in my exchange country. I feel more confident of being able to explain technical and scientific literature in the language of my exchange country than when I began studying in my exchange country. I feel more confident about my knowledge of global issues, processes, trends, and systems than when I began the Atlantis Concurrent Degree Program. I feel more confident about my knowledge of other cultures I am more acceptant of other cultures I feel more confident about interacting with local citizens in another country I feel more confident about my knowledge of world geography and conditions I feel more confident about using local transportation systems in another country I feel more confident about taking excursions in another country I feel more comfortable living in another country I know more about the key cultural and historical events of my exchange countries Earning concurrent degrees, one from my home country and one from an exchange country, has prepared me better to be successful in the workplace. I made the correct decision earning my current masters degrees through the Atlantis program The transition between the three Atlantis universities worked well Equity in cost amongst institutions Although the current exchange rates between the euro and the U.S. dollar has improved, strong consideration needs to be made by the funding agency to adjust the amount of money provided to U.S. students to compensate for the low value of the U.S. dollar in Europe. The allotted $12,000

16 in U.S. dollars is not equivalent to 12,000. The cost difference could greatly impact the level of future students interest and their ability to gain a robust cultural and global experience. Dissemination and reporting The leadership team continues to actively disseminate the results of the STIMS Program. They have established a website for the project. They have teamed to present papers at major professional association conferences, such as the American Society for Engineering Education, SEFI, and are currently working on several other international conference presentations. They are exploring disseminating information about the project in scholarly journals. All institutions have given internal presentations to various university constituencies about the program, including several presentations to potential students. Accomplishments The STIMS Program leadership team has accomplished much through their third year of this initiative. BTILLC, in their discussions with the leadership team and their analysis of pertinent documents, find the following accomplishments were made by the STIMS Program team: graduated their first class of students continued to build trust in the already strong partnership continued to develop and align relevant courses executed and followed a signed MOU and other necessary by all three institutions recruited an appropriate and manageable number of students to begin the program strengthened the communication process and network enhanced the recruiting and marketing materials, including a website implemented further the faculty development and mobility component of the project continued to strengthen the foundation for institutional change through this project continued working on long-term sustainability strategies continued a strong dissemination effort of information about the program to various internal and external constituencies moved in an appropriate pathway toward sustainability Guided by the Outcomes Capability Matrix, and evaluation determined how well the project improved teaching and student achievement and is also addressed the two Government Performance and Results Act of 1993 (GPRA) performance measures established for FIPSE Atlantis: 1. the percentage of students pursuing a joint or dual degree who persist from one academic year to the next (persistence), and 2. the percentage of students who graduate within the project's stated time for completing a joint or dual degree (graduation).

17 A third party evaluation of STIMS provides strong support that the program is making more than adequate progress towards the two GPRA performance measures for the project. The following evidence supports that conclusion: To date, STIMS performance measures for GPRA 1 and 2 are 100 and 87.5 percent respectively. Wilder s Collaborative Factors Inventory shows that the STIMS team works well and is working to sustain the program. The students have adapted well to studying under different educational systems. Of the eight students who began the program, all but one student completed their concurrent dual degree on or before schedule. Of the other 26 students enrolled in STIMS all students are matriculating through the program on schedule. No student, to date has failed a course or had particularly difficulty with a non-native language. The only shortcoming found is the need for more Purdue students to obtain their second masters at UPC. Conclusion The STIMS Program continues to be on a great track for being sustainable after the completion of the grant funding cycle. The leadership team works well together, and they have done an outstanding job building the kind of trust and foundation necessary to make this program highly successful. The experiences gained by the leadership team and those within each institution that has been a part of the development and implementation process will greatly help in Year IV. Having the top university officials involved from the beginning of the initiative and keeping them in the loop established a solid foundation for institutional change and sustainability, a real strength of this program. Therefore, BTILLC sees strong evidence that the right foundation was established for sustainability once the project funding ends. However, sustainability will only occur if the key personnel and administration learn from each year s experiences and modify the upcoming year based on this knowledge. During the third year of the program, the leadership team successfully addressed four major tasks identified in earlier evaluations: create a plan to assess student learning outcomes; implement a proposal to increase faculty mobility; finalize the expectations of the student directed project/thesis; and explore ways to address foreign language preparation. Central to becoming sustainable will be getting more departments, faculty, and students involved in the STIMS Program. The team needs continuing to work on this urgent need. BTILLC is very confident that the program is moving in the right direction and has successfully accomplished its third year goals and objectives. The contents of this paper were developed under an EU-U.S. Atlantis grant from the Fund for the Improvement of Postsecondary Education, (FIPSE), U.S. Department of Education. However,

18 those contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. References [1] Committee on Enhancing the Master s Degree in the Natural Sciences, the Board on Higher Education and Workforce Policy and Global Affairs. (2008). Science Professionals: Master's Education for a Competitive World. Washington, DC: The National Academies Press. [2] National Center for Best Practices. (2013, January). Making Our Future: What States Are Doing to Encourage Growth in Manufacturing through Innovation, Entrepreneurship, and Investment. Washington, DC: National Governor s Association. [3] Committee on the Engineer of 2020, Phase II, Committee on Engineering Education, National Academy of Engineering. (2005). Educating the Engineer of 2020: Adapting Engineering Education to the New Century. Washington, DC: National Academies Press, [4] National Academy of Sciences, National Academy of Engineering, Institute of Medicine. (2007). Rising Above The Gathering Strom: Energizing and Employing America for a Brighter Economic Future. Washington, DC: National Academies Press. [5] National Academy of Sciences, National Academy of Engineering, Institute of Medicine,. (2010). Rising Above The Gathering Strom, Revised: Rapidly Approaching Category 5, Washington, DC: National Academies Press. [6] National Research Council. (2005). Decision Making for the Environment, Washington, DC,: National Academies Press. [7] National Research Council. (2008). Research on Future Skill Demands: A Workshop Summary. Washington, DC,: National Academies Press. [8] Board on Testing and Assessment. (2011). Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: National Academies Press. [9] Asgary, N., & Robert, M. (2010). A Cost-Benefit Analysis of an International Dual Degree Programme, Journal of Higher Education Policy and Management, 32(3) DOI: / [10] Bhandari, R. (2009, February). Key Research in U.S. Study Abroad: Findings from the Institute of International Education's Study Abroad Capacity Series. Paper presented at

19 Emerging Directions in Global Education 2009 conference, Feb 9-11, New Delhi, India: lie (New York). [11] Blumenthal, P., & Laughlin, S. (2009, March). Meeting America's Global Education Challenge: Promoting Study Abroad in Science and Technology Fields. Number 5 in IIE Study Abroad Series of White Papers on / Expanding Capacity and Diversity in Study Abroad. New York: Institute of International Education. [12] Committee on Enhancing the Master's Degree in the Natural Sciences, the Board on Higher Education and Workforce Policy and Global Affairs. (2008). Science Professionals: Master's Education for a Competitive World. Washington, DC: The National Academies Press. [13] Culver, S.M., Puri, I.K., Spinelli, G., DePauw, K.P.K., & Dooley, J.E. (2011, April 21). Collaborative dual-degree programs and value added for students: Lessons learned through the Evaluate-E Project. Journal of Studies in International Education, 20(10) DOI: / [14] Marginson, S., & van der Wende, S. (2006, September). Globalisation and higher education. [draft #2b, prepared for OECD]. Paris, France: OECD. [15] National Association of International Educators (NAFSA). (n.d.). Washington, DC. [16] Obst, D., & Kuder, M. (eds). (2009). Joint and Double Degree Programs, An Emerging Model for Transatlantic Exchange. New York: Institute of International Education. [17] Stufflebeam, D. (2002). CIPP evaluation checklist: A tool for applying the fifth installment of the CIPP model to assess long-term enterprises. Retrieved from [18] Stufflebeam, D. (2004). The 21st Century CIPP Model: Origins, Development, and Use. In M. Akin (Ed.). Evaluation roots: Tracing theorists views and thoughts (pp ). Thousand Oaks, CA: Sage. [19] Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, 50, (2), [20] Mattessich, P. W., Murray-Close, M, & Monsey, B.R. (2001). Collaboration: What makes it work (2nd ed.). St. Paul, MN: Fieldstone Alliance. [21] Meyerson, J. (2010). Smart measurement: Tools and tips for capturing results. Evaluation Series Workshop. Retrieved from