Evaluating an eportfolio System: the Case of a Hybrid University Course Igor Balaban 1,Goran Bubaš 1 1 Faculty of Organization and informatics Varazdin Key words: eportfolio evaluation, Mahara, ELGG Abstract: For users to effectively use an eportfolio it is of great importance to select the system which will support eportfolio features according to their specific needs. For this purpose, a questionnaire which will enable organizations to evaluate eportfolio systems, make comparison between two or more systems or to choose their future eportfolio system according to user requirements was developed. In this paper, beside the questionnaire development, the comparison results between two eportfolio systems: Mahara and ELGG will be shown as well as hybrid course settings in which the eportfolio system was introduced. 1 Introduction to eportfolios Electronic portfolios (eportolios) represent a user-centred environment in e-learning and can be defined as a personal digital record containing artefacts which can be provided to the faculty, peers, friends, prospective employers, or the general public (i.e. to whom the owner has chosen to grant permission). Artefacts stand for a representative collection of someone's work which best shows his/her skills, competencies, and talents. They can be in form of information, links, tools, and all other personal or non-personal records that can be selectively provided by the eportfolio owner. The roles of an eportfolio can be diverse and complex. As a collection of artefacts it can be used in various ways: as an assessment tool or an environment for personal development planning (PDP), for presentation of one s achievements, and lifelong learning. The latter term is especially important for adult education (e.g. [1]). On the other hand a collection of individual eportfolios can be used for presenting organizational potential. One other use of eportfolio could be to enable individuals to become self-directed and reflective learners. Selfreflection is one of the crucial benefits of an eportfolio. To use the potential of reflection in eportfolio, users should be taught on how to become reflective thinkers and reflective writers. However, to fully utilize the possibilities of eportfolio system, students must first be introduced to the technology ([4], p. 165). Considering eportfolio presentation capabilities, three main types of eportfolios can be drawn from literature ([3],[4],[5]) and some of them were used in our course: 1. Assessment Portfolio: Demonstrate student competence and skill for well-defined areas. Purpose is to evaluate student competency as defined by program standards and/or outcomes. Student publishes his/her work and educators as well as peers can leave their feedback. 2. Development Portfolio: Demonstrate the advancement and development of student skills over a period of time. They support Personal Development Planning. ICL 2009 Proceedings - Page 638 1(6)
3. Showcase Portfolio: Demonstrate exemplary work and student skills. Students typically show this portfolio to potential employers or to peers or educators. It can be one's CV for example. 4. Hybrid Portfolio: Combination of all three types. Most widely used today. For users to effectively use an eportfolio it is of great importance to select the system which will support eportfolio features according to their specific needs. Therefore we have developed a questionnaire which will enable organizations to evaluate eportfolio systems, make comparison between two or more systems or to choose their future eportfolio system according to user requirements. 2 Selecting software solutions One of the first steps prior implementation of eportfolio concept is choosing an adequate eportfolio application. Since this was the first case of eportfolio implementation at our faculty and within University of Zagreb as well, the recommendations for implementation neither as the preferred eportfolio application were not proposed. Therefore, three main aspects were taken into consideration: 1. Recommendations for eportfolio specification by IMS/GLC 2. Organizational/course requirements and possibilities 3. Available eportfolio applications The IMS eportfolio specification suggests that the types of information an eportfolio can contain are [7]: about digital and non-digital works created or part-created by the subject about the subject of the eportfolio about activities in which subject has participated, is participating or plans to participate about the competencies of the subject about subject's preferences, goals, plans, interests and values any notes, reflections or assessments relevant to any other part the results of any test or examination of the subject about the creation and ownership of the parts of the eportfolio etc. Organizational and course requirements were not so strict since this was the first case of eportfolio implementation and therefore it was not possible to predict and define them entirely. Furthermore, this research will show us which functionalities we need and they will be identified in the questionnaire. For some basic guidelines we have studied findings from previous research like [1],[2],[6],[7] and others. Most constraints were related to organizational technical requirements. Since it is a growing tendency to work with integrated architecture standards giving attention to open standards and interoperability in Croatia, commercial eportfolio systems were parsed out from our considerations. Three ones that were free of charge and available to our institution for installation were Exabis eportfolio block within Moodle LMS (which is official LMS at our Faculty); ELGG open source social networking and social publishing platform; and Mahara open source eportfolio and social networking software. Since Exabis was still in its early phase of development and "showed serious weaknesses concerning the support of portfolio processes, especially in regard to the design of a presentation portfolio"[2] we decided to introduce only Mahara and ELGG eportfolio system to our students. Mahara is entirely built as an eportfolio application, while ELGG is foremost a social networking platform that supports eportfolio functionalities. Numerous examples of using both of these systems as an eportfolio can be found in literature. Since both of them are open ICL 2009 Proceedings - Page 639 2(6)
source systems, they are being continuously improved by the Community, so new functionalities are added rather frequently. According to literature and users experience, Mahara has a much simpler user interface. In addition, it is impressive how quickly the Mahara developers fix bugs that have been reported. ELGG, on the other hand, has a richer set of functionalities and therefore offers a better support in the community/social network aspect. Mahara has been entirely built as an eportfolio system in accordance with its definitions. Although we primarily wanted a system which would support eportfolio features, we offered the students ELGG, with its social networking functionalities, to see if they will also use those functionalities. Although Mahara supports single-sign-on from Moodle, no such connection has been made in order to preserve the independency of the systems and to avoid giving any advantage to Mahara since our students are using Moodle LMS and would perhaps prefer to use a system that is interoperable with Moodle. 3 Implementation of an eportfolio system in the university course Security of Information Systems During the winter semester in academic year 2008/2009 we decided to implement an eportfolio system in a hybrid course Security of Information Systems at the Faculty of Organization and Informatics, University of Zagreb. The 54 students who were attending this course took regular classes in the classroom and had exercises in a computer lab. Additional teaching materials as well as discussion forums were placed in the Moodle learning management system. By introducing an eportfolio system into this course we had two main goals: 1. To choose which eportfolio system would be best for use at the college regarding course structure since most of the courses at the Faculty of Organization and Informatics are organized in similar manner. For this purpose the questionnaire for evaluating eportfolio system will be developed. 2. To introduce the eportfolio concept to students and educators. Here we followed a common strategy found in literature that it is better to start with small-scale pilots to explore eportfolio in specific context and to train key players rather than to introduce the system to all students at once. The question that needs to be answered is whether it is possible, and if so, what is needed, to implement eportfolio in a hybrid university course? The students in our study were given a quick tutorial on use of the eportfolio systems and they also had assignments which they had to complete at the end of each week simultaneously in both eportfolio systems. At first they needed to create a showcase Portfolio and after they got familiar with the systems they created an assessment Portfolio by placing their working assignments and reflections for assessment on the eportfolio systems. Since eportfolio had not been previously introduced in any of courses at FOI, this was the students first encounter with such a system. In order to avoid possible issues and provide the students with the necessary information, a lecture was given as an introduction to the concept of eportfolio, as well to eportfolio as a tool which they would use in the course. In addition, an agenda with stages of eportfolio implementation and its usage (see Table 1) was given to students. Thus the students had a full insight into the entire process; they knew what their assignments were and what would be expected from them at any moment. In the same week in laboratory classes they were given a quick tutorial on the use of the eportfolio systems Mahara and ELGG. ICL 2009 Proceedings - Page 640 3(6)
Table 1. Stages of the introduction of eportfolio systems and instructions for the students Stage 1 2 3 4 Title and description: Introduction to eportfolio 1. Introduction to the eportfolio concept and systems; The need for an eportfolio; Power of reflection 2. Logging into the systems and artefact upload: a. Upload course related artefacts: seminar, presentation, practical work b. Personal artefact upload (4-5 artefacts) -> "Best of me" section c. Tag the artefacts as follows: i. All course related artefacts are to be tagged with SIS08 ii. Tag all personal related artefacts arbitrarily 3. Fill in personal profile (including resume) and review at least 5 profiles of your peers Reflecting on eportfolio 1. Monitor progress, problem solving... 2. Reflect by answering the questions according to the template: a. What have I learned about the eportfolio? b. What was the most interesting thing about using the eportfolio so far? Explain why. c. What was less interesting in the eportfolio? Why? d. Where can I apply the eportfolio in the process of my Life Long Learning? 3. Split in groups. Make a view available only to peers from your group in which you will include the reflection made in Step 2. Use the eportfolio systems to give feedback on reflections made by other peers within your group. Using eportfolio to make course related reflections 1. Monitor progress, problem solving... 2. Now the set of tasks has been created. Reflect on all 4 major units learned in laboratory exercises. For each of them, answer the following questions by using the given template: a. What have I learned in this unit? b. What was the most interesting part of this unit? Why? c. What was less interesting? Why? d. Where can I apply it in future? Analyzing the results and evaluating the systems 1. Final conversation about experience and impressions. 2. Analyzing and scoring students work in eportfolio. 3. Evaluating the eportfolio systems used during classes. The basic idea was to give the students some tasks for which they would have to use eportfolio. As it can be seen from Stage 1, apart from being introduced to eportfolio, they were also given exercises like artefacts upload and tagging in order to get used to this new system. First they had to create a showcase Portfolio and after they got familiar with the systems they had to create an assessment Portfolio by placing their assignments and reflections for assessment in the eportfolio systems (Stage 3). After the Portfolio has been created and published, other peers could browse it and add their feedback. The potential of reflection and feedback is considered to be the most valuable characteristic of eportfolio. 4 Questionnaire development and results A comprehensive questionnaire was created to enable the evaluation and comparison of both eportfolio systems. Statements in the questionnaire were mostly focused on (a) application features such as application in general, organization of artefacts, file management, communication, security and privacy, as well as (b) technical features. The main objective was not only to choose the eportfolio system which will be used in the future at the Faculty or Organization and Informatics, but also to create an initial form of an eportfolio evaluation ICL 2009 Proceedings - Page 641 4(6)
survey. Furthermore, in order to find out more about the impressions of students regarding the use of an eportfolio as a new way of learning several statements were included in the questionnaire just for that purpose. It is important to mention that the questionnaire was based on an extensive overview of eportfolio literature, as well as on a questionnaire developed for on-line course evaluation in our previous research. The scales for evaluation of eportfolio characteristics of our questionnaire proved to be of acceptable internal consistency both in the case of evaluation of Mahara and ELGG eportfolio systems in our pilot study (see Table 2). Table 2. The reliability (internal consistency) of evaluation scales and the results of evaluation of the of eportfolio systems (N=52) Reliability of scale in evaluation of each eportfolio system* Average evaluation of the eportfolio system with the scale eportfolio evaluation scale Mahara ELGG Number of scale items Mahara ELGG General evaluation 0,91 0,94 18 65,2 61,6 Usability & documentation 0,79 0,79 5 20,4 12,1 Organization of artifacts 0,79 0,85 8 35,0 24,9 Management of digital 0,85 0,88 11 41,3 29,7 repository Communication and collaboration 0,83 0,83 6 26,0 18,6 Privacy and security 0,60 0,78 7 28,2 20,0 Technical advancement 0,85 0,80 13 51,8 39,7 * Cronbach alpha coefficient of internal consistency By analyzing the students responses to the questionnaire interesting results were obtained and both of the goals mentioned earlier were achieved. Considering the effort to select the eportfolio system which will be introduced to the next generation of students, Mahara outperformed ELGG in all categories. The biggest difference was in the category which describes the applications general features (e.g. usability and documentation ) where students stated that Mahara was much easier to use and had a more user friendly interface. Although ELGG has more functionality for social networking, it turned out that this was not at all relevant in the case of eportfolio use for our university course Security of Information Systems. Therefore Mahara was the system that was introduced to the students of several other courses during the summer semester of 2008/2009 academic year. Almost 400 students use Mahara today as an eportfolio system at the faculty. The second goal of introducing eportfolio as a new concept and methodology for teaching and learning has also been fully utilized. Based on the results of the first part of the questionnaire in which students expressed their opinion about eportfolio in general, we can say that they were overwhelmed with this new tool. Most of them intend to use Mahara and eportfolio in general in the future to show their competencies, work results, goals and reflections. The students found that eportfolio was very useful and it was easy for them to learn how to build a personal eportfolio. The questionnaire as well as introduction of eportfolio in courses was only preliminary research prior to developing an extensive instrument which will measure eportfolio success on individual as well as on the organizational level. Parallel to instrument development, an eportfolio success model will also be developed which will include six main constructs for measuring eportfolio success. ICL 2009 Proceedings - Page 642 5(6)
5 Conclusion Pilot studies related to evaluation of eportfolio systems prior to their introduction in large educational or other organizations should be considered as an important phase in their implementation. Even though the ELGG eporfolio system had more features related to social networking, when the results of student evaluation of both systems were compared the Mahara eportfolio dominated in all of the assessed categories. Therefore, we emphasize the importance of the development of evaluation scales for users who have experience in working with eportfolio systems and of the evaluation procedures that are taken into consideration for implementation in concrete organizations, as well as the need for empirically based comparison of at least 2-3 eporfolio systems that are subjectively or on the basis of expert opinion considered as potentially most suitable. References: [1] Bisovsky, G., Schaffert, S.: Learning and teaching with E-Portfolios: Experiences in and Challenges for Adult Education, Proceedings of the ICL conference, Austria, September 2008 [2] Himpsl, K., Baumgartner, P.: Evaluation of E-Portfolio Software, International Journal of Emerging Technologies in Learning (ijet), Vol 4, No 1 (2009) [3] IMS Global Learning Consortium: IMS eportfolio Best Practice and Implementation Guide, IMS/GLC, Inc., June 2005, http://www.imsglobal.org [4] Jafari, A., Kaufman, C.: Handbook of Research on eportfolios, 2006, Idea Group Reference, USA [5] McGrath, S., Molder, M., Quon, P., Trapnell, T., Wilton, D.: Types of eportfolios, eportfolio portal, December 2004, http://www.danwilton.com/eportfolios/types.php, visited in January 2009 [6] Richardson, H.C., Ward, R.: Developing and Implementing a Methodology for Reviewing E-portfolio Products, JISC infonet, Northumbria University, 2005, http://www.jiscinfonet.ac.uk/resources/external-resources/eportfolio-products-crareport [7] Stefani, L., Mason, R., Pegler, C.: The educational potential of e-portfolios, Routledge T&F Group, Great Britain, 2007 Author(s): Igor Balaban, MSc. University of Zagreb, Faculty or Organization and Informatics Varazdin Pavlinska 2, 42000 Varazdin, Croatia igor.balaban@foi.hr Goran Bubaš, PhD. University of Zagreb, Faculty or Organization and Informatics Varazdin Pavlinska 2, 42000 Varazdin, Croatia goran.bubas@foi.hr ICL 2009 Proceedings - Page 643 6(6)