Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language

Transcription

1 TUM Department of Electrical and Computer Engineering Chair of Media Technology Distributed Multimodal Information Processing Group Prof. Dr.-Ing. Eckehard Steinbach Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language Konzept eines Einstufungstests auf Mobiltelefonen für Deutsch als Fremdsprache Andreas Plieninger Bachelor Thesis Author: Address: Andreas Plieninger Matriculation Number: Professor: Prof. Dr.-Ing. Eckehard Steinbach Advisor: Dipl.-Ing. (Univ.) Stefan Diewald Prof. Dr. rer. nat. Matthias Kranz Begin: End:

2 TUM Department of Electrical and Computer Engineering Chair of Media Technology Distributed Multimodal Information Processing Group Prof. Dr.-Ing. Eckehard Steinbach Declaration I declare under penalty of perjury that I wrote this Bachelor Thesis entitled Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language Konzept eines Einstufungstests auf Mobiltelefonen für Deutsch als Fremdsprache by myself and that I used no other than the specified sources and tools. Munich, Andreas Plieninger Andreas Plieninger

3 Abstract Mobile learning is a current trend which is enabled by the increasing improvement and adaption of mobile devices. Applications allow learning at one s own pace, everywhere and at any time. Thereby, these applications, for example, help to effectively make use of available and otherwise unused spare time. In this thesis the focus is on learning a foreign language. Recommending and choosing the ideal content for the user is very important for targeted and effective learning. The difficulty of the learning material should be challenging but not overwhelming. For this purpose it is important to know the user s language level. In this thesis, a placement test on mobile phones for learning German as a foreign language is proposed. The focus is on a user-centered perspective and usability. An overview of findings on user experience for mobile phones forms the background of this thesis. Additionally, several existing language tests and applications were discussed. Using these findings, two existing test concepts and contents were chosen. As a result of an iterative feedback process a prototype concept has been developed with the objective of indicating the learner s German language ability. In order to assess the effectiveness of the approach, an evaluation was conducted. The participants prior taken language test were used as criterion tests. The subjects took the proposed placement test on a mobile phone and afterwards answered a questionnaire with focus on usability and preferred test parts. The scope of this work only allowed a small participant number, which limits the certainty of the conclusions. The results showed that the concept is usable. In addition, they showed that the correlation between the test score and the criterion test is significant. These findings allow the conclusion that the concept can indicate the test taker s German language ability. Based on the questionnaire and observation results, it is suggested that the concept can be further improved by shortening the test, adaption of the difficulty to the user s level, more precise task description and in-place correction. iii

4 Kurzfassung Mobiles Lernen ist ein aktueller Trend, der mit der fortschreitenden Popularisierung von mobilen Endgeräten einhergeht. Mobile Anwendungen erlauben das Lernen mit individueller Geschwindigkeit, überall und zu jeder Zeit. Dadurch können diese Anwendungen beispielsweise helfen verfügbare und ansonsten ungenutzte Zeit effektiv zu nutzen. Der Fokus dieser Bachelorarbeit ist das Erlernen einer Fremdsprache. Das Empfehlen und Auswählen idealer Inhalte für den Nutzer ist für zielgerichtetes und effektives Lernen sehr wichtig. Die Inhalte sollten herausfordern, jedoch nicht überfordern. Für diesen Zweck ist es wichtig, das Sprachniveau des Nutzers zu kennen. Diese Bachelorarbeit schlägt ein Konzept eines Einstufungstest angepasst auf Smartphones für Deutsch als Fremdsprache vor. Der Schwerpunkt dieser Arbeit wurde auf eine nutzerzentrierte Perspektive und Benutzerfreundlichkeit gelegt. Die Grundlage dieser Bachelorarbeit bildet ein Kapitel über Erkenntnisse der Benutzerfreundlichkeit bei Smartphones. Darüber hinaus wurden einige Sprachtests und Anwendungen untersucht. Basierend auf diesen Ergebnissen wurden zwei bereits existierende Tests und entsprechende Inhalte ausgewählt. Es wurde ein Konzept in Form eines Prototypen mit einem iterativen Feedback- Prozess entwickelt, mit dem Ziel, das Deutschniveau des Nutzers angeben zu können. Um die Effektivität des Ansatzes zu untersuchen, wurde eine Auswertung durchgeführt bei dem ein bereits abgelegter Sprachtest der Teilnehmer als Kriteriumstest diente. Die Personen machten den Einstufungstest auf einem Smartphone und beantworteten anschließend einen Fragebogen, der sich besonders auf Nutzerfreundlichkeit und bevorzugte Testteile konzentrierte. Der begrenzte Umfang dieser Arbeit ließ nur eine kleine Teilnehmerzahl zu und limitierte somit die Aussagekraft der Schlussfolgerungen. Das Ergebnis zeigte, dass das Konzept benutzerfreundlich ist. Weiterhin zeigte es, dass die Korrelation zwischen dem Testergebnis und dem Kriteriumstest signifikant ist. Diese Erkenntnisse erlauben die Schlussfolgerung, dass das Konzept das Sprachniveau des Nutzers richtungsweisend angeben kann. iv

5 Contents Contents v 1. Introduction Motivation Statement of the Problem Structure of the Thesis Background and Related Work Definitions of User Experience and Usability User Experience Usability Distinction of User Experience and Usability Mobile Phone User Characteristics Mobile Interruptible Available Contextual Identifiable Understanding the Mobile Device Limited Display Modality Touchscreen Interface Text Input Modalities Connection Characteristics Second Language Testing Linguistic Proficiency Test Selection Criteria Cloze Test C-Test Vocabulary Size Test (VST) TOEFL Related Mobile Applications v

6 CONTENTS vi 3. Concept Choosing a Test Test Contents Source of VST Items Amount of C-Test Items Source of C-Test Items Paper Prototyping Mobile Phone Prototyping Evaluation Research Questions Methodology and Test Setup Test Execution Results and Interpretation Demography and Mobile Phone Experience Usability Test Time Overall Perceived Value Preferred Test Part Open Questions and Observations Proficiency Correlation Conclusion 43 A. Test Items 44 A.1. VST Items A.2. C-Test Items B. Prototypes and Feedback 47 B.1. Paper Prototype B.2. Paper Prototype Feedback B.3. Second Prototype B.4. Feedback for Second Prototype B.5. Third Prototype C. Evaluation 61 C.1. Questionnaire C.2. Answers to Open Questions and Observations C.3. Test and Criterion Test Results

7 CONTENTS vii List of Figures 70 List of Tables 72 Bibliography 73

8 Chapter 1. Introduction 1.1. Motivation Mobile learning is a current trend which is enabled by the increasing adaption and improvement of mobile devices. Many people are carrying around their mobile phones almost all the time. Learning on the phone has some potential advantages: It is possible everywhere, at one s own pace and at any time [1]. Furthermore more people around the world are using mobile technology to access learning resources and applications [2 4]. In this thesis the focus is on learning a foreign language, a sub field of mobile learning. Currently many applications are available to study vocabulary. It is suggested that there should be more research on how to design and deliver learning [2] which could lead to more contents and learning applications. Still, no matter what kind of content, they have one thing in common: Recommending and choosing the optimal content for the user is very important for targeted and effective learning. It could be a step towards the vision of a perfect individual teacher to learn a foreign language Statement of the Problem The best learning effort can only be achieved given the most suitable material. In the beginning when a student starts using a specific application, the learning content must be specified. The knowledge of the learner should match the difficulty of the recommended content. To be more precise, the difficulty of the learning material should be challenging but not overwhelming. In order to achieve this matching, the content as well as the learner must be rated. The first challenge, to rate the content, is not part of this thesis. The second challenge is to rate the learner s language knowledge, the linguistic proficiency. In this thesis we will focus on learning German as a foreign language. The rating with a placement test will be done on a mobile phone. A first concept shall be created to evaluate the placement test and suggest points for further improvement. To summarize, the central research question is: 1

9 Chapter 1. Introduction 2 How could a concept of a language test on a mobile phone which can indicate the test taker s general German language proficiency look like? The following points characterize the research question: The concept is for people who learn German as a foreign language, are no beginners and are using or will probably use a mobile phone application to do so It could be used when a learner tries a new mobile learning application for the first time or, more general, when using a new product/resource (e.g. language course) for the first time The concept is designed to run on a mobile phone (current models: Apple iphone 6, Samsung Galaxy S6 or Xiaomi Redmi 1S) The implementation is prototypical (on paper or an application running on a single specific phone) The prototype can be used by the majority of potential users without further help ( usable ) An economic and scalable way of testing: Computers can calculate and evaluate the test takers results The concept is a step further to provide individuals better learning material and a better learning experience 1.3. Structure of the Thesis The thesis is divided into five chapters. Chapter 2 focuses on the background and related work. Existing placement tests as part of second language testing as well as usability and user experience are examined. Chapter 3 describes the process and results of creating a concept of a placement test. Findings of several feedback rounds are used to improve the concept. In order to analyze the final concept, a user test was designed, executed and finally evaluated. Chapter 4 describes the process, results and interpretation especially regarding the research question. At the end, Chapter 5 summarizes the thesis, points out conclusions and indicates potential points for further improvement.

10 Chapter 2. Background and Related Work This chapter summarizes related literature and works. The purpose is to provide the background and basis for the research undertaken [5]. In order to answer the research question, considering how a placement test on a mobile phone could look like, it is especially important to understand the mobile user, mobile device and resulting design implications. In this way it shall be ensured that the concept can be used in the designated way. In the second part, these design implications are essential to discuss and review different language learner placement concepts from the literature. Especially the pros and cons for an implementation on a mobile device are of interest. This analysis will form the basis to create a draft of the placement concept Definitions of User Experience and Usability User Experience The commonly used definition of user experience according to ISO :2010 is: a person s perceptions and responses that result from the use and/or anticipated use of a product, system or service [6]. This very formal definition is supported by another interpretation: User experience explores how a person feels about using a product, i.e., the experiential, affective, meaningful and valuable aspects of product use [7] Usability The commonly cited definition of usability is from Nielsen: usability is about learnability, efficiency, memorability, errors, and satisfaction [8]. Yet, the ISO definition of usability the extent to which a product can be used by specified users to achieve specified goals with effectiveness, 3

11 Chapter 2. Background and Related Work 4 efficiency and satisfaction in a specified context of use [9] is the main reference and will be used for the purpose of this research [10] Distinction of User Experience and Usability The main difference between the two areas is the point of view. Fig. 2.1 shows this in a pyramid. Usability describes the basic requirement for a product and is quantifiable. If it is not present a positive user experience is almost not possible. Therefore usability can be considered as an important subfield of user experience. With more abstraction subjective perception increases and therefore also the development process is less quantifiable. Thus, a successful product in terms of user experience should fulfill several requirements on different levels. All these levels demand a lot of effort and can therefore not be considered in the same detail in this work. The following chapters focus on the characteristics of the user and device to provide the background to create a usable and convenient product concept. Figure 2.1.: The user experience pyramid with requirements on different levels. This work focuses on Usable and Convenient with the whole picture in mind. Source: D. Traynor,

12 Chapter 2. Background and Related Work Mobile Phone User Characteristics The research goal is to implement a placement concept for mobile phone users. Hence it is important to understand the mobile user characteristics in order to discuss the aspects of existing classification tests regarding their fit in a mobile application. Furthermore usability problems could have a negative impact on the final test performance. Therefore the user characteristics must be considered in detail to prevent negative influence as effectively as possible. Ballard [11] wrote an in-depth analysis of these characteristics. It is the basic source for the overview in the following sections Mobile Mobile phone users may be in any situation anywhere when using their phone. They are in a conversation, commuting, on a plane, waiting for a meeting or just at home [12]. They interact with the environment and the physical as well as the social context changes [11]. In the same time the user tries to use the phone application and has to shift the focus from the device to the environment and back. This is one factor that makes the mobile phone user more interruptible Interruptible Besides being interrupted by context shifts between navigating in the environment and the phone there is another factor: Sitting in front of a computer in comparison to using a phone, people are more likely to interrupt the phone user. They do not have a social clue that suggests that the user is focused on a task [11]. The same clues are missing when the user is, e.g., waiting for a train or in a meeting while using the phone. This more distracting environment results in a shorter interaction period at a time. An average time per interaction on a mobile phone is 17 minutes compared to 39 minutes on a desktop computer [12]. The shorter interaction periods and high probability of an interruption have an impact on the mobile phone application design, which is outlined in Section D. Traynor, 2008 (concept from S.P. Anderson, 2007); last access: 2015/01/13

13 Chapter 2. Background and Related Work Available On the other hand an interruption can be very important to the user. Currently more and more applications utilize notifications as their core service and only way to communicate with the user 2. Mobile phone owners bring their device most of the time with them in order to stay connected and communicate, for example, with friends and be available for any communication attempt almost at any time. This availability allows applications to provide technologies similar to instant messaging with high confidence that the user is present and gets the information immediately [11]. The concept of this work will not depend on this availability. It is mentioned to keep the list of characteristics complete Contextual The environment influences the phone usage. Ideally the device could know and understand the context and adapt to it [11]. This context includes factors such as the location, activity and the corresponding expected behavior. As an example it could be considered that the user is starting to drive. The device notices this new context and automatically changes its interface to car mode as well as connects to the car s media interface. There are first applications which try to implement this, e.g. Google Now is providing information like routes and weather forecast based on the context 3. This aspect of context is mentioned for a better general understanding but will not be considered further in the concept of this work Identifiable The mobile phone is a very personal device and most of the time the user will carry it around and be available [13]. Additionally, the user has his own personal phone number which provides a persistent way of contacting him. The value of the phone number is so high that many countries are regulating the phone carriers to ensure number portability between carriers. Because the phone number is very personal it provides a good way for user identification [11]. This possibility is already used by many applications, e.g. WhatsApp validates the phone number by sending a text message with a verification code 4. 2 For example WhatsApp informs the user about new messages via notifications; last access: 2015/01/23 3 Google Now, last access: 2014/12/18 4 WhatsApp FAQ, last access: 2014/12/18

14 Chapter 2. Background and Related Work 7 Identification is covered to complete the list of user characteristics and will currently not be used in the concept of this work Understanding the Mobile Device Learning the characteristics of mobile devices is essential to understand the mobile experience and therefore to design a user interface and application. In this section we explore the aspects of the hardware and user interface elements that make the devices so different compared to traditional desktop computers. Additionally, the consequences of every aspect and resulting design principles are introduced Limited Display Modality For a mobile device it is essential to be small and easy to carry around. Hence the screen has to be small. The following section summarizes advice from Nielsen [14]: As he stated in his book, there is less information that can be displayed at a time compared to a desktop computer. Therefore the user has to rely on his short-term memory for other necessary information which is not visible at that moment. As a result the interaction gets more difficult. The challenge is to find room for all necessary information on one screen and to make the interaction as easy and pleasurable as possible. Focus on one Task Every application screen must focus on the user s immediate task only. Therefore every screen is designed to fulfill one distinct task and should only display the information needed to perform this task. Only necessary buttons and navigation elements should be available. Thus the information which is displayed on one page must be reduced. The Blackberry UI Guidelines 5 include a good example: A very complex customer relationship management tool provides a massive amount of information. The user only needs a tiny amount of information at a time, e.g. to change the customer s address it is not important to see the customer s orders. In summary, the application should be designed in a way that the user can perform desired tasks easily and quickly. This requirement to focus on simplicity and the most important elements only paired with the ability to install many different applications on one phone results in the goal to design one appli- 5 Blackberry UI Guidelines, Constraints_of_designing_for_mobile_devices_ _11.jsp, last access: 2014/12/16

15 Chapter 2. Background and Related Work 8 cation for one specific user need. In this way one application can do one thing very well and in the same time fulfill the requirements of simplicity and focus. Consequently, the concept of this work will focus on one goal: To perform a placement test on a mobile phone. Smoothen Application Switching On the small screens one application can be displayed at a time. This is very different compared to personal computers where many windows can be arranged next to each other and still be visible. Thus users are required to switch applications quite often [15]. Therefore the application should be designed in a way to handle this events (lose focus, get focus again, exit, open) instantly, without much waiting time and a good strategy for the user after coming back, e.g. to continue with his paused task. For example, a user is writing an . While writing, he receives a phone call. The application will be paused and the phone call will be displayed. After the call is finished the user wants to continue writing his . One good way to handle this is that after he reopens the mail application, the stored draft of the mail is displayed again. He continues writing where he stopped and finally sends the Touchscreen Interface The main interaction element of modern phones is the touchscreen which combines a screen with a touch interface. It seems to be the most natural of all input devices so far because it allows the direct engagement with the displayed information [16]. The main interaction tool is the finger, to be more precise, one-handed thumb interaction [17]. Thus, the main guidance for touchscreen interface design is to optimize for the thumb. Minimal Target Size Apple recommends a button size of at least 44 points x 44 points, which corresponds to about 6 7 mm x 7 mm. Given this size, no target element should have less than 9.7 mm in diagonal which corresponds to the smallest average size of a finger pad 7 and is also recommended by other 6 Apple iphone 4/5/6 have 326 ppi: 44 points x 44 points, retina displays have two pixel per point, therefore 88 px x 88 px which equals 0.27 inch x 0.27 inch = 0.69 cm x 0.69 cm 7 Ubuntu - Designing for finger-driven UIs, DesigningForFingerUIs, last access: 2014/12/18

16 Chapter 2. Background and Related Work 9 research [18, 19]. A recommended padding between targets of at least 2 mm prevents accidental taps 8. Preferred Interaction Areas Depending on the orientation the device is held (horizontal vs. vertical) the fingers can reach some parts of the touchscreen more comfortable. Most of the time the phone is hold vertically (portrait mode), 49% use their phone one-handed, 25% two-handed and with their thumb on the screen 9. Fig. 2.2 describes the dependence between touchscreen area and ease of reach graphically. With increasing screen size it is getting more complicated to reach the whole screen as it is illustrated in Fig Therefore Apple introduced for the iphone 6 a new feature called reachability 10. On a double home button tap the device shifts the entire screen down closer to the thumb, Fig. 2.4 shows the shifted screen. This facilitates the interaction e.g. with the back button in the upper bar. Figure 2.2.: Screen reachability in one-handed and two-handed thumb use as reference for user interface drafts where to position interaction elements - Source: Steven Hoober 11 8 Microsoft Windows - Design touch interactions, apps/hh aspx#touch_targets, last access: 2014/12/18 9 Steven Hoober: How do users really hold mobile devices?, 02/how-do-users-really-hold-mobile-devices.php, last access: 2014/12/18 10 Apple iphone 6 Design, last access: 2014/12/17 11 Steven Hoober, last access: 2014/12/18 12 Kirk McElhearn, last access: 2015/01/14 13 Apple iphone 6, last access: 2015/01/14

17 Chapter 2. Background and Related Work 10 Figure 2.3.: Reachability with one-handed thumb usage on different screen sizes: Larger screens demand an adapted user interface to keep the same level of convenience, e.g. Apple Reachability - Source: Kirk McElhearn 12 Figure 2.4.: Example of Apple s iphone 6 feature to increase reachability: The entire displayed application gets shifted closer to the thumb. This feature can be used in interface drafts for large screens - Source: Apple iphone 6 13 Responsiveness A touchscreen allows the direct engagement with the displayed information. From this seemingly intuitive and natural input method the user expects high responsiveness [20, 21]. It is absolutely critical to respond to an interaction immediately. Users find latencies above 580 ms unacceptable and greatly prefer lower latencies, even below 50 ms [22]. In the case an action takes longer to finish, it is essential to acknowledge the input, e.g. by displaying a progress bar. This is the

18 Chapter 2. Background and Related Work 11 difference between speed and responsiveness [23] Text Input Modalities The main input modalities usually are a keyboard, touchscreen, video/audio recording and speech to text. Video and audio recording are widely used by media applications, such as YouTube and sometimes for direct messaging applications. Automated interpretation by a computer of this content is sometimes inaccurate. One very interesting technology is speech to text, which is already available on the latest generation of mobile phones. The accuracy is getting better but cannot yet be considered mature 14. The main text input modality up to today is still a keyboard. Most of the time this is a software implemented keyboard using the touchscreen for interaction. Considering the small size of these keyboards they are often perceived as impractical and therefore lower rates of text entry compared to personal computers are reasonable [24] and most users are unlikely to write long texts on a phone. Minimizing Text Input Because of the difficulties the user text input should be reduced to a minimum and replaced by other methods. One way is to provide options, e.g. to provide possible answers and the user has to choose one or to provide an adjusted minimal input element containing only the limited needed amount of characters to complete a specific task 15. Another way is to tie in existing phone services, e.g. use a Google account to log in the application 16. Whenever a lot of text input is still necessary it should be considered to support landscape mode in the application. The user can decide to turn the phone and use the larger keyboard which might be preferred. There are also several different keyboard layouts on popular smart phone systems available to support specific content, e.g. text, number or URL input 17. The expected input should be analyzed carefully to choose the corresponding keyboard layout, e.g. whether auto-correct is helpful and in case, the unnecessary input methods should be disabled [23]. 14 Intelligentvoice.com, 46-americans/, last access: 2015/01/14 15 see Memrise, memrisecompanion&hl=de, last access: 2015/01/14 16 Google+ Login in a custom application, de, last access: 2015/01/14 17 ios Developer Library, UITextInputTraits_Protocol/index.html, last access: 2015/01/14

19 Chapter 2. Background and Related Work Connection Characteristics Mobile Internet connections are getting faster and faster. Still, even with 4G, latency is noticeable. Another important point to consider is network coverage. From time to time there is a connection interruption or no coverage at all, e.g. in many subways and on railway lines coverage is partial. In these cases, depending on the application and necessary amount of data to download, loading the next screen on the phone can take too long [14]. A possible solution is to heuristically pre-fetch content. Another solution is to provide an offline mode and synchronize data when an Internet connection is available again. These technical solutions allow to minimize the waiting time and therefore improve the user experience. All potential content in our concept, e.g. text, images, short audio and video, can be pre-fetched and stored offline. Therefore this question is for our goal, a placement concept, mainly a technical implementation aspect and it will not be further discussed in this thesis Second Language Testing This section describes the different basic skills a language test can measure and examines some tests in detail. In order to not exceed the scope of this thesis the following sections will only explain the necessary linguistic background and focus on second language testing for assessment and placement purposes Linguistic Proficiency A common definition of general language or linguistic proficiency is the overall ability of an individual to communicate in the acquired language 18. This concept was further developed in the 1980s in America based on new research insights by the ACTFL 19. Their Proficiency Guidelines defined several levels to classify the learner. The goal of proficiency tests is to provide a fair and standardized classification into e.g. a certain predefined level. In comparison, achievement tests usually try to assess to which amount an individual completed a given learning task. This tests are usually used in the classroom environment and follow the principle of test as you teach [25] and will not be considered any further. The ACTFL proficiency guidelines further broke down the general proficiency to assess four distinct areas: speaking, listening, reading, writing. Based on the assumption that these skills are 18 Wikipedia - Language proficiency, last access: 2015/01/02 19 ACTFL - American Council on The Teaching of Foreign Languages, Proficiency Guidelines, org/publications/guidelines-and-manuals, last access: 2015/01/14

20 Chapter 2. Background and Related Work 13 independent from another they can be assessed separately [26]. For example, teachers can use the result to identify problems of their students in a specific area. These kinds of tests are therefore also a tool for language diagnostics. All individual results together are then used to determine the general proficiency level. The goal of this thesis is to assess the general proficiency (see Section 1.2). Therefore results of more broken down tests on specific areas of proficiency (e.g. grammar, listening) are not required. Hence the focus of the following sections is mainly on relevant, researched and feasible tests which will be discussed for potential usage in a mobile application Test Selection Criteria In this section the criteria to rate the different language tests will be discussed. Beginning with the statement of the problem, to create a concept of a German language placement test for mobile phones, we will further break this down into several criteria. In Section 1.2 we defined our goal to assess the general language proficiency. Hence one criterion will be how good one test correlates with the general language proficiency, which will be called test performance. The placement concept should provide scalability in an economic way, therefore automated test correction by the computer is valuable. Furthermore, test correction by human individuals can be very subjective. Without an exact predefined solution set this unwanted effect can be even stronger. The criterion automated correction will assess, how good a test can be corrected by computers without any large additional effort to create special software. This will be an exclusion criterion, meaning tests which can almost only be corrected with human help are excluded from our concept. In order to create an appealing product, Section 2.2 and Section 2.3 provide the background to user and device characteristics. The screen size on touch phones is limited, the discussed implication in Section is to focus on one task per screen. Some language tests utilize questions where the user has to see and know a lot of context. In order to reduce the short-term memory load of the user, all the information necessary to answer the question should be visible at the same time. Meanwhile, the minimal target size and preferred interaction areas for important content (see Section 2.3.2) should still be respected. This criterion, how easy it is to fit all the information on one screen, will be called screen size compatibility. Another important factor is the text input modality and the resulting implication to minimize text input (see Section 2.3.3). This criterion, how many characters a user has to enter, will be called text input amount. The name mobile phone already implies another important characteristic: mobility (see Sec-

21 Chapter 2. Background and Related Work 14 tion 2.2.1). Considering this, the user can be literally anywhere in any situation when using an application on the phone. Furthermore, it cannot be assumed that audio output is always desired and accepted by the user. Reasons include examples like noisy environment, social expectations (e.g. restaurant) or missing headphones while sitting in the subway. Accordingly, another criterion for tests is no audio dependency. The final criterion is time. A shorter test means less effort for the test taker. Assuming that users actually take this test for a final goal, e.g. to get better content or to know their level, they just want to achieve this goal as fast as possible Cloze Test A cloze test is a text where certain words are removed and replaced by fixed-width gaps. The task of the student is to fill the gaps with the words again. In the example in Fig. 2.5 the words are replaced with a line. This test with an appropriate length, difficulty and every n-th word removed is called cloze test [27]. Education has two goals. First, it means so that we may earn living. Some of these tools are tools, such as reading or writing. are technical skills, such as typing,, and data processing. Still others are professional skills peculiar for example, to, to science, and to teaching. Figure 2.5.: Excerpt of an English cloze test for better understanding, Source: Farhady [28] Creation Guidelines To prepare the test, every n-th word of the text is removed. The value of n has a large influence on the difficulty and length of the test. Common values are in the range from 5 to 8, most of the time 7 is used [29]. If n is small many words will be removed and therefore the gaps will be more difficult to fill again. If n is large the resulting text will be longer in order to keep the same amount of gaps. An additional commonly used term in this context is the frequency, which describes the same as n, the amount of gaps per passage [30]. It is also possible to rationally choose which words should be replaced with gaps. This allows the creator to have more influence on the resulting test. The validity and readability will still be comparable to the fixed-ratio method [31]. Because it is more effort with almost the same outcome compared to the fixed-ratio method, it will not be considered further. Another important factor is the length of the test: the number of gaps to fill in. Researchers suppose numbers from 20 up to a maximum of 50. Commonly 20 to 25 gaps are recommended [32].

22 Chapter 2. Background and Related Work 15 In order to improve the readability of the resulting test the first and last sentence should be without gaps [29]. To be more specific, the first sentence should already contain the paragraph s main idea. The conclusion of the text is in the closing sentence. This intact sentences are important to not distort the student s natural language processing. To sum up, an example cloze test could consist of 25 gaps with every 7-th word removed and an intact initial and closing sentence. This test would have an appropriate length of about 230 words [33]. Finally, the difficulty of the original text must be taken into consideration. One commonly used method is to estimate the readability, e.g. with the Fog s index of readability for English. Correlation with Proficiency Many studies examined the correlation of the cloze test and an established and validated benchmark test for general English proficiency. Saeedi investigated a correlation between the cloze test and TOEFL-Test with 90 Iranian English majors. The result was a statistically significant correlation and therefore the cloze test also correlates to general proficiency (correlation coefficient 0.82, significant at the 0.01 level) [34]. To the same statistically significant correlation between cloze test and language proficiency came other researchers [35 40]. Discussing Selected Criteria Section defined several criteria, which will now be discussed regarding this cloze test. There is no need for audio while taking the test. Additionally, given predefined solutions, automated correction is possible. A full cloze test with the goal for placement would presumably take about 40 minutes of time to finish 20. It can be assumed that the amount of text input is about 800 characters 21. To find the correct solution for a gap, the test taker has to consider the context. Therefore it is important to see the text before and after a gap. text into separate screens. This prevents to split one A single test with about 230 words 22 will probably not fit in an appropriate manner on one smart phone screen. However, scrolling seems to be a possible, but not a perfect solution. Furthermore, every time the user types a word, the onscreen keyboard will use approximately 30% of the screen 23. Because less context is visible in that moment, the test taker has to rely more on his short-term memory. In summary the screen compatibility is still good. 20 Suppose that a placement test consist of eight texts with the same amount of gaps like the ondaf (20 each), which is 40 minutes long [41] 21 Considering eight texts with 20 gaps each and an average word length of five characters, the test taker has to write 800 characters. 22 see Section 2.4.3: Creation Guidelines 23 Considering a mobile phone screen shot (Android phone viewport, 360 px x 640 px, see Figure 3.2)

23 Chapter 2. Background and Related Work C-Test Based on the cloze test (see Section 2.4.3) researchers developed the C-Test in the 1980s. In an initial text every second word is omitted: The second half of the word is removed and replaced by a gap [42]. In such a resulting test like in Fig. 2.6 the student has to fill in the gaps in order to reproduce the original text. In comparison to the cloze test, not the whole word, just the second half of the word is removed. Nothing beats the heat like a refreshing dip in a swimming pool. But wh it co _ to wa _, both ki and adu _ need t_ be car. Susan King s daug _ - Alison, 12, a Christy, 9, a in th _ grandparents po every d. King s gi _ have ma pool ru _, including n being all in t pool ar without a_ adult, n_ jumping i_ the sha end, n_ running around the pool and no holding each other under water. "Kids drown quickly and quietly" caution Jen Costello of the National safe kids campaign. Figure 2.6.: Excerpt of an English C-Test for better illustration, Source: Saeedi et al. [34] Test Construction According to Grotjahn [43] the test should follow these construction guidelines: As a basis for the test several short texts (60 to 80 words) with different topics are chosen. Beginning with the second sentence in every second word the second half is omitted and replaced by a gap. If the word is a proper noun the word will not be omitted. If the word is only a single character it will also not be omitted. In the case the length of the word is odd the larger part will be omitted. Every text should contain the same amount of gaps. The last part of the text should remain intact in order to provide more context for the student to solve the test. To format the gaps, constructors can use lines of the same length or dashes according to the number of removed characters. The former method results in a more difficult test. All the resulting texts are arranged by the perceived difficulty, beginning with the easiest. For every exact solution the student gets one point. Validity regarding Proficiency The validity of the test was subject of many studies: For example, in a study with 102 Hungarian English majors the C-Test was statistically significant correlating with the TOEIC 24 [39]. Another study with 470 participants showed a statistically significant correlation between a C-Test and 24 TOEIC: Test of English for International Communication; it is considered to be highly reliable and valid to measure English proficiency

24 Chapter 2. Background and Related Work 17 the TestDaF 25, especially regarding the factor general proficiency [44]. Yet another study with 223 students of German as a second language showed statistically significant correlation between the C-Test of ondaf 26 and DIALANG 27 [41]. Furthermore, latest research suggests that the test not only measures on the micro level but also on a larger more general level [45 50]. Therefore this test is among others used as assessment test for universities and language schools [51, 52]. In conclusion, it is shown that the C-Test is a reliable method to measure the German language proficiency. Discussing Selected Criteria As Section defined, several criteria besides the test performance are important to assess: Automated correction is possible due to the distinct solution. Additionally, the test requires no audio. The amount of text input can be estimated with 400 characters 28. The time can be assessed accordingly to the cloze test 29 with about 40 minutes. The last criterion is the screen compatibility: In comparison to the cloze test, the C-Test has less text between the gaps and is therefore much shorter. It could maybe allow to show 20 gaps on one screen 30 while still respecting touchscreen interaction guidelines (see Chapter 2.3.2). Therefore the screen compatibility is good (and slightly better than the cloze test) Vocabulary Size Test (VST) Compared to a traditional placement test (e.g. grammar/listening/reading comprehension questions) a vocabulary size test is quite different: The goal is to measure the number of vocabulary a student knows, which is often referred as breath of vocabulary knowledge. Researchers discovered that the vocabulary knowledge is essential for many language skills. The vocabulary size is especially a very good predictor for reading comprehension [53]. In general, students with a large vocabulary are more proficient [54]. Therefore, one can use a test to measure a person s vocabulary size in order to estimate his proficiency. Because quite a large sample of answers is required to reliably estimate one s vocabulary size, simple test formats are often preferred: One way to achieve this is the so called Yes/No Format: 25 TestDaF: Test of German as a foreign language; a standardized and well-recognized valid test to measure German proficiency 26 ondaf: Online-Einstufungstest Deutsch als Fremdsprache; standardized language test of German proficiency which uses A2-C1 levels from the CEFR (Common European Framework of Reference for Languages) 27 DIALANG is an online language diagnostic system to assess a person s proficiency in 14 European languages, last access: 2015/01/14 28 The gaps are missing half a word. Therefore half the amount of characters compared to the cloze test have to be typed in: same amount of gaps and slightly less text to read, see Section Considering 20 gaps: every second word has a gap; First and last sentences are intact; 20 2 words+15 words = 55 words: It seems possible to fit 55 words with a reasonable large font on one screen.

25 Chapter 2. Background and Related Work 18 The test taker is presented with a list of words where he has to indicate whether he knows the word or not. Sometimes the student overestimates his knowledge and answers too many words with yes. In order to prevent a wrong result, invented non-words are added to the real word list [55]. Figure 2.7.: Yes/No Format: Vocabulary Size Placement Test example to facilitate the estimation of compatibility with a small screen, Source: DIALANG 31 Validity regarding Proficiency One well-regarded example of this test format is DIALANG 32 [41]. In the introduction the student is asked, as Fig. 2.7 shows, to take a vocabulary test. This result is then used to estimate his general proficiency and adapt further specific skill tests to his level. Analysis of DIALANG s data suggest a significant statistical correlation between the vocabulary test result and specific measured skills (reading, grammar, writing listening correlation each above 0.6) [56]. Another significant example is the Swansea test, where the software programs X_LEX and Y_LEX provide vocabulary tests for the most frequent English words [57]. To conclude, there are established and valid methods to measure the test taker s vocabulary size and therefore estimate his general language proficiency. Discussing Selected Criteria In this section the missing criteria (see Section 2.4.2) regarding this test are discussed: To answer a yes/no question one simple tap on the interaction area is necessary. Accordingly, the amount of 31 DIALANG online test, last access: 2015/01/08 32 DIALANG is an online language diagnostic system to assess a person s proficiency in 14 European languages, last access: 2015/01/14

26 Chapter 2. Background and Related Work 19 text input is zero. The test taking process has no need of audio. Furthermore, an automated correction of the answers is possible. It takes approximately 10 minutes of time to answer all questions 33. To answer an individual question, meaning one word and the possible answers yes/no, no other information is needed and therefore also independent from other questions. They can be split over several screens without any negative influence of the test taker s short-term memory load. Therefore even larger amount of these questions can be implemented in the user interface. In conclusion, the screen compatibility is very good TOEFL The TOEFL 34 is a standardized test which measures English language proficiency and is an accepted certificate by most American Universities for non-native English speakers who try to study there 35. It is also regarded as a well-respected criterion test to benchmark new tests and used by several researchers [58]. In the following section a TOEFL ibt sample test 36, which is administered via computer, will be discussed. The test consists of a speaking part (20 minutes), writing part (50 minutes), listening part (60-90 minutes) and reading part (60-90 minutes) 37. Speaking Task The speaking section measures the ability to talk about various topic. Typical tasks include to talk into the computer microphone about a pleasant moment or answer questions about a written/oral conversation or text. These responses are rated by humans 38. In Section one of our criteria is automated correction, which cannot be achieved using this kind of questions. Therefore this method will not be considered any further. Writing Task The writing task consists usually of two essays: One essay with personal opinion about a controversial topic and a second one without personal opinion about an academic topic. The rating is 33 DIALANG has 75 yes/no questions; Assumption: A non-native speaker needs up to ten minutes to answer them. If the assumption was about right can be validated in the evaluation of the concept 34 TOEFL: Test of English as a Foreign Language 35 TOEFL ibt: About the Test, last access: 2015/01/15 36 TOEFL Sample Test Questions, last access: 2015/01/15 37 TOEFL Test Content, last access: 2015/01/15 38 TOEFL Scores, last access: 2015/01/15

27 Chapter 2. Background and Related Work 20 done again by humans, therefore the criterion automated correction (see Section 2.4.2) cannot be achieved and this kind of task will not be considered further. Reading Task The last task is to read three or four passages (about 700 words each) from academic texts and answer about 40 multiple choice questions. These questions have usually four possible answers with a very detailed description (mostly words long). In the following section the criteria (see Section 2.4.2) regarding this question type will be discussed: automated correction is possible because the answers are given on multiple choice questions. Therefore also the amount of text input is zero. All the questions and corresponding contents are text-based. As a result there is no need for audio. The screen compatibility requires a more detailed investigation: The test taker will read the given text, which is about 700 words or one and a half DIN A4 pages long. This text must be displayed on a modern mobile phone. Considering 250 words as the maximum of visible text at a time 39 the user would have to scroll almost two pages down to read the full text. In the next step multiple choice questions are shown, each at a time. Every time test takers are not sure about an answer they will want to take a second look at the original text. For this purpose users have to e.g. tap on a button to change the screen to the text and then scroll to the appropriate position. Then they have to interact a second time to see the question again. This back and forth will be especially necessary given several very detailed and similar choices. Not only might this feel inconvenient, it also puts a high load on the test taker s short-term memory. For the stated reasons it appears difficult to implement this task on small screens in an acceptable manner and therefore the screen compatibility is rated as satisfying. Listening Task One of the longest parts, the listening, focuses usually on two tasks: Listen to a conversation and to a class transcript and answer six multiple choice questions about each topic. This multiple choice questions have four possible answers, where each is about one sentence long. For this task it is necessary to use audio output and be in a suitable environment (see Section 2.4.2). While listening to the audio files, which are sometimes five minutes long 40, the listener is advised to take notes. Then the multiple choice questions have to be answered. Next the second audio will be played and so forth. One cannot assume that the user has the possibility to take notes while using the application. An implementation of note-taking capabilities into the application 39 Screenshots of the Spiegel online application show about 250 words on the screen, com/store/apps/details?id=de.spiegel.android.app.spon&hl=de, last access: 2015/01/16 40 TOEFL ibt Quick Prep Examples, last access: 2015/01/16

28 Chapter 2. Background and Related Work 21 is quite problematic: it requires fast typing and has to be visible while answering the questions. The idea to take shorter audio files would completely change this task, which tests if the user can understand the general idea and details by listening to e.g. longer conversations. Therefore the screen compatibility of this task is rated as satisfying. Because of the similarity to the reading task, the other criteria can be answered accordingly: Automated correction is possible and the amount of text input is zero Related Mobile Applications In this section two related mobile applications which include a placement test will be described. For each one the most important points will be discussed and what can be learned from them. OpenLanguage English Placement Test The application has the goal to help test takers to determine which level of course they should take. The information is from the Android application, which can be downloaded from the Android store Google Play multiple choice questions with different level of difficulty have to be answered. The question consists of a sentence where one word is removed. It has four possible choices to fill the gap. This concept is very similar to a cloze test (see Chapter 2.4.3). The key difference is, that every sentence is independent. Therefore the advantage is that every question can be displayed on a new screen as no additional information is required. This could basically also be gained by using many short C- or cloze tests. The content and the context change for every question. The user has to understand this every time again, has to refocus. Therefore this could maybe require more time and effort compared to a normal cloze test. Another positive aspect is that no text input is required. So it is basically a tradeoff between no text input and less change of context. Because the VST (see Chapter 2.4.5) also requires no text input and seems superior to this test (e.g. shorter, less context to read) and on the other hand maybe some amount of text input is not perceived as inconvenient 42 this application will not be considered further. Also the interface design does not really convince, especially the small text and advertising feels stressful. 41 English Placement Test mobile application, openlanguage.placementtest_english, last access: 2015/01/29 42 Which a concept would have to evaluate

29 Chapter 2. Background and Related Work 22 Duolingo Placement Test The Duolingo Android application 43 is the source of information for this section. It is described as one of the best language learning applications and offers courses for free and in many languages. On the first usage the learner can choose whether to start as beginner or take a placement test. This test will be discussed in the following section. The placement test consists of the same elements which are used to teach the language in the main part of the application 44 : Translating sentences by tapping on text parts in the right order ( Translate this sentence ), by typing in the translation ( Type what you hear ) and by writing down dictated text. As the application provides one predefined course to learn a language, this placement test seems to take some questions from different levels. The learner answers these questions. Then it can be assessed where the test taker s level and the course level might eventually intersect. The learner can now skip the course up to this level. Because the course already has some very specific learning questions, these can be also reused for the placement test. However, it is not sure if these questions correlate to general language proficiency and to completely different course material. Because it cannot be assessed in the scope of this thesis, the approach will not be considered for our placement concept. Besides the placement method the interface design can be inspected. It feels really polished and sophisticated: Every screen has one distinct task, it is not overfilled, it always seems clear what to do and where to tap and the chosen colors and fonts look nice. In the author s opinion it might be considered as good example of the mentioned characteristics and design implications in Chapter Duolingo - Google Play, last access: 2015/01/29 44 Duolingo Review - Lingholic, last access: 2015/01/29

30 Chapter 3. Concept 3.1. Choosing a Test Table 3.1 is a summary of the discussed tests regarding their fit for the placement concept on a smart phone (see Section 2.4). The goal of this section is to compare these results and choose a test as basis for our concept. Table 3.1.: Short summary of the discussed tests regarding their fit for the placement concept on a mobile phone (see Section 2.4) C-Test Cloze Test TOEFL VST Reading Writing Listening Speaking Test good good good N.A. good N.A. satisf. Performance Automated yes yes yes no yes no yes Correction Screen good good satisf. N.A. satisf. N.A. very good Compatibility Text Input N.A. 0 N.A. 0 Amount No Audio yes yes yes N.A. no N.A. yes Time 40 min 40 min min N.A min N.A. 10 min The VST (Vocabulary Size Test) is very good in every requirement expect the test performance. On a closer look, the statistical correlation to a criterion test is 0.6 (see Section 2.4.5). Compared to the other discussed tests this is weaker. But the short test time, very good screen compatibility and no text input are several advantages. Additionally, researchers state, that this kind of test is very suited for placement purposes [59]. In summary, the VST seems to be a suitable test for a 23

31 Chapter 3. Concept 24 mobile phone placement application. To lower the potential risk of a bad correlating VST, another test will be chosen. There does not seem to be a huge disadvantage in including two different kinds in the concept. Therefore, in the case one test performance is low, the results from the other will be a good basis to iterate on the concept. To choose a second test, a closer look is necessary. The C-Test is slightly better than the cloze test: The amount of text input is lower and the correlation to a criterion is slightly better [34]. Looking at the TOEFL subtests, Reading has no need for audio compared to Listening. Consequently, Reading is slightly better. TOEFL Reading versus C-Test Now the TOEFL Reading test can be compared to the C-Test. The TOEFL Reading has no text input. The C-Test has 500 characters of text input. On the other hand the C-Test has better screen compatibility (good versus satisfying). In conclusion, two factors have to be balanced against each other: screen compatibility versus text input. The screen compatibility of the reading test can only be significantly improved by drastically reducing the amount of text to read so it can fit on one screen together with the question. This would also mean that the character of the task changes from testing the understanding of long complex texts to short excerpts. On the other hand, the text input amount of the C-Test can be lowered by reducing the number of gaps. Because 40 minutes for a mobile placement test are probably too long, this shortening might be an unavoidable constraint. In summary, it seems that the C-Test can be optimized more easily without changing the nature of the test for a good user experience. Therefore the C-Test is chosen together with the VST as a basis for this concept Test Contents The creation of test contents is a complex process: It includes writing the items using a framework, prototyping and field testing [60]. The item creation is a lot of effort and would burst the scope of this work. Additionally, after evaluating a concept based on existing test items the contents still can be changed or improved. Therefore this thesis will use existing contents as a starting point for the concept.

32 Chapter 3. Concept Source of VST Items The first section of the DIALANG is a Vocabulary Size Test (VST). The result is used as an initial placement. Based on this the difficulty of the other sections are adapted. The test consists of 75 Yes/No items and was examined in detail in Section The DIALANG VST was also validated for German [41] and can therefore be used as a very solid basis for our concept. The 75 test items in Appendix A.1 were retrieved from the DIALANG online system Amount of C-Test Items An average C-Test with 20 to 25 gaps will take up to 5 minutes [43]. Four C-Tests would take at maximum 20 minutes. Considering that our concept has also VST items it seems plausible to take an upper maximum of 20 minutes for this part. This assumption will be evaluated with the help of the user experience feedback (see Chapter 4). In summary the concept will consist of four C-Tests Source of C-Test Items Two German C-Tests (see Appendix A.1) with an average level of difficulty are from Krekeler [61]. In his experience they proved to be suitable because they could differentiate the test taker s level distinctly. Because of the restricted amount of space on a mobile phone, only the shorter test of the two will be used. Another resource is the ondaf-website. Eckes developed several C-Test items for the ondaf-test of which four easier items can be found on the corresponding website 2 [41]. Three of them (see Appendix A.2) will be used in our concept Paper Prototyping In order to start developing a concept and get first feedback as soon as possible, a paper prototype was designed. Table 3.2 summarizes the seven screens of the first planned prototype. These screens were designed with the help of the online wireframing software Balsamiq. It allows effective and fast creation of interface drafts which can be printed on paper 3. Appendix B.1 shows the resulting screens which were used as starting point to iterate on the concept. In order to finish the first iteration feedback needed to be collected. The goal was to get answers for the questions in Table 3.3 and identify other potential interaction problems. 1 DIALANG VST items, last access: 2015/01/19 2 ondaf, last access: 2015/01/20 3 Balsamiq, last access: 2015/01/20

33 Chapter 3. Concept 26 Table 3.2.: A summary of the planned application screens for the first paper prototype Screen Name Description 01: Loading Screen Title of concept which will be shown while the application is loading 02: Introduction A short introduction of the application for the user 03: Vocabulary Instructions and items of the first vocabulary size test (VST) including submit button 04: C-Test Instructions and first C-Test 05: C-Test Keyboard C-Test screen when a text input field is selected and a keyboard displayed 06: C-Test Final C-Test completely filled out and submit button visible 07: Last Screen Thank you message for finishing the placement test Figure 3.1.: The Vocabulary Screen and C-Test Keyboard Screen of the first paper prototype to get initial feedback Four persons tried the paper prototype while thinking aloud and stating their experience. Additionally, their interaction with the prototype was observed and notes were taken. The most important points of the results (see Appendix B.2) which require changes are summarized in Table 3.4. All the other questions (see Table 3.3) can be, according to the small sample set, answered positive. Especially, the testers knew what they are expected to do, where they can tap and how to enter a solution. This first feedback allowed to refine the prototype in a second

34 Chapter 3. Concept 27 version. Because of the positive results the second version is an interactive application displayed on a phone to test the concept in more detail. Table 3.3.: Questions for the first paper prototype and how to answer them Question Measurement Do users know where they can tap? Observe interactions Number of questions for help Do users know what they are expected to do? Observe interactions Do users know how they can enter their solution? Observe interactions Question: How can you enter your solution? Is the behavior of the prototype expected by the users? Question: What do you expect to happen after the tap? Table 3.4.: Excerpt from the most important feedback for the first paper prototype which require changes Feedback Potential Solution Name Linudo perceived as strange Change name to Shnay (similar to nude ) Expected result of answers (right or wrong) Add screen at the end to show result/answers Screen 03: tap on word button Indicate button as marked, e.g. change the color Button to exit test Implement a button on every screen Screen 04: Submit solution with empty gaps Submit button must be visible all the time Interview introduction tells user Change second screen to be able to to take placement test, which choose a placement test so the user is also explained in second screen knows what is going to happen 3.4. Mobile Phone Prototyping A concept implementation which can be used on the phone can better explore the user experience. It is more similar to the final product. Especially as the concept uses text input, the effects must be investigated in more detail (see Chapter 2.3.3). A phone application prototype enables that because the user can actually enter text by typing on the keyboard. Furthermore, all learnings from the paper prototype (see Table 3.4) were integrated and tested. A short overview of the important questions: Can user enter solutions in the gaps? Can user do the full test without further help? Are there any other unexpected interface or test problems?

35 Chapter 3. Concept 28 What do the users like or dislike? The prototype on the phone does not require to be a final product, meaning a native phone application. In order to further evaluate the user experience and gather user data, a preferably fast implementation method was required. In this work website technology (HTML5/CSS3/JavaScript) was chosen to implement the prototype. Two excerpts of the finished screens can be seen in Figure 3.2. One can see the difference and progress in comparison to the paper prototype (see Figure 3.1): Straight lines and colors are used and the screen elements are interactive. The prototype is working, meaning one can enter the solutions, click on Submit and then see the result. Figure 3.2.: The interactive vocabulary screen (left) and C-Test keyboard screen (right) of the second prototype on a phone to gather further feedback This second prototype was used to gather qualitative feedback again. Short iteration cycles are very helpful for fast user interface development. One can see which elements are not working before a lot of energy is spent to make them perfect. The point of view of a tester can help to detect potential problems earlier and fix them [62]. Two persons were given the prototype on a phone while thinking aloud and given them instructions according to Buxton [63]. Their interaction was observed and their comments written down (see Appendix B.4). It is important to note that both users could enter the solutions and finish the whole test without any help. Based on this qualitative analysis one fundamental goal of this app, essential usability, seemed to be

36 Chapter 3. Concept 29 achieved. Besides from that, a detailed further evaluation of the feedback in Table 3.5 showed several points which required changes. Those were implemented in the third prototype. The same technology, a website, was used because it proofed to work as excepted: Interface changes can be implemented fast and reliable. Furthermore it supports the collection of realistic user feedback and the prototype does not feel extremely slow compared to native applications on the phone. Table 3.5.: Second Prototype: Summary of the tester feedback as final step towards the third prototype Feedback Potential Solution Where can I see how many more Change progress bar to have questions I have to answer? several distinct steps, one for each test; Add a short animation to make the change of progress more visible Result Screen: What is Show level and short description the test taker s level? e.g. Level 6: Native Speaker Two or more tries to tap Enlarge gaps visually or the interactive the right area to select a gap area around a gap add button to focus next gap Figure 3.3 shows two screens of the third prototype. The changes according to Table 3.5 have been implemented. Furthermore, the Finished-Screen was combined with the Result-Screen as one tap can be removed to make the interaction faster. This third prototype (see Appendix B.5) was used to conduct a more detailed evaluation.

37 Chapter 3. Concept 30 Figure 3.3.: The interactive vocabulary screen (left) and C-Test keyboard screen (right) of the third prototype, which will be used for the evaluation

38 Chapter 4. Evaluation The focus of this chapter is on evaluating the concept. The first section will develop the research questions concerning functionality and user experience. Afterwards, the methodology and test execution will be described. The last section will discuss the results and interpret them Research Questions The evaluation will be used to answer the central question of this thesis (see Section 1.2): How could a concept of a language test on a mobile phone which can indicate the test taker s general German language proficiency look like? Therefore, one important question to answer is whether this concept can be used as a placement test. More precisely, how much does the test correlate with a criterion test? To estimate the overall test performance, it is important to know how long users need to complete the test. Furthermore, the user experience is of interest. Especially, how do users think about the amount of time it takes to complete the test? Which sub-test did they prefer? Do they think that the placement test is valuable? In addition, do test takers have any interaction problems? In summary the questions are: How much does the test correlate with a criterion test? How much time do users need to finish the test? Is the user interface easy to understand? Which sub-test do users prefer? How is the keyboard input perceived? Which interaction problems do user encounter? 31

39 Chapter 4. Evaluation 32 What are the strengths and weaknesses of the current implementation? 4.2. Methodology and Test Setup The study will use within-subjects design: Every participant will perform the same tasks. To collect test data a questionnaire will be used. Questions which rate a statement will use a Likert scale from Strongly disagree, Disagree, Neither agree nor disagree, Agree to Strongly agree. Open questions will require the participants to write own answers. Further test data will be obtained from the mobile phone. The placement test prototype records every interaction (e.g. tap on a button) and stores this data for further processing. The hardware setup includes a mobile phone 1. On the phone the placement test will already be opened, displaying the first screen. The test supervisor will give the phone to the participant. The supervisor will observe the participant and take notes of extraordinary occurrences (e.g. participant fails to focus a gap five times in a row) and questions. In case the result of the DIALANG criterion test is needed, the participant will be given a laptop which displays the introduction of the test. The result will be stored on the laptop using a screen recorder Test Execution In the beginning the participants had to read an introduction. Then they were given the mobile phone. The supervisor observed them and took notes. After they finished the placement test on the phone, they were given the questionnaire (see Appendix C.1). They had to answer all questions regarding their experience with the prototype. The second part of the questionnaire tried to get the overall impression of the participants. They were asked to name strengths and weaknesses of the placement test. Finally, they were asked if they took a criterion test in the last two years. If not, they would take the DIALANG test on a laptop. In order to not exceed the scope of this evaluation, accepted criterion tests have been the Test Deutsch als Fremdsprache (TestDaF) and final exams of the Technische Universität München language courses Results and Interpretation The results are presented in a similar structure and order like in the questionnaire. For better understanding and in order to be able to follow the interpretations more easily, they are presented 1 Xiaomi Redmi 1S

40 Chapter 4. Evaluation 33 in the context of the results. Accordingly, at first the results are shown and directly afterwards their interpretation is given Demography and Mobile Phone Experience In total, 30 participants took part in the evaluation and completed the test. Out of the 30 participants, 14 were male and 16 were female. Therefore, the gender is distributed almost equally. The test takers average age was 24.1 years. Fig 4.1 shows the age distribution of the subjects. The youngest participant s age was 20 years, the oldest participant s age was 33 years. Almost all of the subjects were students (28 out of 30) and learning German as a foreign language (28 out of 30). Two participants were speaking German as a first language. 8 7 Number of Participants Age Figure 4.1.: Age distribution of the participants in the user test Another important question is the experience of the subjects with mobile phones. All of the participants regularly use smart phones. 15 participants own and regularly use an Apple iphone, 14 an Android phone and one participant owns another phone. It can be noted that the type of phone is almost equally distributed between iphone and Android. To conclude, the participants seem to be used to utilize applications and modern phones in their everyday life. In summary, the majority of participants were part of the target group (see Section 1.2): They regularly use smart phones, learn German as a foreign language and it can be assumed that they possibly use an application on their phone to learn German as a foreign language. Therefore it is expected that the results are a valid representation of the target audience.

41 Chapter 4. Evaluation Usability At first it should be stated that 28 out of 30 participants could successfully finish the given task on the phone without any further help. Therefore, the application concept fulfills basic usability requirements and achieves one part of the problem statement: The aim that the majority can use the concept without further help (see Section 1.2). The two participants who had problems will be discussed in Section For a more detailed evaluation, the gathered answers from the questionnaire in Table 4.1 will be used. Table 4.1.: Summary of statements concerning the general usability of the application. The possible ratings were Strongly agree (SA), Agree (A), Neither agree nor disagree (N), Disagree (D) to Strongly disagree (SD). # Statement SA A N D SD a.1 I found the application unnecessarily complex a.2 I thought the application was easy to use a.3 I think the application is graphically appealing a.4 I felt very confident using the application a.5 I found easily the button to submit my solution a.6 I felt always sure where to click a.7 I thought the phone was unfamiliar to use The vast majority of the subjects thought that the application was easy to use (statement #a.2). Nobody disagreed with that statement. Furthermore, the majority indicated that they felt confident using the application and were sure where to click (#a.4 and #a.6). This is also supported by the fact, that the vast majority found the submit button easily (#a.5). Moreover, the vast majority disagreed that the application was unnecessarily complex (#a.1). Additionally, a large majority stated that the phone was not unfamiliar to use (#a.7). Therefore the specific phone model used as test hardware does not seem to have negative influence on the test results. Furthermore, the answers to the open question what they liked in the application counts many positive items regarding the usage. Subjects stated, among others, that it was really easy to use and straightforward (...) (see Appendix C.2).

42 Chapter 4. Evaluation 35 In summary, the participants stated that the application was easy to use. Thus, general usability of the application is given. However, there is still room for improvement to get even better feedback. Especially, the graphical appearance was not appealing to 11 subjects (#a.3). As a consequence, the graphical presentation should be improved in future versions Test Time On average, it took the participants 2.2 minutes to finish the VST and 10.6 minutes to finish the C-Test. In the questionnaire they were asked to indicate their thoughts about the duration. Table 4.2 shows the summary of the statements. Table 4.2.: Summary of statements concerning the length of the test. The possible ratings were Strongly agree (SA), Agree (A), Neither agree nor disagree (N), Disagree (D) to Strongly disagree (SD). # Statement SA A N D SD b.1 Overall, I am satisfied with the amount of time to finish the test b.2 I found the vocabulary test had too many questions b.3 I thought the gap test had too many gaps to fill in b.4 I felt the time to finish the vocabulary test was too long b.5 I found the time to finish the gap test was too long Overall, all participants agreed that the test time was appropriate or were neutral (#b.1). On the other hand, for the specific test parts there was no clear preference in the answers. The statement that the VST had too many questions was negated by half of the subjects (#b.2). Similarly, less than half disagreed with the statement, that the VST was too long (#b.4). Only three participants rated clearly different between the two statements 2. Thus, this could indicate that the length of the VST both in time and number of questions was indeed perceived as too long. The second part, the C-Test, was rated very similar. Less than half disagreed that the duration was too long (#b.5). About half disagreed that there were too many gaps to fill in (#b.3). Only four participants rated clearly different between the two statements. Therefore, many subjects 2 The difference in rating of a single subject between statements #b.2 and #b.4 was more than one.

43 Chapter 4. Evaluation 36 seem to implicitly agree. Hence, the C-Test might also be perceived as too long both in time and number of questions. In summary, this allows three conclusions: The general statement (#b.1) was not rated honestly because both of the individual tests were perceived as too long (time and number of questions) and therefore both tests are too long. One possible solution might be to shorten both tests. A further possibility is that overall the time was okay (#b.1), both tests were considered too long (time and number of questions) and the participants might prefer more variety. This could for example be taken into account by adding another kind of question. This was proposed by four participants in the open questions and is a good option for further development. The third possible interpretation, which is that individual subjects only perceived one test as too long (time and number of questions) but were overall satisfied with the time, could also be true: Seven subjects clearly 3 indicated that only the C-Test was too long 4. One person clearly stated, that only the number of questions of the VST was too much. In summary, it seems that primarily the C-Test is perceived as too long and could be considered to be shortened Overall Perceived Value In this section, the overall perceived value of the application by the subjects is analyzed. Table 4.3 summarizes the results. Table 4.3.: Summary of statements concerning the value of the test. The possible ratings were Strongly agree (SA), Agree (A), Neither agree nor disagree (N), Disagree (D) to Strongly disagree (SD). # Statement SA A N D SD c.1 Overall, I would take the test again to get better learning material c.2 I think the placement test is useful in a learning application The statement #c.1 was agreed by the vast majority of participants. Even more positive, all of the participants agreed that the prototype is useful in a learning application (#c.2). To sum up, the subjects were very positive regarding the potential value of such a placement test in an application. 3 The difference in the rating of a single subject between question #b.2 and #b.3 respectively #b.4 and #b.5 was more than one. 4 Three subjects indicated both in time and number of questions, two indicated only too many questions and two other indicated in the comments that it is too long.

44 Chapter 4. Evaluation Preferred Test Part This section will present the results which part the participants preferred: VST or C-Test. Table 4.4 sums up the result of the questionnaire. Table 4.4.: Summary of statements concerning the preferred part of the test. The possible ratings were gap test (VST) or vocabulary test (C-Test). # Statement VST C-Test b.6 Which part did you prefer? The result is equal and no real preference neither for the VST nor for the C-Test is identifiable in the sum of the answers (#b.6). Table 4.5.: Summary of statements concerning the difficulty of the test. The possible ratings were Strongly agree (SA), Agree (A), Neither agree nor disagree (N), Disagree (D) to Strongly disagree (SD). # Statement SA A N D SD b.7 I found it easy to find a solution for the gaps b.8 I felt very confident using the keyboard to fill in the gaps b.9 I found the vocabulary questions complicated Also the subjects thoughts about the difficulty, as can be seen in Table 4.5, are overall quite similar. 12 participants stated that the C-Test was easy whereas five disagreed (#b.7). Even more balanced was the answer for the statement, that the VST was complicated (#b.9): 11 participants agreed and 12 disagreed. The question does not clearly specify whether it meant the difficulty of the question or the usage. However, it seems that some subjects found the questions difficult. Furthermore, some could only answer a very low percentage correctly (see Section 4.4.7) what was actually intended in the first place. The answers to the open question about possible improvements (see Appendix C.2) provided further insight: Five participants said that the VST and/or C-Test was really difficult. One participant commented that every placement test I took was very intimidating [including this]. Another one proposed to add (...) vocabulary tests on different levels. To sum up, a future version could maybe provide a less intimidating experience by e.g. using questions adapted to the test taker s language level. Section 3.1 tried to minimize the amount of characters which the test takers have to type. As

45 Chapter 4. Evaluation 38 Table 4.5 shows, the vast majority of subjects were very confident using the keyboard (#b.8). Therefore, the keyboard usage does not seem to be a problem. Still, one participant noted that the C-Test was too much to type (see Appendix C.2). Additionally, three participants noted in the open questions that they preferred the VST because it was easier to use (see Appendix C.2), which confirms earlier assumptions (see Section 3.1). On the other hand, also the VST had some issues: One participant commented, that the vocabulary question was strange and there were too many words on [the] screen. Furthermore, two subjects did not understand the VST in the correct way: They thought they should choose only words (...) with the meaning exactly (see Appendix C.2). Thus it seems very important to consider to e.g. change the task description in a future version. Another idea would be to change the task completely to e.g. Choose the real German words. To conclude this section, there is no clear distinguishable preference. It seems that the VST is better especially, in terms of usability, than the C-Test. On the other hand, the C-Test was sometimes preferred despite its usability Open Questions and Observations This Section will only describe the notes and comments which have not been discussed yet. The full list of responses can be found in the Appendix C.2. Four subjects noted that they would like to see a correction to know exactly what they did right and wrong. In the author s opinion, this seems a reasonable idea to consider. It could potentially improve the user experience as they directly retrieve more value out of answering questions. By using the feedback they can learn and improve their language knowledge. In addition, two technical problems were found during the evaluation. It was observed and/or noted seven times, that scrolling to the top in the C-Test was not possible. Moreover, it was observed five times that participants had to click twice on the submit button of the C-Test to continue. They tapped once, waited for about one second and then realized that nothing had happened. Then they tapped again and the next test showed up. Both issues should be solved in an improved version to guarantee better user experience. Adding elements of gamification could further enhance the test takers motivation [64, 65] Proficiency Correlation This section will answer the question how much the test correlates with a criterion test. Both test parts will be analyzed in respect of correlation. As stated in Section 4.3, accepted criterion tests were, among others, Test Deutsch als Fremdsprache and final exams of the Technische Universität

46 Chapter 4. Evaluation 39 München language courses. As every participant has taken one of the accepted criterion tests, there was no need to conduct a DIALANG test. Score Calculation The score of the C-Test is the percentage of the correctly filled gaps (see Equation 4.1). The calculation for the VST result is more difficult. Since 25 of the 75 items are wrong, not selected by default and therefore without further change automatically correct, they are not included in the score. To be more precise, 25 items are subtracted from the correct ones, as Equation 4.2 shows. p C T EST = n correct 78 p V ST = n correct (4.1) (4.2) Correlation and Linear Regression The following paragraphs will analyze the results and try to fit a linear regression model. To facilitate the calculations, the criterion test results, which are using the Common European Framework of Reference for Languages (CEFR), were mapped to numbers: The result is a scale from one ( A1 ) to six ( C2 ). Table 4.6.: Quantity and variance of VST and C-Test grouped by the criterion test result. Criterion Test Quantity VST Variance C-Test Variance The first step is to test for homoscedasticity (lit. same variance): For every criterion test level the variance of all the C-Test and VST results was calculated. Table 4.6 shows that the variances are not consistently rising. However, the variances of level four and five are quite large, especially for the C-Test.

47 Chapter 4. Evaluation VST Percent C-Test Percent Criterion Test Criterion Test Figure 4.2.: Boxplot of VST results in Figure 4.3.: Boxplot of C-Test results in percent grouped by criterion percent grouped by criterion test test In order to figure out a possible reason, two boxplots for the C-Test and VST results were created. The VST boxplot in Fig. 4.2 shows two potential outliers in level four and five (red cross). Whereas the C-Test boxplot in Fig. 4.3 does not show any extreme outlier C-Test C-Test Regression VST VST Regression Percent Criterion Test Figure 4.4.: The results of the C-Test and VST including simple linear regressions versus the criterion test.

48 Chapter 4. Evaluation 41 The scatter plot of the C-Test and VST results including both simple linear regressions in Fig. 4.4 shows the two outliers of the C-Test results again: They are marked with a black circle. The questionnaire for these two subjects contained an interesting note: Both did not understand the task description of the VST. They thought they should only select words which mean exactly in German. As a result they have not selected almost anything. Therefore the percentage of their VST result was very low. In order to not disturb the correlation analysis by this issue, those participants were removed from the regression analysis. Table 4.7.: Quantity and variance of VST and C-Test grouped by the criterion test result without the two outliers. Criterion Test Quantity VST Variance C-Test Variance Table 4.7 shows the re-calculated variance of each group after the two outliers have been discarded. There are still differences in the variance, which are most likely caused by the small number of subjects per group. However, they are not constantly rising with the criterion test level. Therefore, we will assume that homoscedasticity is given, which allows us to continue with the linear regression analysis. The scatter plot in Fig. 4.5 shows the results of the C-Test and VST including their simple linear regression against the criterion test. The highlighted areas mark the 95% confidence intervals of each regression line. The confidence interval of the C-Test is smaller than the VST interval. The VST confidence interval is broader and more formed like a cone, opening with rising values. A reason could be the larger variance of the results in level four, especially one possible outlier at the top. This possibility could be analyzed in a further evaluation with more subjects. To get a better understanding of the quality of the simple linear regressions, the determination coefficients were calculated using Pearson s correlation coefficient (r). The determination coefficient of the VST regression is r 2 = 0.87 and of the C-Test regression is r 2 = Both are significant at α = p avg = a C T EST + a V ST 2 L criterion + b C T EST + b V ST 2 = L criterion 3.35 (4.3) L criterion,est = round( p avg ) (4.4)

49 Chapter 4. Evaluation C-Test C-Test Regression VST VST Regression Percent Criterion Test Figure 4.5.: The results of the C-Test and VST including a simple linear regression against the criterion test, where the two outliers have been removed. The highlighted areas are the 95% confidence intervals of each regression. In summary, in the author s personal opinion, the correlation turned out to be really positive. In addition, the slopes of the linear regressions (C-Test and VST) are very similar. These resulting regression functions allow to map the test results to an according criterion test level (see Equation 4.3 and Equation 4.4) and thereby to the more commonly used Common European Framework of Reference for Languages (CEFR). Using the test subjects and Equation 4.4, the rate of correctly indicated CEFR is 79%. According to the evaluation the used concept can indicate the test taker s general German language proficiency with a certain probability. This answers the central research question in Section 1.2.

50 Chapter 5. Conclusion The aim of this thesis was to find a concept of a placement test on mobile phones for German as a foreign language. The final prototype was highly correlating with a criterion test and provides insight on how a mobile placement test might look like. In the beginning, Chapter 1 developed a clear statement of the problem and defined the solution space. Important background knowledge was given in Chapter 2. It outlined findings in user experience for mobile phones. This knowledge allowed to create a usable concept. Additionally, based on the developed criteria several existing language tests and existing applications were discussed. Using these findings, two existing test concepts and contents were chosen. In addition, the iterative process of the prototype development was described. It included findings from several small user tests and lead to a promising prototype. The evaluation with a small user group validated the concept regarding the requirements and usability. Moreover, the findings of a linear regression model and the significant correlation to a criterion test was presented. These findings allowed the conclusion that the placement concept can in fact indicate the test taker s German proficiency. As the scope of this thesis was limited to find and validate a concept, further work and research is needed. In the author s opinion this especially includes the significant reduction of the test length, an improved first part (sharpened description or new task), adaption of the questions to the user s level and in-place feedback with the correct answer for each mistake. The user experience and overall value of the placement test must be extraordinary to clearly outweigh the effort which is needed to take it. Traditionally many placements tests are required by universities, employers and language courses. That is why the users have no choice but to take them. Hence, research most of all focuses on the test performance and not on user experience. However, in the rising field of mobile learning, users can easily quit the application. This is where the findings of this thesis provide a first step towards an improved learning experience. By combining the fields of ubiquitous computing, user experience and applied linguistics, improved language acquisition applications will one day hopefully provide the perfect individual teacher for the maximum learning outcome. 43

51 Appendix A. Test Items A.1. VST Items Full list of vocabulary size test items, Source: DIALANG 1 : schwören, abkratzen, vereinlichen, zurückstecken, erdulden, seitern, schuttern, schildieren, festlegen, umstellen, halbieren, sträubern, aggressieren, bauen, kleben, probieren, bremsen, glasieren, stillen, hineinbekommen, pöhlen, fliehen, zermalmen, bemollen, schmieren, verhutzeln, einarbeiten, zerdrücken, herauspasten, verfeinern, eindräuen, erkrellen, wählen, dreien, hochjagen, herstellen, binden, herausfinden, einpfauen, verderben, krönen, vertrenken, erinnern, abstitzen, leisten, schweiken, schlopfen, geschehen, ausblintern, entfremden, orientieren, chiffrieren, lecken, mögen, steuern, leihen, heraufsetzen, hinstürzen, beherrschen, hineinstopfen, umhaupten, zählen, plaschen, umrahmen, entklupfern, entlaben, gemaunen, mindern, vergessen, hinhauen, ansprechen, fordern, niederkämpfen, mitlaufen, verballen A.2. C-Test Items 1 Source: DIALANG VST items, last access: 2015/01/19 2 ondaf, last access: 2015/01/20 44

52 Appendix A. Test Items 45 Bedeutung der Lesefähigkeit Die Lesefähigkeit trägt ihren Wert natürlich in sich, hat aber auch ökonomische Auswirkungen. Erwac Leser, d besser le als d Durchschnitt, üb mit größ Wahrscheinlichkeit gutbe Berufe a. Die wach Spezialisierung i der Gesell erfordert me Bildung, ei Forderung, d vor al an d Schulen geri wird. Du die erhö Anforderungen a das Bildungsniveau, die heute in den westlichen Gesellschaften gestellt werden, ist die Lesefähigkeit des Einzelnen immer wichtiger geworden. Naturkatastrophen Neben den plötzlich auftretenden Naturkatastrophen gibt es natürliche Risiken, die kontinuierlich vorhanden und schwer erkennbar sind: etwa die natürlich vorkommende Radioaktivität oder natürliche toxische Metallvorkommen in der Umwelt. Zu können ein natürliche Ris durch d Eingriffe d Menschen verschlimmert wer : etwa Überschw aufgrund d Zerstörung v Wäldern. F die Erfor dieser Gefa sind des die grundl Erkenntnisse d Umweltwissenschaften v zentraler Bede. Die schwe Risiken du Naturkatastrophen best in den wirtschaftlich noch wenig entwickelten Staaten. Dies liegt teils an den klimatischen Bedingungen der Tropen, teils an der Lage innerhalb geologischer Schwäche- oder Gefahrenzonen und schließlich an der noch gering ausgebauten Infrastruktur bezüglich Schutzmaßnahmen für Mensch und Umwelt. Figure A.1.: Two C-Tests with average level of difficulty, Source: [61]

53 Appendix A. Test Items 46 Fragen zur Berufswahl Alte Berufe verschwinden, neue kommen hinzu: Bei d Berufswahl ha Schüler im wieder Fra oder Prob. Denn e_ gibt ei große Anz sehr versch Berufe, u es i nicht ein, die rich Wahl z_ treffen. D berufliche Zuk sollte m rechtzeitig pla. Dabei ka es sinn sein, sich beim Arbeitsamt beraten zu lassen. Manchmal hilft auch ein Test zu den persönlichen Berufsinteressen. Gesunde Ernährung In mehreren europäischen Ländern haben sich die Ernährungsgewohnheiten der Menschen in den letzten Jahren verändert. Ein aktu Bericht zei, dass d Europäer heu mehr a Ernährung u Gesundheit ach. Sie es mehr Ob und Gem, mehr Fi und vi weniger Flei. Obwohl man Speisen im noch ziem süß, fet oder sal sind, i ein kla Trend zu gesünderer Ernährung erkennbar. Vermutlich wird sich dieser Trend noch fortsetzen. Geschicktes Zeitmanagement Zeitmanagement: In den Ohren vieler Studenten klingt dieses Wort eher abschreckend. Dabei kön gerade ei gute Pla viele ne Freiräume scha. Mit Hi eines gen Arbeitsplans ka man näm die fr Zeit oh schlechtes Gewi genießen, u der unange Lernstress v den Prüf wäre vor. Die Lös ist ga einfach: Sei Arbeit systematisch in Zeitblöcke aufteilen. Auch Pausen sollten im Arbeitsplan aufgeführt werden. Figure A.2.: Three C-Tests with easy to average level of difficulty, Source: ondaf 2

54 Appendix B. Prototypes and Feedback B.1. Paper Prototype All screens of the first paper prototype have been created with a tool called Balsamiq 1 : Figure B.1.: First and second screen of the paper prototype 1 Balsamiq, last access: 2015/01/20 47

55 Appendix B. Prototypes and Feedback 48 Figure B.2.: Third and fourth screen of the paper prototype Figure B.3.: Fifth and sixth screen of the paper prototype

56 Appendix B. Prototypes and Feedback 49 Figure B.4.: Seventh screen of the paper prototype

57 Appendix B. Prototypes and Feedback 50 B.2. Paper Prototype Feedback Notes of Paper Prototype Interview: Person 1 Name: Dominik Student age Owns an android phone Introduction: Imagine to open an app the first time to study German. You are asked to take a placement, you tap on start. Screen 01 Observation: Tapped on center to continue Screen 02 Observation: Tapped on button to continue Screen 03 Observation: Tapped on each word which he did know; finally tapped on submit; noted that he was not sure whether dreien is a valid word in German or not How can you enter your solution? Tapping on the word What do you expect to happen after the tap? He expects that the button changes the color or something similar Screen 04 Observation: Tapped on gap How can you enter your solution? Tapping on the gap What do you expect to happen after the tap? That a keyboard will be displayed Screen 05 Observation: Screen was as expected How can you submit your solution? Fill all gaps, then automatically done or maybe button will be displayed Screen 06 Observation: Tap on submit button Screen 07 No comment Number of questions for help: 0 Notes of Paper Prototype Interview: Person 2 Name: Alexander Student age Owns an android phone

58 Appendix B. Prototypes and Feedback 51 Introduction: Imagine to open an app the first time to study German. You are asked to take a placement, you tap on start. Screen 01 Observation: Waited first, then tapped on center to continue Screen 02 Observation: Tapped on button to continue Screen 03 Observation: Tapped on each word which he did know; then tapped on submit How can you enter your solution? Tapping on the word What do you expect to happen after the tap? He expects to see a check box checked or something similar Screen 04 Observation: Tapped on gap while stating solution aloud How can you enter your solution? Tapping on the gap What do you expect to happen after the tap? That a keyboard will be displayed Screen 05 Observation: Screen was as expected How can you submit your solution? Submit button expected after finishing all gaps or at the bottom Screen 06 Observation: Tap on submit button Screen 07 No comment Number of questions for help: 0 Notes of Paper Prototype Interview: Person 3 Name: Anna Student age Owns an android phone Introduction: Imagine to open an app the first time to study German. You are asked to take a placement, you tap on start. Screen 01 Observation: Noted that she did not like the name ( sounds like nude ); tapped on center to continue Screen 02 Observation: Tapped on button to continue Screen 03

59 Appendix B. Prototypes and Feedback 52 Observation: Tapped on every word which she did know; then tapped on submit button How can you enter your solution? Tapping on the word What do you expect to happen after the tap? She expects to see whether her answer was right or wrong Screen 04 Observation: Noted that fill in the gaps would be enough explanation; tapped on gap while stating solution aloud How can you enter your solution? Tapping on the gap What do you expect to happen after the tap? That a keyboard will be displayed Screen 05 Observation: Screen was as expected How can you submit your solution? Submit button expected at the end of the test or after filling in all gaps Screen 06 Observation: Tap on submit button Screen 07 Note: She expected to see her result (what was right/wrong) Number of questions for help: 0 Notes of Paper Prototype Interview: Person 4 Name: Ode Student age Owns an android phone Introduction: Imagine to open an app the first time to study German. You are asked to take a placement, you tap on start. Screen 01 Observation: Tapped on center to continue Screen 02 Observation: Tapped on button Take Example Test Screen 03 Observation: Tapped on each word which she did know (two she did not know); Finally tapped on submit; She suggested a stop test button to exit the placement test How can you enter your solution? Tapping on the word What do you expect to happen after the tap? She expects to see a pop up or overlay which indicates if answer is correct or that it was tapped Screen 04 Observation: Tapped on first gap

60 Appendix B. Prototypes and Feedback 53 How can you enter your solution? Tapping on the gap What do you expect to happen after the tap? She expected that a keyboard will be displayed Screen 05 Observation: Screen was as expected How can you submit your solution? Fill all gaps, then it will be submitted; She noted, that maybe one does not know all solutions and therefore can not enter every gap: How can one then submit the solutions?! Screen 06 Observation: Tap on submit button Screen 07 Note: She asked, how to get the result and proposed a statistic of right/wrong answers. Number of questions for help: 0 B.3. Second Prototype All screen of the second prototype: Figure B.5.: First and second screen of the second prototype

61 Appendix B. Prototypes and Feedback 54 Figure B.6.: Third and fourth screen of the second prototype Figure B.7.: Fifth and sixth screen of the second prototype

62 Appendix B. Prototypes and Feedback 55 Figure B.8.: Seventh and eighth screen of the second prototype; The Details button on the right screen is there to investigate potential technical problems

63 Appendix B. Prototypes and Feedback 56 B.4. Feedback for Second Prototype Notes of Prototype 02 Interview: Person 1 Name: Lukas Student age Owns an Apple iphone Introduction: You help us to test an application prototype. We want to find potential problems with the product. If you have trouble, it is the product s fault. We want to test the product and not you. Here is the phone with the application. Please think aloud while using it. Tell me what you think, what your problems are and what you find confusing. Now, imagine you just downloaded the application and open it the first time to study German. You already know some German and are asked to take a placement. Screen 01 Observation: Tapped on Take Placement Test Screen 02 to Screen 05 Observation: Read the words, waited for some seconds and then tapped on the first word. Tapped on some more, then Submit Screen 06 Observation: Tapped on gap and used keyboard to enter the characters, tapped on all others and entered words; Sometimes he needed two or three tries to tap the right area to select the gap; Finally tapped on Submit Question: How many more tests do I have to fill out? Screen 07 Observation: Tapped on first See Result Screen 08 Question: What does mean this now? How do I know my level now? What can be improved?: The test is quite long. Maybe less questions. What did the blue progress show?: The remaining time, according to the timer on top. Number of questions for help: 0 General Summary Can user enter solutions in the gaps? Yes Can user do the full test without further help? Yes

64 Appendix B. Prototypes and Feedback 57 Notes of Prototype 02 Interview: Person 2 Name: Ali Student age Owns an Apple iphone Introduction: You help us to test an application prototype. We want to find potential problems with the product. If you have trouble, it is the product s fault. We want to test the product and not you. Here is the phone with the application. Please think aloud while using it. Tell me what you think, what your problems are and what you find confusing. Now, imagine you just downloaded the application application and open it the first time to study German. You already know some German and are asked to take a placement. Screen 01 Observation: Tapped on Take Placement Test Screen 02 to Screen 05 Observation: Read the words, waited for about one second in the beginning, and then tapped on the first word. Tapped on some more, then Submit Screen 06 Observation: Short waiting (half second?); Tapped on gap and used keyboard to enter the characters, tapped on all others and entered words; Often she needed two or three tries to tap the right area to select the gap; Finally tapped on Submit Screen 07 Observation: Tapped on See Result Screen 08 Question: So how do I know how good I was? And where Can I see the mistakes? What can be improved?: Sometimes it was hard to tap on gap. But maybe because it was not my phone. What did the blue progress show?: I thought it shows the remaining time, how much time I have left in this task. Number of questions for help: 0 General Summary Can user enter solutions in the gaps? Yes Can user do the full test without further help? Yes

65 Appendix B. Prototypes and Feedback 58 B.5. Third Prototype All screenshots of the third prototype: Figure B.9.: First and second screen of the third prototype Figure B.10.: Third and fourth screen of the third prototype

66 Appendix B. Prototypes and Feedback 59 Figure B.11.: Fifth and sixth screen of the third prototype Figure B.12.: Seventh and eighth screen of the third prototype

67 Appendix B. Prototypes and Feedback 60 Figure B.13.: Ninth and tenth screen of the third prototype; The Details button on the right screen is there to investigate potential technical problems

68 Appendix C. Evaluation C.1. Questionnaire 61

69 Appendix C. Evaluation 62 Participant Questionnaire Incremental Number Page 1/3 Participant Questionnaire Feedback on your experience on the application is extremely valuable and will help us to make improvements. Please rate your agreement with these statements. How old are you? male female Do you regularly use a smart phone? yes no Which kind of smart phone do you own? ios Android other Please mark chosen answer with an X Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree a. general 1. I found the application unnecessarily complex 2. I thought the application was easy to use 3. I think the application is graphically appealing 4. I felt very confident using the application 5. I found easily the button to submit my solution 6. I felt always sure where to click 7. I thought the phone was unfamiliar to use b. test parts 1. Overall, I am satisfied with the amount of time to finish the test 2. I found the vocabulary test had too many questions 3. I thought the gap test had too many gaps to fill in 4. I felt the time to finish the vocabulary test was too long 5. I found the time to finish the gap test was too long 6. Which part did you prefer? vocabulary test gap test 7. I found it easy to find a solution for the gaps 8. I felt very confident using the keyboard to fill in the gaps 9. I found the vocabulary questions complicated Figure C.1.: The questionnaire used in the evaluation (page one)

70 Appendix C. Evaluation 63 Participant Questionnaire Incremental Number Page 2/3 Please mark chosen answer with an X Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree c. overall 1. Overall, I would take the test again to get better learning material 2. I think the placement test is useful in a learning application d. What did you like in the application? Please feel free to comment here on any aspect. e. What could be improved in the application? Please feel free to comment here on any aspect. Thank you for taking part in the evaluation! Figure C.2.: The questionnaire used in the evaluation (page two)

71 Appendix C. Evaluation 64 Participant Questionnaire Incremental Number Page 3/3 Introduction We are looking for places where the application may be difficult to use. If you have trouble with some of the tasks, it is the product's fault, not yours. Don't feel bad, that is exactly what we are looking for. If we can locate the trouble spots then we can improve the application. Remember, we are testing the application, not you! As you work through the task, I will not be able to answer any questions. This is because we want to create a realistic situation. After the task I will answer any questions you still have. The task: Imagine to open an app the first time to study German. You skipped the introduction for the language learning application. Now you are asked to take a placement test. Please take the mobile phone and start. Language Test Did you take a German language test in the last two years? And if yes, which? Figure C.3.: The questionnaire used in the evaluation (page three)

72 Appendix C. Evaluation 65 C.2. Answers to Open Questions and Observations What did you like in the application? Vocabulary part feels nice to fill out Easiness of use I liked the UI as it is very simple and nice looking The easy access + design A lot of vocabulary and different kind of sentences Gap test feels like one can learn something / more than in vocabulary Easy Was really easy to use Straightforward, easy to use Easy for user Very convenient, time is OK It probably gives a frank result, because it s not possible to guess during gaps part The global results at the end Easy and quick to use Good texts Not distracting, i.e. lots of crazy colors/images/etc. Clear interface More convenient / faster than other traditional placement test like in German courses (30min) I find it useful The variety of the test What could be improved in the application? Gap test is very long Question: Are you a beginner? is not really clear. What should I choose here?

73 Appendix C. Evaluation 66 More types of questions Too much to type, maybe implement a question where to choose an answer for the gap It is missing e.g. how to conjugate verbs Some more questions, e.g. like Artikel I think there should be more vocabulary tests on different levels The connections of the test The scrolling in the gap test is not smooth Being able to see the answers on the gap test to see what was right and what was wrong See correction of gaps Scrolling did not work good Scrolling to top was not really working For beginner gaps were intimidating, because very hard Too much gaps /text to fill out (maximum 3) Show results what was right/wrong I think in the beginning the instructions were written grammatically wrong Separate vocabulary / grammar Maybe measurement not as good as longer test Vocabulary question was strange In vocabulary task: e.g. explain what some words mean Maybe less gaps but more difficult Time optimization (shorter) Add some grammar tests (or some grammar gaps in gap test) Add the translations of the vocabulary test See exact results what was right / wrong Explain what to do in gap test Every placement test I took is very intimidating. It makes me feel bad and that my German is not good and so hard to learn. It does not feel rewarding (like gap test here) More different tasks

74 Appendix C. Evaluation 67 Felt overwhelmed because so difficult For the vocab part a person could choose words even if they don t know the meaning Not satisfied because not so easy (really hard, felt overwhelmed) Another exercise (e.g. verbs + make present perfect) Explanation of words or something (to get some reward out of it ) Vocabulary test was new (never done before) and therefore more difficult because of many unknown words Too many words on screen (e.g. max 10 words at once) Vocabulary was really difficult and only knew some Observations and Comments by the Supervisor Sometimes participant were also asked a specific question after the questionnaire to further investigate potential issues. The answers are noted here: First thought exactly should be translated, so only chose words which mean exactly in German Did not really understand what to do in the vocabulary part; Thought only to choose words with the meaning exactly Understood question #c.1 like: Would I take this test now again? Scrolling to top broken Two taps on submit necessary (waited after first tap, then nothing happened and tried again) Two taps on submit necessary (waited after first tap, then nothing happened and tried again) Vocabulary part felt easier to use (just a click) Considers himself a beginner, but not sure No problem to use with big thumbs First question was not clear ( what should I take? ) Two taps on submit necessary (waited after first tap, then nothing happened and tried again) Could almost not speak English Two taps on submit necessary (waited after first tap, then nothing happened and tried again) Two taps on submit necessary (waited after first tap, then nothing happened and tried again)

75 Appendix C. Evaluation 68 Scrolling to top broken Scrolling to top broken Gap part was kind of surprising, because no separate introduction or what to do Vocabulary test felt most convenient to use Scrolling to top broken

76 Appendix C. Evaluation 69 C.3. Test and Criterion Test Results Table C.1.: Summary of the test results and criterion test level, which is scaled from one ( A1 ) to six ( C2 ). # Crit. Test VST (%) C-Test (%)

77 List of Figures 2.1. User experience pyramid Phone screen reachability in thumb usage Reachability of thumb usage on different screen sizes Example of Apple s iphone 6 reachability feature Excerpt of an English cloze test Excerpt of an English C-Test Vocabulary Size Placement Test example Two screens of first paper prototype Vocabulary and C-Test screen of the second prototype Vocabulary and C-Test screen of the third prototype Age distribution of the participants in the user test Boxplot of VST results in percent grouped by criterion test Boxplot of C-Test results in percent grouped by criterion test Results of test versus criterion test Results of linear regression and confidence intervals A.1. Two C-Tests with average level of difficulty A.2. Three C-Tests with easy to average level of difficulty B.1. First and second screen of the paper prototype B.2. Third and fourth screen of the paper prototype B.3. Fifth and sixth screen of the paper prototype B.4. Seventh screen of the paper prototype B.5. First and second screen of the second prototype B.6. Third and fourth screen of the second prototype B.7. Fifth and sixth screen of the second prototype B.8. Seventh and eighth screen of the second prototype B.9. First and second screen of the third prototype B.10.Third and fourth screen of the third prototype B.11.Fifth and sixth screen of the third prototype

78 LIST OF FIGURES 71 B.12.Seventh and eighth screen of the third prototype B.13.Ninth and tenth screen of the third prototype C.1. The questionnaire used in the evaluation (page one) C.2. The questionnaire used in the evaluation (page two) C.3. The questionnaire used in the evaluation (page three)

79 List of Tables 3.1. Summary of the discussed language tests Summary of the planned application screens for the first paper prototype Questions for the first paper prototype Excerpt of important feedback for the paper prototype Summary of feedback for the second prototype Summary of statements concerning the general usability of the application Summary of statements concerning the length of the test Summary of statements concerning the value of the test Summary of statements concerning the preferred part of the test Summary of statements concerning the difficulty of the test Quantity and variance of VST and C-Test grouped by the criterion test result Quantity and variance of VST and C-Test excluding outliers C.1. Summary of the test results and criterion test level

80 Bibliography [1] Mohamed Ally and Avgoustos Tsinakos. Increasing access through mobile learning. Commonwealth of Learning, [2] Mohamed Ally and Josep Prieto-Blázquez. What is the future of mobile learning in education? RUSC. Universities and Knowledge Society Journal, 11(1): , [3] Andreas Möller, Barbara Beege, Stefan Diewald, Luis Roalter, and Matthias Kranz. MobiDics: Cooperative Mobile E-Learning for Teachers. In Marcus Specht, Jari Multisilta, and Mike Sharples, editors, Proceedings of the 11th World Conference on Mobile and Contextual Learning (mlearn), pages , Helsinki, Finland, Oct [4] Stefan Diewald, Andreas Möller, Luis Roalter, and Matthias Kranz. Today, you walk! When Physical Fitness Influences Trip Planning. In Andreas Butz, Michael Koch, and Johann Schlichter, editors, Mensch & Computer 2014 Tagungsband, pages , Berlin, September De Gruyter Oldenbourg. [5] Christine Bruce. Research students early experiences of the dissertation literature review. Studies in Higher Education, 19(2): , [6] International Organization for Standardization. ISO : Ergonomics of human-system interaction part 210: Human-centred design for interactive systems. International Organization for Standardization, [7] Arnold Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. User experience evaluation methods: Current state and development needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, pages ACM, [8] Jakob Nielsen. Usability Engineering. Morgan Kaufmann, San Francisco, 1st edition, [9] John Karat. Evolving the scope of user-centered design. Communications of the ACM, 40(7):33 38, [10] Timo Jokela, Netta Iivari, Juha Matero, and Minna Karukka. The standard of user-centered design and the standard definition of usability: Analyzing ISO against ISO In 73

81 BIBLIOGRAPHY 74 Proceedings of the Latin American conference on Human-computer interaction, pages ACM, [11] Barbara Ballard. Designing the mobile user experience. John Wiley & Sons, [12] Adrian Mendoza. Mobile user experience: Patterns to make sense of it all. Newnes, [13] Steven Hoober and Eric Berkman. Designing mobile interfaces. O Reilly Media, [14] Jakob Nielsen and Raluca Budiu. Mobile Usability. New Riders, [15] Jason Morris. Android User Interface Development: Beginner s Guide. Packt Publishing, [16] Andreas Holzinger. Finger instead of mouse: Touch screens as a means of enhancing universal access. In Universal access theoretical perspectives, practice, and experience, pages Springer, [17] Yong Park and Sung Han. One-handed thumb interaction of mobile devices from the input accuracy perspective. International Journal of Industrial Ergonomics, 40(6): , [18] Dan Saffer. Designing gestural interfaces: Touchscreens and interactive devices. O Reilly Media, [19] Pekka Parhi, Amy Karlson, and Benjamin Bederson. Target size study for one-handed thumb use on small touchscreen devices. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, pages ACM, [20] Klaas Bollhoefer, Kerstin Meyer, and Rosina Witzsche. Microsoft Surface und das Natural User Interface (NUI). Technical report, Pixelpark, [21] Glen Anderson, Rina Doherty, and Subhashini Ganapathy. User perception of touch screen latency. In Design, User Experience, and Usability. Theory, Methods, Tools and Practice, pages Springer, [22] Ricardo Jota, Albert Ng, Paul Dietz, and Daniel Wigdor. How fast is fast enough? A study of the effects of latency in direct-touch pointing tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages ACM, [23] Jonathan Stark. Building iphone Apps with HTML, CSS, and JavaScript: Making App Store Apps Without Objective-C or Cocoa. O Reilly Media, [24] Steven Castellucci and Scott MacKenzie. Gathering text entry metrics on android devices. In CHI 11 Extended Abstracts on Human Factors in Computing Systems, pages ACM, 2011.

82 BIBLIOGRAPHY 75 [25] Helmut Vollmer. Leistungsmessung, Lernerfolgskontrolle, Selbstbeurteilung: Überblick. Handbuch Fremdsprachenunterricht, 4: , [26] Robert Lado. Language Testing: The Construction and Use of Foreign Language Tests. A Teacher s Book. ERIC, [27] Wilson Taylor. Cloze procedure: A new tool for measuring readability. Journalism quarterly, [28] Hossein Farhady. Varieties of cloze procedure in EFL education. Roshd Foreign Language Teaching, 12(44): , [29] Christine Klein-Braley and Ulrich Raatz. A survey of research on the C-Test. Language Testing, 1(2): , [30] Hossein Farhady, Abdoljavad Jafarpur, and Parviz Birjandi. Testing language skills: From theory to practice. SAMT Publications, [31] Lyle Bachman. Performance on cloze tests with fixed-ratio and rational deletions. Tesol Quarterly, 19(3): , [32] Andrew Cohen. Testing Language Ability in the Classroom. ERIC, [33] Parviz Birjandi, Hamid Farhadi, and Ali Jafarpour. Testing language skills from theory to practice, [34] Masoud Saeedi, Mansour Tavakoli, Kazerooni Rahimi, and Vahid Parvaresh. Do C-Test and cloze procedure measure what they purport to be measuring?: A case of criterion-related validity. International Journal of Human and Social Sciences, 6:2 99, [35] Charles Alderson. The cloze procedure and proficiency in English as a foreign language. Tesol Quarterly, pages , [36] Andrew Cohen. On taking tests: What the students report. Language Testing, 1:70 81, [37] Giuseppe Sciarone and Jeannette Schoorl. The cloze test: Or why small isn t always beautiful. Language Learning, 39(3): , [38] Carol Chapelle and Roberta Abraham. Cloze method: What difference does it make? Language Testing, 7(2): , [39] Zoltán Dörnyei and Lucy Katona. Validation of the C-Test amongst hungarian EFL learners. Language Testing, 9(2): , [40] Ari Huhta. Validating an EFL C-Test for students of English philology. Der C-Test. Theoretische Grundlagen und praktische Anwendungen, 3: , 1996.

83 BIBLIOGRAPHY 76 [41] Thomas Eckes. Der Online-Einstufungstest Deutsch als Fremdsprache (ondaf): Theoretische Grundlagen, Konstruktion und Validierung. Rüdiger Grotjahn, pages , [42] Ulhich Raatz and Christine Klein-Braley. The C-Test: A Modification of the Cloze Procedure. ERIC, [43] Rüdiger Grotjahn. Konstruktion und Einsatz von C-Tests: Ein Leitfaden für die Praxis. Der C-Test. Theoretische Grundlagen und praktische Anwendungen, 4: , [44] Thomas Eckes and Rüdiger Grotjahn. A closer look at the construct validity of C-Tests. Language Testing, 23(3): , [45] Rüdiger Grotjahn. Integration durch Sprache: Beiträge der 31. Jahrestagung DaF 2003, chapter Der C-Test: Aktuelle Entwicklungen, pages Fachverband Deutsch als Fremdsprache, [46] Rüdiger Grotjahn, Christine Klein-Braley, and Ulrich Raatz. C-tests: An overview. University language testing and the C-Test, pages , [47] Rüdiger Grotjahn and Brigitte Stemmer. C-Tests and language processing. University language testing and the C-Test, pages , [48] Günther Sigott. High-level processes in C-Test taking. Der C-Test. Theoretische Grundlagen und praktische Anwendungen, 4:67 82, [49] Christine Klein-Braley. Language testing with the C-Test: A linguistic and statistical investigation into the strategies used by C-Test takers, and the prediction of C-Test difficulty. PhD thesis, Duisburg, [50] Ulrich Raatz and Christine Klein-Braley. Introduction to language testing and to C-Tests. University language testing and the C-test, pages 75 91, [51] Rüdiger Grotjahn, Christine Klein-Braley, and Ulrich Raatz. C-Tests in der praktischen Anwendung. Erfahrungen beim Bundeswettbewerb Fremdsprachen. Der C-Test. Theoretische Grundlagen und praktische Anwendungen, 1: , [52] Christine Klein-Braley and Rüdiger Grotjahn. C-Tests in der Schule? Praxis des neusprachlichen Unterrichts, 45 (4): , [53] Zhifa Shen. The roles of depth and breadth of vocabulary knowledge in EFL reading performance. Asian Social Science, 4(12): , [54] Richard Anderson and Peter Freebody. Comprehension and teaching: Vocabulary knowledge. International Reading Association, pages , [55] Richard Anderson and Peter Freebody. Reading comprehension and the assessment and acquisition of word knowledge. Advances in reading/language research, 1983.

84 BIBLIOGRAPHY 77 [56] Charles Alderson. Diagnosing foreign language proficiency: The interface between learning and assessment. Bloomsbury Publishing, [57] Paul Meara and Imma Miralpeix. Y_lex: The Swansea advanced vocabulary levels test, [58] Rüdiger Grotjahn. C-Test: Contributions from current research, volume 18. Peter Lang, [59] Christine Coombe, Peter Davidson, Barry O Sullivan, and Stephen Stoynoff. The Cambridge guide to second language assessment. Cambridge University Press, [60] Glenn Fulcher and Fred Davidson. Language testing and assessment. Routledge London, NY, [61] Christian Krekeler. Grammatik und Fachbezug in Sprachtests für den Hochschulzugang. PhD thesis, Universität Duisburg-Essen, Fakultät für Bildungswissenschaften, [62] Stefan Diewald, Luis Roalter, Andreas Möller, and Matthias Kranz. Towards a Holistic Approach for Mobile Application Development in Intelligent Environments. In Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia, MUM 11, pages 73 80, New York, NY, USA, December ACM. [63] Bill Buxton. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann, [64] Stefan Diewald, Andreas Möller, Luis Roalter, Tobias Stockinger, and Matthias Kranz. Gameful design in the automotive domain: review, outlook and challenges. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 13, pages , New York, NY, USA, ACM. [65] Stefan Diewald, Patrick Lindemann, Andreas Möller, Tobias Stockinger, Marion Koelle, and Matthias Kranz. Gamified Training for Vehicular User Interfaces - Effects on Drivers Behavior. In Proceedings of the 3rd International Conference on Connected Vehicles & Expo, ICCVE 14, pages IEEE, Nov 2014.