Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language

Transcription

1 TUM Department of Electrical and Computer Engineering Chair of Media Technology Distributed Multimodal Information Processing Group Prof. Dr.-Ing. Eckehard Steinbach Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language Konzept eines Einstufungstests auf Mobiltelefonen für Deutsch als Fremdsprache Andreas Plieninger Bachelor Thesis Author: Address: Andreas Plieninger Matriculation Number: Professor: Prof. Dr.-Ing. Eckehard Steinbach Advisor: Dipl.-Ing. (Univ.) Stefan Diewald Prof. Dr. rer. nat. Matthias Kranz Begin: End:

2 TUM Department of Electrical and Computer Engineering Chair of Media Technology Distributed Multimodal Information Processing Group Prof. Dr.-Ing. Eckehard Steinbach Declaration I declare under penalty of perjury that I wrote this Bachelor Thesis entitled Concept of a Placement Test on Mobile Phones for Learning German as a Foreign Language Konzept eines Einstufungstests auf Mobiltelefonen für Deutsch als Fremdsprache by myself and that I used no other than the specified sources and tools. Munich, Andreas Plieninger Andreas Plieninger

3 Abstract Mobile learning is a current trend which is enabled by the increasing improvement and adaption of mobile devices. Applications allow learning at one s own pace, everywhere and at any time. Thereby, these applications, for example, help to effectively make use of available and otherwise unused spare time. In this thesis the focus is on learning a foreign language. Recommending and choosing the ideal content for the user is very important for targeted and effective learning. The difficulty of the learning material should be challenging but not overwhelming. For this purpose it is important to know the user s language level. In this thesis, a placement test on mobile phones for learning German as a foreign language is proposed. The focus is on a user-centered perspective and usability. An overview of findings on user experience for mobile phones forms the background of this thesis. Additionally, several existing language tests and applications were discussed. Using these findings, two existing test concepts and contents were chosen. As a result of an iterative feedback process a prototype concept has been developed with the objective of indicating the learner s German language ability. In order to assess the effectiveness of the approach, an evaluation was conducted. The participants prior taken language test were used as criterion tests. The subjects took the proposed placement test on a mobile phone and afterwards answered a questionnaire with focus on usability and preferred test parts. The scope of this work only allowed a small participant number, which limits the certainty of the conclusions. The results showed that the concept is usable. In addition, they showed that the correlation between the test score and the criterion test is significant. These findings allow the conclusion that the concept can indicate the test taker s German language ability. Based on the questionnaire and observation results, it is suggested that the concept can be further improved by shortening the test, adaption of the difficulty to the user s level, more precise task description and in-place correction. iii

4 Kurzfassung Mobiles Lernen ist ein aktueller Trend, der mit der fortschreitenden Popularisierung von mobilen Endgeräten einhergeht. Mobile Anwendungen erlauben das Lernen mit individueller Geschwindigkeit, überall und zu jeder Zeit. Dadurch können diese Anwendungen beispielsweise helfen verfügbare und ansonsten ungenutzte Zeit effektiv zu nutzen. Der Fokus dieser Bachelorarbeit ist das Erlernen einer Fremdsprache. Das Empfehlen und Auswählen idealer Inhalte für den Nutzer ist für zielgerichtetes und effektives Lernen sehr wichtig. Die Inhalte sollten herausfordern, jedoch nicht überfordern. Für diesen Zweck ist es wichtig, das Sprachniveau des Nutzers zu kennen. Diese Bachelorarbeit schlägt ein Konzept eines Einstufungstest angepasst auf Smartphones für Deutsch als Fremdsprache vor. Der Schwerpunkt dieser Arbeit wurde auf eine nutzerzentrierte Perspektive und Benutzerfreundlichkeit gelegt. Die Grundlage dieser Bachelorarbeit bildet ein Kapitel über Erkenntnisse der Benutzerfreundlichkeit bei Smartphones. Darüber hinaus wurden einige Sprachtests und Anwendungen untersucht. Basierend auf diesen Ergebnissen wurden zwei bereits existierende Tests und entsprechende Inhalte ausgewählt. Es wurde ein Konzept in Form eines Prototypen mit einem iterativen Feedback- Prozess entwickelt, mit dem Ziel, das Deutschniveau des Nutzers angeben zu können. Um die Effektivität des Ansatzes zu untersuchen, wurde eine Auswertung durchgeführt bei dem ein bereits abgelegter Sprachtest der Teilnehmer als Kriteriumstest diente. Die Personen machten den Einstufungstest auf einem Smartphone und beantworteten anschließend einen Fragebogen, der sich besonders auf Nutzerfreundlichkeit und bevorzugte Testteile konzentrierte. Der begrenzte Umfang dieser Arbeit ließ nur eine kleine Teilnehmerzahl zu und limitierte somit die Aussagekraft der Schlussfolgerungen. Das Ergebnis zeigte, dass das Konzept benutzerfreundlich ist. Weiterhin zeigte es, dass die Korrelation zwischen dem Testergebnis und dem Kriteriumstest signifikant ist. Diese Erkenntnisse erlauben die Schlussfolgerung, dass das Konzept das Sprachniveau des Nutzers richtungsweisend angeben kann. iv

5 Contents Contents v 1. Introduction Motivation Statement of the Problem Structure of the Thesis Background and Related Work Definitions of User Experience and Usability User Experience Usability Distinction of User Experience and Usability Mobile Phone User Characteristics Mobile Interruptible Available Contextual Identifiable Understanding the Mobile Device Limited Display Modality Touchscreen Interface Text Input Modalities Connection Characteristics Second Language Testing Linguistic Proficiency Test Selection Criteria Cloze Test C-Test Vocabulary Size Test (VST) TOEFL Related Mobile Applications v

6 CONTENTS vi 3. Concept Choosing a Test Test Contents Source of VST Items Amount of C-Test Items Source of C-Test Items Paper Prototyping Mobile Phone Prototyping Evaluation Research Questions Methodology and Test Setup Test Execution Results and Interpretation Demography and Mobile Phone Experience Usability Test Time Overall Perceived Value Preferred Test Part Open Questions and Observations Proficiency Correlation Conclusion 43 A. Test Items 44 A.1. VST Items A.2. C-Test Items B. Prototypes and Feedback 47 B.1. Paper Prototype B.2. Paper Prototype Feedback B.3. Second Prototype B.4. Feedback for Second Prototype B.5. Third Prototype C. Evaluation 61 C.1. Questionnaire C.2. Answers to Open Questions and Observations C.3. Test and Criterion Test Results

7 CONTENTS vii List of Figures 70 List of Tables 72 Bibliography 73

8 Chapter 1. Introduction 1.1. Motivation Mobile learning is a current trend which is enabled by the increasing adaption and improvement of mobile devices. Many people are carrying around their mobile phones almost all the time. Learning on the phone has some potential advantages: It is possible everywhere, at one s own pace and at any time [1]. Furthermore more people around the world are using mobile technology to access learning resources and applications [2 4]. In this thesis the focus is on learning a foreign language, a sub field of mobile learning. Currently many applications are available to study vocabulary. It is suggested that there should be more research on how to design and deliver learning [2] which could lead to more contents and learning applications. Still, no matter what kind of content, they have one thing in common: Recommending and choosing the optimal content for the user is very important for targeted and effective learning. It could be a step towards the vision of a perfect individual teacher to learn a foreign language Statement of the Problem The best learning effort can only be achieved given the most suitable material. In the beginning when a student starts using a specific application, the learning content must be specified. The knowledge of the learner should match the difficulty of the recommended content. To be more precise, the difficulty of the learning material should be challenging but not overwhelming. In order to achieve this matching, the content as well as the learner must be rated. The first challenge, to rate the content, is not part of this thesis. The second challenge is to rate the learner s language knowledge, the linguistic proficiency. In this thesis we will focus on learning German as a foreign language. The rating with a placement test will be done on a mobile phone. A first concept shall be created to evaluate the placement test and suggest points for further improvement. To summarize, the central research question is: 1

9 Chapter 1. Introduction 2 How could a concept of a language test on a mobile phone which can indicate the test taker s general German language proficiency look like? The following points characterize the research question: The concept is for people who learn German as a foreign language, are no beginners and are using or will probably use a mobile phone application to do so It could be used when a learner tries a new mobile learning application for the first time or, more general, when using a new product/resource (e.g. language course) for the first time The concept is designed to run on a mobile phone (current models: Apple iphone 6, Samsung Galaxy S6 or Xiaomi Redmi 1S) The implementation is prototypical (on paper or an application running on a single specific phone) The prototype can be used by the majority of potential users without further help ( usable ) An economic and scalable way of testing: Computers can calculate and evaluate the test takers results The concept is a step further to provide individuals better learning material and a better learning experience 1.3. Structure of the Thesis The thesis is divided into five chapters. Chapter 2 focuses on the background and related work. Existing placement tests as part of second language testing as well as usability and user experience are examined. Chapter 3 describes the process and results of creating a concept of a placement test. Findings of several feedback rounds are used to improve the concept. In order to analyze the final concept, a user test was designed, executed and finally evaluated. Chapter 4 describes the process, results and interpretation especially regarding the research question. At the end, Chapter 5 summarizes the thesis, points out conclusions and indicates potential points for further improvement.

10 Chapter 2. Background and Related Work This chapter summarizes related literature and works. The purpose is to provide the background and basis for the research undertaken [5]. In order to answer the research question, considering how a placement test on a mobile phone could look like, it is especially important to understand the mobile user, mobile device and resulting design implications. In this way it shall be ensured that the concept can be used in the designated way. In the second part, these design implications are essential to discuss and review different language learner placement concepts from the literature. Especially the pros and cons for an implementation on a mobile device are of interest. This analysis will form the basis to create a draft of the placement concept Definitions of User Experience and Usability User Experience The commonly used definition of user experience according to ISO :2010 is: a person s perceptions and responses that result from the use and/or anticipated use of a product, system or service [6]. This very formal definition is supported by another interpretation: User experience explores how a person feels about using a product, i.e., the experiential, affective, meaningful and valuable aspects of product use [7] Usability The commonly cited definition of usability is from Nielsen: usability is about learnability, efficiency, memorability, errors, and satisfaction [8]. Yet, the ISO definition of usability the extent to which a product can be used by specified users to achieve specified goals with effectiveness, 3

11 Chapter 2. Background and Related Work 4 efficiency and satisfaction in a specified context of use [9] is the main reference and will be used for the purpose of this research [10] Distinction of User Experience and Usability The main difference between the two areas is the point of view. Fig. 2.1 shows this in a pyramid. Usability describes the basic requirement for a product and is quantifiable. If it is not present a positive user experience is almost not possible. Therefore usability can be considered as an important subfield of user experience. With more abstraction subjective perception increases and therefore also the development process is less quantifiable. Thus, a successful product in terms of user experience should fulfill several requirements on different levels. All these levels demand a lot of effort and can therefore not be considered in the same detail in this work. The following chapters focus on the characteristics of the user and device to provide the background to create a usable and convenient product concept. Figure 2.1.: The user experience pyramid with requirements on different levels. This work focuses on Usable and Convenient with the whole picture in mind. Source: D. Traynor,

12 Chapter 2. Background and Related Work Mobile Phone User Characteristics The research goal is to implement a placement concept for mobile phone users. Hence it is important to understand the mobile user characteristics in order to discuss the aspects of existing classification tests regarding their fit in a mobile application. Furthermore usability problems could have a negative impact on the final test performance. Therefore the user characteristics must be considered in detail to prevent negative influence as effectively as possible. Ballard [11] wrote an in-depth analysis of these characteristics. It is the basic source for the overview in the following sections Mobile Mobile phone users may be in any situation anywhere when using their phone. They are in a conversation, commuting, on a plane, waiting for a meeting or just at home [12]. They interact with the environment and the physical as well as the social context changes [11]. In the same time the user tries to use the phone application and has to shift the focus from the device to the environment and back. This is one factor that makes the mobile phone user more interruptible Interruptible Besides being interrupted by context shifts between navigating in the environment and the phone there is another factor: Sitting in front of a computer in comparison to using a phone, people are more likely to interrupt the phone user. They do not have a social clue that suggests that the user is focused on a task [11]. The same clues are missing when the user is, e.g., waiting for a train or in a meeting while using the phone. This more distracting environment results in a shorter interaction period at a time. An average time per interaction on a mobile phone is 17 minutes compared to 39 minutes on a desktop computer [12]. The shorter interaction periods and high probability of an interruption have an impact on the mobile phone application design, which is outlined in Section D. Traynor, 2008 (concept from S.P. Anderson, 2007); last access: 2015/01/13

13 Chapter 2. Background and Related Work Available On the other hand an interruption can be very important to the user. Currently more and more applications utilize notifications as their core service and only way to communicate with the user 2. Mobile phone owners bring their device most of the time with them in order to stay connected and communicate, for example, with friends and be available for any communication attempt almost at any time. This availability allows applications to provide technologies similar to instant messaging with high confidence that the user is present and gets the information immediately [11]. The concept of this work will not depend on this availability. It is mentioned to keep the list of characteristics complete Contextual The environment influences the phone usage. Ideally the device could know and understand the context and adapt to it [11]. This context includes factors such as the location, activity and the corresponding expected behavior. As an example it could be considered that the user is starting to drive. The device notices this new context and automatically changes its interface to car mode as well as connects to the car s media interface. There are first applications which try to implement this, e.g. Google Now is providing information like routes and weather forecast based on the context 3. This aspect of context is mentioned for a better general understanding but will not be considered further in the concept of this work Identifiable The mobile phone is a very personal device and most of the time the user will carry it around and be available [13]. Additionally, the user has his own personal phone number which provides a persistent way of contacting him. The value of the phone number is so high that many countries are regulating the phone carriers to ensure number portability between carriers. Because the phone number is very personal it provides a good way for user identification [11]. This possibility is already used by many applications, e.g. WhatsApp validates the phone number by sending a text message with a verification code 4. 2 For example WhatsApp informs the user about new messages via notifications; last access: 2015/01/23 3 Google Now, last access: 2014/12/18 4 WhatsApp FAQ, last access: 2014/12/18

14 Chapter 2. Background and Related Work 7 Identification is covered to complete the list of user characteristics and will currently not be used in the concept of this work Understanding the Mobile Device Learning the characteristics of mobile devices is essential to understand the mobile experience and therefore to design a user interface and application. In this section we explore the aspects of the hardware and user interface elements that make the devices so different compared to traditional desktop computers. Additionally, the consequences of every aspect and resulting design principles are introduced Limited Display Modality For a mobile device it is essential to be small and easy to carry around. Hence the screen has to be small. The following section summarizes advice from Nielsen [14]: As he stated in his book, there is less information that can be displayed at a time compared to a desktop computer. Therefore the user has to rely on his short-term memory for other necessary information which is not visible at that moment. As a result the interaction gets more difficult. The challenge is to find room for all necessary information on one screen and to make the interaction as easy and pleasurable as possible. Focus on one Task Every application screen must focus on the user s immediate task only. Therefore every screen is designed to fulfill one distinct task and should only display the information needed to perform this task. Only necessary buttons and navigation elements should be available. Thus the information which is displayed on one page must be reduced. The Blackberry UI Guidelines 5 include a good example: A very complex customer relationship management tool provides a massive amount of information. The user only needs a tiny amount of information at a time, e.g. to change the customer s address it is not important to see the customer s orders. In summary, the application should be designed in a way that the user can perform desired tasks easily and quickly. This requirement to focus on simplicity and the most important elements only paired with the ability to install many different applications on one phone results in the goal to design one appli- 5 Blackberry UI Guidelines, Constraints_of_designing_for_mobile_devices_ _11.jsp, last access: 2014/12/16

15 Chapter 2. Background and Related Work 8 cation for one specific user need. In this way one application can do one thing very well and in the same time fulfill the requirements of simplicity and focus. Consequently, the concept of this work will focus on one goal: To perform a placement test on a mobile phone. Smoothen Application Switching On the small screens one application can be displayed at a time. This is very different compared to personal computers where many windows can be arranged next to each other and still be visible. Thus users are required to switch applications quite often [15]. Therefore the application should be designed in a way to handle this events (lose focus, get focus again, exit, open) instantly, without much waiting time and a good strategy for the user after coming back, e.g. to continue with his paused task. For example, a user is writing an . While writing, he receives a phone call. The application will be paused and the phone call will be displayed. After the call is finished the user wants to continue writing his . One good way to handle this is that after he reopens the mail application, the stored draft of the mail is displayed again. He continues writing where he stopped and finally sends the Touchscreen Interface The main interaction element of modern phones is the touchscreen which combines a screen with a touch interface. It seems to be the most natural of all input devices so far because it allows the direct engagement with the displayed information [16]. The main interaction tool is the finger, to be more precise, one-handed thumb interaction [17]. Thus, the main guidance for touchscreen interface design is to optimize for the thumb. Minimal Target Size Apple recommends a button size of at least 44 points x 44 points, which corresponds to about 6 7 mm x 7 mm. Given this size, no target element should have less than 9.7 mm in diagonal which corresponds to the smallest average size of a finger pad 7 and is also recommended by other 6 Apple iphone 4/5/6 have 326 ppi: 44 points x 44 points, retina displays have two pixel per point, therefore 88 px x 88 px which equals 0.27 inch x 0.27 inch = 0.69 cm x 0.69 cm 7 Ubuntu - Designing for finger-driven UIs, DesigningForFingerUIs, last access: 2014/12/18

16 Chapter 2. Background and Related Work 9 research [18, 19]. A recommended padding between targets of at least 2 mm prevents accidental taps 8. Preferred Interaction Areas Depending on the orientation the device is held (horizontal vs. vertical) the fingers can reach some parts of the touchscreen more comfortable. Most of the time the phone is hold vertically (portrait mode), 49% use their phone one-handed, 25% two-handed and with their thumb on the screen 9. Fig. 2.2 describes the dependence between touchscreen area and ease of reach graphically. With increasing screen size it is getting more complicated to reach the whole screen as it is illustrated in Fig Therefore Apple introduced for the iphone 6 a new feature called reachability 10. On a double home button tap the device shifts the entire screen down closer to the thumb, Fig. 2.4 shows the shifted screen. This facilitates the interaction e.g. with the back button in the upper bar. Figure 2.2.: Screen reachability in one-handed and two-handed thumb use as reference for user interface drafts where to position interaction elements - Source: Steven Hoober 11 8 Microsoft Windows - Design touch interactions, apps/hh aspx#touch_targets, last access: 2014/12/18 9 Steven Hoober: How do users really hold mobile devices?, 02/how-do-users-really-hold-mobile-devices.php, last access: 2014/12/18 10 Apple iphone 6 Design, last access: 2014/12/17 11 Steven Hoober, last access: 2014/12/18 12 Kirk McElhearn, last access: 2015/01/14 13 Apple iphone 6, last access: 2015/01/14

17 Chapter 2. Background and Related Work 10 Figure 2.3.: Reachability with one-handed thumb usage on different screen sizes: Larger screens demand an adapted user interface to keep the same level of convenience, e.g. Apple Reachability - Source: Kirk McElhearn 12 Figure 2.4.: Example of Apple s iphone 6 feature to increase reachability: The entire displayed application gets shifted closer to the thumb. This feature can be used in interface drafts for large screens - Source: Apple iphone 6 13 Responsiveness A touchscreen allows the direct engagement with the displayed information. From this seemingly intuitive and natural input method the user expects high responsiveness [20, 21]. It is absolutely critical to respond to an interaction immediately. Users find latencies above 580 ms unacceptable and greatly prefer lower latencies, even below 50 ms [22]. In the case an action takes longer to finish, it is essential to acknowledge the input, e.g. by displaying a progress bar. This is the

18 Chapter 2. Background and Related Work 11 difference between speed and responsiveness [23] Text Input Modalities The main input modalities usually are a keyboard, touchscreen, video/audio recording and speech to text. Video and audio recording are widely used by media applications, such as YouTube and sometimes for direct messaging applications. Automated interpretation by a computer of this content is sometimes inaccurate. One very interesting technology is speech to text, which is already available on the latest generation of mobile phones. The accuracy is getting better but cannot yet be considered mature 14. The main text input modality up to today is still a keyboard. Most of the time this is a software implemented keyboard using the touchscreen for interaction. Considering the small size of these keyboards they are often perceived as impractical and therefore lower rates of text entry compared to personal computers are reasonable [24] and most users are unlikely to write long texts on a phone. Minimizing Text Input Because of the difficulties the user text input should be reduced to a minimum and replaced by other methods. One way is to provide options, e.g. to provide possible answers and the user has to choose one or to provide an adjusted minimal input element containing only the limited needed amount of characters to complete a specific task 15. Another way is to tie in existing phone services, e.g. use a Google account to log in the application 16. Whenever a lot of text input is still necessary it should be considered to support landscape mode in the application. The user can decide to turn the phone and use the larger keyboard which might be preferred. There are also several different keyboard layouts on popular smart phone systems available to support specific content, e.g. text, number or URL input 17. The expected input should be analyzed carefully to choose the corresponding keyboard layout, e.g. whether auto-correct is helpful and in case, the unnecessary input methods should be disabled [23]. 14 Intelligentvoice.com, 46-americans/, last access: 2015/01/14 15 see Memrise, memrisecompanion&hl=de, last access: 2015/01/14 16 Google+ Login in a custom application, de, last access: 2015/01/14 17 ios Developer Library, UITextInputTraits_Protocol/index.html, last access: 2015/01/14

19 Chapter 2. Background and Related Work Connection Characteristics Mobile Internet connections are getting faster and faster. Still, even with 4G, latency is noticeable. Another important point to consider is network coverage. From time to time there is a connection interruption or no coverage at all, e.g. in many subways and on railway lines coverage is partial. In these cases, depending on the application and necessary amount of data to download, loading the next screen on the phone can take too long [14]. A possible solution is to heuristically pre-fetch content. Another solution is to provide an offline mode and synchronize data when an Internet connection is available again. These technical solutions allow to minimize the waiting time and therefore improve the user experience. All potential content in our concept, e.g. text, images, short audio and video, can be pre-fetched and stored offline. Therefore this question is for our goal, a placement concept, mainly a technical implementation aspect and it will not be further discussed in this thesis Second Language Testing This section describes the different basic skills a language test can measure and examines some tests in detail. In order to not exceed the scope of this thesis the following sections will only explain the necessary linguistic background and focus on second language testing for assessment and placement purposes Linguistic Proficiency A common definition of general language or linguistic proficiency is the overall ability of an individual to communicate in the acquired language 18. This concept was further developed in the 1980s in America based on new research insights by the ACTFL 19. Their Proficiency Guidelines defined several levels to classify the learner. The goal of proficiency tests is to provide a fair and standardized classification into e.g. a certain predefined level. In comparison, achievement tests usually try to assess to which amount an individual completed a given learning task. This tests are usually used in the classroom environment and follow the principle of test as you teach [25] and will not be considered any further. The ACTFL proficiency guidelines further broke down the general proficiency to assess four distinct areas: speaking, listening, reading, writing. Based on the assumption that these skills are 18 Wikipedia - Language proficiency, last access: 2015/01/02 19 ACTFL - American Council on The Teaching of Foreign Languages, Proficiency Guidelines, org/publications/guidelines-and-manuals, last access: 2015/01/14

20 Chapter 2. Background and Related Work 13 independent from another they can be assessed separately [26]. For example, teachers can use the result to identify problems of their students in a specific area. These kinds of tests are therefore also a tool for language diagnostics. All individual results together are then used to determine the general proficiency level. The goal of this thesis is to assess the general proficiency (see Section 1.2). Therefore results of more broken down tests on specific areas of proficiency (e.g. grammar, listening) are not required. Hence the focus of the following sections is mainly on relevant, researched and feasible tests which will be discussed for potential usage in a mobile application Test Selection Criteria In this section the criteria to rate the different language tests will be discussed. Beginning with the statement of the problem, to create a concept of a German language placement test for mobile phones, we will further break this down into several criteria. In Section 1.2 we defined our goal to assess the general language proficiency. Hence one criterion will be how good one test correlates with the general language proficiency, which will be called test performance. The placement concept should provide scalability in an economic way, therefore automated test correction by the computer is valuable. Furthermore, test correction by human individuals can be very subjective. Without an exact predefined solution set this unwanted effect can be even stronger. The criterion automated correction will assess, how good a test can be corrected by computers without any large additional effort to create special software. This will be an exclusion criterion, meaning tests which can almost only be corrected with human help are excluded from our concept. In order to create an appealing product, Section 2.2 and Section 2.3 provide the background to user and device characteristics. The screen size on touch phones is limited, the discussed implication in Section is to focus on one task per screen. Some language tests utilize questions where the user has to see and know a lot of context. In order to reduce the short-term memory load of the user, all the information necessary to answer the question should be visible at the same time. Meanwhile, the minimal target size and preferred interaction areas for important content (see Section 2.3.2) should still be respected. This criterion, how easy it is to fit all the information on one screen, will be called screen size compatibility. Another important factor is the text input modality and the resulting implication to minimize text input (see Section 2.3.3). This criterion, how many characters a user has to enter, will be called text input amount. The name mobile phone already implies another important characteristic: mobility (see Sec-

21 Chapter 2. Background and Related Work 14 tion 2.2.1). Considering this, the user can be literally anywhere in any situation when using an application on the phone. Furthermore, it cannot be assumed that audio output is always desired and accepted by the user. Reasons include examples like noisy environment, social expectations (e.g. restaurant) or missing headphones while sitting in the subway. Accordingly, another criterion for tests is no audio dependency. The final criterion is time. A shorter test means less effort for the test taker. Assuming that users actually take this test for a final goal, e.g. to get better content or to know their level, they just want to achieve this goal as fast as possible Cloze Test A cloze test is a text where certain words are removed and replaced by fixed-width gaps. The task of the student is to fill the gaps with the words again. In the example in Fig. 2.5 the words are replaced with a line. This test with an appropriate length, difficulty and every n-th word removed is called cloze test [27]. Education has two goals. First, it means so that we may earn living. Some of these tools are tools, such as reading or writing. are technical skills, such as typing,, and data processing. Still others are professional skills peculiar for example, to, to science, and to teaching. Figure 2.5.: Excerpt of an English cloze test for better understanding, Source: Farhady [28] Creation Guidelines To prepare the test, every n-th word of the text is removed. The value of n has a large influence on the difficulty and length of the test. Common values are in the range from 5 to 8, most of the time 7 is used [29]. If n is small many words will be removed and therefore the gaps will be more difficult to fill again. If n is large the resulting text will be longer in order to keep the same amount of gaps. An additional commonly used term in this context is the frequency, which describes the same as n, the amount of gaps per passage [30]. It is also possible to rationally choose which words should be replaced with gaps. This allows the creator to have more influence on the resulting test. The validity and readability will still be comparable to the fixed-ratio method [31]. Because it is more effort with almost the same outcome compared to the fixed-ratio method, it will not be considered further. Another important factor is the length of the test: the number of gaps to fill in. Researchers suppose numbers from 20 up to a maximum of 50. Commonly 20 to 25 gaps are recommended [32].

22 Chapter 2. Background and Related Work 15 In order to improve the readability of the resulting test the first and last sentence should be without gaps [29]. To be more specific, the first sentence should already contain the paragraph s main idea. The conclusion of the text is in the closing sentence. This intact sentences are important to not distort the student s natural language processing. To sum up, an example cloze test could consist of 25 gaps with every 7-th word removed and an intact initial and closing sentence. This test would have an appropriate length of about 230 words [33]. Finally, the difficulty of the original text must be taken into consideration. One commonly used method is to estimate the readability, e.g. with the Fog s index of readability for English. Correlation with Proficiency Many studies examined the correlation of the cloze test and an established and validated benchmark test for general English proficiency. Saeedi investigated a correlation between the cloze test and TOEFL-Test with 90 Iranian English majors. The result was a statistically significant correlation and therefore the cloze test also correlates to general proficiency (correlation coefficient 0.82, significant at the 0.01 level) [34]. To the same statistically significant correlation between cloze test and language proficiency came other researchers [35 40]. Discussing Selected Criteria Section defined several criteria, which will now be discussed regarding this cloze test. There is no need for audio while taking the test. Additionally, given predefined solutions, automated correction is possible. A full cloze test with the goal for placement would presumably take about 40 minutes of time to finish 20. It can be assumed that the amount of text input is about 800 characters 21. To find the correct solution for a gap, the test taker has to consider the context. Therefore it is important to see the text before and after a gap. text into separate screens. This prevents to split one A single test with about 230 words 22 will probably not fit in an appropriate manner on one smart phone screen. However, scrolling seems to be a possible, but not a perfect solution. Furthermore, every time the user types a word, the onscreen keyboard will use approximately 30% of the screen 23. Because less context is visible in that moment, the test taker has to rely more on his short-term memory. In summary the screen compatibility is still good. 20 Suppose that a placement test consist of eight texts with the same amount of gaps like the ondaf (20 each), which is 40 minutes long [41] 21 Considering eight texts with 20 gaps each and an average word length of five characters, the test taker has to write 800 characters. 22 see Section 2.4.3: Creation Guidelines 23 Considering a mobile phone screen shot (Android phone viewport, 360 px x 640 px, see Figure 3.2)

23 Chapter 2. Background and Related Work C-Test Based on the cloze test (see Section 2.4.3) researchers developed the C-Test in the 1980s. In an initial text every second word is omitted: The second half of the word is removed and replaced by a gap [42]. In such a resulting test like in Fig. 2.6 the student has to fill in the gaps in order to reproduce the original text. In comparison to the cloze test, not the whole word, just the second half of the word is removed. Nothing beats the heat like a refreshing dip in a swimming pool. But wh it co _ to wa _, both ki and adu _ need t_ be car. Susan King s daug _ - Alison, 12, a Christy, 9, a in th _ grandparents po every d. King s gi _ have ma pool ru _, including n being all in t pool ar without a_ adult, n_ jumping i_ the sha end, n_ running around the pool and no holding each other under water. "Kids drown quickly and quietly" caution Jen Costello of the National safe kids campaign. Figure 2.6.: Excerpt of an English C-Test for better illustration, Source: Saeedi et al. [34] Test Construction According to Grotjahn [43] the test should follow these construction guidelines: As a basis for the test several short texts (60 to 80 words) with different topics are chosen. Beginning with the second sentence in every second word the second half is omitted and replaced by a gap. If the word is a proper noun the word will not be omitted. If the word is only a single character it will also not be omitted. In the case the length of the word is odd the larger part will be omitted. Every text should contain the same amount of gaps. The last part of the text should remain intact in order to provide more context for the student to solve the test. To format the gaps, constructors can use lines of the same length or dashes according to the number of removed characters. The former method results in a more difficult test. All the resulting texts are arranged by the perceived difficulty, beginning with the easiest. For every exact solution the student gets one point. Validity regarding Proficiency The validity of the test was subject of many studies: For example, in a study with 102 Hungarian English majors the C-Test was statistically significant correlating with the TOEIC 24 [39]. Another study with 470 participants showed a statistically significant correlation between a C-Test and 24 TOEIC: Test of English for International Communication; it is considered to be highly reliable and valid to measure English proficiency

24 Chapter 2. Background and Related Work 17 the TestDaF 25, especially regarding the factor general proficiency [44]. Yet another study with 223 students of German as a second language showed statistically significant correlation between the C-Test of ondaf 26 and DIALANG 27 [41]. Furthermore, latest research suggests that the test not only measures on the micro level but also on a larger more general level [45 50]. Therefore this test is among others used as assessment test for universities and language schools [51, 52]. In conclusion, it is shown that the C-Test is a reliable method to measure the German language proficiency. Discussing Selected Criteria As Section defined, several criteria besides the test performance are important to assess: Automated correction is possible due to the distinct solution. Additionally, the test requires no audio. The amount of text input can be estimated with 400 characters 28. The time can be assessed accordingly to the cloze test 29 with about 40 minutes. The last criterion is the screen compatibility: In comparison to the cloze test, the C-Test has less text between the gaps and is therefore much shorter. It could maybe allow to show 20 gaps on one screen 30 while still respecting touchscreen interaction guidelines (see Chapter 2.3.2). Therefore the screen compatibility is good (and slightly better than the cloze test) Vocabulary Size Test (VST) Compared to a traditional placement test (e.g. grammar/listening/reading comprehension questions) a vocabulary size test is quite different: The goal is to measure the number of vocabulary a student knows, which is often referred as breath of vocabulary knowledge. Researchers discovered that the vocabulary knowledge is essential for many language skills. The vocabulary size is especially a very good predictor for reading comprehension [53]. In general, students with a large vocabulary are more proficient [54]. Therefore, one can use a test to measure a person s vocabulary size in order to estimate his proficiency. Because quite a large sample of answers is required to reliably estimate one s vocabulary size, simple test formats are often preferred: One way to achieve this is the so called Yes/No Format: 25 TestDaF: Test of German as a foreign language; a standardized and well-recognized valid test to measure German proficiency 26 ondaf: Online-Einstufungstest Deutsch als Fremdsprache; standardized language test of German proficiency which uses A2-C1 levels from the CEFR (Common European Framework of Reference for Languages) 27 DIALANG is an online language diagnostic system to assess a person s proficiency in 14 European languages, last access: 2015/01/14 28 The gaps are missing half a word. Therefore half the amount of characters compared to the cloze test have to be typed in: same amount of gaps and slightly less text to read, see Section Considering 20 gaps: every second word has a gap; First and last sentences are intact; 20 2 words+15 words = 55 words: It seems possible to fit 55 words with a reasonable large font on one screen.

25 Chapter 2. Background and Related Work 18 The test taker is presented with a list of words where he has to indicate whether he knows the word or not. Sometimes the student overestimates his knowledge and answers too many words with yes. In order to prevent a wrong result, invented non-words are added to the real word list [55]. Figure 2.7.: Yes/No Format: Vocabulary Size Placement Test example to facilitate the estimation of compatibility with a small screen, Source: DIALANG 31 Validity regarding Proficiency One well-regarded example of this test format is DIALANG 32 [41]. In the introduction the student is asked, as Fig. 2.7 shows, to take a vocabulary test. This result is then used to estimate his general proficiency and adapt further specific skill tests to his level. Analysis of DIALANG s data suggest a significant statistical correlation between the vocabulary test result and specific measured skills (reading, grammar, writing listening correlation each above 0.6) [56]. Another significant example is the Swansea test, where the software programs X_LEX and Y_LEX provide vocabulary tests for the most frequent English words [57]. To conclude, there are established and valid methods to measure the test taker s vocabulary size and therefore estimate his general language proficiency. Discussing Selected Criteria In this section the missing criteria (see Section 2.4.2) regarding this test are discussed: To answer a yes/no question one simple tap on the interaction area is necessary. Accordingly, the amount of 31 DIALANG online test, last access: 2015/01/08 32 DIALANG is an online language diagnostic system to assess a person s proficiency in 14 European languages, last access: 2015/01/14

26 Chapter 2. Background and Related Work 19 text input is zero. The test taking process has no need of audio. Furthermore, an automated correction of the answers is possible. It takes approximately 10 minutes of time to answer all questions 33. To answer an individual question, meaning one word and the possible answers yes/no, no other information is needed and therefore also independent from other questions. They can be split over several screens without any negative influence of the test taker s short-term memory load. Therefore even larger amount of these questions can be implemented in the user interface. In conclusion, the screen compatibility is very good TOEFL The TOEFL 34 is a standardized test which measures English language proficiency and is an accepted certificate by most American Universities for non-native English speakers who try to study there 35. It is also regarded as a well-respected criterion test to benchmark new tests and used by several researchers [58]. In the following section a TOEFL ibt sample test 36, which is administered via computer, will be discussed. The test consists of a speaking part (20 minutes), writing part (50 minutes), listening part (60-90 minutes) and reading part (60-90 minutes) 37. Speaking Task The speaking section measures the ability to talk about various topic. Typical tasks include to talk into the computer microphone about a pleasant moment or answer questions about a written/oral conversation or text. These responses are rated by humans 38. In Section one of our criteria is automated correction, which cannot be achieved using this kind of questions. Therefore this method will not be considered any further. Writing Task The writing task consists usually of two essays: One essay with personal opinion about a controversial topic and a second one without personal opinion about an academic topic. The rating is 33 DIALANG has 75 yes/no questions; Assumption: A non-native speaker needs up to ten minutes to answer them. If the assumption was about right can be validated in the evaluation of the concept 34 TOEFL: Test of English as a Foreign Language 35 TOEFL ibt: About the Test, last access: 2015/01/15 36 TOEFL Sample Test Questions, last access: 2015/01/15 37 TOEFL Test Content, last access: 2015/01/15 38 TOEFL Scores, last access: 2015/01/15