TECHNICS TECHNOLOGIES EDUCATION MANAGEMENT JOURNAL OF SOCIETY FOR DEVELOPMENT OF TEACHING AND BUSINESS PROCESSES IN NEW NET ENVIRONMENT IN B&H

Transcription

1 Vol. 9, No. 3, ISSN T TE M TECHNICS TECHNOLOGIES EDUCATION MANAGEMENT JOURNAL OF SOCIETY FOR DEVELOPMENT OF TEACHING AND BUSINESS PROCESSES IN NEW NET ENVIRONMENT IN B&H

2 The services of Android-based devices increasing their accessibility for the blind Jolanta Brzostek-Pawlowska, Daniel Kocielinski Institute of Mathematical Machines, Warsaw, Poland. Abstract Touchscreens are hardly accessible to the blind. Assistive technologies offered by mobile device manufacturers do not eliminate all the obstacles. We propose new solutions that would improve the accessibility of touch interfaces in Android-based devices. We present the concept of services that provide functions and resources of the touchable Android-based devices for blind with Braille notetakers. Key words: User Interfaces Input devices and strategies, User-centered design, assistive technology, mobile accessibility, touch screens, blind users 1. Introduction Advanced functions of Android-based devices are hardly accessible to the blind using a touchscreen. One example of barrier is the virtual QWERTY keyboard, other examples include lack of haptic points on the screen (these would improve spatial orientation) and poor suitability of touchscreen gestures for the blind (preferred gestures start on the edge of the screen or in its close vicinity). The concept of our solution improving the accessibility is based on two assumptions, tested positively for example in [1] and [2]: most of the tested blind smartphone users deem notetakers to be indispensable, especially for taking notes quickly; the use of a linear-sequential Braille interface by the blind, especially for entering text using a physical Braille keyboard, is much more efficient than the use of a virtual keyboard (QWERTY or Braille). Advanced smartphone functions are made available through software that we developed as part of concept implementation. This part operates under the Android system and provides services for the notetaker operating under Windows CE like most ones. These services are called by specific software Volume 9 / Number 3 / 2014 operated under Windows CE and developed as second part of the concept. Both of these parts of software also support the application layer protocols and transmission layer. The Bluetooth it is transmission layer protocol and it is supported by the lowest level of communication software. Bluetooth protocol software runs on both devices and connects either the smartphone to the notetaker or the other way. And the application layer XML-based protocol has been developed by us. The protocol that transmits data from the notetaker of the service to be performed with the necessary parameters. As well as transfers to notetaker result of the service. Pilot implementation of the concept involved a Samsung Galaxy S III smartphone and a Polish notetaker Kajetek SD, see [3]; it included access to contact management and message composing services. Results of pilot implementation tests conducted on 5 blind users with different Braille and technical experience indicate a need for further development of the concept. 2. Related Work There are many different ways to interact with interfaces of touchscreen-based devices: a simple single tap, several simultaneous or successive taps, directional and scanning gestures as well as fixed and adaptive layouts of function fields comprising the screen. However, software developers often neglect to adapt their methods to the needs of a blind user. This is because researches are focused on overcoming the two greatest difficulties faced by the blind when using touchscreen-based mobile devices: enabling efficient navigation through graphical elements and their correct selection and developing a method for quick and correct text input. Findings of Kane et al. presented in [4] point to differences in preferred gestures and ways of making them between sighted and blind users. Oliviera 521

3 et al. demonstrated in [5] that the blind input text using different methods more or less efficiently depending on their personality traits and personal experience. D Andrea found out that most of the tested blind students prefer using smartphones with Braille notetakers and value their Braille skills, especially for the possibility of taking notes, see [1]. Southern et al. in turn noticed lower erroneousness in case of entering text in Braille, using a physical or virtual keyboard, see [2] and [5]; according to 4., the use of a physical Braille keyboard is the most efficient method. Azenkot et al. confirmed these findings in [6] and [2]. The virtual QWERTY keyboard has been enhanced e.g. by Findlater et al. in [7], applying extensions available through specific user gestures. In [8] Costagliola and Capua suggest implementing into the virtual keyboard an additional menu displayed upon making certain gestures around a given character. In [9] Ruamviboonsuk et al. suggest entering numbers sightlessly (to enable quicker dialling), using specific multi-touch combinations. Improvements suggested in [7], [8] and [9] have not been tested by the blind nor designed for this group of users. In [4] the authors suggest a fixed virtual Braille keyboard that would have 6 keys: 3 on the left and 3 on the right side of the screen, near its edges.[6] in turn suggests a dynamic virtual Braille keyboard appearing where 3 fingers of one hand or 6 fingers of two hands are placed. In [10] Kocieliński and Brzostek-Pawłowska have proposed multi-touch gestures preferred by blind to speed up navigation through the lists of items. 3. Concept of increasing the accessibility through the facilitated access to specialized services Our goal is allowing blind people to easily use such advanced functions of smartphones as speech recognition, cloud data (e.g. contacts) management, file sharing (e.g. Dropbox), web-based search engines, instant messaging, phonecalls or GSM and GPS navigation. Our concept makes it possible blind user to effective use all the functions of a smartphone using developed specialized services launched with Braille notetaker. Our solution eliminates the need for development aimed at implementing smartphone functions into notetakers. This way the blind would be able to use common mobile devices, not only to selected functions of smartphones developed in Braille notetaker. This goal we will achieve using integration of the two environments Windows CE of the notetaker and Android of the smartphone. In this research we have applied the results of our earlier researches presented in[10]. Their aim was to find ways of transforming the graphical objects on a Braille interface. As a results we have formulated basic assumptions concerning Versatile Multimodal Linear Interface (VMLI) and we have tested its pilot version. VMLI transforms a graphical objects and their features into the hierarchy of linear lists. In research in on the integration of smartphones and notetakers presented in this paper we have used the VMLI to be able generate a lists of items transmitted to the notetaker as an output parameters of executed services. We developed set of Android services and the protocol of access to services (application layer protocol) and of providing results of services. By integrating user will have easier access to advanced functionalities of smartphone using the physical Braille keyboard. Operation of notetakers is based mainly on Braille keystrokes and sequential-hierarchical access to smartphone functions. Preferred interaction methods, which speed up and facilitate blind users action, include: function selection lists activated by pressing appropriate navigation keys, handy menus available through hierarchical navigation or called using keystrokes, sets of Braille keystrokes that provide access to all the important notetaker functions at any time, editing functions that allow using six- and eight-dot Braille and navigating through the text quickly. 4. Description of Braille notekers The Braille notetakers are provided with Braille keyboard (6 or 8 keys) and with synthesis speech module which allow a user to access notetaker s functionalities. Less advanced notetakers are mainly used to edit text notes with the aid of Braille keyboard and to record voice notes. More 522 Volume 9 / Number 3 / 2014

4 advanced notetakers also have Braille displays which allow a blind user to read and edit notes in a more comfortable way. Notetakers can also be provided with complex editor, calculator, media player, Internet browser, mail client etc. Those notetakers are similar to the smartphones considering their functionality. The most advanced ones are similar to laptops of sighted users. On figure 1 there are examples of the notetakers: from simplest to most complex ones. function selection lists activated by pressing appropriate navigation keys, handy menus available through hierarchical navigation or called using keystrokes, sets of Braille keystrokes that provide access to all the important notetaker functions at any time, editing functions that allow using six- and eight-dot Braille and navigating through the text quickly. 5. Principles of communication between an Android-based device and a Braille notetaker Figure 1. a. A simple Braille notetaker which allows to edit text with the aid of Braille keyboard. b. A Braille notetaker which is provided with Braille keyboard and Braille display. c. A complex Braille notetaker which is provided with advanced functionality, such as Internet browser, media player etc. In general the Braille notetakers are operated under mobile operating systems (such as Windows CE, Windows Mobile), they can operate external SD/SDHC memory cards and (through buildin USB ports) another external memory drivers, printers etc. Simple Braille notetakers usually have limited editing functions; on the other hand more complex devices can be provided with MS Office package (Word, Excel), Internet browser, mail client, synchronizing functions with PC (for example to synchronize personal documents and contacts) etc. Braille notetakers can have a number of additional attractive functions such as e-book s player, dictaphone, speaking colour tester, stopwatch etc.. As opposed to the case of graphical smartphone interfaces, where users interact with applications by making touch gestures on items displayed on the screen, operation of notetakers is based mainly on Braille keystrokes and sequential-hierarchical access to functions. Preferred interaction methods, which speed up and facilitate blind users actions and are usually integrated in notetakers, include: Volume 9 / Number 3 / 2014 Our concept assumes that the Android-based device and the Braille notetaker communicate via Bluetooth, which is common in smartphones, programmable via API (Application Interface Programming) using the android.bluetooth package and supported by many Braille notetakers. Notetakers not equipped with an integrated Bluetooth adapter may be expanded with an external Bluetooth adapter. The Bluetooth protocol is a carrier layer for commands (queries) sent from the notetaker and for answers sent from the smartphone. A layer for the developed communication application protocol is provided by the Bluetooth protocol. The XML format is used for application protocol writing. The application used for communication between the two devices consists of two modules: Commander, which operates in Windows CE environment of the Braille notetaker, and Controller, which operates in Android environment. Commands generated by Commander and sent to Controller as well as answers generated by Controller, including any requested data and sent to Commander are based on the XML format. The notetaker sends queries with identifiers of services to be provided by the Controller module, which works as a service server. Figure 2 shows an example of a request-related XML command for service execution Create- ContactEntry. Figure 3 shows an example of an XML answer (transmitting service result.) 523

5 Figure 2. Example of a command with a request sent from the notetaker to the smartphone Figure 3. Example of an answer with the data sent from the smartphone to the notetaker after service execution The Controller module, developed in its pilot version as an Android service, can connect with the notetaker via Bluetooth in two ways: by manually choosing the notetaker from a list generated by Controller and sending a connection request to Commander or by accepting a Bluetooth connection request incoming from the notetaker s Commander module. Controller generates a list of devices available via Bluetooth as a result of detecting them by scanning, using the BluetoothAdapter class instance provided in the Android system. A link between the two devices is established upon successfully confirming the connection request sent to the notetaker or upon accepting the connection incoming from the notetaker. Then the Controller module waits for Commander commands and executes them. The smartphone s Controller module works in the following modes: sending Bluetooth connection requests to the notetaker (if chosen from the list by the user), waiting for Bluetooth connection requests incoming from the notetaker, accepting a Bluetooth connection request incoming from the notetaker, receiving and sending text and binary data (executing commands incoming from the notetaker). For the purposes of providing services requested by the notetaker, the Controller module works in server mode in relation to the Commander module. For running in server mode the Controller module keeps an open server socket (an instance of the Android BluetoothServerSocket class), which waits for incoming queries. The connection is accepted only if the notetaker has sent a query with a specific UUID value of ControllerService. The Commander module performs the following functions for the purposes of smartphone access: detecting Android-based devices (with the Controller module running) present within the range of Bluetooth communication, receiving text data from the paired Androidbased device, sending the input text commands to the Android-based device. The essential function of the Commander module is to send commands and data to the Androidbased device with a request of service execution. The transmitted commands they begin execution of services that allow e.g. employing the wireless keyboard functionality when the user inputs characters using a Braille keyboard. Characters 524 Volume 9 / Number 3 / 2014

6 are then sent as corresponding commands to the smartphone s Controller module via Bluetooth. Besides transmitting commands the Commander module receives answers as a result of services. All text data contained in answers is sent to the speech synthesiser and/or a Braille line. The following types of commands have been implemented in the Commander module for the purposes of sending: character a command of this type simulates inputting a given character by the user; A command corresponding to inputting the letter a by the user will have the following form: <command type= character >a</command> navigating a command of this type simulates pressing a certain navigation key (routing); A command corresponding to pressing the left arrow will have the following form: <command type= navigating >left</command>, function a command of this type simulates pressing a certain function key. Such command will have the following form: <command command> type= function >function_key</ Figure 4 shows an overview of the proposed communication concept. Figure 4. Communication layers between an Android-based device and a Braille note taker 6. Methodology In order to test the concept we have developed communication protocol as a sequence of XML query-response messages; the Bluetooth protocol serves as the transport layer. We have developed also two interacting module. One module runs on an Android-based smartphone and one on a notetaker (Kajetek SD - Polish product) operated under Windows CE 5. Both modules support wireless Bluetooth connectivity. The module on Android system has been implemented in Java, using Android API level 17, and tested on a Samsung Galaxy S III smartphone. Feedback is realised using text-to-speech, with the TalkBack screen reader. The notetaker software is a C++ library that extends Kajetek application with support for a protocol used to communicate with the Android module. During the research the basic concept assumptions have been tested by managing contacts using the remote interface of the notetaker. The pilot software allows the user to operate the smartphone using Kajetek as a physical Braille keyboard; this way he or she can quickly navigate through smartphone interface items using notetaker keystrokes. The software also provides easy wireless access to contact management functions of the smartphone. The following contact management services have been implemented in the Kajetek notetaker: browsing contact lists (using specific Braille keystrokes), reviewing details of selected contacts, adding new contacts (using the editing functions integrated in Kajetek software), deleting contacts. Several methods of inputting text into a smartphone have also been tested during the research. The test application provides three ways of composing messages to selected recipients: using a virtual QWERTY keyboard, a virtual Braille keyboard or the editing functions of the notetaker. Practical tests of text input speed have been conducted on a virtual QWERTY keyboard and on the physical keyboard of the Kajetek notetaker. The tests have been conducted on a group of 5 users with different experience in the use of smartphones: from a few weeks to a couple months. The Volume 9 / Number 3 /

7 test was carried out on the iphone with VoiceOver screen reader and on the smartphones Androidbased (version 4.02 or later) equipped with the TalkBack screen reader. All the testers knew Braille well and they were ones with experience in the practical use of Braille notetakers. 7. Results Practical tests of pilot application have shown that the use of a notetaker to operate the interface of a smartphone is not only a solution that improves a blind user s work efficiency and convenience, but also a practical alternative for those for whom a touch interface is manually inaccessible. Feedback from the test group confirms that the existing accessibility mechanisms of the Android system (such as TalkBack and BrailleBack), especially the ones related to navigation based on the available gestures (e.g. implemented through TalkBack) and text input using a virtual QWERTY keyboard, are subject to significant limitations. The tests have proven that managing contacts with the interface of the Kajetek notetaker and its physical Braille keyboard is more efficient than doing so using the standard smartphone application for managing contacts combined with the TalkBack screen reader, gestures and the virtual QWERTY keyboard. Selecting desired contacts from the list using Kajetek is more efficient primarily because of lesser erroneousness of browsing actions; operating a touch interface requires greater manual precision. One of the testers suggested extending the list item selection functionality of Kajetek e.g. with a quick search function looking for the first few entered letters of a contact. The received feedback also confirms that the existing smartphone accessibility mechanisms related to text input are insufficient. Message composing tests indicated a significant advantage of the physical Braille keyboard of a notetaker over a virtual QWERTY keyboard. The use of a Braille notetaker with its editing functions is much more efficient than using a virtual QWERTY keyboard: the slowest and the fastest tester reached a text input speed of 11 and 38 characters per minute (CPM) in the case of the latter and respectively 87 and 160 CPM in the case of the former. 8. Conclusions The results of preliminary research on pilot implementation into the Android system, concerning functions like managing contacts and composing messages to selected recipients, confirm that the adopted concept of using interaction mechanisms for touchscreen-based mobile devices and Braille notetakers is a promising direction for further research and give grounds for continuing works. The research suggests new ways of solving the essential problem, which is the inaccessibility of new, advanced technologies to the blind. The main cause of this problem is that hardware manufacturers and software developers usually marginalise this group of users by providing (mostly behindhand) only accessibility tools (such as screen readers) that rather adapt methods of operating graphical touch interfaces than provide other, suitable ones. The proposed solution for improving the accessibility of touch interfaces is quite unusual compared to those currently provided in smartphones: it is based on using a Braille interface, smart notetaker mechanisms and a dedicated communication protocol for convenient operation of advanced mobile device functions and services. The feedback from blind users is very positive. Currently we develop that conception and its application in national research project concerning the PlatMat platform [11] which aims is increasing efficiency of real-time math-related communication between teacher and blind students using mobile devices. References 1. d Andrea FM. Preferences and Practices Among Students Who Read Braille and Use Assistive Technology. Journal of Visual Impairment & Blindness, Year 2012, October-November 2012; 106: Southern C, Clawson J, Frey B, Abowd GD, Romero M. An Evaluation of BrailleTouch: Mobile Touchscreen Text Entry for the Visually Impaired. Proc. MobileHCI 12, ACM 2012; Information about Kajetek SD on the manufacturer s website: php?option=com_virtuemart&page=shop. browse&category_id=5&itemid=2 4. Kane SK, Wobbrock JO, Ladner RE. Usable gestures for blind people: Understanding preference and performance. Proc. CHI 11, ACM, 2011; Volume 9 / Number 3 / 2014

8 5. Oliveira J, Guerreiro T, Nicolau H, Jorge J, Gonçalves D. Blind people and mobile touch-based textentry: acknowledging the need for different flavors. Proc. ASSETS 11, ACM 2011; Azenkot S, Wobbrock JO, Prasain S, Ladner RE. Input finger detection for nonvisual touch screen text entry in Perkinput. Proc. Graphics Interface, Canadian Information Processing Society 2012; Findlater L, Lee BQ, Wobbrock JO. Beyond QWERTY: Augmenting Touch-Screen Keyboards with Multi-Touch Gestures for Non-Alphanumeric Input. Proc. CHI 21, ACM 2012; Costagliola G, Fuccella V, Di Capua M. Text Entry with KeyScretch. Proc. IUI, ACM 2011; Ruamviboonsuk V, Azenkot S, Ladner RE. Tapulator: A Non-Visual Calculator using Natural Prefix-Free Codes. Proc. ASSETS 12, ACM 2012; Kocieliński D, Brzostek-Pawłowska J. Linear Interface for Graphical Interface of Touch-Screen: a Pilot Study on Improving Accessibility of the Android-Based Mobile Devices, Proc. Mobile HCI 2013, ACM 2013; Brzostek-Pawłowska J, Mikułowski D. A Concept of Mobile Technology for Remotely Supporting Mathematical Education of the blind, Proc. REV 2014, 2014 IEEE, 2014; technics technologies education management Corresponding Author Jolanta Brzostek-Pawłowska, Institute of Mathematical Machines, Warsaw, Poland, j.brzostek@imm.org.pl Volume 9 / Number 3 /