Digitizer Auditory Graph: Making Graphs Accessible to the Visually Impaired

Transcription

1 Digitizer Auditory Graph: Making Graphs Accessible to the Visually Impaired Stephen H. Choi Sonification Lab, School of Interactive Computing Georgia Institute of Technology 654 Cherry St. Atlanta, GA USA Bruce N. Walker Sonification Lab, Schools of Psychology and Interactive Computing Georgia Institute of Technology 654 Cherry St. Atlanta, GA USA Copyright is held by the author/owner(s). CHI 2010, April 10 15, 2010, Atlanta, Georgia, USA. ACM /10/04. Abstract This paper describes the design goal, design approach, and user testing of an assistive technology called Digitizer Auditory Graph a sonification software tool that allows users to upload or take an image of a line graph with an optical input device (e.g., webcam, digital camera, cell phone camera) and then hear an auditory graph of the digitized graph image. This technique enables visually impaired students to have a multimodal display of the information in a graph. Preliminary evaluation results indicate that both visually impaired and sighted people can understand the patterns of graphs by listening to auditory graph, and optical input allows them to have simple and fast output results. Keywords Sonification, auditory graph, assistive technology, visually impaired ACM Classification Keywords H5.2 User Interfaces: Auditory (non-speech) feedback. K4.2 Social Issues: Assistive technologies for persons with disabilities 3445

2 General Terms Human Factors Introduction Learning to interpret graphs of functions is a central task in the high school math curriculum. These graphs contain a concise but complete summary of the data, employing ordered pairs, axes, origin, grid lines, tick marks, labels, etc. Obviously, these components are all visual. As a result, for students with vision impairments, understanding data patterns or learning the concept of a graph becomes an extremely difficult task. Teachers working with visually impaired students also struggle to convey the concept of a line graph to their students. An alternative to the visual graph is to use sound to create an auditory graph, a type of sonification (the use of non-speech audio to convey information or perceptualize data). Walker, among others, has pointed out the benefits of auditory graphs in science and education, for both visually impaired and sighted users [7]. Despite the potential scientific and educational benefits, there are few available tools for creating auditory graphs. Further, the few toolkits available do not provide enough accessibility to the visually impaired. A significant barrier is that data input and graph creation methods are limited to the typical mouse and keyboard interface. To provide easier access for visually impaired students and their teachers to create auditory graphs, better input devices and simple interfaces are needed. The objective of this project is to process the image of a graph by uploading or capturing it with an optical input device, then to create an auditory graph that is accessible to visually impaired students and their teachers. This will allow them to make auditory graphs from input ranging from the graphs in text books and online, to their own handdrawn or hand-made graphs. Related Work Auditory Graphs An auditory graph is a sonified display that uses nonspeech sound to explore quantitative data. The most commonly-used representational technique in auditory graphs is to have changes in data values (i.e., changes along the visual y-axis) correspond to change in the pitch of sounds. Each data point is played by a separate musical note. Several researchers, such as Brewster [1], have shown that blind and visually impaired individuals can access information from these sonified line graphs. The idea of using sound for data display inspired the development of different types of auditory graph software. The Sonification Sandbox [8] is a widely used auditory graph toolkit that integrates contemporary auditory graph knowledge with modular software design. The current release of the Sonification Sandbox allows developers to use its Auditory Graph Model (AGM) library to import data and create auditory graphs in their own software. We leverage the AGM in the current project. Some have provided additional tactile or haptic feedback for auditory graphs. For example Graph Builder [4] allows blind people to construct bar graphs using a force feedback device. Assistive technology using optical input devices Assistive technologies for helping visually impaired users access printed materials and documents are divided into three categories: Braille, visual magnification, and speech. Optical input is mainly used for magnifying tools, but also for optical character recognition (OCR) software. Such software uses OCR 3446

3 Figure 1. Examples of using Wikki Stix on a raised line graph paper. Picture taken at the Georgia Academy for the Blind. and speech synthesizers to read scanned material aloud. Examples of such software include knfbreader Mobile [3] and the recently released Intel Reader [2]. One of the other notable examples is the voice [5] an augmented reality and soundscape-based synthetic vision software for the blind. It rasterizes across arbitrary video images and translate them into sounds to provide scene information. The system can also be used to read printed graphs; however, the voice differs from other auditory graph tools, in that it does not create any kind of intermediary data set that would allow for more flexible output of sophisticated auditory graphs based on musical notes. Background Information Graphs A graph is a visually compressed quantitative data representation and there are many different types of graphs point graph, bar graph, line graph, and box graph. In this paper, however, we aimed to help the visually impaired students who want to learn math; therefore, we are particularly interested in graphs that high school students learn. According to the Math I curriculum described in Mathematics Georgia Performance Standard [6], students must learn how to n graph basic functions f ( x) = x, where n = 1 to 3, 1 f ( x) = x, f ( x) = x, and f ( x) =. Graphs of basic x functions are all line graphs, which are very suitable for display via auditory graphs. Current Teaching Method In our collaboration visits to the Georgia Academy for the Blind (GAB), we have learned that there are no really satisfactory teaching methods for visually impaired students to learn how to produce a graph. Moreover, the ways students can draw or interact with graphs are limited. The teachers need to be very creative and try many approaches, hoping that the students connect with one of the methods. Some students learn their materials, including functions, in Braille, and type equations using a Braille embosser. However, reading and creating a Braille graph is difficult and burdensome. The net result is that in order to explain how a graph looks, teachers either describe it with words or take the student s hand and draw the shape of the graph in the air or on the desk. It can easily take up a whole class period to explain a single graph. Furthermore, teachers have expressed that they do not have much time for preparation in general, and creating tactile graphs is difficult in such limited time. To produce a graph, visually impaired students often use low-tech but effective tangible materials. Wikki Stix and Bendaroos are non-toxic twistable wax-covered strings that easily stick to each other and to any smooth surface. Using Wikki Stix on raised-line graph paper is one of the few successful methods for visually impaired students to form a graph with points and lines (Figure 1), and is often used in schools such as the GAB and the Texas School for the Blind and Visually Impaired. Another similar approach is to use push pins and rubber bands on a rubber graph board. The problem with these purely tactile tools is that students do not get any feedback about their graphs, or the data they represent, other than teachers comments. Many of the graphs that students make are incorrect, incomplete, or do not convey all the concepts that the curriculum specifies. 3447

4 Figure 2. Physical setup of Digitizer Auditory Graph Design Approach Physical Setup Visually impaired students tend to be enthusiastic when it comes to trying new technology, in part because of their currently limited alternatives. Like most teenagers, they get comfortable with new devices easily. Teachers, on the other hand, are not all as comfortable adopting new technologies. Many of them still prefer reliable lowtech devices. Therefore, we designed the physical setup of our tool to closely match normal educational devices. One of the classic and popular devices used in a classroom is the overhead projector. The main purpose is to capture and then project images onto a screen or wall for viewing. The same configuration is used in closed-circuit television systems for enlarging text. The Digitizer Auditory Graph system leverages this familiar form factor, with a webcam extending from an arm above a table (Figure 2). Bumps are placed at the corner of the board to help visually impaired students know where to put their paper, text book, or Wikki Stix figure. Digitizing The goal behind the system is to capture the pattern of the graph and to digitize data points from an image. Image processing can involve boundless tasks such as rectifying tilted or skewed images, processing OCR to read labels, distinguishing between two different lines of functions, etc. For the initial phase of this project, however, we focused on parsing line data on a contrasting background or paper as our first step. This phase is acceptable for digitizing graphs made out of Wikki Stix on a raised-line graph paper, or a line drawn on a blank paper or on whiteboard material. Assuming that the line of the graph is distinct from the background, we can perform edge detection on the image. Once the edge has been found, we can detect pixel coordinates of the line graph (Figure 3). The pixel coordinate values can be directly mapped to actual data points with a certain ratio, thereby representing the pattern of the graph. Starting from the leftmost coordinate to the rightmost coordinate, we record 40 x,y data points into a data file format such as a comma-separated values (CSV) file. Currently, the system records the top-most pixels along the x-axes, which limits the system to digitizing a single line Auditory Graph Model The newest version of the Sonification Sandbox handles the user interface (UI) and auditory graph and data model separately. The Auditory Graph Model (AGM) library in the Sonification Sandbox allows developers to control every single element of auditory graph rendering. These elements include pitch, timbre, polarity, pan, and volume along with graph contexts analogous to visual graphs. One of the features of the AGM library is importing a CSV file and creating a Sonification Sandbox project (SSP) file. Once a CSV file has been imported, the AGM library determines the minimum and maximum pitch of the auditory graph, in order to create a pleasant and comprehensible sound. With the default settings, the auditory graph plays discrete piano notes using middle C note as the middle point, with 10 second graph duration. It uses positive pitch polarity, meaning it plays high pitches for high data values and vice versa. The frequencies of notes are rounded to notes on the musical scale when played on the piano. Panning and volume control are disabled by default. When the pan option is enabled, it will play 3448

5 Figure 3. Result of edge detection of a graph image. the auditory graph in stereo, starting from left to right. Researchers are still determining the optimal standard settings for creating auditory graphs [9]; however, by using the AGM library in the Digitizer Auditory Graph, we have full control over how it can sound, thus allowing us to incorporate the best mappings as they are determined via empirical research. User Testing To begin to evaluate the utility and usability of the Digitizer Auditory Graph, we recruited four visually impaired adults from the Center for the Visually Impaired and four sighted graduate students from the Georgia Institute of Technology. The purpose of having two separate groups was to represent both visually impaired students and (sighted) teachers. We asked both groups to create four graphs of basic functions using Wikki Stix and raised-line graph paper, capture the image, listen to the output for each, and fill out a survey at the end of the study. Likert-style survey questions assessed predictability (i.e., To what extent did the flow of the tasks involved match your expectations?), overall experience (i.e., To what extent did you enjoy using the interface?), responsiveness and task conformance (i.e., To what extent did you feel that the interface responded in a desired manner?), and general issues on sound. Finally, we asked open-ended questions for both positive and negative feedback. The average rating (out of 7) for the visually impaired participants was 5.5 for predictability, 6.5 for overall experience, and 6 for responsiveness and task conformance. Also the average score on appropriateness of sound was 5.75, functionality of sound was 6, and likeability of sound was Some positive comments praised the system s quickness, the analogy of using pitch to data, simple and easy interface, and validation of hearing. Negative feedbacks include no text-to-speech description, difficulty in distinguish between a line and a curve, and no confirmation alert. The procedure for the sighted group added a few extra steps, such as those that might be completed by a teacher. Once they captured the image, we asked them to select the region of interest (ROI) with the mouse before producing the auditory graph. The average ratings for sighted participants were similar to those of the visually impaired: 6 on predictability, 5.5 on overall experience, and 5.5 on responsiveness and task conformance. Ratings on sound design were also similar. The average score for appropriateness of sound was 5.75, functionality of sound was 6, and likeability of sound was Positive feedback from sighted people included, simple and easy interface, intuitive, and fun to use tangible device. Some participants did not like selecting ROI because it made the operation a little complicated. A couple of participants thought that playing data in equally distributed time frame was not helpful in some cases. Overall, both groups were able to understand the pattern of graphs by listening to the auditory output and felt that the outcome matched what they had produced on a raised line graph paper. The result between the two groups showed no significant difference; however, visually impaired individuals appreciated the system slightly more. Moreover, some interesting discoveries were made. Despite the visually impaired students preferring the discrete tones overall, some expressed that they might prefer a continuous 3449

6 sound. Also, the lack of context information (e.g., axes, origin, tick marks) led to difficulty in figuring out detailed information such as the difference between two adjacent data points, slope, and point estimation. Discussion and Conclusion In this paper, we presented a novel interaction method of optical input for an auditory graph. We visited a school for the blind and found that teachers have a difficult time explaining graphs; and students have limited ways to access graphs. Based on these findings, we have developed an enhanced auditory graph tool with optical input to reduce the time and effort to create graphs and produce appropriate auditory feedback. The results of preliminary evaluations suggest that both visually impaired and sighted people can understand auditory output, and using optical input lets them create graphs easier and faster. There are many ways we plan to extend Digitizer Auditory Graph. One is to improve the design of the auditory graphs. As participants have indicated, the current implementation does a good job of presenting the pattern of data, but lacks in conveying context information such as axes, intercepts, tick marks and so on. These are all known issues that need to be addressed when creating auditory graphs, and as such the Auditory Graph Model and the Sonification Sandbox definitely include the capability to render context, axes, tick marks, etc. The software is also able to produce spoken descriptions of the graphs, based on the Math Description Engine. Thus, it will be straightforward to address these concerns in the Digitizer Auditory Graph system, simply by leveraging the existing functionality in the Auditory Graph Model. We also plan to expand the optical input devices from stationary webcams to mobile devices (e.g., mobile phones). Then we will perform a formal evaluation with both sighted and blind students and teachers. References [1] Brewster, S. A. (2002). Visualization tool for blind people using multiple modalities. Disability and Rehabilitation Technology, 24(11-12), [2] Intel Corporation. (2009). Intel Reader. Retrieved December, 2009, from [3] knfb Reading Technology Inc. KnfbReader Mobile. Retrieved December, 2009, from [4] McGookin, D.K. & Brewster, S.A. (2006). Graph Builder: Constructing Non-Visual Visualizations. In Proc. BCS HCI 2006 [5] Meijer, P. (2009). The voice. Retrieved December, 2009, from [6] Mathematics Georgia Performance Standards. Georgia Department of Education. Retrieved December, 2009, from 20Performance%20Standards/Math-I-Stds.pdf [7] Walker, B. N., & Lane, D. M. (2001). Psychophysical scaling of sonification mappings: A comparison of visually impaired and sighted listeners. In Proc. ICAD 2001, [8] Walker, B. N., & Cothran, J. T. (2003). Sonification Sandbox: A graphical toolkit for auditory graphs. In Proc. ICAD 2003, [9] Walker, B. N., & Kramer, G. (2005). Mappings and metaphors in auditory displays: An experimental assessment. ACM Transactions on Applied Perception, 2(4),