Let s get usabe! Usabiity studies for indexes Susan C. Oason The artice discusses a series of usabiity studies on indexes from a systems engineering and human factors perspective. The purpose of these studies is to estabish a set of user requirements that indexes need to satisfy. The resuts from the first set of studies are presented and suggestions made as to how these can be appied to improve the usabiity and quaity of indexes. Over the past few years, when I expained my new indexing career, a common thread kept surfacing. Most peope I spoke with referred to indexes as confusing for accessing information. Not wanting to perpetuate confusing indexes, I started researching index user requirements. Features and capabiities of products (indexes) are usuay derived from specific user requirements. I expected to find these user requirements or some sort of usabiity testing to point me away from confusing. What I actuay found in my research was a number of opinions on indexing, but no set of user requirements and very itte usabiity testing. Eizabeth Liddy and Corinne Jörgensen have done outstanding work in this area (Liddy and Jörgensen, 1993, 1996), but I was sti unabe to find a set of index requirements. I became concerned that we may have faen into the trap of indexing for indexers at the expense of our. In systems engineering, the most critica factor for ensuring quaity products is user invovement throughout the ife-cyce of that product. 1 In the best of a circumstances, conducting forma usabiity tests on a aspects of a product is the idea. But this can very rapidy become overwheming. Sophisticated equipment is required to track and record study participants. Human factors is necessary for designing and interpreting tests. On occasions, bypassing usabiity testing is rationaized in various ways: we assume we understand what the user wants, we caim ack of time or money, or we beieve that usabiity tests are just too compicated to conduct. However, whenever user invovement is bypassed, whether it be forma usabiity tests or usabiity studies, the product is ony marginay accepted or is rejected outright by the user. As professiona indexers we cannot afford to use such rationaizations to avoid conducting some sort of usabiity testing on our indexes. We are continuay striving to improve our indexing skis by attending workshops and conferences, studying books and artices, and querying d indexers. Usabiity studies are just one more too for earning how to improve our indexes. If we do not take advantage of this too, we run the risk of perpetuating incorrect assumptions of how peope access information using indexes. My background in systems engineering and my human factors give me a unique perspective of how to study access and use of information. In order to vaidate whether index features and capabiities satisfy actua user requirements, a reverse engineering approach was adopted. A method for discovering any unmet user requirements was aso deveoped. The foowing sections detai the study pan, the study packages, the study participants profie and the study resuts from the usabiity studies on back-of-the-book indexes. 2 The study resuts are transated into a practica rue-set for indexing. Study pan The primary purpose of an index is to support the user in practica appication of knowedge. Indexes accompish this by providing the most efficient access map to information or data pus context contained within the materia. The search for index requirements was begun by assessing the impact of the foowing index features on efficiency: run-on versus indented stye; sub-entries beginning with prepositions or conjunctions; access paths. One of the most straightforward methods for measuring efficiency is to time the competion of a task. The chaenge was to deveop tasks that tested the entire index without eading the study participant. Experiments with a question ist simpy resuted in a keyword search, since the answer was contained within the question; the user woud merey use a term from the question to access the index. Experiments with detaied task statements aso resuted in keyword searches, with the task statement eading the study participant to specific areas of the index. For exampe: Saes from your website have been disappointing. How woud you redesign your website, increase repeat traffic to your website, improve ead generation and quaification, and advertise offine and onine? This kind of task statement eads the study participant to website design, traffic generation, ead generation and quaification and advertising. In order to deve much deeper into access paths than just keyword searches, task statements were generaized. To compete the task, the study participant woud have to test the index fuy. The foowing tasks were deveoped to test each index: The Indexer Vo. 22 No. 2 October 2000 91
Oason: Let s get usabe! You want to compete in endurance riding events. How woud you seect a horse suitabe for endurance riding? Deveop a conditioning pan for you and your horse. You have finay decided to eave the city and move into the country. What business options are avaiabe? Choose one and deveop a pan for impementing that option. Saes from your Web Site have been disappointing. What steps wi you take to remedy this? To gain a better understanding of how an index is used in order to discover any unmet user requirements, it was necessary to capture the access paths taken during task competion for ater anaysis. This had to be accompished without actuay tracking the study participants eye movements. Severa months of experimenting with different methods ed to the use of finger pointing. Each study participant was asked to use their index finger to indicate where they were ooking. Dragging the index finger indicated scanning; moving it back and forth indicated reading. Access paths were then recorded, together with any comments made as the study participant competed the task. After this timed portion of the study was competed; each participant was asked why he or she had chosen particuar access paths. Study packages The study was based on three indexes with indexing densities of 8 14 entries per page. Each was modified to refect the variabe being studied. Each study consisted of a tota of 126 indexes: 63 (3 indexes, 21 copies of each) with the variabe being studied, 63 (3 indexes, 21 copies of each) without the variabe being studied. Each study package consisted of: Instructions: This is a study to determine how peope access information. You wi be asked to compete a task using the materia provided. Pease use your index finger to point to where you are ooking, side your index finger in a downward motion whenever you are scanning, and move your index finger back and forth whenever you are reading. I wi record your finger movements, any comments you wish to make, and wi time the task competion. Tabe of contents, front matter, page proofs, and index (in three-ring binder) Task to be competed (timed portion of study) Index usefuness ranking (circe one): Useess Neutra User-friendy matter (circe one): Novice Famiiar Expert When seecting a book for purchase, ist 3 factors that infuence your decision. Study participants profie Tabe 1. Characteristics of study participants (%) Age 17 20 15 21 40 35 41 60 35 61 80 15 Education High-schoo graduates 100 Coege graduates 50 Advanced degrees 20 Profession Students 10 Home-makers 20 Bue coar 30 White coar 35 Retirees 5 Race Caucasian 54 Minority 46 The characteristics of the study participants are shown in Tabe 1. The same group of 126 study participants were used for each of the three studies. Representing a good crosssection of index, they were assembed from friends and friends-of-friends-of-friends. The majority were uncompensated; ony 8 percent received engineering work, horseback riding essons, or horse-training sessions in return for their participation. Each study participant used each index ony once to prevent earning of tasks. Resuts The resuts from each study are transated into a practica rue-set for indexing at the end of this artice; the numbers in brackets in the foowing sections provide traceabiity to the rue-set. Study 1. Measure the impact on efficiency of run-on versus indented stye Run-on exampe: Water suppy, 35 50: chorination, 45; emergencies, 47; feeding and, 22; fitration systems, 36 38; pump use, 35 37; shows, 100; ; testing quaity, 46; traveing with, 95 96 Indented exampe: Water suppy, 35 50 chorination, 45 emergencies, 47 feeding and, 22 fitration systems, 36 38 pump use, 35 37 shows, 100 testing quaity, 46 traveing with, 95 96 Indented-stye indexes had overa higher efficiencies and higher usefuness rankings compared to run-on-stye indexes (Tabe 2), the overa efficiency gain of indented stye being 60 percent over run-on indexes [1.1]. 3 Indented indexes were ranked as user-friendy (90 percent of indexes) whie run-on indexes were never ranked in this category [1.2]. Indented-stye indexes were never ranked as useess; conversey, 70 percent of run-on indexes were ranked as 92 The Indexer Vo. 22 No. 2 October 2000
Oason: Let s get usabe! Tabe 2. Reative efficiency of run-on and indented index styes (%) useess [1.3]. Comments about run-on indexes incuded frustration about being forced to read rather than scan, confusion about sorting and confusion about which page references went with which sub-entry. There appeared to be no correation with user subject. Study 2. Measure the impact on efficiency of subentries beginning with prefix words Sub-entries with prefix words exampe: Water suppy, 35 50 chorination, 45 in emergencies, 47 and feeding, 22 fitration systems, 36 38 using pumps, 35 37 at shows, 100 testing quaity, 46 when traveing, 95 96 Sub-entries without prefix words exampe: Water suppy, 35 50 chorination, 45 emergencies, 47 feeding and, 22 fitration systems, 36 38 pump use, 35 37 shows, 100 testing quaity, 46 traveing with, 95 96 Run-on Efficiency +60 Novice 25 19 Famiiar 45 43 Expert 30 38 Indented Indexes that did not begin sub-entries with prefix words (prepositions or conjunctions) had higher efficiencies and higher usefuness rankings (Tabe 3), the efficiency gain being 50 percent [2.1]. Indexes were ranked (80 percent) as user-friendy when they did not use sub-entry prefix words compared to ony 40 percent for those indexes that did use sub-entry prefix words [2.2]. No index was ranked as useess. Comments about prefix words incuded frustration about being forced to read rather than scan, confusion about sorting ( did not reaize that prefix words were ignored in sort), and confusion about their purpose (did not carify the main entry/sub-entry reationship). Again, there was no correation with user subject. Tabe 3. Reative efficiency of entries beginning with prefix words (%) Subentries beginning with prefix words Efficiency +50 Usefuness ranking Useess Study participants were encouraged to use anything in the study package for competing the tasks to prevent bias toward index use over using other parts of the materia. The instructions simpy stated: This is a study to determine how peope access information. You wi be asked to compete a task using the materia provided. After the competion of the first two studies, 252 access paths were avaiabe for anaysis. Three main patterns of access were taken in competing the tasks. 1. If the study participants knew what term they were ooking for, they woud go directy to that specific term or concept [3.1]. These participants cassified themseves as famiiar or expert with the subject area. 2. If the participants did not know the specific term, they woud rey on past with simiar materia to seect a main entry, and scan the sub-entries unti they found a term that seemed to jog their memories. If they were unabe to find a famiiar sub-entry, they assumed the information was not there [3.2]. These participants cassified themseves as famiiar with the subject area. 3. If the study participants had no idea how to compete the task, they woud first ook for a main entry that refected the tite of the book. For exampe: if the book tite was Internet marketing made easy, they woud ook under Internet or marketing, much as one might use a Tabe of Contents. Athough a Tabe of Contents was avaiabe for their use, they tried to find this type of information in the index. After they were unabe to find this information in the index, they then went to the Tabe of Contents for cues. These study participants cassified themseves as novice or famiiar with the subject area. Indexes shoud support a eves of user subject. This study appears to suggest that an unmet user requirement has been identified: a tabe-of-contents-ike main entry to hep adopt more focused searches. The next stage of the investigation was to determine whether a tabe-of-contents-ike main entry with a see or see aso refer- Usefuness ranking Useess 70 Neutra 30 10 Userfriendy 90 Neutra 60 20 Userfriendy 40 80 Novice 22 30 Famiiar 50 50 Expert 48 20 Study 3. Access path anaysis and study Subentries not beginning with prefix words The Indexer Vo. 22 No. 2 October 2000 93
Oason: Let s get usabe! Tabe 4. Efficiency of indexes with tabe-of-contents-type main entries (%) ence to the other primary main entries (usuay chapter tites) woud hep the novice and famiiar. Tabe-of-contents entry exampe: Internet marketing, 28-47. See aso Advertising; Customer service; E-mai; Events and meetings; Fufiment; Lead generation; Newsgroups; Offine and onine marketing; Search toos; Seing onine; Web sites Indexes that used a tabe-of-contents-ike main entry had both higher efficiencies (55 percent efficiency gain) and higher usefuness rankings (Tabe 4) [3.3]. Indexes were ranked (75 percent) as user-friendy when they used a tabeof-contents-ike main entry, compared to ony 35 percent of those indexes that did not [3.4]. No index was ranked as useess. Users with a eves of subject, from novice to expert, commented that the tabe-of-contents-ike main entry heped them to define their search. Other comments from The foowing comments from the 378 index tasks studied are potentia index requirements. 4 Usabiity studies are continuing in these areas to determine if these index requirements reay are based on actua user requirements. Efficiency +55 Usefuness ranking Useess Indexes with tabe-of-contents main entry Neutra 25 65 Userfriendy 75 35 Novice 27 20 Famiiar 33 50 Expert 40 30 Indexes without tabe-of-contents main entry 90% stated that the index adequatey covered the subject matter [A]. 60% stated that combining a sub-entry page references for every mention of that sub-entry was miseading [B]. 55% stated that main entries that used undifferentiated page references pus sub-entries ooked unfinished, wasted their time by forcing them to check these page references, or these page references were ignored and the study participant possiby missed information [C]. 45% stated that main entries with chapter spreads were hepfu because they indicated a primary discussion [D]. 40% stated that undifferentiated page references (more than three) wasted their time by forcing them to check a page references [E]. 35% stated that commas in entries were confusing, that they expected page references to foow a comma [F]. 30% stated that turnover ines were confusing [G]. 20% stated they did not ike entries that continued for more than three coumns [H]. 15% stated that three index coumns per page was too busy [I]. <1% stated no sorting preference (the indexes were a mixture of etter-by-etter or word-by-word sort). <1% stated no preference for cross-entry use or crossentry pacement (the indexes were a mixture of cross references run-on from the main entry or as the ast subentry). <1% stated no preference for capitaization of main entries. <1% stated no opinion on the use of bod or itaic page references to indicate charts or iustrations. Transation of study findings to rue-set As professiona indexers, it is our responsibiity to suggest improvements and increase awareness of usabiity factors, but utimatey the editor or pubisher defines what a quaity index is. Whenever I see a potentia confict with my rue-set, I dipomaticay expain my rationae, but if required I aways foow the editor s or pubisher s specifications. Line-imited indexes wi aso impact the rue-set, but prioritization with the editor or pubisher wi usuay produce a workabe compromise. In the foowing rue-set, sha statements indicate index requirements based on actua user requirements that have been estabished by the usabiity studies aready undertaken. Shoud statements indicate potentia requirements. The number or etter in brackets foowing each rue inks it to specific study resuts or to other comments made (see previous section). The index shoud incude chapter tites, concepts, proper names, terms, tites, reationships, and subheads as entries [A]. Formatting rues: The index sha be in indented stye, rather than run-on stye [1.1, 1.2, 1.3]. The index shoud avoid turnover ines, if possibe [G]. The index shoud be in two rather than three coumns [I]. Main entry rues: Main entries sha incude a items (as sub-entries) discussed under main entries (exceptions: tites, proper names); anciary entries sha not be excuded [3.2]. There sha be a main entry that refects the book tite with see or see aso references to other primary main entries (tabe-of-contents entry) [3.3, 3.4]. Main entries shoud avoid undifferentiated page references when sub-entries are present [C]. Main entries with sub-entries shoud incude chapter spreads to indicate primary discussions [D]. Main entries shoud avoid undifferentiated page references with more than 3 5 page references; sub-entries shoud be created [E]. Main entries shoud be imited to no more than three coumns in ength [H]. 94 The Indexer Vo. 22 No. 2 October 2000
Oason: Let s get usabe! Tabe 5. Factors infuencing book seection (%) Book seection factors Main entries shoud be created for detaied discussions continuing for 10 or more pages with a see aso reference from the primary main entry [H]. Sub-entry rues: Sub-entries sha avoid prefix words (prepositions/ conjunctions); if necessary, prefix words sha be put at end of the sub-entry [2.1, 2.2]. Sub-entries sha be doube-posted (exceptions: defined, overview ), in addition to doube-posting acronyms and common/latin names [3.1]. Main entries shoud determine which sub-entry page references to incude [B]. For exampe: a main entry of Snowboarding with sub-entry Aspen, shoud ony incude those page references to snowboarding in Aspen and excude page references about horseback riding in Aspen or skiing in Aspen. Sub-entries shoud avoid commas, if possibe [F]. For exampe: business pan, deveoping shoud be rephrased as business pan deveopment. Concuding remarks Novice Famiiar Author 32 29 35 Index comparison 47 48 52 Price 20 15 10 Other factors 1 8 3 Expert When seecting a book for purchase, the three factors most often isted were as shown in Tabe 5. This emphasizes the vita importance of creating the most compete, most userfriendy indexes possibe. Indexes directy affect pubishers profits and add vaue to authors reputations by making their knowedge accessibe in a user-friendy manner. It is a chaenge for professiona indexers to ensure that we understand how indexes are used and continuay ook for ways to improve their usabiity. Conducting forma usabiity tests woud go a ong way toward that goa. In the absence of forma testing, usabiity studies such as that reported here can provide vauabe insight. Acknowedgements I woud ike to thank Nancy Muvany and Richard Evans for their vauabe insights. Notes 1. I confronted this fact time after time during my 20 years as a systems engineer/manager. 2. Usabiity studies shoud aso be conducted on website indexes. We may not be taking into account the impact of expectations or the fact that cicking gives utimate contro over what they view. 3. A percentages are rounded to the nearest whoe number. 4. The etters in brackets provide traceabiity to the rue-set outined beow. References Liddy, Eizabeth D. and Jörgensen, Corinne L. (1993) Reaity check! Book index characteristics that faciitate information access. In Proceedings from the American Society of Indexers 25th Annua Meeting, pp. 125 38. Port Aransas, TX: American Society of Indexers. Liddy, Eizabeth D. and Jörgensen, Corinne L. (1996) Information access or information anxiety? An exporatory evauation of book index features. The Indexer 20(2), 64 8. Susan C. Oason is the soe proprietor of Indexes & Knowedge Maps, providing professiona services in indexing books, corporate materia and websites. She speciaizes in business, computer science, equestrian and outdoor-ife subject areas. Susan has a Master s Degree in Business Management, is a past-president of the Coorado Chapter of the American Society of Indexers, and deveoped the Let s Get Profitabe! workshop. Emai: oeduke@ix.netcom.com Wondrous taxonomy Foucaut famousy opens The Order of Things by describing a certain Chinese encycopaedia, drawn from a story by Borges, in which it is written that animas are divided into: a) beonging to the Emperor, b) embamed, c) tame, d) sucking pigs, e) sirens, f) fabuous, g) stray dogs, h) incuded in the present cassification, i) frenzied, j) innumerabe, k) drawn with a very fine camehair brush, ) et cetera, m) having just broken the water pitcher, n) that from a ong way off ook ike fies. In the wonderment of this taxonomy, Foucaut continues, the thing we apprehend in one great eap, the thing that, by means of the fabe, is demonstrated as the exotic charm of another system of thought, is the imitation of our own, the stark impossibiity of thinking that. What the encounter with otherness forces us to rethink most insistenty are not so much the formaities and customs beoved of anthropoogica specuation as the itte things those patternings of mind so naturaized and automatic that we never think about them at a. In seeing the way they order things abroad, what we are brought up short against are the very things we do absent-mindedy the way we pocket a handkerchief or bear a grudge, crack an egg or, for that matter, watch a pay. (From Catherine Bates, Ceaning up Caiban, book review, Times Literary Suppement, 19 Nov. 1999) The Indexer Vo. 22 No. 2 October 2000 95