Crowdsourcing for Speech Processing

Transcription

1 Crowdsourcing for Speech Processing Applications to Data Collection, Transcription and Assessment Editors Maxine Eskénazi Gina-Anne Levow Helen Meng Gabriel Parent David Suendermann

2

3 CROWDSOURCING FOR SPEECH PROCESSING

4

5 CROWDSOURCING FOR SPEECH PROCESSING APPLICATIONS TO DATA COLLECTION, TRANSCRIPTION AND ASSESSMENT Editors Maxine Eskénazi Carnegie Mellon University, USA Gina-Anne Levow University of Washington, USA Helen Meng The Chinese University of Hong Kong, SAR of China Gabriel Parent Carnegie Mellon University, USA David Suendermann DHBW Stuttgart, Germany A John Wiley & Sons, Ltd., Publication

6 This edition first published 2013 C 2013 John Wiley & Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Eskénazi, Maxine. Crowdsourcing for speech processing : applications to data collection, transcription, and assessment / Maxine Eskénazi, Gina-Anne Levow, Helen Meng, Gabriel Parent, David Suendermann. pages cm Includes bibliographical references and index. ISBN (hardback : alk. paper) ISBN (ebook/epdf) ISBN (epub) ISBN (mobi) ISBN Speech processing systems Research. 2. Human computation. 3. Data mining. I. Title. II. Title: Crowd sourcing for speech processing. TK7882.S65E dc A catalogue record for this book is available from the British Library. ISBN: Typeset in 10/12pt Times by Aptara Inc., New Delhi, India

7 Contents List of Contributors Preface xiii xv 1 An Overview 1 Maxine Eskénazi 1.1 Origins of Crowdsourcing Operational Definition of Crowdsourcing Functional Definition of Crowdsourcing Some Issues Some Terminology Acknowledgments 6 References 6 2 The Basics 8 Maxine Eskénazi 2.1 An Overview of the Literature on Crowdsourcing for Speech Processing Evolution of the Use of Crowdsourcing for Speech Geographic Locations of Crowdsourcing for Speech Specific Areas of Research Alternative Solutions Some Ready-Made Platforms for Crowdsourcing Making Task Creation Easier Getting Down to Brass Tacks Hearing and Being Heard over the Web Prequalification Native Language of the Workers Payment Choice of Platform in the Literature The Complexity of the Task Quality Control 29

8 vi Contents Was That Worker a Bot? Quality Control in the Literature Judging the Quality of the Literature Some Quick Tips Acknowledgments 33 References 33 Further reading 35 3 Collecting Speech from Crowds 37 Ian McGraw 3.1 A Short History of Speech Collection Speech Corpora Spoken Language Systems User-Configured Recording Environments Technology for Web-Based Audio Collection Silverlight Java Flash HTML and JavaScript Example: WAMI Recorder The JavaScript API Audio Formats Example: The WAMI Server PHP Script Google App Engine Server Configuration Details Example: Speech Collection on Amazon Mechanical Turk Server Setup Deploying to Amazon Mechanical Turk The Command-Line Interface Using the Platform Purely for Payment Advanced Methods of Crowdsourced Audio Collection Collecting Dialog Interactions Human Computation Summary Acknowledgments 69 References 70 4 Crowdsourcing for Speech Transcription 72 Gabriel Parent 4.1 Introduction Terminology Transcribing Speech The Need for Speech Transcription Quantifying Speech Transcription 75

9 Contents vii Brief History Is Crowdsourcing Well Suited to My Needs? Preparing the Data Preparing the Audio Clips Preprocessing the Data with a Speech Recognizer Creating a Gold-Standard Dataset Setting Up the Task Creating Your Task with the Platform Template Editor Creating Your Task on Your Own Server Instruction Design Know the Workers Game Interface Submitting the Open Call Payment Number of Distinct Judgments Quality Control Normalization Unsupervised Filters Supervised Filters Aggregation Techniques Quality Control Using Multiple Passes Conclusion Acknowledgments 103 References How to Control and Utilize Crowd-Collected Speech 106 Ian McGraw and Joseph Polifroni 5.1 Read Speech Collection Procedure Corpus Overview Multimodal Dialog Interactions System Design Scenario Creation Data Collection Data Transcription Data Analysis Games for Speech Collection Quizlet Voice Race Self-Transcribed Data Simplified Crowdsourced Transcription Data Analysis Human Transcription Automatic Transcription Self-Supervised Acoustic Model Adaptation 127

10 viii Contents 5.6 Voice Scatter Corpus Overview Crowdsourced Transcription Filtering for Accurate Hypotheses Self-Supervised Acoustic Model Adaptation Summary Acknowledgments 135 References Crowdsourcing in Speech Perception 137 Martin Cooke, Jon Barker, and Maria Luisa Garcia Lecumberri 6.1 Introduction Previous Use of Crowdsourcing in Speech and Hearing Challenges Control of the Environment Participants Stimuli Tasks Speech Intelligibility, Quality and Naturalness Accent Evaluation Perceptual Salience and Listener Acuity Phonological Systems BigListen: A Case Study in the Use of Crowdsourcing to Identify Words in Noise The Problem Speech and Noise Tokens The Client-Side Experience Technical Architecture Respondents Analysis of Responses Lessons from the BigListen Crowdsourcing Test Issues for Further Exploration Conclusions 169 References Crowdsourced Assessment of Speech Synthesis 173 Sabine Buchholz, Javier Latorre, and Kayoko Yanagisawa 7.1 Introduction Human Assessment of TTS Crowdsourcing for TTS: What Worked and What Did Not Related Work: Crowdsourced Listening Tests Problem and Solutions: Audio on the Web Problem and Solution: Test of Significance What Assessment Types Worked What Did Not Work 186

11 Contents ix Problem and Solutions: Recruiting Native Speakers of Various Languages Conclusion Related Work: Detecting and Preventing Spamming Our Experiences: Detecting and Preventing Spamming Optional Playback Interface Investigating the Metrics Further: Mandatory Playback Interface The Prosecutor s Fallacy Conclusions and Discussion 212 References Crowdsourcing for Spoken Dialog System Evaluation 217 Zhaojun Yang, Gina-Anne Levow, and Helen Meng 8.1 Introduction Prior Work on Crowdsourcing: Dialog and Speech Assessment Prior Work on Crowdsourcing for Dialog Systems Prior Work on Crowdsourcing for Speech Assessment Prior Work in SDS Evaluation Subjective User Judgments Interaction Metrics PARADISE Framework Alternative Approach to Crowdsourcing for SDS Evaluation Experimental Corpus and Automatic Dialog Classification Collecting User Judgments on Spoken Dialogs with Crowdsourcing Tasks for Dialog Evaluation Tasks for Interannotator Agreement Approval of Ratings Collected Data and Analysis Approval Rates and Comments from Workers Consistency between Automatic Dialog Classification and Manual Ratings Interannotator Agreement among Workers Interannotator Agreement on the Let s Go! System Consistency between Expert and Nonexpert Annotations Conclusions and Future Work Acknowledgments 238 References Interfaces for Crowdsourcing Platforms 241 Christoph Draxler 9.1 Introduction Technology TinyTask Web Page World Wide Web Hypertext Transfer Protocol 243

12 x Contents Hypertext Markup Language Cascading Style Sheets JavaScript JavaScript Object Notation Extensible Markup Language Asynchronous JavaScript and XML Flash SOAP and REST Section Summary Crowdsourcing Platforms Crowdsourcing Platform Workflow Amazon Mechanical Turk CrowdFlower Clickworker WikiSpeech Interfaces to Crowdsourcing Platforms Implementing Tasks Using a GUI on the CrowdFlower Platform Implementing Tasks Using the Command-Line Interface in MTurk Implementing a Task Using a RESTful Web Service in Clickworker Defining Tasks via Configuration Files in WikiSpeech Summary 278 References Crowdsourcing for Industrial Spoken Dialog Systems 280 David Suendermann and Roberto Pieraccini 10.1 Introduction Industry s Willful Ignorance Crowdsourcing in Industrial Speech Applications Public versus Private Crowd Architecture Transcription Semantic Annotation Subjective Evaluation of Spoken Dialog Systems Conclusion 300 References Economic and Ethical Background of Crowdsourcing for Speech 303 Gilles Adda, Joseph J. Mariani, Laurent Besacier, and Hadrien Gelas 11.1 Introduction The Crowdsourcing Fauna The Crowdsourcing Services Landscape Who Are the Workers? Ethics and Economics in Crowdsourcing: How to Proceed? Economic and Ethical Issues What Are the Problems for the Workers? 309

13 Contents xi Crowdsourcing and Labor Laws Which Economic Model Is Sustainable for Crowdsourcing? Under-Resourced Languages: A Case Study Under-Resourced Languages Definition and Issues Collecting Annotated Speech for African Languages Using Crowdsourcing Experiment Description Results Discussion and Lessons Learned Toward Ethically Produced Language Resources Defining a Fair Compensation for Work Done Impact of Crowdsourcing on the Ecology of Linguistic Resources Defining an Ethical Framework: Some Solutions Conclusion 330 Disclaimer 331 References 331 Index 335

14

15 List of Contributors Gilles Adda LIMSI-CNRS, France Jon Barker University of Sheffield, UK Laurent Besacier LIG-CNRS, France Sabine Buchholz SynapseWork Ltd, UK Martin Cooke Ikerbasque, Spain University of the Basque Country, Spain Christoph Draxler Ludwig-Maximilian University, Germany Maxine Eskénazi Carnegie Mellon University, USA Hadrien Gelas LIG-CNRS, France DDL-CNRS, France Javier Latorre Toshiba Research Europe Ltd, UK Gina-Anne Levow University of Washington, USA Maria Luisa Garcia Lecumberri University of the Basque Country, Spain Joseph J. Mariani LIMSI-CNRS, France IMMI-CNRS, France