Predictive Analytics, Data Mining and Big Data

Transcription

1

2 Predictive Analytics, Data Mining and Big Data

3 This page intentionally left blank

4 Predictive Analytics, Data Mining and Big Data Myths, Misconceptions and Methods Steven Finlay

5 Steven Finlay 2014 All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission. No portion of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, Saffron House, 6 10 Kirby Street, London EC1N 8TS. Any person who does any unauthorized act in relation to this publication may be liable to criminal prosecution and civil claims for damages. The author has asserted his right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act First published 2014 by PALGRAVE MACMILLAN Palgrave Macmillan in the UK is an imprint of Macmillan Publishers Limited, registered in England, company number , of Houndmills, Basingstoke, Hampshire RG21 6XS. Palgrave Macmillan in the US is a division of St Martin s Press LLC, 175 Fifth Avenue, New York, NY Palgrave Macmillan is the global academic imprint of the above companies and has companies and representatives throughout the world. Palgrave and Macmillan are registered trademarks in the United States, the United Kingdom, Europe and other countries. ISBN This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. Logging, pulping and manufacturing processes are expected to conform to the environmental regulations of the country of origin. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. Typeset by MPS Limited, Chennai, India.

6 To Ruby and Samantha

7 This page intentionally left blank

8 Contents Figures and Tables Acknowledgments x xii 1 Introduction What are data mining and predictive analytics? How good are models at predicting behavior? What are the benefits of predictive models? Applications of predictive analytics Reaping the benefits, avoiding the pitfalls What is Big Data? How much value does Big Data add? The rest of the book 19 2 Using Predictive Models What are your objectives? Decision making The next challenge Discussion Override rules (business rules) 36 3 Analytics, Organization and Culture Embedded analytics Learning from failure A lack of motivation A slight misunderstanding Predictive, but not precise Great expectations Understanding cultural resistance to predictive analytics The impact of predictive analytics 60 vii

9 viii Contents 3.9 Combining model-based predictions and human judgment 62 4 The Value of Data What type of data is predictive of behavior? Added value is what s important Where does the data to build predictive models come from? The right data at the right time How much data do I need to build a predictive model? 79 5 Ethics and Legislation A brief introduction to ethics Ethics in practice The relevance of ethics in a Big Data world Privacy and data ownership Data security Anonymity Decision making 99 6 Types of Predictive Models Linear models Decision trees (classification and regression trees) (Artificial) neural networks Support vector machines (SVMs) Clustering Expert systems (knowledge-based systems) What type of model is best? Ensemble (fusion or combination) systems How much benefit can I expect to get from using an ensemble? The prospects for better types of predictive models in the future The Predictive Analytics Process Project initiation Project requirements Is predictive analytics the right tool for the job? Model building and business evaluation Implementation 145

10 Contents ix 7.6 Monitoring and redevelopment How long should a predictive analytics project take? How to Build a Predictive Model Exploring the data landscape Sampling and shaping the development sample Data preparation (data cleaning) Creating derived data Understanding the data Preliminary variable selection (data reduction) Pre-processing (data transformation) Model construction (modeling) Validation Selling models into the business The rise of the regulator Text Mining and Social Network Analysis Text mining Using text analytics to create predictor variables Within document predictors Sentiment analysis Across document predictors Social network analysis Mapping a social network Hardware, Software and All that Jazz Relational databases Hadoop The limitations of Hadoop Do I need a Big Data solution to do predictive analytics? Software for predictive analytics 206 Appendix A. Glossary of Terms 209 Appendix B. Further Sources of Information 218 Appendix C. Lift Charts and Gain Charts 223 Notes 227 Index 246