Multivariate Data Analysis In Practice 5th Edition

Transcription

1 Multivariate Data Analysis In Practice 5th Edition An Introduction to Multivariate Data Analysis and Experimental Design Kim H. Esbensen Ålborg University, Esbjerg with contributions from Dominique Guyot Frank Westad Lars P. Houmøller CAMO Software AS. Nedre Vollgate 8, N-0158, Oslo, NORWAY Tel: (47) Fax: (47) CAMO Software Inc. One Woodbridge Center, Suite 319, Woodbridge, NJ 07095, USA Tel: (732) Fax: (973) CAMO Software India Pvt. Ltd. 14 & 15, Krishna Reddy Colony Domlur Layout, Bangalore , INDIA Tel: (91) Fax: (91)

2 This book was produced using Doc-to-Help together with Microsoft Word. Visio and Excel were used to make some of the illustrations. The screen captures were taken with Paint Shop Pro. Trademark Acknowledgments Doc-To-Help is a trademark of WexTech Systems, Inc. Microsoft is a registered trademark and Windows 95, Windows NT, Excel and Word are trademarks of the Microsoft Corporation. PaintShop Pro is a trademark of JASC, Inc. Visio is a trademark of the Shapeware Corporation. Information in this book is subject to change without notice. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of CAMO Process AS. ISBN CAMO Process AS All rights reserved. 5th edition. Re-print December 2004

3 Preface iii Preface October 2001 Learning to do multivariate data analysis is in many ways like learning to drive a car: You are not let loose on the road without mandatory training, theoretical and practical, as required by current concern for traffic safety. As a minimum you need to know how a car functions and you need to know the traffic code. On the other hand, everybody would agree that it is first after having obtained your drivers license that the real practical learning begins. This is when your personal experience really starts to accumulate. There is a strong interaction between the theory absorbed and the practice gained in this secondary, personal training period. Please substitute multivariate data analysis for driving a car in all of the above. Neither in this context are you let out on the data analytical road without mandatory training, theoretical and practical. The analogy is actually very apt! This book presents a basic theoretical foundation for bilinear (projection-based) multivariate data modeling and gives a conceptual framework for starting to do your own data modeling on the data sets provided. There are some 25 data sets included in this training package. By doing all exercises included you re off to a flying start! Driving your newly acquired multivariate data analysis car is very much an evolutionary process: this introductory textbook is filled with illustrative examples, many practical exercises and a full set of selfexamination real-world data analysis problems (with corresponding data sets). If, after all of this, you are able to work confidently on your own applications, you ll have reached the goal set for this book.

4 iv Preface This is the 5 th revised edition of this book. The three first editions were mainly reprints, the only major change being the inclusion of a completely revised chapter on Introduction to experimental design, which first appeared in the 3 rd edition (CAMO). The 4 th revised edition however (published March 2000) saw very many major extensions and improvements: Text completely rewritten by the senior author, based on five years of extensive use in teaching at both university and dedicated course levels. More than copies in use. 30% new theory & text material added, reflecting extensive student response, full integration of PCA, PLS1 & PLS2 NIPALS algorithms and explanations. Text revised with an augmented self-learning objective throughout. Four new master data sets added (with extended self-exercise potential): 1. Master violin data (PCA/PLS) 2. Norwegian car dealerships (PCA/PLS) 3. Vintages (PCA/PLS) 4. Acoustic chemometric calibration (PCR/PLS) Additional chapter on experimental design: new features include mixture designs and D-optimal designs. New chapter on the powerful, novel: Martens Uncertainty Test. Comprehensive glossary of terms. This 5th edition also includes essential additional revisions and improvements: Lars P. Houmøller, Ålborg University Esbjerg, has carried out a complete work-through of all demonstrations and exercises. Many of these had not been updated with respect to several of the intervening UNSCRAMBLER software versions. We are happy to have finally eliminated this most frustrating nuisance.

5 Preface v About the authors Kim H. Esbensen, Ph.D., has more than 20 years of experience in multivariate data analysis and applied chemometrics. He was professor in chemometrics at the Norwegian Telemark Institute of Technology (HIT/TF), Institute of Process Technology (PT) , where he was also head of the Chemometrics Department Tel-Tek, Telemark Industrial R&D Center, Porsgrunn. Between these institutions he founded ACRG: the Applied Chemometrics Research Group, HIT/TF- Tel-Tek, which a.o. hosted SSC6, the 6 th Scandinavian Symposium on Chemometrics, August 1999 as well as numerous other international courses, workshops and meetings. July 1 st, 2001 he moved to a position as research professor in Applied Chemometrics at Ålborg University, Esbjerg, Denmark (AUE), where he is currently leading ACACSRG: the Applied Chemometrics, Analytical Chemistry and Sampling Research Group. As the name implies, applied chemometrics activities continue in Esbjerg while new activities are added most notably through close collaboration with assoc. prof. Lars P. Houmøller, who independently built up the area of analytical chemistry/chemometrics at AUE before Prof. Esbensen s arrival. Most recently the discipline of sampling (proper sampling) has been added, in recognition of the immense importance of sampling in any data analytical discipline, including chemometrics. Kim H. Esbensen has published more than 60 papers and technical reports on a wide range of chemical, geochemical, industrial, technological, remote sensing, image analytic and acoustic chemometric applications. Together with Paul Geladi he has been instrumental in codeveloping the concept of Multivariate Image Analysis (MIA); with ACRG he pioneered the development of the novel area of acoustic chemometrics. His M. Sc. is from the University of Aarhus, Denmark in 1978 (geology, geochemistry), while a Ph.D. was conferred him by the Technical University of Denmark (DTH) in 1981 within the areas of metallurgy, meteoritics and multivariate data analysis. He then did post-doctoral work for two years with the Research Group for Chemometrics at the University of Umeå , after which he worked in a Swedish geochemical exploration company, Terra Swede, for two more years. Moving to Norway, this was followed by eight years as data analytical research scientist at the Norwegian Computing Center (NCC), Oslo,

6 vi Preface after which he became a senior research scientist at SINTEF, the Norwegian Foundation for Industrial and Technological Research for four additional years. In between these two assignments he was a visiting guest professor at Norsk Hydro s Research Center in Bergen, Norway. He also holds a position as Chercheur associé (now Chercheur affilié) du Centre de Recherche en Géomatique, Université Laval, Quebec. He is a member of the editorial board of Journal of Chemometrics, Wiley Publishers, and is a member of ICS, AGU and several other geological, data analytical and statistical associations. Dominique Guyot, educated in Statistics, Economics and Biomathematics (ENSAE and Université de Paris 7, France), has 15 years of experience in the field of chemometrics. She gained industrial experience from her work in the pharmaceutical and cosmetic industries, before joining CAMO from 1995 until With CAMO, Dominique worked as a Senior Consultant, and was particularly involved in food applications. She put together a practical strategy for efficient product development, based on experimental design and multivariate data analysis. This strategy was implemented in the Guideline + software package, complemented by an integrated training course focusing on multivariate methods for food product developers. Dominique is now studying music and singing at the Conservatoire of Trondheim, Norway. Frank Westad has a M. Sc. in physical chemistry from the University of Trondheim, Norway. He has 13 years experience in applied multivariate data analysis, and he completed a Ph.D. in multivariate regression in Frank has given numerous courses in experimental design and multivariate analysis for companies in Europe and in the U.S.A. His main research fields include variable selection, shift modelling and image analysis. Lars P. Houmøller has a M.Sc. in chemistry and physics from the University of Aarhus, Denmark. He has 12 years of experience in analytical chemistry and has worked 5-7 years with chemometrics. His teaching experiences include chemometrics, analytical chemistry, spectroscopy, physical chemistry, general and technical chemistry, organic and inorganic chemistry, unit operations and fluid dynamics. His research field covers NIR spectroscopic applications over a very broad industrial spectrum. He also has experience from working in the Danish food production industry.

7 Preface vii interaction with the authors: Kim Esbensen Dominique Guyot Frank Westad Lars P. Houmøller About this book Since 1986, when CAMO ASA first commercialized and started marketing THE UNSCRAMBLER, many customers have asked for basic, easy-to-understand literature on chemometrics. In 1993 a group of data analysts at different competence levels was invited to a one-day seminar at CAMO, Trondheim, for discussing their experience from both learning and teaching chemometrics. The result was a blue-print outline for what came to be this introductory book: the specifications called for a comprehensive training-package, involving basic, practical, easy-to-read, largely non-mathematical theory, with plenty of hands-on examples and exercises on real-world data sets. CAMO contracted SINTEF to write this book (first three editions), and the parties agreed to cooperate on the completion of the complete training package. In the intervening years, this book was published in some copies and was used for the introductory basic training in some 15 universities and in several hundred industrial companies; reactions were many and largely constructive. We learned a lot from these criticisms; we thank all who contributed! Came 1999, the time was ripe for a complete revision of the entire package. This was undertaken by the senior author in the summer 1999 with significant assistance from his then Ph.D. student Jun Huang (now with CAMO, Norway); Frank Westad (Matforsk) who wrote chapter 14 (Martens Uncertainty Test), Dominique Guyot (CAMO) who wrote the original new entire chapter 17 (Complex Experimental Design Problems), and with further invaluable editorial and managerical contributions from Michael Byström (CAMO) and Valérie Lengard (CAMO). A most sincere thank you goes to Peter Hindmarch (CAMO, UK) for very effective linguistic streamlining of the 4 th edition! The authors and CAMO also take this opportunity to acknowledge Suzanne Schönkopf s (CAMO) contribution to editions previous to the 4 th one.

8 viii Preface The present edition of this book still bears the fruit of her very important past efforts. The publication of the 4 th edition, in March 2000, was unfortunately somewhat marred by a less than complete revision of the exercises and illustrative UNSCRAMBLER runs in the book, which was not considered fatal at the time This soon proved to be a serious mistake; disapointment and frustration from several generations of students, who wanted to follow all the exercises closely, followed rapidly. A Danish university teacher, who had himself experienced this frustration close up when using the book for his own teachings, assoc. prof. Lars P. Houmøller at the University of Ålborg, Esbjerg voluntarily took it upon himself to carry out a complete work-through of this essential didactic aspect of the book. His very valuable demo and exercise revisions, as well as a very thorough text consistency check, have now been included in toto in the 5 th edition. Today, this book is a collaborative effort between the senior author and CAMO Process AS; the tie with SINTEF is now defunct. There is little academic glamour in writing an introductory level textbook, as the senior author has well experienced - which was never the goal anyway. But on the other hand, the introductory level is definitely where the largest audience and potential market exist, as CAMO has well experienced. The senior author has used the book for six consecutive years teaching introductory chemometrics largely to engineering (M.Sc.) students, as well as for extensive course work in industrial and foreign university environments. The response from some accumulated 500 students has made this author happy, while some 5500 sales have made CAMO equally satisfied. Thus all is well with the training package! We hope that this revised 5 th edition will continue to meet the challenging demands of the market, hopefully now in an improved form. Writing for precisely this introductory audience/market constitutes the highest scientific and didactic challenge, and is thus (still) irresistible!

9 Preface ix Acknowledgements The authors wish to thank the following persons, institutions and companies for their very valuable help in the preparation of this training package: Hans Blom, Østlandskonsult AS, Fredrikstad, Norway Frode Brakstad, Norsk Hydro F-Center, Porsgrunn, Norway Rolf Carlson, Department of Chemistry, University of Tromsø, Norway Chevron Research & Technology Co, Richmond, CA, USA Lennart Eriksson, Dept. of Organic Chemistry, University of Umeå, Sweden (now with Umetrics, Inc.) Professor Magni Martens, The Royal Vetarinary & Agricultural University, Denmark Geological Survey of Greenland, Denmark IKU, Institute for Petroleum Research, Trondhein, Norway Norwegian Food Research Institute (MATFORSK), Ås, Norway Norwegian Society of Process Control Norwegian Chemometrics Society International Chemometrics Society UOP Guided Wave, CA, USA Pierre Gy, Cannes, France (for a gentleman s introduction to the finest French wines) Zander & Ingerstrõm, Oslo, Norway Tomas Õberg Konsult AB, Karlskoga, Sweden KAPITAL (weekly Norwegian economic magazine), no 14/1994, p50-55 Hlif Sigurjonsdottir, Reykjavik, Iceland (owner of G. Sgarabotto violin no 9 ) Birgitta Spur, LSO, Reykjavik, Iceland (permission to use the Sgarabotto oeuvre data) Sensorteknikk A/S, Bærum, Oslo (Bjørn Hope: sensor technology entrepreneur extraordinaire; Evy: for innumerable occasions: warm company, coffee and waffles, waffles, waffles) Thorbjørn T. Lied, Maths Halstensen, Tore Gravermoen, Rune Mathisen a.o. (for enormous help in developing acoustic chemometrics) Anonymous wine importer, Odense, Denmark. Helpful wine assessors (partly anonymous), Manson, Wa, USA. Finally the author(s) and CAMO wish to thank all THE UNSCRAMBLER users during the last seven years for their close relationships with us, which have given us so much added experience in

10 x Preface teaching multivariate data analysis. And thanks for all the constructive criticism to the earlier editions of this book. Last, but certainly not least, a warm thank you to all the students at HIT/TF, at Ålborg University, Esbjerg and many, many others, who have been associated with the teachings of the authors, nearly all of whom have been very constructive in their ongoing criticism of the entire teaching system embedded in this training package. We even learned from the occasional not-so-friendly criticisms Communication The period of seven years that has been the formative period for the training package has come of age. By now we are actually beginning to be rather satisfied with it! And yet: The author(s) and CAMO always welcome all critical responses to the present text. They are seriously needed in order for this work to be continually improving.

11 Contents xi Contents 1. Introduction to Multivariate Data Analysis - Overview Indirect Observations and Correlation Hidden Data Structures Multivariate Data Analysis vs. Multivariate Statistics Main Objectives of Multivariate Data Analytical Techniques Multivariate Techniques as Projections Getting Started - with Descriptive Statistics Purpose Data Set 1: Quality of Green Peas Data set 2: Economic Characteristics of Car Dealerships in Norway Principal Component Analysis (PCA) Introduction Representing the Data as a Matrix The Variable Space - Plotting Objects in p Dimensions Plotting Objects in Variable Space Exercise - Plotting Raw Data (People) The First Principal Component Extension to Higher-Order Principal Components Principal Component Models - Scores and Loadings Model Center Loadings - Relations Between X and PCs Scores - Coordinates in PC Space Object Residuals Objectives of PCA Score Plot - Map of Samples Loading Plot - Map of Variables 40

12 xii Contents 3.10 Exercise: Plotting and Interpreting a PCA-Model (People) PC-Models The PC Model: X = TP T + E = Structure + Noise Residuals - The E-Matrix How Many PCs to Use? Variable Residuals More about Variances - Modeling Error Variance Exercise - Interpreting a PCA Model (Peas) Exercise - PCA Modeling (Car Dealerships) PCA Modeling The NIPALS Algorithm Principal Component Analysis (PCA) - In Practice Scaling or Weighting Outliers Scaling, Transformation and Normalization are Highly Problem Dependent Issues PCA Step by Step The Unscrambler and PCA Summary of PCA Interpretation of PCA-Models Interpretation of Score Plots Look for Patterns Summary - Interpretation of Score Plots Summary - Interpretation of Loading Plots PCA - What Can Go Wrong? Exercise - Detecting Outliers (Troodos) PCA Exercises Real-World Application Examples Exercise - Find Clusters (Iris Species Discrimination) Exercise - PCA for Experimental Design (Lewis Acids) Exercise - Mud Samples Exercise - Scaling (Troodos) Multivariate Calibration (PCR/PLS) Multivariate Modeling (X,Y): The Calibration Stage Multivariate Modeling (X, Y): The Prediction Stage Calibration Set Requirements (Training Data Set) Introduction to Validation Number of Components (Model Dimensionality) Univariate Regression (y x) and MLR 124

13 Contents xiii Univariate Regression (y x) Multiple Linear Regression, MLR Collinearity PCR - Principal Component Regression Exercise - Interpretation of Jam (PCR) Weaknesses of PCR PLS- Regression (PLS-R) PLS - A Powerful Alternative to PCR PLS (X,Y): Initial Comparison with PCA(X), PCA(Y) PLS2 NIPALS Algorithm Interpretation of PLS Models The PLS1 NIPALS Algorithm Exercise - Interpretation of PLS1 (Jam) Exercise - Interpretation PLS2 (Jam) When to Use which Method? Exercise - Compare PCR and PLS1 (Jam) Summary Validation: Mandatory Performance Testing The Concept of Test Set Validation Calculating the Calibration Variance (Modeling Error) Calculating the Validation Variance (Prediction Error) Studying the Calibration and Validation Variances Requirements for the Test Set Cross Validation Leverage Corrected Validation How to Perform PCR and PLS-R PLS and PCR - Step by Step Optimal Number of Components in Modeling Information in Later PCs Exercises on PLS and PCR: the Heart-of-the-Matter! Exercise - PLS2 (Peas) Exercise - PLS1 or PLS2? (Peas) Exercise - Is PCR better than PLS? (Peas) Multivariate Data Analysis in Practice: Miscellaneous Issues Data Constraints 181

14 xiv Contents Data Matrix Dimensions Missing Data Data Collection Use Historical Data Monitoring Data from an On-Going Process Data Generated by Planned Experiments Perform Experiments or Collect Data - Always by Careful Reflection The Random Design A Powerful Alternative Selecting from Abundant Data Selecting a Calibration Data Set from Abundant Training Data Selecting a Validation Data Set Error Sources Replicates - A Means to Quantify Errors Estimates of Experimental - and Measurement Errors Error in Y (Reference Method): Reproducibility Stability over Consecutive Measurements: Repeatability Handling Replicates in Multivariate Modeling Validation in Practice Test Set Cross Validation Leverage Correction The Multivariate Model Validation Alternatives How Good is the Model: RMSEP and Other Measures Residuals Residual Variances (Calibration, Prediction) Correction for Degrees of Freedom RMSEP and RMSEC - Average, Representative Errors in Original Units RMSEP, SEP and Bias Comparison Between Prediction Error and Measurement Error Compare RMSEP for Different Models Compare Results with Other Methods Other Measures of Errors Prediction of New Data Getting Reliable Prediction Results How Does Prediction Work? Prediction Used as Validation 210

15 Contents xv Uncertainty at Prediction Study Prediction Objects and Training Objects in the Same Plot Coding Category Variables: PLS-DISCRIM Scaling or Weighting Variables Using the B- and the Bw-Coefficients Calibration of Spectroscopic Data Spectroscopic Data: Calibration Options Interpretation of Spectroscopic Calibration Models Choosing Wavelengths PLS (PCR) Exercises: Real-World Application Examples - I Exercise - Prediction of Gasoline Octane Number Exercise - Water Quality Exercise - Freezing Point of Jet Fuel Exercise - Paper PLS (PCR) Multivariate Calibration In Practice Outliers and Subgroups Scores X-Y Relation Outlier Plots (T vs. U Scores) Residuals Dangerous Outliers or Interesting Extremes? Systematic Errors Y-Residuals Plotted Against Objects Residuals Plotted Against Predicted Values Normal Probability Plot of Residuals Transformations Logarithmic Transformations Spectroscopic Transformations Multiplicative Scatter Correction Differentiation Averaging Normalization Non-Linearities How to Handle Non-Linearities? Deleting Variables Procedure for Refining Models 264

16 xvi Contents 11.6 Precise Measurements vs. Noisy Measurements How to Interpret the Residual Variance Plot Summary: The Unscrambler Plots Revealing Problems PLS (PCR) Exercises: Real-World Applications - II Exercise ~ Log-Transformation (Dioxin) Exercise - Multiplicative Scatter Correction (Alcohol) Exercise Dirty Data (Geologic Data with Severe Uncertainties) Exercise - Spectroscopy Calibration (Wheat) Exercise QSAR (Cytotoxicity) Master Data Sets: Interim Examination Sgarabotto Master Violin Data Set Norwegian Car Dealerships - Revisited Vintages Acoustic Chemometrics (a. c.) Uncertainty Estimates, Significance and Stability (Martens Uncertainty Test) Uncertainty Estimates in Regression Coefficients, b Rotation of Perturbed Models Variable Selection Model Stability Introduction An Example Using the Paper Data Exercise - Paper - Uncertainty Test and Model Stability SIMCA: An Introduction to Classification SIMCA - Fields of Use How to Make SIMCA Class-Models? Basic SIMCA Steps: A Standard Flow-Sheet How Do we Classify new Samples? Classification Results Statistical Significance Level and its Use: An Introduction Graphical Interpretation of Classification Results The Coomans Plot The Si vs. Hi Plot (Distance vs. Leverage) 345

17 Contents xvii Si/S0 vs. Hi Model Distance Variable Discrimination Power Modeling Power SIMCA-Exercise IRIS Classification Introduction to Experimental Design Experimental Design Screening Designs Full Factorial Designs Fractional Factorial Designs Plackett-Burman Designs Analyzing a Screening Design Significant effects Using F-Test and P-Values to Determine Significant Effects Exercise - Willgerodt-Kindler Reaction Optimization Designs Central Composite Designs Box-Behnken Designs Analyzing an Optimization Design Exercise - Optimization of Enamine Synthesis Practical Aspects of Making an Experimental Design Extending a Design Validation of Designed Data Sets Problems in Designed Data Sets Detect and Interpret Effects How to Separate Confounded Effects? Blocking and Repeated Response Measurements Fold-Over Designs What Do We Do if We Cannot Keep to the Planned Variable Settings? A Random Design Modeling Uncoded Data Exercise - Designed Data with Non-Stipulated Values (Lacotid) Experimental Design Procedure in The Unscrambler Complex Experimental Design Problems 447

18 xviii Contents 17.1 Introduction to Complex Experimental Design Problems Constraints Between the Levels of Several Design Variables A Special Case: Mixture Situations Alternative Solutions The Mixture Situation An Example of Mixture Design Screening Designs for Mixtures Optimization Designs for Mixtures Designs that Cover a Mixture Region Evenly How To Deal With Constraints Introduction to the D-Optimal Principle Non-Mixture D-Optimal Designs Mixture D-Optimal Designs Advanced Topics How To Analyze Results From Constrained Experiments Use of PLS Regression For Constrained Designs Relevant Regression Models The Mixture Response Surface Plot Exercise ~ Build a Mixture Design - Wines Comparison of Methods for Multivariate Data Analysis - And their Validation Comparison of Selected Multivariate Methods Principal Component Analysis (PCA) Factor Analysis (FA) Cluster Analysis (CA) Linear Discriminant Analysis (LDA) Comparison: Projection Dimensionality in Multivariate Data Analysis Multiple Linear Regression, (MLR) Principal Component Regression (PCR) Partial Least Squares Regression (PLS-R) Increasing Projection Dimensionality in Regression Modeling Choosing Multivariate Methods Is Not Optional! Problem Formulation Unsupervised Methods Supervised Methods 503

19 Contents xix 18.5 A Final Discussion about Validation Test Set Validation Cross Validation Leverage Corrected Validation Selecting a Validation Approach in Practice Summary of Basic Rules for Success From Here You Are on Your Own. Good Luck! Literature Appendix: Algorithms PCA PCR PLS PLS Appendix: Software Installation and User Interface Welcome to The Unscrambler How to Install and Configure The Unscrambler Problems You Can Solve with The Unscrambler The Unscrambler Workplace The Editor The Viewer Dockable Views Dialogs The Help System Tooltips Using The Unscrambler Efficiently Analyses Some Tips to Make Your Work Easier 545 Glossary of Terms 549 Index 587