Computational Statistics Handbook with MATLAB

Transcription

1 «H Computer Science and Data Analysis Series Computational Statistics Handbook with MATLAB Second Edition Wendy L. Martinez The Office of Naval Research Arlington, Virginia, U.S.A. Angel R. Martinez Naval Surface Warfare Center Dahlgren, Virginia, U.S.A. Chapman &. Hall/CRC Taylor & Francis Group Boca Raton London New York Chapman & Hall/CRC is an imprint of the Taylor & Francis Group, an informa business

2 Table ofcontents Preface to the Second Edition Preface to the First Edition xvii xxi Chapter 1 Introduction 1.1 What Is Computational Statistics? An Overview of the Book 2 Philosophy 2 What Is Covered 3 A Word About Notation MATLAB Code 6 Computational Statistics Toolbox 7 Internet Resources Further Reading 9 Chapter 2 Probability Concepts 2.1 Introduction Probability 12 Background 12 Probability 14 Axioms of Probability Conditional Probability and Independence 17 Conditional Probability 17 Independence 18 Bayes' Theorem Expectation 21 Mean and Variance 21 Skewness 23 Kurtosis Common Distributions 24 Binomial 24 Poisson 26 Uniform 29 Normal 31 vii

3 viii Computational Statistics Handbook with MATLAB, 2 ND Edition Exponential 34 Gamma 36 Chi-Square 37 Weibull 38 Beta 40 Student's t Distribution 41 Multivariate Normal 44 Multivariate t Distribution MATLAB Code Further Reading 49 Exercises 52 Chapter 3 Sampling Concepts 3.1 Introduction Sampling Terminology and Concepts 55 Sample Mean and Sample Variance 57 Sample Moments 58 Covariance Sampling Distributions Parameter Estimation 65 Bias 66 MeanSquared Error 66 Relative Efficiency 67 Standard Error 67 Maximum Likelihood Estimation 68 Method of Moments Empirical Distribution Function 72 Quantiles MATLAB Code Further Reading 78 Exercises 80 Chapter 4 Generating Random Variables 4.1 Introduction General Techniques for Generating Random Variables 83 Uniform Random Numbers 83 Inverse Transform Method 86 Acceptance-Rejection Method Generating Continuous Random Variables 93 Normal Distribution 93 Exponential Distribution 94 Gamma 95

4 Table ofcontents ix Chi-Square 98 Beta 99 Multivariate Normal 101 Multivariate Student's t Distribution 103 Generating Variates on a Sphere Generating Discrete Random Variables 107 Binomial 107 Poisson 108 Discrete Uniform MATLAB Code Further Reading 113 Exercises 115 Chapter 5 Exploratory Data Analysis 5.1 Introduction Exploring Univariate Data 119 Histograms 119 Stem-and-Leaf 122 Quantile-Based Plots - Continuous Distributions 124 Quantile Plots - Discrete Distributions 132 Box Plots Exploring Bivariate and Trivariate Data 145 Scatterplots 145 Surface Plots 146 Contour Plots 148 Bivariate Histogram D Scatterplot Exploring Multi-Dimensional Data 158 Scatterplot Matrix 158 Slices and Isosurfaces 160 Glyphs 166 Andrews Curves 168 Parallel Coordinates MATLAB Code Further Reading 181 Exercises 183 Chapter 6 Finding Structure 6.1 Introduction Projecting Data Principal Component Analysis Projection Pursuit EDA 195

5 x Computational Statistics Handbook with MATLAB, 2 ND Edition Projection Pursuit Index 197 Finding the Structure 198 Structure Removal Independent Component Analysis Grand Tour Nonlinear Dimensionality Reduction 216 Multidimensional Scaling 216 Isometric Feature Mapping - ISOMAP MATLAB Code Further Reading 227 Exercises 230 Chapter 7 Monte Carlo Methods for Inferential Statistics 7.1 Introduction Classical Inferential Statistics 234 Hypothesis Testing 234 Confidence Intervals Monte Carlo Methods for Inferential Statistics 246 Basic Monte Carlo Procedure 246 Monte Carlo Hypothesis Testing 247 Monte Carlo Assessment of Hypothesis Testing Bootstrap Methods 256 General Bootstrap Methodology 256 Bootstrap Estimate of Standard Error 258 Bootstrap Estimate of Bias 260 Bootstrap Confidence Intervals MATLAB Code Further Reading 269 Exercises 271 Chapter 8 Data Partitioning 8.1 Introduction Cross-Validation Jackknife Better Bootstrap Confidence Intervals Jackknife-After-Bootstrap MATLAB Code Further Reading 296 Exercises 298

6 Table of Contents xi Chapter 9 Probability Density Estimation 9.1 Introduction Histograms D Histograms 303 Multivariate Histograms 309 Frequency Polygons 311 Averaged Shifted Histograms Kernel Density Estimation 322 Univariate Kernel Estimators 322 Multivariate Kernel Estimators Finite Mixtures 329 Univariate Finite Mixtures 331 Visualizing Finite Mixtures 333 Multivariate Finite Mixtures 335 EM Algorithm for Estimating the Parameters 338 Adaptive Mixtures Generating Random Variables MATLAB Code Further Reading 357 Exercises 359 Chapter 10 Supervised Learning 10.1 Introduction Bayes Decision Theory 365 Estimating Class-Conditional Probabilities: Parametric Method 367 Estimating Class-Conditional Probabilities: Nonparametric 369 Bayes Decision Rule 370 Likelihood Ratio Approach Evaluating the Classifier 380 Independent Test Sample 380 Cross-Validation 382 Receiver Operating Characteristic (ROC) Curve Classification Trees 390 Growing the Tree 394 Pruning the Tree 399 Choosing the Best Tree 403 Other Tree Methods Combining Classifiers 414 Bagging 415 Boosting 417 Arcing Classifiers 420 Random Forests MATLAB Code 423

7 xii Computational Statistics Handbook with MATLAB 9, 2 ND Edition 10.7 Further Reading 424 Exercises 428 Chapter 11 Unsupervised Learning 11.1 Introduction Measuresof Distance Hierarchical Clustering K-Means Clustering Model-Based Clustering 445 Finite Mixture Models and the EM Algorithm 446 Model-Based Agglomerative Clustering 450 Bayesian Information Criterion 453 Model-Based Clustering Procedure Assessing Cluster Results 458 Mojena - Upper Tail Rule 458 Silhouette Statistic 459 Other Methods for Evaluating Clusters MATLAB Code Further Reading 466 Exercises 469 Chapter 12 Parametric Models 12.1 Introduction Spline Regression Models Logistic Regression 482 Creating the Model 482 Interpreting the Model Parameters Generalized Linear Models 488 Exponential Family Form 489 Generalized Linear Model 494 Model Checking MATLAB Code Further Reading 509 Exercises 511 Chapter 13 Nonparametric Models 13.1 Introduction Some Smoothing Methods 514 Bin Smoothing 515 RunningMean 517

8 Table ofcontents xiii Running Line 518 Local Polynomial Regression - Loess 519 Robust Loess Kernel Methods 528 Nadaraya-Watson Estimator 531 Local Linear Kernel Estimator Smoothing Splines 534 Natural Cubic Splines 536 Reinsch Method for Finding Smoothing Splines 537 Values for a Cubic Smoothing Spline 540 Weighted Smoothing Spline Nonparametric Regression - Other Details 542 Choosing the Smoothing Parameter 542 Estimation of the Residual Variance 547 Variability of Smooths Regression Trees 551 Growing a Regression Tree 553 Pruning a Regression Tree 557 Selecting a Tree Additive Models MATLAB Code Further Reading 570 Exercises 573 Chapter 14 Markov Chain Monte Carlo Methods 14.1 Introduction Background 576 Bayesian Inference 576 Monte Carlo Integration 577 Markov Chains 579 Analyzing the Output Metropolis-Hastings Algorithms 580 Metropolis-Hastings Sampler 581 Metropolis Sampler 584 Independence Sampler 587 Autoregressive Generating Density The Gibbs Sampler Convergence Monitoring 602 Gelman and Rubin Method 604 Raftery and Lewis Method MATLAB Code Further Reading 610 Exercises 612

9 xiv Computational Statistics Handbook with MATLAB, 2 ND Edition Chapter 15 Spatial Statistics 15.1 Introduction 617 What Is Spatial Statistics? 617 Types of Spatial Data 618 Spatial Point Patterns 619 Complete Spatial Randomness Visualizing Spatial Point Processes Exploring First-order and Second-order Properties 627 Estimating the Intensity 627 Estimating the Spatial Dependence Modeling Spatial Point Processes 638 Nearest Neighbor Distances 638 IC-Function Simulating Spatial Point Processes 646 Homogeneous Poisson Process 647 Binomial Process 650 Poisson Cluster Process 651 Inhibition Process 654 Strauss Process MATLAB Code Further Reading 659 Exercises 661 Appendix A Introduction to MATLAB A.l What Is MATLAB? 663 A.2 Getting Help in MATLAB 664 A.3 File and Workspace Management 664 A.4 Punctuation in MATLAB 666 A.5 Arithmetic Operators 666 A.6 Data Constructs in MATLAB 668 Basic Data Constructs 668 Building Arrays 668 CellArrays 669 A.7 Script Files and Functions 670 A.8 Control Flow 672 For Loop 672 WhileLoop 672 If-Else Statements 673 Switch Statement 673 A.9 Simple Plotting 673 A.10 Contact Information 676

10 Table ofcontents xv Appendix B Projection Pursuit Indexes B.l Indexes 677 Friedman-Tukey Index 677 Entropy Index 678 Moment Index 678 L 2 Distances 679 B.2 MATLAB Source Code 680 Appendix C MATLAB Statistics Toolbox File I/O 687 Dataset Arrays 687 GroupedData 687 Descriptive Statistics 688 Statistical Visualization 688 Probability Density Functions 689 Cumulative Distribution Functions 690 Inverse Cumulative Distribution Functions 691 Distribution Statistics Functions 691 Distribution Fitting Functions 692 Negative Log-Likelihood Functions 692 Random Number Generators 693 Hypothesis Tests 694 Analysis of Variance 694 Regression Analysis 694 Multivariate Methods 695 Cluster Analysis 696 Classification 696 Markov Models 696 Design of Experiments 697 Statistical Process Control 697 Graphical User Interfaces 697 Appendix D Computational Statistics Toolbox Probability Distributions 699 Statistics 699 Random Number Generation 700 Exploratory Data Analysis 700 Bootstrap and Jackknife 701 Probability Density Estimation 701 Supervised Learning 701 Unsupervised Learning 701

11 xvi Computational Statistics Handbook with MATLAB, 2 ND Edition Parametric and Nonparametric Models 702 Markov Chain Monte Carlo 702 Spatial Statistics 702 Appendix E Exploratory Data Analysis Toolboxes E.l Introduction 703 E.2 Exploratory Data Analysis Toolbox 704 E.3 EDA GUI Toolbox 705 Appendix F Data Sets Introduction 719 Appendix G Notation Overview 727 ObservedData 727 Greek Letters 728 Functions and Distributions 728 Matrix Notation 729 Statistics 729 References 731 Author Index 751 Subject Index 757