Stochastic Approximation Algorithms and Applications

Transcription

1 Harold J. Kushner G. George Yin Stochastic Approximation Algorithms and Applications With 24 Figures Springer

2 Contents Preface and Introduction xiii 1 Introduction: Applications and Issues Outline of Chapter The Robbins-Monro Algorithm Introduction Finding the Zeros of an Unknown Function A Linear Pattern Classifier: Best Linear Least Squares Fit Minimization by Recursive Monte Carlo The Kiefer-Wolfowitz Procedure The Basic Procedure Random Directions Extensions of the Algorithms: Variance Reduction, Robustness, Iterate Averaging, Constraints, and Convex Optimization A Variance Reduction Method Constraints Averaging of the Iterates: "Polyak Averaging" Robust Algorithms Nonexistence of the Derivative at Some Applications to Learning, State Dependent Noise, and Queueing Outline of Chapter 25

3 viii Contents 2.1 An Animal Learning Model A Neural Network Q-Learning State Dependent Noise: A Motivational Example : Optimization of a GI/G/1 Queue Derivative Estimation and Infinitesimal Perturbation Analysis: A Brief Review The Derivative Estimate for the Queueing Problem Passive Stochastic Approximation 45 3 Applications in Signal Processing and Adaptive Control Outline of Chapter Parameter Identification and Tracking The Classical Model ARMA and ARMAX Models Tracking Time Varying Systems: An Adaptive Step Size Algorithm The Algorithm Some Data Feedback and Averaging in the Identification Algorithm Applications in Communication Theory Adaptive Noise Cancellation and Disturbance Rejection Adaptive Equalizers 62 4 Mathematical Background Outline of Chapter Martingales, Submartingales, and Inequalities Ordinary Differential Equations Limits of a Sequence of Continuous Functions Stability of Ordinary Differential Equations Projected ODE, Stochastic Stability and Perturbed Stochastic Liapunov Functions 80 5 Convergence with Probability One: Martingale Difference Noise Outline of Chapter Truncated Algorithms: Introduction The ODE Method: A Basic Convergence Theorem Assumptions and the Main Convergence Theorem Chain Recurrence A General Compactness Method The Basic Convergence Theorem 107

4 Contents ix Sufficient Conditions for the Rate of Change Condition The Kiefer-Wolfowitz Algorithm Stability and Stability-ODE Methods Soft Constraints Random Directions, Subgradients, and Differential Inclusions Convergence for the Lizard Learning and Pattern Classification Problems The Lizard Learning Problem The Pattern Classification Problem Convergence to a Local Minimum: A Perturbation Method 127 Convergence with Probability One: Correlated Noise Outline of Chapter A General Compactness Method Introduction and General Assumptions The Basic Convergence Theorem Local Convergence Results Sufficient Conditions for the Rate of Change Assumptions: Laws of Large Numbers Perturbed State Criteria for the Rate of Change Assumptions Introduction to Perturbed Test Functions General Conditions for the Asymptotic Rate of Change Alternative Perturbations Examples Using State Perturbation Kiefer-Wolfowitz Algorithms A State Perturbation Method and State Dependent Noise Stability Methods Differential Inclusions and the Parameter Identification Problem State Perturbation-Large Deviations Methods Large Deviations Estimates Two-Sided Estimates Upper Bounds and Weaker Conditions Escape Times 182 Weak Convergence: Introduction Outline of Chapter Introduction Martingale Difference Noise 189

5 x Contents 7.3 Weak Convergence Definitions Basic Convergence Theorems Martingale Limit Processes and the Wiener Process Verifying that a Process Is a Martingale The Wiener Process A Perturbed Test Function Method for Verifying Tightness and the Wiener Process Weak Convergence Methods for General Algorithms Outline of Chapter Assumptions: Exogenous Noise and Constant Step Size Convergence: Exogenous Noise Constant Step Size: Martingale Difference Noise Correlated Noise Step Size e n -» Random e n Differential Inclusions The Kiefer-Wolfowitz Algorithm Martingale Difference Noise Correlated Noise Markov State Dependent Noise Constant Step Size Decreasing Step Size e n -* The Invariant Measure Method: Constant Step Size An Alternative Form Unconstrained Algorithms Applications: Proofs of Convergence Outline of Chapter Average Cost per Unit Time Criteria: Introduction General Comments A Simple Illustrative SDE Example A Continuous Time Stochastic Differential Equation Example A Discrete Example: A GI/G/1 Queue Signal Processing Problems Rate of Convergence Outline of Chapter Exogenous Noise: Constant Step Size Martingale Difference Noise Correlated Noise Exogenous Noise: Decreasing Step Size 286

6 Contents xi 10.3 The Kiefer-Wolfowitz Algorithm Martingale Difference Noise Correlated Noise Tightness of the Normalized Iterates: Decreasing Step Size, W.P.I Convergence Martingale Difference Noise: Robbins-Monro Algorithm Correlated Noise The Kiefer-Wolfowitz Algorithm Tightness of the Normalized Iterates: Weak Convergence The Unconstrained Algorithm The Constrained Algorithm and Local Methods Weak Convergence to a Wiener Process Random Directions: Martingale Difference Noise Comparison of Algorithms State Dependent Noise Averaging of the Iterates Outline of Chapter Rate of Convergence of the Averaged Iterates: Minimal Window of Averaging The Robbins-Monro Algorithm: Decreasing Step Size Constant Step Size The Kiefer-Wolfowitz Algorithm A Two Time Scale Interpretation Maximal Window of Averaging The Parameter Identification Problem: An Optimal Algorithm Distributed/Decentralized and Asynchronous Algorithms Outline of Chapter Examples Introductory Comments Pipelined Computations A Distributed and Decentralized Network Model Multiaccess Communications Introduction: Real-Time Scale The Basic Algorithms Constant Step Size: Introduction Martingale Difference Noise Correlated Noise Infinite Time Analysis Decreasing Step Size State Dependent Noise 378

7 xii Contents 12.6 Rate of Convergence: The Limit Rate Equations Stability and Tightness of the Normalized Iterates The Unconstrained Algorithm Convergence for Q-Learning: Discounted Cost 390 References 393 Symbol Index 409 Index 413