Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis and Computation

Transcription

1 Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis and Computation Floyd B. Hanson University of Illinois Chicago, Illinois, USA Cover, Front Matter, Preface, References, Index Copyright c 2006 by the Society for Industrial and Applied Mathematics. April 9, 2006 page 1

2 2 page 2

3 Preface Contents 0 Preliminaries in Probability and Analysis Distributions for Continuous Random Variables Probability Distribution and Density Functions Expectations and Higher Moments Uniform Distribution Normal Distribution and Gaussian Processes Simple Gaussian Processes Lognormal Distribution Exponential Distribution Distributions of Discrete Random Variables Poisson Distribution and Poisson Process Joint and Conditional Distribution Definitions Conditional Distributions and Expectations Law of Total Probability Probability Distribution of a Sum: Convolutions Characteristic Functions Sample Mean and Variance: Sums of IID Random Variables Law of Large Numbers Weak Law of Large Numbers (WLLN) Strong Law of Large Numbers (SLLN) Central Limit Theorem Matrix Algebra and Analysis Some Multivariate Distributions Multivariate Normal Distribution Multinomial Distribution Basic Asymptotic Notation and Results Generalized Functions: Combined Continuous and Discrete Fundamental Properties of Stochastic and Markov Processes Basic Classification of Stochastic Processes Markov Processes and Markov Chains Stationary Markov Processes and Markov Chains Continuity, Jump Discontinuity and Non-Smoothness i xv page i

4 ii Contents Beyond Continuity Properties Taylor Approximations of Composite Functions Extremal Principles Exercises Stochastic Jump and Diffusion Processes Poisson and Wiener Processes Basics Wiener Process Basic Properties More Wiener Process Moments Wiener Process Non-Differentiability Wiener Process Expectations Conditioned on Past Poisson Process Basic Properties More Poisson Process Moments Poisson Process Expectations Conditioned on Past Exercises Stochastic Integration for Diffusions Ordinary or Riemann Integration Stochastic Integration in W(t): The Foundations Stratonovich and other Stochastic Integration Rules Conclusion Exercises Stochastic Integration for Jumps Stochastic Integration in P(t): The Foundations Stochastic Jump Integration Rules and Expectations: Conclusion Exercises Stochastic Calculus for Jump-Diffusions Diffusion Process Calculus Rules Functions of Diffusions Alone, G(W(t)) Functions of Diffusions and Time Itô Stochastic Natural Exponential Construction Transformations of Linear Diffusion SDEs: Functions of General Diffusion States and Time Poisson Jump Process Calculus Rules Jump Calculus Rule for h(dp(t)) Jump Calculus Rule for H(P(t), t) Jump Calculus Rule with General State Transformations of Linear Jump with Drift SDEs Jump-Diffusion Rules and SDEs Jump-Diffusion Conditional Infinitesimal Moments Stochastic Jump-Diffusion Chain Rule Linear Jump-Diffusion SDEs SDE Models Exactly Transformable page ii

5 Contents 4.4 Poisson Noise is White Noise Too! Exercises Stochastic Calculus for General Markov SDEs Space-Time Poisson Process State-Dependent Generalizations State-Dependent Poisson Processes State-Dependent Jump-Diffusion SDEs Linear State-Dependent SDEs Multi-Dimensional Markov SDE Conditional Infinitesimal Moments Stochastic Chain Rule in Multi-Dimensions Distributed Jump SDE Models Exactly Transformable Jump SDE Models Exactly Transformable Vector Jump SDE Models Exactly Transformable Exercises Deterministic Optimal Control Hamilton s Equations Deterministic Computational Complexity Optimum Principles: The Basic Principles Approach Linear Quadratic (LQ) Canonical Models Scalar, Linear Dynamics, Quadratic Costs (LQ) Matrix, Linear Dynamics, Quadratic Costs (LQ) Deterministic Dynamic Programming (DDP) Deterministic Principle of Optimality Hamilton-Jacobi-Bellman (HJB) Equation of DDP Computational Complexity for DDP Linear Quadratic (LQ) Problem by DDP Differential Dynamic Programming Control of PDE Driven Dynamics (DPS) DPS Optimal Control Problem DPS Hamiltonian Extended Space Formulation DPS Optimal State, Co-State and Control PDEs Exercises Stochastic Dynamic Programming Stochastic Optimal Control Problem Bellman s Principle of Optimality HJB Equation of Stochastic Dynamic Programming Linear Quadratic Jump-Diffusion (LQJD) Problem LQJD in Control Only (LQJD/U) Problem LLJD/U or the Case C 2 0: Canonical LQJD Problem Exercises iii page iii

6 iv Contents 8 Kolmogorov Equations Dynkin s Formula and the Backward Operator Backward Kolmogorov Equations Forward Kolmogorov Equations Multi-dimensional Backward and Forward Equations Chapman-Kolmogorov Equation for Markov Processes Jump-Diffusion Boundary Conditions Absorbing Boundary Condition Reflecting Boundary Conditions Stopping Times: Expected Exit and First Passage Times Expected Stochastic Exit Time Diffusion Approximation Basis Exercises Computational Stochastic Control Methods Finite Difference PDE Methods of SDP Linear Dynamics and Quadratic Control Costs Crank-Nicolson, Prediction-Correction for SDP Upwinding If Not Diffusion-Dominated Multi-state Systems andcurse of Dimensionality Markov Chain Approximation for SDP The MCA Formulation for Stochastic Diffusions MCA Local Diffusion Consistency Conditions MCA Numerical Finite Differenced Derivatives MCA Extensions to Include Jump Processes Stochastic Simulations SDE Simulation Methods Convergence and Stability for Stochastic Simulations Stochastic Diffusion Euler Simulations Milstein s Higher Order Diffusion Simulations Convergence of Jump-Diffusion Simulations Jump-Diffusion Simulation Procedures Monte Carlo Methods Monte Carlo Basics Inverse Generation for Non-Uniform Variates Acceptance and Rejection Method of von Neumann Importance Sampling Stratified Sampling Antithetic Variates Control Variates Applications in Financial Engineering Classical Black-Scholes Option Pricing Model Merton s Three Asset Option Pricing Model PDE of Option Pricing page iv

7 Contents Final and Boundary Conditions for Option Pricing Transforming PDE to Standard Diffusion PDE Jump-Diffusion Option Pricing Jump-Diffusions with Normal Jump-Amplitudes Risk-Neutral Option Pricing for Jump-Diffusions Optimal Portfolio and Consumption Models Log-Uniform Jump-Diffusion for Log-Return Log-Uniform Jump-Amplitude Model Optimal Portfolio and Consumption Policies CRRA Utility and Canonical Solution Reduction: Important Financial Events Model: The Greenspan Process Scheduled and Unscheduled Events Model Properties of Scheduled Event Processes Optimal Utility, Stock Fraction and Consumption Canonical CRRA Model Solution Exercises Applications in Mathematical Biology Stochastic Bioeconomics: Optimal Harvesting Applications Optimal Harvesting of Logistically Population Optimal Harvesting with Random Price Dynamics Stochastic Biomedical Applications Tumor Doubling Time Application Optimal Drug Delivery to the Brain Applied Guide to Abstract Stochastic Processes () Martingale Methods Useful Theoretical Results: Girsanov s Theorem and more Itô, Lévy and Other Stochastic Processes Itô Processes and Jump-Diffusion Processes H Control Methods A Appendix: MATLAB Programs 473 A.1 Program: Uniform Distribution Simulation Histograms A.2 Program: Normal Distribution Simulation Histograms A.3 Program: Lognormal Distribution Simulation Histograms A.4 Program: Exponential Distribution Simulation Histograms A.5 Program: Poisson Distribution versus Jump Counter k A.6 Program: Binomial Distribution versus Binomial Frequency f A.7 Program: Simulated Diffusion W(t) Sample Paths A.8 Program: Diffusion Sample Paths Time Step Variation A.9 Program: Simulated Simple Poisson P(t) Sample Paths A.10 Program: Simulated Incremental Poisson P(t) Sample Paths. 484 A.11 Program: Simulated Diffusion Integrals!(dW) A.12 Program: Simulated Diffusion Integrals g(w, t)dw A.13 Program: Simulated Diffusion Integrals g(w, t)dw: Chain Rule 488 v page v

8 vi Contents A.14 Program: Simulated Linear Jump-Diffusion Sample Paths A.15 Program: Simulated Linear Mark-Jump-Diffusion Sample Paths 492 A.16 Program: Euler-Maruyama Simulations for Linear Diffusion SDE 495 A.17 Program: Milstein Simulations for Linear Diffusion SDE A.18 Program: Monte Carlo Simulation Comparing Uniform and Normal Errors A.19 Program: Monte Carlo Simulation Comparing Uniform and Normal Errors A.20 Program: Monte Carlo Acceptance-Rejection Technique A.21 Program: Monte Carlo Multidimensional Integration Bibliography 509 Index 529 page vi

9 List of Figures 0.1 Histograms of simulations of uniform distribution on (0, 1) using MAT- LAB [198] for two different sample sizes N Histograms of simulations of the standard normal distribution with mean 0 and variance 1 using MATLAB [198] with 50 bins for two sample sizes N. The histogram for the large sample size of N = 10 5 in Fig. 2(b) exhibits a better approximation to the theoretical normal density φ n(x;0,1) Histograms of simulations of the lognormal distribution with mean µ n = 0 and variance σ n = 0.5 using MATLAB [198] normal distribution simulations, x = exp(mu*ones(n,1) + sigma*randn(n,1)), with 150 bins for two sample sizes. The histogram for the large sample size of N = 10 5 in Fig. 3(b) exhibits a better approximation to the theoretical lognormal density φ n(x;0, 1) than the one in Fig. 3(a) Histograms of simulations of the standard exponential distribution, with mean taken to be mu = 1, using MATLAB s hist function [198] with 50 bins for two sample sizes N, generated by x = mu log(rand(n,1)) in MATLAB. The histogram for the large sample size of N = 10 5 in Fig. 4(b) exhibits a better approximation to the standard theoretical exponential density φ e(x;1) Poisson distributions with respect to the Poisson counter variable k for parameter values Λ = 0.2, 1.0, 2.0 and 5.0. These represent discrete distributions, but discrete values are connected by dashed, dotted and dash-dotted lines only to help visualize the distribution form for each parameter value Binomial distributions with respect to the binomial frequency f 1 with N = 10 for values of the probability parameter, π 1 = 0.25, 0.5 and These represent discrete distributions, but discrete values are connected by dashed, dotted and dash-dotted lines only to help visualize the distribution form for each parameter value In Figure 1.1(a), paths were simulated using MATLAB [198] with N = 1000 sample points, four randn states and maximum time T = 1.0. In Figure 1.1(b), paths were simulated using subsets of the same random state of randn used for the finer grid vii page vii

10 viii List of Figures 1.2 In Figure 1.2(a), Simulated sample paths for the simple Poisson Process P(t) versus the dimension-less time λt using four different MATLAB [198] random states for four different sample paths and the exponential distribution of the time between jumps. In Figure 1.2(b) is a similar illustration for the incremental simple Poisson process simulations versus t with λ = 1.0 and t = 0.05, based upon the zero-one jump law implemented with a uniform distribution.paths were simulated using subsets of the same random state of randn used for the finer grid Simulated sample path for the Itô forward integration approximating sum of R (dw) 2 (t) ims = t P i ( Wi)2 for n = 10 4 MATLAB randn sample size Example of a simulated Itô discrete approximation to the stochastic diffusion integral I n [g](t i+1 ) = i j=0 g j W j for i = 0 : n, using the MATLAB randn with sample size n = 10, 000 on 0 t 2.0. Presented are the simulated Itô partial sums S i+1, the simulated noise W i+1 and the error E i+1 relative to the exact integral, I (ims) [g](t i+1 ) ims = exp(w i+1 t i+1 /2) 1, in the Itô mean square sense Example of a simulated Itô discrete approximation to the stochastic diffusion integral I n [g](t i+1 ) = i j=0 g j W j for i = 0 : n, using the MATLAB randn with sample size n + 1 = 10, 001 on 0 t 2.0. Presented are the simulated Itô partial sums S i+1, the simulated noise W i+1 and the error E i+1 relative to the stochastic chain rule partially integrated form, I i+1 given in the text (4.23) Four linear jump-diffusion sample paths for constant coefficients are simulated using MATLAB [198] with N = 1000 sample points, maximum time T = 1.0 and four randn and four rand states. Parameter values are µ 0 = 0.5, σ 0 = 0.10, ν 0 = 0.10, λ 0 = 3.0 and x 0 = 1.0. In addition to the four simulated states, the expected state E[X(t)] and two deviation measures E[X(t)] V (t) and E[X(t)]/V (t), where the factor V (t) is based on the standard deviation of the state exponent Y (t) Four linear pure diffusion sample paths for constant coefficients are simulated using MATLAB [198] with N = 1000 sample points, maximum time T = 1.0 and four randn states. Parameter values are µ 0 = 0.5, σ 0 = 0.10, ν 0 = 0.0, and x 0 = 1.0. In addition to the four simulated states, the expected state E[X(t)] and two deviation measures E[X(t)] V (t) and E[X(t)]/V (t), where the factor V (t) is based on the standard deviation of the state exponent Y (t) page viii

11 List of Figures 4.5 Four linear pure jump with drift sample paths for constant coefficients are simulated using MATLAB [198] with N = 1000 sample points, maximum time T = 1.0 and four randn states. Parameter values are µ 0 = 0.5, σ 0 = 0.0, ν 0 = 0.10, and x 0 = 1.0. In addition to the four simulated states, the expected state E[X(t)] and two deviation measures E[X(t)] V (t) and E[X(t)]/V (t), where the factor V (t) is based on the standard deviation of the state exponent Y (t) Four linear mark-jump-diffusion sample paths for time-dependent coefficients are simulated using MATLAB [198] with N = 1, 000 sample points, maximum time T = 2.0 and four randn and four rand states. Initially, x 0 = 1.0. Parameter values are given in vectorized functions using vector functions and dot-element operations, µ d (t) = 0.1 sin(t), σ d (t) = 1.5 exp( 0.01 t) and λ = 3.0 exp( t. t). In addition to the four simulated states, the expected state E[X(t)] is calculated using quasi-deterministic equivalence (5.44) of Hanson and Ryan [108] Hamitonian and optimal solutions for regular control problem example from (6.29) for X (t) and (6.30) for λ (t). Note that the γ = 2.0 power utility is only for illustration purposes Hamiltonian and optimal solutions for bang control problem example from (6.29) for X (t) and (6.30) for λ (t). Note that the γ = 2.0 power utility is only for illustration purposes Optimal solutions for a simple, static optimal control problem represented by (6.34) and (6.35), respectively Optimal control, state and switch time multiplier sum are shown for bang-bang control example with sample parameter values t 0 = 0, t f = 2.0, a = 0.6, M = 2, K = 2.4 and x 0 = 1.0. The computed switch time t s is also indicated Optimal state solutions for singular control example leading to a bang-singular-bang trajectory represented by (6.59). Subfigure (a) yields a maximal bang trajectory from x 0 using U (max), where as Subfigure (b) yields a minimal bang trajectory from x 0 using U (min) Multibody Stochastic Dynamical System Under Feedback Control Estimate of the logarithm to the base 2 of the order of the growth of memory and computing demands using 8 byte words to illustrate the curse of dimensionality in the diagonal Hessian case for n x = 1 : 10 dimensions and N x = 1 : 64 = 1 : 2 6 nodes per dimension. Note that 1KB or one kilobyte is 10 = log 2 (2 10 ), 1MB is 20, 1GB is 40 and 1TB is 60 on the log 2 scale ix page ix

12 x List of Figures 10.1 Comparison of coarse Euler-Maruyama and fine exact paths, simulated using MATLAB with N t = 1024 fine sample points for the exact path (10.15) and N t /8 = 128 coarse points for the Euler path (10.13), initial time t 0 = 0, final time t f = 5 and initial state x 0 = 1.0. Time-dependent parameter values are µ(t) = 0.5/( t) 2 and σ(t) = Error in coarse Euler-Maruyama and fine exact paths using the coarse discrete time points. The simulations use MATLAB the same values and time-dependent coefficients as in Fig The Euler maximal-absolute error for this example is t/8, while for N t = 4096 the maximal error is better at t/ Comparison of coarse Milstein and fine exact paths, simulated using MATLAB with N t = 1024 fine sample points for the exact path (10.15) and N t /8 = 128 coarse points for the Milstein path (10.23), initial time t 0 = 0, final time t f = 5 and initial state x 0 = 1.0 as in Fig Time-dependent parameter values are µ(t) = 0.5/( t) 2 and σ(t) = Error in coarse Milstein and fine exact paths using the coarse discrete time points. The simulations use MATLAB the same values and time-dependent coefficients as in Fig The Milstein maximal-absolute error for this example is 1.2, while for N t = 4096 the maximal error is better at Difference in coarse Milstein and Euler paths using the coarse discrete time points. The simulations use MATLAB the same values and time-dependent coefficients as in Fig The Milstein- Euler maximal-absolute difference for this example is 0.19, while for N t = 4096 the maximal difference is comparable at Monte Carlo simulations for testing use of the uniform distribution to approximate the integral of for the integrand F(x) = 1 x 2 on (a, b) = (0, 1) using MATLAB code A.19 on p. 502 for n = 10 k, k = 1: Monte Carlo simulations shown apply the acceptance and rejection technique and the normal distribution to compute the estimates for the mean bµ n and the magnified standard error 10 bσ n/ n for the integral of the truncated normal distribution with F(x) = φ n(x) on [a, b] = [ 2,2] using MATLAB code A.20 on p. 504 for n = 10 k, k = 1: Monte Carlo simulations for estimating multi-dimensional integrals of for the n x-dimension normal integrand F(x) = φ n(x) on [a,b] = [ 2,2] nx using MATLAB code A.21 on p. 506 for n = 10 k, k = 1 : 6. The acceptance-rejection technique is used to handle the finite domain Optimal portfolio stock fraction policy u (t) on t [0, 12] subject to the control constraint set [U (min) 0, U (max) 0 ] = [ 10, 10] Optimal consumption policy c (t, w) for (t, w) [0, 12] [0, 100]. 433 page x

13 List of Figures 12.1 Optimal tumor density Y1 (x 1, t) in the one-dimensional case with time as a parameter rounded at quartile values {0, t q1 = t f /4, t mid = t f /2, t q3 = 3t f /4, t f }, where t f = 5 days. The total tumor density integral is reduced by 29% in the for 5-day simulated drug treatment trial xi page xi

14 xii List of Figures page xii

15 List of Tables 0.1 Some expected moments of bivariate normal distribution Some expected moments (powers) of absolute value of the Wiener increments Some expected moments (powers) of Poisson increments and their deviations Some Itô stochastic diffusion differentials with an accuracy with error o(dt) as dt Some stochastic jump integrals of powers with an accuracy with error o(dt) as dt Some Itô stochastic jump differentials with an accuracy with error o(dt) as dt Table of Example Transforms Listing Original Coefficients in terms of Target and Transform Coefficients: Some final conditions for deterministic optimal control Some Simple jump amplitude models and inverses xiii page xiii

16 xiv List of Tables page xiv

17 Overview of This Book Preface Everything should be as simple as it is, but not simpler. Albert Einstein ( ). A mathematical theory is not to be considered complete until you have made it so clear that you can explain it to the first man whom you meet on the street. David Hilbert ( ). Always take a pragmatic view in applied mathematics: the proof of the pudding is in the eating. N. H. Bingham and R udiger Kiesel (2004) [30]. The aim of this book is to be a self-contained, practical, entry level text on jumpdiffusions for graduate students as well as a research monograph for researchers in applied mathematics, computational science and engineering. Also, the book may be useful for practicianers of financial engineering who need fast and efficient answers to stochastic financial problems. Hence, the exposition is based upon basic principles of applied mathematics and applied probability. The target audience includes mathematical modelers and students in many areas of science and engineering seeking to construct models for scientific applications subject to uncertain environments. The prime interest is in modeling and problem solving. The utility of the exposition, based upon systematic derivations in the spirit of classical applied mathematics, is more important to setting up a stochastic model of an application than abstract theory. More rigorous theorem formulation and proving is not of immediate importance compared to modeling and solving an applied problem. Many research problems deal with new applications and often these new applications require models beyond those in the existing literature, so it is important to have a reasonably understandable derivation for a nearby model that can be perturbed to obtain a xv page xv

18 xvi Preface proper new model. The level of rigor here is embodied in correct and systematic derivations under reasonable conditions, not necessarily the tightest possible conditions. In fact, much of this book and the theory of Markov processes in continuous time is based upon modifying the formulations for continuous function in calculus to extend them to the discontinuous and non-smooth functions of stochastic calculus. Origin of the Book The book is based upon the first author s (FBH s at the University of Illinois at Chicago) courses Math 574 Applied Optimal Control, Math 590 Special Topics: Applied Stochastic Control, MCS 507 Mathematical, Statistical and Scientific Software for Industry and partly on MCS 571 Numerical Methods for Partial Differential Equations, in addition to the many research papers on computational stochastic dynamic programming of both authors. Courses in numerical analysis and asymptotic analysis play a role as well. However, as with our lectures, every attempt is made to keep the book self-contained, without strong emphasis on prerequisites. This book will integrate many of the research and exposition advances made in computational stochastic dynamic programming, and exhibit the broader impact of our applications and computationally oriented approach. Our stochastic applications are wide-ranging, including population extinctions in uncertain environments, bio-medical applications, stochastic bio-economics, ground water remediation, stochastic manufacturing systems and financial engineering. Scope of This Book The scope of the book will be Markov processes in continuous time, namely stochastic diffusions and jump processes, i.e., general jump-diffusion processes. There will be particular emphasis on the balanced use of jump processes and their calculus in modeling applications. Most texts on stochastic processes almost exclusively emphasize Wiener (Brownian motion or diffusion) processes; the calculus of these processes are better developed and relatively easier to analyze. Later chapters will introduce more advanced analytical methods and results such as Martingale methods, Dynkin s formula, Girsanov s theorem and other topics, to prepare the reader with sufficient insight to move on to the more advanced theoretical treatments with a hope for increased understanding. The main problem is composed of several parts: 1. modeling of stochastic dynamics with jump-diffusion processes and background continuous noise, 2. setting up the corresponding stochastic control model, 3. analyzing the stochastic control model to obtain a stochastic dynamic programming formulation or a reduced canonical formulation of the problem, and 4. discussing computational techniques for solving the problem formulation, approximately. page xvi

19 Preface The book will strive to systematically establish the partial differential equation of stochastic dynamic programming for Markov processes in continuous time to facilitate the modeling of real applications by entry-level graduate students as well as scientists and engineers in related areas. Distinct Features of This Book Both analytical and computational methods are emphasized based on the utility, with respect to the computational complexity, of the problems. The book is based upon a number of distinct features: 1. using basic principles of probability theory in the spirit of classical applied mathematics to set up the practical foundations through clear and systematic derivations, making the book accessible as a research monograph to many who work with applications, 2. having a particular emphasis on space-time Poisson jump processes to balance an essentially exclusive treatment of diffusion processes elsewhere, 3. showing the strong role the discontinuous as well as non-smooth properties of stochastic processes play compared to the random properties emphasizing jump calculus, without much reliance on measure-theoretic constructs, 4. showing how canonical models such as the Linear Quadratic Gaussian Poisson (LQGP) problem, least squares approximations and financial risk-adverse power utilities can be used to reduce computational dimensional complexity of approximate solutions along with other computational techniques. 5. having insightful and useful material so that the book can be readily used to model real applications and even modify the derivations when new applications do not quite fit the old stochastic model. Target Audience Colleagues and students have requested a more accessible, practical treatment of these topics. They are interested in learning about stochastic calculus and optimal stochastic control in continuous time, but reluctant to invest time to learn it from more advanced treatments relying heavily on measure-theoretic concepts, Hence, this book should be of interest to an interdisciplinary audience of applied mathematician, applied probabilists, engineers (including control engineers dealing with deterministic problems and financial engineers needing fast as well as useful methods for modeling rapidly changing market developments), statisticians and other scientists. After this primary audience, a secondary audience would be mathematicians, engineers and scientists, using this book as a research monograph, seeking more intuition to help more fully understand stochastic processes and how the more advanced analytical approaches fit in with important applications like financial market modeling. xvii page xvii

20 xviii Prerequisites Preface For optimal use of this book, it would be helpful to have had prior introduction to applied probability theory including continuous random variables, mathematical analysis at least at the level of advanced calculus, ordinary differential equations, partial differential equations, and basic computational methods. In other words, the usual preparation for students of applied mathematics, science and engineering. However, the authors have strived to make this book as self-contained as practical, not strongly relying on prior knowledge and explaining or reviewing the prerequisite knowledge at the point it is needed to justify a step in the systematic derivation of some mathematical result. More About This Book This book strongly emphasizes the stochastic differential equation formulation as a logical transition from modeling with deterministic ordinary differential equations to modeling with stochastic differential equations. A more concrete derivation is given of the Principle of Optimality and the PDE of stochastic dynamic programming than is available in most texts. Also, special computational techniques are given for handing the globally functional terms that arise due to the jumps of state-space Poisson processes, since techniques for those terms due to diffusion processes are better understood. The use of jump processes and their calculus are emphasized in modeling for a variety of applications such as bio-economics, manufacturing, and finance. Many problems in nature exhibit catastrophic or abrupt changes for which purely continuous models are inappropriate. The unique emphasis on jump processes sets this book apart from the multitude of books on diffusion processes, and will be more generally useful in solving a wider range of problems. Later chapters introduce more advanced methods and results such as Martingale methods, Dynkin s formula, Girsanov s theorem, optimal stopping problems and other topics, showing how they can be useful in analysis for applications. However, probability spaces are not used in any serious way and measure-theoretic concepts are not discussed, except for Poisson random measure due to its utility in understanding the concept of space-time Poisson processes. There is a heavier emphasis on computational methods since computational methods are more useful in problem solving than are measure-theoretic methods for many problems in stochastic optimal control where there are few exact solutions. Besides, there are plenty of books and papers on the pure analysis of stochastic control papers than on computational stochastic control. How This Book is Organized This book begins with Preliminaries in Chapter 1 to bring up all readers up to the same basic concepts and notations of probability and analysis needed for jumpdiffusion processes and their deviations from continuity. From the beginning, an effort is made to make the presentation as self-contained as possible. Chapter 2 in- page xviii

21 Preface troduces elementary stochastic differential equations with standard Wiener and simple Poisson processes, i.e., simple jump-diffusions, while introducing the Markovian stochastic calculus. Chapter 3 introduces Space-time Poisson processes at a more advanced level with Poisson random measure or distributed Poisson jump amplitudes, and state-dependent generalizations. More general formulations of stochastic differential equations and the stochastic calculus are given for multi-dimensional Markov processes in continuous time in Chapter 4. Chapter 5 gives a summary of deterministic optimal control results to provide a background for comparisons to stochastic optimal control results. In Chapter 6 stochastic optimal control problems are introduced and stochastic dynamic programming is systematically derived. In Chapter 7 computational models are discussed to permit solving the many stochastic control problems that do not possess exact analytical solutions. In Chapters 8, 9 and 10, applications of stochastic processes are treated for mathematical biology, stochastic manufacturing systems and financial engineering, respectively. Finally in Chapter 11 more advanced methods such as Martingale methods are introduced from the applied point of view to prepare the readers for the many advanced books and other literature in the field. Structure of This Sample Book Material Brief descriptions of the content are given for all chapters. However, some chapter descriptions have much more detail for stochastic differential equations and stochastic control models to give the reviewers an idea of the scope of the full book and the scope of models considered. In lieu of presenting full sample chapters, the thread running through the core material of the book is presented. In addition, a few sample exercises are included. The full book will have many more details, justifications and exercises. Acknowledgments We are grateful to a number of co-workers and students who helped as reviewers or contributed to this applied stochastic book through research contributions, as well as other authors and agencies giving grant support for computational stochastic dynamic programming: My research assistants, Siddhartha Pratim Chakrabarty and Zongwu Zhu, have helped review drafts of this book with the the keen eye of an applied mathematics and computer science graduate students to make sure that it would be useful and understandable to other graduate students. Over the years many have helped develop pieces of the underlying applied theory or model applications: Abdul Majid Wazwaz, Dennis Ryan, Kumarss Naimipour, Siu-Leung Chung, Huihuang (Howard) Xu, Dennis J. Jarvis, Christopher J. Pratico, Michael S. Vetter, Raghib abu-saris, and Daniel L. Kern. This work has been influenced from books and related works by many authors such as Ludwig [175], Clark [51], Gihman and Skorohod [88, 89], Parzen [212], xix page xix

22 xx Preface Feller [77, 78], Jazwinski [144], Kushner [161, 163], Wonham [270], Arnold [13], Çinlar [50], Schuss [231], Fleming and Rishel [79], Kirk [152], Snyder and Miller [238], Karlin and Taylor [150, 151, 250] Tuckwell [255], Merton [191], Kushner and Dupuis [167], Øksendal [210], Mikosch [197], Steele [241], D. Higham [131], and others. Although this influence may not be directly apparent here, some have shown how to make the presentation much simpler, while others have supplied the motivation to simplify the presentation, making it more accessible to a more general audience and other applications. This material is based upon work supported by the National Science Foundation under Grants No , , , , and in the Computational Mathematics program entitled: Advanced Computational Stochastic Dynamic Programming for Continuous Time Problems at the University of Illinois at Chicago. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Argonne National Laboratory Advanced Computing Research Facility supplied parallel processing training for FBH through summer and sabbatical support that enabled the development of large scale computational stochastic applications from Many of our national supercomputing centers have provided supercomputing time to FBH on the the currently most powerful supercomputers for continuing research on Advanced Computational Stochastic Dynamic Programming in addition to Argonne National Laboratory: National Center for Supercomputing Applications, Los Alamos National Laboratory s Advanced Computing Laboratory, Cornell Theory Center CNSF, Pittsburgh Supercomputing Center and the San Diego Supercomputing Center (NPACI). At the University of Illinois Chicago, the Laboratory of Advanced Computing and associate centers have supplied us with cluster computing and the Electronic Visualization supplied a master graduate student, Chris Pratico, with facilities for the developing a multi-dimensional computational control visualization system using a real-time socket feed from our Los Alamos National Laboratory account. page xx

23 Bibliography [1] M. L. Abell and J. P. Braselton, The Maple V Handbook, Academic Press, New York, NY, [2] M. Abramowiitz and I. A. Stegun, (eds.), Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, Applied Mathematics Series 55, National Bureau of Standards, Washington, DC, [3] R. M. Abu-Saris and F. B. Hanson, Computational Suboptimal Filter for a Class of Wiener-Poisson Driven Stochastic Process, Dynamics and Control, vol. 7, no. 3, 1997, pp [4] N. Ahmed and K. Teo, Optimal Control of Distributed Parameter Systems, North Holland, New York, NY, [5] Y. Aït-Sahalia, Disentangling Diffusion from Jumps, J. Financial Economics, vol. 74, 2004, pp [6] T. G. Andersen, L. Benzoni and J. Lund, An Empirical Investigation of Continuous-Time Equity Return Models, J. Finance, vol. 57, no. 3, 2002, pp [7] B. D. O. Anderson and J. B. Moore, Optimal Filtering, Prentice-Hall, Englewood Cliffs, NJ, [8] B. D. O. Anderson and J. B. Moore, Optimal Control: Linear Quadratic Methods, Prentice-Hall, Englewood Cliffs, NJ, [9] P. Andersen and J. G. Sutinen, Stochastic Bioeconomics: A Review of Basic Methods and Results, Marine Res. Econ., vol. 1, 1982, pp [10] H. L. Anderson, Metropolis, Monte Carlo Method, and the MANIAC, Los Alamos Science, Fall 1986, pp [11] C. A. Aourir, D. Okuyama, C. Lott and C. Eglinton, Exchanges - Circuit Breakers, Curbs, and Other Trading Restrictions, com/articles/exch-circuit-brkr.html. [12] D. Applebaum, Lévy Processes and Stochastic Calculus, Cambridge University Press, Cambridge, UK, page 509

24 510 Bibliography [13] L. Arnold, Stochastic Equations: Theory and Applications, John Wiley, New York, NY, [14] Y. Ashkenazy, J. M. Hausdorff, P. C. Ivanov and H. E. Stanley, A Stochastic Model of Human Gait Dynamics, Physica A, vol. 316, 2002, pp [15] M. Athans and P. L. Falb, Optimal Control: An Introduction to the Theory and Its Applications, McGraw-Hill Book Co., New York, NY, [16] L. Bachelier, Théorie de la Spéculation, Annales de l Ecole Normale Supérieure, vol. 17, 1900, pp English translation by A. J. Boness in The Random Character of Stock Market Prices, P. H. Cootner (Editor), MIT Press, Cambridge, MA, 1967, pp [17] C. A. Ball and W. N. Torous, On Jumps in Common Stock Prices and Their Impact on Call Option Prices, J. Finance, vol. 40, 1985, pp [18] M. S. Bartlett, Stochastic Equations: Theory and Applications, 3rd ed., Cambridge University Press, Cambridge, UK, [19] T. Basar, Twenty-five Seminal Papers in Control, IEEE Control Systems Magazine, vol. 20, no. 1, February 2000, pp [20] M. Baxter and A. Rennie, Financial Calculus: An Introduction to Derivative Pricing, Cambridge University Press, Cambridge, UK, [21] I. Beichl and F. Sullivan, The Metropolis Algorithm, Computng in Sci. & Engineering, vol. 2, no. 1, 2000, pp [22] D. J. Bell and D. H. Jacobson, Singular Optimal Control Problems, Academic Press, New York, NY, [23] R. E. Bellman, Dynamic Programming, Princeton University Press, Princeton, NJ, [24] R. E. Bellman, Adaptive Control Processes: A Guided Tour, Princeton University Press, Princeton, NJ, [25] R. Bellman and R. Kalaba, Selected Papers on Mathematical Trends in Control Theory, Dover Publications, New York, NY, [26] C. M. Bender and S. A. Orszag, Advanced Mathematical Methods for Scientists and Engineers, McGraw-Hill Book Co., New York, NY, [27] D. S. Bernstein, Feedback Control: An Invisible Thread in the History of Technology, IEEE Control Systems Magazine, vol. 22, no. 2, April 2002, pp [28] J. T. Betts, Practical Methods for Optimal Control Using Nonlinear Programming, SIAM, Philadelphia, PA, page 510

25 Bibliography 511 [29] A. T. Bharucha-Reid, Elements of the Theory of Markov Processes and their Applications, McGraw-Hill Book Co., New York, NY, [30] N. H. Bingham and R. Kiesel, Risk-Neutral Valuation: Pricing and Hedging of Financial Derivatives, Springer-Verlag, New York, NY, [31] F. Black and M. Scholes, The Pricing of Options and Corporate Liabilities, J. Political Economy, vol. 81, 1973, pp [32] F. Black, How We Came Up with the Option Formula, J. Portfolio Mgmt., vol. 15, Winter 1989, pp [33] A. Boker, C. J. Haberman, L. Girling, R. P.Guzman, G. Louridas, J. R. Tanner, M. Cheang. M. Math, B. W. Maycher, D. D. Bell and G. J. Doak, Variable Ventilation Improves Perioperative Lung Function in Patients Undergoing Abdominal Aortic Aneurysmectomy, Anesthesiology, vol. 100, no. 3, 2004, pp [34] G. E. P. Box and M. E. Muller, A Note on the Generation of Random Normal Deviates, Ann. Math. Stat., vol. 29, 1958, pp [35] P. Boyle, Options: A Monte Carlo Approach, J. Financial Economics, vol. 21, 1977, pp [36] P. Boyle, M. Broadie and P. Glasserman, Monte Carlo Methods for Security Pricing, J. Economic Dynamics and Control, vol. 21, 1997, pp [37] G. A. Bliss, Lectures on the Calculus of Variations, University Chicago Press, Chicago, IL, [38] P. Bossaerts, The Paradox of Asset Pricing, Princeton University Press, Princeton, NJ, [39] P. W. Bridgeman, Dimensional Analysis, Yale University Press, New Haven, CT, [40] A. E. Bryson and Y. C. Ho, Applied Optimal Control, John Wiley, New York, NY, [41] T. G. Buchman, Nonlinear Dynamics, Complex Systems, and the Pathobiology of Critical Illness, Curr. Opin. Crit. Care, vol. 10, no. 5, 2004, pp [42] P. Carr and D. B. Madan, Option Valuation Using the Fast Fourier Transform, J. Comp. Fin., vol. 2, 1999, pp [43] S. P. Chakrabarty and F. B. Hanson, Optimal Control of Drug Delivery to Brain Tumors for a Distributed Parameters Model, in Proc American Control Conference, 2005, pp page 511

26 512 Bibliography [44] S. P. Chakrabarty and F. B. Hanson, Optimal Control of Drug Delivery to Brain Tumors for a PDE Driven Model Using the Galerkin Finite Element Method, in Proc. 44nd IEEE Conference on Decision and Control and European Control Conference 2005, 2005, pp [45] S. P. Chakrabarty and F. B. Hanson, Cancer Drug Delivery in Three Dimensions For a Distributed Parameter Control Model Using Finite Elements, in Proc. 45th IEEE Conference on Decision, 23 February 2006, pp. 1-8, submitted for review; URL: cdc06spcfhweb.pdf. [46] G. Chichilnisky, Fischer Black: The Mathematics of Uncertainty, Notices of the AMS, vol. 43 (3), 1996, pp [47] S.-L. Chung and F. B. Hanson, Optimization Techniques for Stochastic Dynamic Programming, in Proc. 29th IEEE Conference on Decision and Control, vol. 4, 1990, pp [48] S.-L. Chung and F. B. Hanson, Parallel Optimizations for Computational Stochastic Dynamic Programming, in Proc International Conference on Parallel Processing, vol. 3: Algorithms and Applications, P.-C. Yew, ed., 1990, pp [49] S.-L. Chung, F. B. Hanson and H. H. Xu, Parallel Stochastic Dynamic Programming: Finite Element Methods, Lin. Alg. Applic., vol. 172, 1992, pp [50] E. Çinlar, Introduction to Stochastic Processes, Prentice-Hall, Englewood Cliffs, NJ, [51] C. W. Clark, Mathematical Bioeconomics: The Optimal Management of Renewable Resources, 1st and 2nd Editions, John Wiley, New York, NY, 1976 and [52] C. W. Clark and R. Lamberson, An Economic History and Analysis of Pelagic Whaling, Marine Policy, vol. 6, 1982, pp [53] J. E. Cohen, Mathematics Is Biology s Next Microscope, Only Better; Biology Is Mathematics Next Physics, Only Better, PLoS Biology, vol. 2, issue 12, 2004, p [54] R. Cont and P. Tankov, Financial Modelling with Jump Processes, Chapman & Hall/CRC, Boca Raton, FL, [55] E. T. Copson, Asymptotic Expansions, Cambridge University Press, Cambridge, UK, [56] J. M. Courtault, Y. Kabanov, B. Bru, P. Crépel, I. Lebon and A. L. Marchand, Louis Bachelier on the centenary of Théorie De La Spéculation, Mathematical Finance, vol. 10, No. 3, 2000, pp page 512

27 Bibliography 513 [57] D. R. Cox and H. D. Miller, The Theory of Stochastic Processes, Chapman and Hall, London, UK, [58] J. C. Cox and M. Rubinstein, Options Markets, Prentice-Hall, Englewood Cliffs, NJ, [59] S. Cyganowski, L. Grüne and P. Kloeden, Maple for Jump-Diffusion Stochastic Differential Equations in Finance, Programming Languages and Systems in Computational Economics and Finance, S. S. Nielsen (Editor), Kluwer Academic Publishers, Amsterdam, 2002, pp ; URL: jumpfin.html. [60] S. Cyganowski and P. Kloeden, Maple Schemes for Jump-Diffusion Stochastic Differential Equations, Proc. 16th IMACS World Congress, Lausanne 2000, M. Deville, R. Owens (Editors), Dept. Computer Science, Rutgers University, 2000, CD-ROM Paper 216-9, pp. 1-16; uni-frankfurt.de/~numerik/maplestoch/jumpdiff.pdf. [61] S. Cyganowski, P. Kloeden and J. Ombach, Elementary Probability to Stochastic Differential Equations with Maple, Springer-Verlag, New York, NY, [62] B. N. Datta, Numerical Linear Algebra and Applications, Brooks/Cole, New York, NY, [63] J. Dongarra and F. Sullivan, Guest Editor s Introduction: The Top Ten Algorithms, Computng in Sci. & Engineering, vol. 2, no. 1, 2000, pp [64] J. L. Doob, Stochastic Processes, John Wiley, New York, NY, [65] G. D. Doolen and John Hendricks, Monte Carlo at Work, Los Alamos Science, Special Issue Dedicated to S. Ulam, 1987, pp composite [66] P. Dorato, C. Abdallah and V. Cerone, Linear-Quadratic Control: An Introduction, Prentice-Hall, Englewood Cliffs, NJ, [67] J. Douglas Jr. and T. Dupont, Galerkin Methods of Parabolic Equations, SIAM J. Numerical Analysis, vol. 7, 1970, pp [68] J. Douglas Jr., Time Step Procedures of Nonlinear Parabolic PDEs in Mathematics of Finite Elements and Applications, MAFELAP, J. Whiteman (Editor), Academic Press, London, 1979, pp [69] D. Duffie, Dynamic Asset Pricing Theory, Princeton University Press, Princeton, NJ, page 513

28 514 Bibliography [70] D. Düvelmeyer Untersuchungen zu Chancen und Risiken von Anti-Trend- Strategien am Beispiel des DAX-Futures, Thesis, Facultät für Mathematik, Technische Universität Chemnitz, Chemnitz, 2001; URL: www-usercgi.tu-chemnitz.de/{$\sim$}dana\diplom\_pdf\_dd.zip. [71] P. Dyer and S. R. McReynolds, The Computation and Theory of Optimal Control, Academic Press, New York, NY, [72] E. B. Dykin, Markov Processes I and II, Academic Press, New York, NY, [73] R. Eckhardt, Ulam, John von Neumann, and the Monte Carlo Method, Los Alamos Science, Special Issue Dedicated to S. Ulam, 1987, pp http: // [74] P. Embrechts, Multivariate Extremes and Market Risk Scenarios, Bachelior Finance Society Third World Congress, Chicago, July [75] H. H. Engelhard, Brain Tumors and the Blood-Brain Barrier, Neuro- Oncology: The Essentials, Thieme Medical Publishers, Inc., New York 2000, pp [76] M. Evans, N. Hastings and B. Peacock, Statistical Distributions, 3rd ed., John Wiley, New York, NY, [77] W. Feller, An Introduction to Probability Theory and Its Application, vol. 1, 3rd ed., John Wiley, New York, NY, [78] W. Feller, An Introduction to Probability Theory and Its Application, vol. 2, 2nd ed., John Wiley, New York, NY, [79] W. H. Fleming and R. W. Rishel, Deterministic and Stochastic Optimal Control, Springer-Verlag, New York, NY, [80] J. J. Florentin, Optimal Control of Systems with Generalized Poisson Inputs, ASME Trans., vol. 85D (also J. Basic Engr., vol. 2), 1963, pp [81] G. E. Forsythe, M. A. Malcolm and C. Moler, Computer Methods for Mathematical Computations, Prentice-Hall, Englewood Cliffs, NJ, [82] B. Friedman, Principles and Techniques of Applied Mathematics, John Wiley, New York, NY, [83] Future Directions in Control Theory Panel, Report of the Panel on Future Directions in Control Theory: A Mathematical Perspective, W. H. Fleming (Chairman), Society for Industrial and Applied Mathematics, Philadelphia, PA, [84] Future Directions in Control, Dynamics, and Systems Panel, Future Directions in Control in an Information Rich World: Report of the Panel on Future Directions in Control, Dynamics, and Systems, R. M. Murray (Editor), Society for Industrial and Applied Mathematics, Philadelphia, PA, July page 514