On Solving Integral Equations using. Markov Chain Monte Carlo Methods

Transcription

1 On Solving Integral quations using Markov Chain Monte Carlo Methods Arnaud Doucet Department of Statistics and Department of Computer Science, University of British Columbia, Vancouver, BC, Canada mail: Vladislav B. Tadić Department of Mathematics, University of Bristol, Bristol, UK mail: Technical Report CUD-F-INFNG Cambridge University no. 444 First version December, Updated November 4 & January 7

2 ABSTRACT In this report, we propose an original approach to solve Fredholm equations of the second kind. We interpret the standard von Neumann expansion of the solution as an expectation with respect to a probability distribution de ned on an union of subspaces of variable dimension. Based on this representation, it is possible to use trans-dimensional Markov Chain Monte Carlo (MCMC) methods such as Reversible Jump MCMC to approximate the solution numerically. This can be an attractive alternative to standard Sequential Importance Sampling (SIS) methods routinely used in this context. We sketch an application to value function estimation for a Markov decision process. Keywords: Fredholm equation, Trans-dimensional Markov Chain Monte Carlo, Sequential Importance Sampling, Sequential Monte Carlo.

3 I. Fredholm equations and Von Neumann s expansion Fredholm equations of the second kind and related problems appear in many scienti c elds including optimal control [], molecular population genetics [6] and physics [9]. Formally, we are interested in solving the integral equation f (x ) = K (x ; x ) f (x ) dx + g (x ) () where g :! R and K :! R are known and f :! R is unknown. Let us de ne K (x; y), ; K (x; y), K (x; y) and K n (x; y), K (x; z) K n (z; y) dz: If X jk n (x ; x n ) g (x n )j dx n < () n= then the solution of the Fredholm equation () admits the following Von Neumann s series representation; see [], [9] for details: f (x ) = K (x ; x ) f (x ) dx + g (x ) = K (x ; x ) K (x ; x ) f (x ) dx + g (x ) dx + g (x ) = K (x ; x ) K (x ; x ) f (x ) dx dx + K (x ; x ) g (x ) dy + g (x ) and, by iterating X! ny f (x ) = g (x ) + K (x k ; x k ) g (x n ) dx :n (3) n= n k= where we use the notation x i:j, (x i ; : : : ; x j ) for i j: By introducing the notation f (x ) = g (x ) (4) and setting for n ; ny f n (x :n ) = g (x n ) K (x k ; x k ) (5) then it is possible to rewrite (3) as X f (x ) = f (x ) + f n (x :n ) dx :n : n (6) n= We will address two problems in this paper: how to estimate the function f (x ) over the set and how to estimate this function pointwise. k=

4 II. Monte Carlo Methods to Solve Fredholm quations Computing (3) is challenging as it involves an in nite sum of integrals of increasing dimension. Monte Carlo methods are a natural approach to this problem. A. Sequential Importance Sampling. Algorithm The use of Monte Carlo methods to solve problems of this type can be traced back 5 years. The standard approach consists of using Sequential Importance Sampling (SIS) to numerically approximate (3); see for example [], [9]. Consider a Markov chain with initial distribution/density (x) on and a transition kernel M (x; y) which gives the probability or probability density of moving to state y when the current state is x. We select and M such that (x) > over and M (x; y) > if K (x; y) 6=. Moreover, M is chosen to have an absorbing/cemetery state, say fyg =, such that M (x; fyg) = for any x. The algorithm proceeds as follows to approximate the function f: Simulate N independent Markov chain paths y). n o N + until absorption (i.e. X(i) i= k (i) + = Calculate the associated importance weights k(i) Q K W k= M = >: g k (i) ;X(i) k (i) k (i) ;X(i) k (i) A g k (i) if k (i) ; if k (i) = : (7) The empirical measure bf (x ) = N i= W x h R is an unbiased Monte Carlo approximation of the function f (i.e. for any set A, b i f (x A ) dx = R A f (x ) dx ). () If we are interested in estimating the function f (x ) at a given point say x = x, then we n o N simulate paths starting from + = x according to M until absorption/death and i= 3

5 the importance weights become W = >< >: k(i) K k= M k (i) ;X(i) k (i) k (i) ;X(i) k (i) A g k (i) if k (i) ; g(x) if k (i) = : (9) We obtain the following unbiased estimate of f (x) bf (x) = N i= W x; :k (i) : (). Importance Sampling on Path Space To check the unbiasedness of the estimates () and (), we use a slightly non-standard argument which will later prove useful. The rst method to estimate the function f through () can be interpreted as an importance sampling technique using an importance distribution (n; x :n ) de ned on the path space F, U k= fkg k+ where (n; x :n ) = p ;n ;n (x :n ) () with p ;n the probability that the simulated path is of length n + (i.e. X :n n+ and X n+ = fyg) and ;n (x :n ) the probability or probability density of a path conditional upon this path being of length n +. We have p ;n = Pr X :n n+ ; X n+ = fyg = ( ) n ; () and Q (x ) n M (x k ; x k ) k= ;n (x :n ) = ( ) n. (3) Now using (6) and importance sampling, this yields By sampling f (x ) = f (x ) (; x ) (; x ) + = fk (X :k ) (k; X :k ) X n= n f n (x :n) (n; x :n ) (n; x :n ) dx :n (4) n k (i) ; o (i = ; : : : ; N) according to, we can obtain the following approximation bf (x ) = N i= f k (i) k (i) ; (i) X (x ) : (5) 4

6 It is straightforward to check using (4), (5), (7), (), () and (3) that f k (i) thus establishing the unbiasedness of (). k (i) ; = W ; Similarly, the second method to compute f (x) pointwise using () corresponds to an importance sampling method on the space F, U k= fkg k. The importance distribution is given by (; x : ), () = and for n (n; x :n ) = p ;n ;n (x :n ) with p ;n = Pr (X :n n ; X n+ = fg) = ( ) n ; (6) and Q M (x; x ) n M (x k ; x k ) k= ;n (x :n ) = ( ) n. (7) Using the importance sampling identity f (x) = f (x) () () + X n= = fk (x; X ::k ) (k; X :k ) n f n (x; x :n) (n; x :n ) (n; x :n ) dx :n () n o then sampling k (i) ; (i = ; : : : ; N) according to, we obtain the following approximation bf (x) = N i= Using (4), (5), (9), (6) and (7), we have f n x; :k (i) f k (i) k (i) ; :k (i) = W thus establishing the unbiasedness of (). x; :k (i) k (i) ; :k (i) : (9) x; :k (i) 3. Limitations of SIS The estimates (5) and (9) will have a reasonable Monte Carlo variance if the variance of the absolute value of the weights is small. However, this can be di cult to ensure using the standard SIS approach. First, it imposes an arbitrary geometric distribution for the simulated paths length (), 5

7 (6) which might be inappropriate. Second, a product of terms K k ; X(i) k =M k ; X(i) k appears in the expression of the weights if M 6= K ; its variance typically increases approximately exponentially fast with the length of the paths. Third, if we are interested in estimating the function on using (5), the initial distribution appears in the denominator of (7). This distribution has to be selected very carefully to ensure that the variance of the resulting weights will be nite. The performance of SIS algorithms can usually be dramatically improved by introducing a resampling step [], [3]. The basic idea is to monitor the variance of importance weights over time and to discard paths with small weights and multiply paths with high weights; resampled paths are given an equal weight. However, even with an incorporated resampling step, SIS might still be ine cient in this context as we are interested in estimating a function at time. ach time it is used, the resampling step decreases the diversity in the number of paths left from time to the current time index. B. Importance Sampling using Trans-dimensional MCMC In this paper, we propose an alternative approach in which we do not limit ourselves to imputing paths sequentially. The importance sampling identity (4) is valid for any distribution such that R f n n (x :n ) dx :n 6= ) p ;n > and f n (x :n ) 6= ) ;n (x :n ) 6=. Similarly () is valid when R f n n (x; x :n ) dx :n 6= ) p ;n > and f n (x; x :n ) 6= ) ;n (x :n ) 6=. We now show how it is possible to structure e cient importance distributions which can be sampled from using trans-dimensional MCMC methods.. Optimal Importance Distributions We propose selecting importance distributions (n; x :n ) [resp. (n; x :n )] which minimize the variance of the absolute value of the importance weights in (5) [resp. (9)] in order to reduce the Monte Carlo variance of these estimates. Let us rst consider the case (5). We de ne (n; x :n ) on F as follows. The renormalized In many applications K is not a Markov kernel and it is impossible to select M = K: 6

8 version of the absolute value of f n (x :n ) is given by ;n (x :n ) = c ;n jf n (x :n )j () with c ;n = jf n (x :n )j dx :n : n+ Note that if g (x) and K (x; y) for any x; y, then assumption () ensures c n <. However, in the more general case, we need to make the additional assumption c n < for any n. We also consider p ;n = c c ;n () where X c = c ;n : () n= It is assumed here that c < ; this is true if () holds. In this case, where f (x ) = c ; sgn (f (x )) ; (x ) + X c ;n sgn (f n (x :n )) ;n (x :n ) dx :n n n= X = c sgn (f (x )) (; x ) + c Given N random samples approximate (3) by n= >< if u ; sgn (u) = >: if u < : n sgn (f n (x :n )) (n; x :n ) dx :n n k (i) ; o distributed according to, it is possible to bf (x ) = c N i= sgn f k (i) x : This is clearly the optimal importance distribution as the variance of the absolute values of the importance weights is equal to zero. However, it is usually impossible to sample from (n; x :n ) exactly and to compute c in closed-form. We claim that these two problems can be satisfactorily solved in most cases using transdimensional MCMC. To sample from, which is a distribution de ned on a union of subspaces of di erent dimensions, we can use any trans-dimensional MCMC method such as the popular Reversible Jump MCMC (RJMCMC) algorithm [4], [5]. This idea involves building a F -valued 7

9 n ergodic Markov chain k (i) ; which admits :k oi (i) as an invariant distribution. This is a generalization of the standard Metropolis-Hastings algorithm. As i!, one obtains (correlated) samples distributed according to. Moreover, under the standard and realistic assumption that c ; = jg (x)j dx is known or can be estimated numerically; then we can obtain the following estimate of c namely bc = c ; bp ; where bp ; is the proportion of random samples such that k (i) = ; i.e. bp ; = N k (i) : (3) i= Let us now consider the case (9). The importance distribution is de ned on F = U k= fkg k with (n; x :n ) = p ;n ;n (x :n ) (4) where and ;n (x :n ) = c;n n (x; x :n )j ; (5) c ;n = jf n (x; x :n )j dx :n n p ;n = c c ;n; (6) c = X c ;n : (7) n= It is assumed that c < ; this is satis ed if () holds. In this case, f (x) = f (x) + Given N random samples X c ;n sgn (f n (x; x :n )) n (x :n ) dx :n n n= X = f (x) + c n= n sgn (f n (x; x :n )) (n; x :n ) dx :n : n k (i) ; :k (i) o N i= distributed according to, it is possible to approximate (3) by bf (x) = f (x) + c N i= sgn f k (i) x; : :k (i)

10 To sample from, we can use trans-dimensional MCMC. To estimate c, we use the fact that if c ; = jf (x; x )j dx = jg (x ) K (x; x )j dx is known or can be estimated numerically then we can obtain the following estimate of c bc = c ; bp ; where bp ; is the proportion of random samples such that k (i) = ; i.e. bp ; = N k (i) : () i=. A Reversible Jump Markov chain Monte Carlo algorithm For sake of completeness, we describe here a simple RJMCMC algorithm to sample from as de ned by (), () and (). A very similar algorithm could be propose to sample from as de ned by (4), (5), (6) and (7). More elaborate algorithms are discussed in [5]. This algorithm is based on update, birth and death moves. ach move is respectively selected with probability u k (i), b k (i) and d k (i) where u k (i) + b k (i) + d k (i) = at iteration i. We also introduce two proposal distributions on denoted by q u (x; ) and q b () : We denote the uniform distribution on A by U (A). Initialization. Set k () ; X () :k () randomly or deterministically. Iteration i : Sample U U [; ] : If U u k Update move Set k (i) = k, sample J U ; ; : : : ; k (i) and X J q u X J ;. With probability < min : ; set = X :J ; X J k (i) ; :J ; X J X k (i) ; X ) ; X(i q J+:k (i) u q u X ; X(i ) J+:k (i) ; otherwise set J XJ ) ; X(i J ; X J = X : :k 9 = ; (9) 9

11 lse If U u k + b k, Birth move Sample J U ; ; : : : ; k, sample X J q b (). With probability < min : ; k + ; set k (i) = k +, :k = X = X : :k X :J ; X ) J ; X(i k ; X q :k b (XJ ) b k :J ; X J 9 d J:k k + = ; (3) ) ; X(i ; otherwise set k (i) = k, J:k lse Death move Sample J U ; ; : : : ; k. With probability < min : ; k ; set k (i) = k, = = X : :k ) X :J ; X(i q J+:k b X k ; X d :k k X :J ; X(i ) J+:k (i J b k 9 = ; (3) ) ; otherwise set k (i) = k, To compute (9), (3) and (3), one needs to be able to compute ratios of the form (l; x :l ) (k; x :k ) = c l ;l (x :l ) c k ;k (x :k ) = f l (x :l ) f k (x :k ) : This can be performed easily as f l (x :l ) and f k (x :k ) are given by (5). It is easy to check that the invariant distribution of this Markov chain is. stablishing ergodicity has to be performed on a case by case basis. III. Application to Value Function stimation A. Model Our motivating application is related to control. We consider a Markov process fx k g k on with transition kernel P. Let us introduce a reward function r :! R + and a discount factor

12 (; ). When the process is in state x at time k it accumulates a reward k r (x). Thus the expected reward starting from X = x is given by " # X V (x) = X=x k r (X k ) : The expected reward is called the value function in optimal control []. Under standard regularity k= assumptions, it can be established that the value function satis es V (x) = P (x; y) V (y) dy + r (x) ; that is a Fredholm equation of the second kind () where f (x) = V (x), K (x; y) = P (x; y) and g (x) = r (x). B. xample We present here a simple example where all calculations can be performed analytically which emphasizes the limitations of SIS in this context. We denote by N m; the Gaussian distribution of mean m and variance and N x; m; = p exp! (x m) : We set P (x; y) = N y; x; (with jj < ) and r (x) = N x; ; r. In this case, one has X k j (X = x) N m k (x) ; k with m (x) = x; = and for k m k (x) = k x, k =! kx : i= It follows that X f (x) = k N m k (x) ; ; k + r : k= Assume one considers a SIS method to solve this problem. A sensible choice for M is M (x; y) = ( ) P (x; y) + (x ) : If one is interested in estimating the function at a given point x = x, then the importance weights are given by (9); that is >< W = >: k (i) g k (i) ( ) if k (i) ; g(x) if k (i) = :

13 The variance of the importance weights is given by i var hw x; = :k (i) p r This variance (3) will only be nite if X k= k N m k (x) ; ; k + r= f (x) : (3) <. In this case, the optimal importance function ;n can easily be computed in closed-form as p ;n is known and ;n (x :n ) is a Gaussian; the variance of the associated estimate is zero. When estimating the function f (x ), we consider the importance weights (7) given by W = >< >: ( ) g The variance of the importance weights is equal to i var hw = k (i) g k (i) if k (i) ; if k (i) = : X k p r (x ) N m k (x ) ; ; k + r=! dx k= f (x ) dx : (33) Assume we consider (x ) = N x ; ;, then to ensure that the variance (33) is nite, it requires < and > + r : For more complex problems, it could be impossible to determine analytically what are the necessary conditions on to ensure the variance is nite. In this case, the optimal importance function ;n admits a closed-form and the variance of the associated estimate is zero. IV. Discussion A similar methodology can be developed in any situation where we face a convergent series of integrals/sums; each term being possibly of di erent dimension. In particular, this could be applied to the solving of large linear/nonlinear systems of equations and would be an alternative to [7]. V. Note The rst version of this report was written in. After having updated this report in 4, we realized that a related idea had previously appeared in computer graphics []. However, it

14 is di cult to follow the developments in this paper for people who are not familiar with this eld, including the authors of this report. We believe that this report provides a more general methodology and might also be more accessible. In the 7 update, we have added the reference [7], corrected a couple of typos and updated (once more) the a liations of both authors. References [] D.P. Bertsekas, Dynamic Programming and Optimal Control, Athena Publishers, 99. [] P. Del Moral and L. Miclo, Branching and interacting particle systems approximations of Feynman-Kac formulae with applications to non-linear ltering. In Séminaire de Probabilités XXXIV, ds. J. Azéma, M. mery, M. Ledoux, and M. Yor, Lecture Notes in Mathematics, Berlin: Springer-Verlag, vol. 79, -45,. [3] A. Doucet, J.F.G. de Freitas and N.J. Gordon (eds.), Sequential Monte Carlo Methods in Practice. New York: Springer-Verlag,. [4] P.J. Green, Reversible jump MCMC computation and Bayesian model determination, Biometrika, vol., pp. 7-73, 995. [5] P.J. Green, Trans-dimensional Markov chain Monte Carlo, in Highly Structured Stochastic Systems, Oxford University Press, 3. [6] R.C. Gri ths and S. Tavaré, Simulating probability distributions in the coalescent, Theoretical Population Biology, vol. 46, pp. 3-59, 994. [7] J.H. Halton, Sequential Monte Carlo techniques for solving non-linear systems, Monte Carlo Methods and Applications, vol., no., pp. 3-4, 6. [] R. Rubinstein, Simulation and the Monte Carlo Method, New York: Wiley, 9. [9] J. Spanier and.m. Gelbard, Monte Carlo Principles and Neutron Transportation Problems. Reading, Massachusetts: Addison-Wesley, 969. []. Veach and L.J. Guibas, Metropolis light transport, SIGGRAPH 97 Proceedings, Addison- Wesley, pp ,