Selection and implementation of high-performance. platforms in nance: The end-user's point of view. Manchester, January University of Manchester

Transcription

1 University of Manchester School of Computer Science Manchester, January 2015 Grzegorz Kozikowski RP11 REPORT ENTITLED Selection and implementation of high-performance platforms in nance: The end-user's point of view Professor John Keane Doctor Xiaojun Zeng Main Supervisor Co-Supervisor Doctor Suzanne Embury Advisor 1 address: kozikowg@cs.man.ac.uk

2 Abstract This RP considers the advantages and disadvantages of competing HPC technologies. The RP is hosted by University of Manchester in collaboration with SWIP. The project focuses on the identication of technologies on the market and technology partners, with a specic intent of cross-fertilising nancial practice with best practices from other industries (oil &gas, climate modelling, medical imaging); selection and formulation of test-cases for HPC in nance (in conjunction with RP2 and RP3), including identication of comparative metrics; and implementation of test cases from RP2 and RP3 on cloud, grid, GPU and FPGA and comparison of these technologies in practice. This document includes a set of working papers concerning applications of dierent HPC platforms in nance. The research leading to these results has received funding from the People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/ / under REA grant agreement

3 Towards Real-time Risk Management: evaluating rst and second-order Greeks using Automatic Dierentiation on HPC Grzegorz Kozikowski a, Erik Vynckier b,1, Xiao-Jun Zeng a, John Keane a a School of Computer Science, University of Manchester, Manchester, United Kingdom b Alliance Bernstein, London, United Kingdom Abstract Performance is critical in nancial analytics to price options using stochastic dierential models. Of particular concern are the rst and second-order sensitivities commonly used to neutralize the nancial risk of traded commodities as well as model calibration. This paper proposes a more ecient Greeks calculation using optimal differentiation algorithms on multi- and many-core High Performance Computing architectures: GPUs, Intel Xeon, Intel Xeon-Phi Coprocessor supported by the frameworks CUDA, OpenCL and OpenMP with Xeon-Phi. Automatic Dierentiation for the rst- and second-order unbiased Greeks has been implemented. The Greeks are computed through a single Monte-Carlo simulation by the Adjoint with the cost of gradient computation three times that of function evaluation. Experimentation consider both the Black-Scholes and the Heston models, and shows accuracy and performance improvement compared to nite dierences or pathwise methods. Performance is roughly equivalent when more complicated discretization schemes for SDE models are utilized. Computation is reduced from tens of hours to tens of seconds and oers the promise of real-time risk addresses: kozikowg@cs.man.ac.uk (Grzegorz Kozikowski), erik.vynckier@alliancebernstein.com (Erik Vynckier), x.zeng@manchester.ac.uk (Xiao-Jun Zeng), john.keane@manchester.ac.uk (John Keane) 1 SWIP (July August 2013), Alliance Bernstein (since September 2013)

4 management analysis of investment portfolios consisting of many thousands of assets. Keywords: option pricing, the Greeks, the Adjoint, CUDA, OpenCL, Xeon-Phi 1. Introduction To avoid nancial risk and further market movements, nancial institutions invest money on the derivatives market - the total amount of outstanding positions in the US is close to $700 trillion [1] in The fundamental traded instrument is an option that gives the client the right to buy or sell the given asset at a certain time with the price negotiated at the present time [2]. There are two types of options: call, which gives the investor the opportunity to buy the asset, and put, which is to sell the traded instrument [2]. Several models have been derived, describing to a degree the dynamics of options [2]. Among the most common models for option pricing are the Black-Scholes and the Heston model based on stochastic calculus [3]. Black-Scholes simplies nancial reality by assuming that some parameters are constant and some properties are not observable in the market [2]. This is an underlying model used in exotic/basket options [4]; Heston proposes that option dynamics are dependent on the additional, non-constant parameter, volatility [3]. This more accurately expresses options with various expiration time (the time up to when the investor can buy or sell an underlying asset at a previously negotiated price). Both models are usually solved via Monte-Carlo simulation as their semi-closed solution only exists for European options (where the investor can only make a trade at expiration time and not before) and this does not take into account correlation between various underlying assets [4]. These models are useful for investors to price options as well as to calculate the rst- and second-order derivatives of option price with respect to model parameters. These values, known as the Greeks, are used by traders to manage and hedge (compensate loss on the market by gains made on another market and stabilize portfolio value) their investment 2

5 portfolios [2]. As an example, consider an investor wishing to neutralize his portfolio from further commodity price movements by delta-hedging. For this purpose, he calculates the Greek that measures how the change of the underlying asset price aects the option price (delta). Based on this value, he constantly adjusts his investment positions to cover further gains (or loss) on the stock market by loss (or gains) on derivatives market/options. This means that his overall portfolio value will remain constant for a short time (as option prices and stock prices permanently uctuate). To mitigate risk of movements on his portfolio, he must constantly re-calculate the Greeks and rebalance the positions. For portfolios consisting of thousands of underlying assets and options traded by banks every day, frequent rebalancing based on the Greeks (via Monte-Carlo simulation) is computationally demanding. Further, the option model calibration is analytically expensive as this often requires the gradient calculation (the rst-order Greeks) to t hundreds of market quotes and thousands of dierent underlying assets every day [5]. Most existing solutions for the Greeks calculation are based on the pathwise, likelihood or nite dierences methods whose computational cost of the gradient is at least proportional to the number of model parameters x the time of a single Monte-Carlo simulation [4]. This paper proposes a more ecient Greeks calculation using optimal dierentiation algorithms on High Performance Computing (HPC) such as CUDA, OpenCL, OpenMP and Xeon-Phi[6]. The Adjoint method reduces overall computational eort for the rst- and second-order Greeks (the second-order have not been previously addressed in a HPC context) calculation and improves accuracy compared to the pathwise, nite and likelihood methods used in commercial products. In this approach, the Adjoint requires fewer arithmetic operations to evaluate the unbiased gradient and Hessian of functions than standard pathwise methods (a single Monte-Carlo) [6]. Further, combination of the Adjoint and HPC boosts the performance of Monte-Carlo simulation, an embarrassinglyparallel problem, by towards three orders of magnitude. Hence, the approach oers potential for real-time risk management. 3

6 2. Related Work There are several dierentiation methods that can be combined with Monte- Carlo simulation to evaluate the Greeks. The simplest, the nite dierence method, requires two Monte-Carlo simulations that dier in a small change in a single input model parameter [6]. Based on these results, the sensitivity is evaluated as a ratio of the output change to the small change in the input parameter. Unfortunately, this method is both computationally expensive (the rst-order Greeks require (2 x model parameters) x the cost of a single Monte- Carlo) and inaccurate if the change is too large [7]. The likelihood method utilizes probability density functions of the Greeks. Therefore, they can be applied to non-smooth models, but often produce inaccurate results with large variance [8]. Pathwise methods are based on Automatic Dierentiation (AD) with the forward order. They require as many Monte-Carlo simulations as the number of model parameters to evaluate the Greeks [6]. The pathwise method for the unbiased Greeks has a computational cost generally smaller than nite dierences [4]. The main drawback of this technique is complexity in the case of more complex option models. Previous work applying the Adjoint to the LIBOR market models for the rst-order Greeks show improved accuracy of the LIBOR market Greeks of two order of magnitudes in comparison to nite dierences (bumping methods) [7] [9]. For the LIBOR market model, the Greeks required twice as much computation as the model function evaluation. Work in [10] and [11] considers application of the Adjoint to Monte-Carlo simulation for the Greeks: gamma and deltas of cancellable swaptions and Bermudan options. The codes were written in C++ language. Computation of the above case-studies take several seconds on an Intel Core I5 with 4 GB RAM. Work [12] investigates derivations for the rst and second-order unbiased Greeks for the Heston model using AD. The gradient and Hessian are computed on a sequential machine using pathwise methods. The dierentiation routines 4

7 were written in C++ language. They consider European and Asian options as case-studies. Studies have been carried out on an Intel Core Intel Core 2 Duo 3.16 Ghz with 4 Gb of RAM. Computation of the rst and second-order sensitivities with MC simulation using 64 timesteps and 65,535 scenarios takes approximately 4 seconds. Work [13] investigates parallel approaches to the unbiased gradient and Hessian computation using AD. This compares sequential and parallel methods using OpenCL with both pathwise and Adjoint algorithms. The speed-up of the Adjoint gradient calculation is about 180x vs CPU (the Hessian calculation were 200x faster on GPU vs. CPU). Analysis was performed on an AMD Athlon X2 with 2.8 Ghz and 2 GB RAM memory, parallel implementations have been tested on an NVIDIA GF 9800 GT. Work [14] examines application of the Adjoint to evaluation of the rst-order Greeks using Monte-Carlo simulation. This utilizes parallel GPU architectures such as NVIDIA CUDA to boost performance. The rst-order Greeks are calculated through Monte-Carlo simulation for the Black-Scholes and the Heston model. The parallel experiments are compared with a sequential implementation and run on an NVIDIA Tesla C2070 with 448 cores. They show an improvement of approximately two orders of magnitude for both the Black-Scholes and the Heston rst-order Greeks. This paper extends the above work by computing the second-order Greeks and utilizing dierent HPC frameworks such as CUDA, OpenCL, OpenMP, Xeon-Phi. Further, an accuracy and performance analysis of the Adjoint against pathwise and nite dierence methods is reported. Both sequential and parallel implementations are optimized in terms of performance and data-transfer (OpenMP and Xeon-Phi codes fully exploit vectorization techniques). The timing results show performance improvement when compared to the gradient and Hessian calculation for the Heston model presented in [12]. Application of various discretization schemes as quadratic/lognormal/double gamma and integrated double gamma via overloaded operators with the Adjoint [12] to HPC solution presented here will not signicantly aect overall performance results. 5

8 120 Performance boost (HPC vs. CPU) is roughly equivalent for more complicated discretization formulas, thus, this work produces faster Greeks than in [12], [9], [7], [10] [11]. 3. Denitions 3.1. Option Pricing and the Greeks Background. An option is a contract that allows trading underlying assets at the xed time (dened as the expiration time) for the specied price (known as the strike price). Therefore, in order to price options, we forecast the future commodity price to see how it diers from the strike price. Most option pricing models take into account several factors aecting the future underlying asset price as [2]: 1. current commodity price and strike price as the option value is the dierence between them; 2. volatility of commodity price that measures how unpredictable are future underlying asset price movements; 3. time to expiration is fundamental for specic options giving the right to buy assets at any time until expiration time; 4. interest rate, as interest rates increase, the expected growth rate of the commodity price tends to increase; Black-Scholes Model. The model forecasting the behaviour of commodity price is often expressed as a stochastic process (changes of the value are uncertain over time and are dependent on some probability). Besides the commodity price, this process enables prediction of the further volatility of the asset. This is called the stochastic volatility model. According to this model, the further commodity price S satises the following stochastic dierential equation [2]: ds(t) = αs(t)dt + σ(t)s(t)dw (t) (1) where α is its expected rate of return, σ(t) denotes a volatility and W (t) is a Wiener process (known as a Brownian motion - a stochastic process whose 6

9 140 increments over time have normal distribution and are independent of the previous process evolution). If σ(t) is constant over time, then this is called the Black Scholes model. Heston Model. The Heston model is an extension of the Black Scholes model that proposes the underlying asset price follows the various volatilities over time (σ t is a stochastic process) [15]. The non-constant volatility over time is observable in the option market. The commodity price of Heston satises the following stochastic equation [3] ds t+1 = µs t dt + σ t S t Wt 1. (2) The non-constant volatility is expressed as a mean reverting stochastic process of the form [15]: dσ t+1 = κ (θ σ t ) dt + ξ σ t dwt 2. (3) where κ denotes the mean reversion of volatility, θ the long-term variance, ξ the volatility of volatility and σ 0 the initial volatility. dwt 1 and dwt 2 are correlated random variables with normal distribution (the correlation factor is equal to ρ). Discretization schemes. The above nancial models express the dynamics of the underlying price and volatility driven by continuous stochastic processes. Nevertheless, numerical methods only operate on discrete models. Hence, a continuous stochastic process must be transformed into a discrete time model. In order to approximate the numerical solution of a stochastic dierential equation for option pricing we use Euler-Maruyama formulas [16] and discretize the Black- Scholes model. As result, we have: S t+dt = S t + αs(t)dt + σ(t)s(t) dtz (4) 145 Using the Euler-Maruyama scheme [16], the recursive discrete form of the Heston Model can be obtained as below [16]: S t+1 = S t e (r 1 2 σt)dt+ σ tdt Z 1 σ t+1 = σ t + κ (θ σ t )dt + ξ σ t dtz 2 7

10 150 Monte Carlo simulation. Most problems in risk management entail evaluation of stochastic functions involving multi-dimensional integrals with a high level of computational complexity. Monte Carlo simulation is the most ecient approach to determining the results of integral functions that are too complicated to be solved analytically, for example the option pricing model. Its computational eort increases approximately linearly with the number of random samples, while the complexity of analytical solutions tends to increase exponentially [2]. The key idea of Monte-Carlo simulation is production of many random different scenarios (paths) and evaluation of the further expected value. In this way, the expected value converges to the correct result when the number of paths increases. For nancial models describing the evolution of an underlying asset price or volatility, the Monte Carlo method assumes that each scenario is a sample payo calculated and discounted at the interest rate: C i = e rt max(s T K, 0) (5) 155 where C i is the payo the option price along the i-th path, S T is a commodity price at time T according to the i-th scenario, and K denotes the strike price (the previously negotiated price at which the commodity is traded at time T ). The expected value of the option price is equal to the average of all the discounted payos (M denotes the number of dierent payo scenarios paths), as below [4]: M i=0 v M = E(Φ(C i )) Φ(C i) (6) M Further, if we perform dierentiation operations for the expected value, we must take into account all sample paths [4]: dv M dθ = E(dΦ(C i) ) dθ M i=0 dφ(c i) dθ M. (7) These values (known as the Greeks) measure the impact of model factors on the option price and are fundamental in risk management and hedging. 8

11 Name the Greek Description Delta dv M ds0 measures the rate of change of option value with respect to changes in the current underlying asset price Vega dv M dσ0 measures sensitivity of the option value with respect to initial volatility. Table 1: The Greeks Description The Greeks. For hedging nancial instruments, option traders evaluate the Greeks (to measure how the model parameters aect further commodity price and, the option price) to adjust their portfolios [2]. The impact on the option position of alternative future scenarios is evaluated to quantify the dierent aspects of risk. If the risk is acceptable, no adjustment is made to the portfolio; if unacceptable, an appropriate position for either the underlying asset or option contract is taken Table (see 1) introduces several rst-order Greeks Automatic dierentiation 170 Background. Automatic Dierentiation (AD) is a set of algorithmic routines to accurately and eciently compute derivatives of a composite function[6] [13]. AD exploits the fact that each composite function can be interpreted as a sequence (chain-rule) of the elementary operators (such as addition, multiplication or exponential, etc.) required to evaluate its value. To explain AD, the Black- Scholes formula is used [6]. Example. Consider the Black-Scholes model and its recursive dependency of commodity price from time t to time t + 1. S t+1 = S t (1 + r dt + σ dt Z) (8) 175 Suppose, we want to evaluate the Black-Scholes single path from time t to t + 1. For this purpose, we derive a sequence (see Table 1a) (chain-rule) of elementary operations needed to evaluate the underlying asset price at time t + 1. This sequence can be expressed graphically using a Directed Acyclic Graph (DAG) (see Figure 1b) [6]. The edges represent a dependency between subsequent elementary operators or input parameters; the nodes are identied by f i 180 intermediate functions to evaluate the nal result. As can be seen, each f i 9

12 f 1 = r dt f 2 = 1 + f 1 f 3 = dt f 4 = σ f 3 f 5 = f 4 Z f 6 = f 2 + f 5 f 7 = S t f 6 (b) Chain-rule of the Black-Scholes model as a DAG (a) Chain-rule of the Black-Scholes model 185 is dependent on some previously computed f j or model parameters (i > j). It is worth noting that Black-Scholes requires 7 elementary operations to evaluate the underlying asset price at time t + 1 from time t with the forward order. Intuitively, in order to calculate the composite function, we begin computations from independent variables (S t,..., Z) through intermediate operations 190 (f 1,...,f 6 ) to the nal operator (f 7 ). By interpreting such a chain-rule with forward-order (or reverse) and applying basic dierentiation routines for elementary operators, AD computes derivatives of the composite function. These derivatives are accurately evaluated and subject only to rounding and not discretization error. This makes AD particularly attractive compared to standard numerical dierentiation methods, such as nite-dierences [9]. AD has two basic modes: Forward (Pathwise) and Reverse (the Adjoint) [6] Pathwise methods The pathwise method computes derivatives with the forward order and calls 195 dierentiation routines for elementary functions. Example. To explain, we perform dierentiation of the function from f 1 to f 7 (with the forward order). First, the derivatives df 1 and df 3 are equal to zero as f 1 and f 3 are independent of σ. Next, the derivative of df 4 is dσ f 3 + σ df 3. because f 4 contains 200 σ. Note, that this derivation includes previously evaluated and stored df 3. In 10

13 f 1 = r dt df 1 = 0 f 2 = 1 + f 1 df 2 = df 1 f 3 = dt df 3 = 0 f 4 = σ f 3 df 4 = dσ f 3 + σ df 3 f 5 = f 4 Z df 5 = df 4 Z f 6 = f 2 + f 5 df 6 = df 2 + df 5 f 7 = S t f 6 df 7 = S t df 6 (a) Chain-rule of Black-Scholes (Pathwise(b) Chain-rule of Black Scholes (Adjoint methods) methods) f 1 = r dt f 7 = df 7 df 7 = 1 f 2 = 1 + f 1 f 6 = df 7 df 6 f 7 = S t f 7 f 3 = dt S t = df 7 ds t f 7 = f 6 f 7 f 4 = σ f 3 f 2 = df 6 df 2 f 6 = f 6 f 5 = f 4 Z f 5 = df 6 df 5 f 6 = f 6 f 6 = f 2 + f 5 f 4 = df 5 df 4 f 5 = Z f 5 f 7 = S t f 6 Z = df 5 dz f 5 = f 5 f 3 = df 4 df 3 f 4 = σ f 4 σ = df 4 dσ f 4 = f 3 f 4 dt = df 3 ddt f 3 = 1 2 dt f 3 f 1 = df 2 df 1 f 2 = f 2 dt+ = df 1 ddt f 1 = r f 1 r = df 1 dr f 1 = dt f the next step we dierentiate f 5 with respect to f 4 (previously evaluated). Processing this sequence, the nal partial derivative of underlying asset price (df 7 ) with respect to σ is computed (see Table 1a). The gradient computation by the pathwise method is proportional to the number of independent input parameters The Adjoint method Adjoint performs computations in a reverse manner starting from the nal operation. This approach requires evaluation and storage of the function value and all intermediate results (DAG nodes). 210 Consider the following using the above DAG to evaluate the gradient of f 7 Example. We dierentiate the nal operation f 7 with respect to f 7. As expected, df7 df 7 is equal to 1. Assuming the values of all intrinsic functions (f 1,..., 215 f 6 ) are known, f 7 can be dierentiated with respect to S t the left branch and f 6 the right branch of the DAG graph. The rst derivative: Analogously: df 6 df 2 = d(f2+f5) df 2 df 7 df 6 = d(f6 St) ds t = 1 and df6 df 5 = S t In the next stage, df6 df 2 = d(f2+f5) df 5 df 7 ds t = d(f6 St) ds t = f 6 : and df6 df 5 are calculated: = 1 By multiplying the above results by the previously evaluated df7 df df 6 we have: 6 df 2 df7 df 6 = df7 df df 2 = S 6 t df 5 df7 df 6 = df7 df 5 = S t In the third step, we compute derivatives f 5 with respect to f 4 and f 5 with respect 11

14 Evaluation Pathwise Adjoint = = 20 Table 2: Number of arithmetic operations required to evaluate the gradient of the Black Scholes model to Z. Then: df 5 df 4 = d(f4 Z) df 4 = Z and df5 dz we multiply the above results by value df6 df 5 df 5 df 4 df6 df 5 df7 df 6 = df7 df 4 = Z S t df 5 dz df6 df 5 df7 df 6 = d(f4 Z) dz df7 df 6 = df7 dz and df7 dz = f 4 To obtain df7 df 4 from the previous step, then: = f 4 S t In the next stage, we dierentiate f 4 with respect to σ and f 4 with respect to f 3 in an analogous manner and multiply by the above results, giving: df 4 dσ df5 df 4 df6 df 5 df7 df 6 = df7 dσ = Z S t f 3 = Z S t dt and df4 df 3 df5 df 4 df6 df 5 df7 df 6 = df7 df 3 = σ Z S t Repeating this processing ow for all DAG nodes, the gradient of f 7 is evaluated. As can be seen, this procedure requires only one sweep through the chain-rule to calculate the gradient, thus, involving fewer computations than the Pathwise method Table 1b shows a complete chain-rule for the gradient of the Black-Scholes by the Adjoint (f i denotes the partial derivative df7 df i ): Table 2 gives the number of necessary arithmetic operations for the Pathwise and the Adjoint method (including the cost of function evaluation) Mathematical fundamentals In order to present the principles of the AD algorithms [17], we assume that: 235 x 1, x 2,..., x n denote independent variables, y i = f i (fleft i, f right i ) = f left i f right i a dierentiable function f i considered as a composition of two operands fleft i, f right i where 1 left, right i and an intrinsic operator as +,,, /, etc. If is a unary operator (sin, cos, ln, etc.) then f i is only dependent on fleft i 240 f 1,..., f k 1 intermediate intrinsic functions, f k the nal intrinsic function required to evaluate f, y i intermediate results, y i = f i (f i left, f i right ) or y i = f i (f i left ) for 1 i n. 12

15 x 1 x 2. x n y 1. y k = x 1 x 2.. x n f 1 (fleft 1, f right 1 ). f k (fleft k, f right k ) (9) 245 Evaluation of the gradient of f relies on dierentiation of the subsequent intrinsic functions with respect to each input variable. Reverse mode (the Adjoint method) gradient. The second AD method for the gradient calculation is known as the Adjoint Mode performed in a reverse manner. This is better suited to functions of many input variables. To explain the Adjoint method, we consider the relation below: f curr = f curr f i f curr, (10) where f i = f k f i and f curr = f k f curr. The index curr corresponds to the function which is directly dependent on the operations denoted by sub-index i (as can be seen in the DAG 1b), additionally for curr = k we assume: f curr = fcurr f curr = 1 Taking into account the previous equation, the formulae for left and right partial derivatives can be derived. f k = f curr f k, f left f left f curr f k = f curr f k. (11) f right f right f curr 250 Having carried out the above operations, the values of the partial derivatives are represented by nodes of the independent variables. As a result, we evaluate the gradient in a single DAG sweep. 13

16 Reverse Mode (the Adjoint method) Hessian matrix. To evaluate the second derivative of the function f k, we dierentiate both formulae for the gradient (11) with respect to the second variable x i : 2 f k = ( f curr f left x i x i f left f k f curr ) (12) = 2 f curr f left x i 2 f k = f right x i x i = 2 f curr f right x i f k f curr + ( f curr f right 2 f k f curr, f curr x i f left ) (13) f k (14) f curr f k f curr + 2 f k f curr x i f curr f right (15) 255 Obviously, this expression is equal to: 2 f k f k x i = 0. The obtained formu- lae require additional partial derivatives: 2 f curr f left x i and 2 f curr f right x i. These value introduced in [13] High Performance Computing (HPC) CUDA. CUDA is a parallel computing platform and programming model exploiting GPUs [18]. This technology gives performance of up to 4 teraflops ( FLOPS). OpenCL. OpenCL is a framework designed to simplify cross-platform programming, providing a common low-level API for many-core architectures, GPUs and eld-programmable gateway arrays (FPGA) supporting parallel processing [19]. Xeon-Phi. Xeon-Phi is a many-core hardware architecture designed for highly parallel problems. Xeon Phi coprocessors provide up to 61 cores that can compute 244 tasks (threads) at once, with a peak of about 1.2 teraflops ( FLOPs) [20]. 14

17 The Greek Recursive equation (the rst-order) ds t+1 df t (S t ) ds t ds 0 ds t ds 0 ds t+1 df t (S t ) ds t dr ds t dr + df t (r) dr ds t+1 df t (σ t ) dσ t + df t (S t ) ds t dσ t dσ t dσ 0 ds t dσ 0 ds t+1 df t (σ t ) dσ t dκ dσ t dκ + df t (S t ) ds t ds t dκ (a) Formulas for rst-order Greeks of the un-(b) Formulas for rst-order Greeks of the derlying asset The Greek Recursive equation (the rst-order) dσ t+1 dg t (σ t ) dσ t dσ 0 dσ t dσ 0 dσ t+1 dg t (σ t ) dσ t dξ dσ t dξ + dg t (ξ) dξ dσ t+1 dg t (σ t ) dσ t dθ dσ t dθ + dg t (θ) dθ dσ t+1 dg t (σ t ) dσ t dκ dσ t dκ + dg t (κ) dκ volatility Library for rst and second-order Greeks using HPC 4.1. Overview The idea underlying the rst and second-order Greeks calculation combines symbolic dierentiation and the Adjoint method. Considering the rst-order Greeks, the recursive formulas (via Monte-Carlo simulation) must be derived. 275 S t+1 = F t (S t, µ, σ t ) = S t e (µ 1 2 σt)dt+ σ tdt Z 1 σ t+1 = G t (κ, θ, ξ, σ t ) = σ t + κ (θ σ t ) dt + ξ σ t dtz Tables 1a and 1b present the recursive formulas for the rst-order Greeks through a single path of the Heston model. As can be observed, the Greeks at time t+1 are dependent only on the Greeks at time t and the partial derivatives of G and F. Based on this recursive dependency of underlying asset price and volatility, AD exploits the chain-rule representing functions F and G. These express the dependency of subsequent underlying asset prices and volatilities at time t and t + 1. The Adjoint is employed to calculate all partial derivatives of F and G. Further, these results are utilized to update the nal rst Greeks. The second-order Greeks are computed by combination of the Adjoint and pathwise methods. First, the DAG is processed with the forward-order to compute the rst-derivative of F and G. Further, the Adjoint algorithm process each partial derivative of F and G in the reverse manner to evaluate the second-order Greeks. Tables 1a and 1b show derivations of the formulas for the second-order Greeks. Performance is further improved by processing independent Monte Carlo paths in parallel. 15

18 The Greek Recursive equation (the second-order) d 2 S t+1 d 2 F t (S t ) d 2 S 0 d 2 ( ds t ) S ds 2 + df t (S t ) t 0 ds t d 2 S t d 2 S 0 d 2 F t (S t ) d 2 S t+1 d 2 r d 2 r d 2 F t (S t ) d 2 S t + 2 d2 F t (S t ) drds t ds t dr + ( ds t dr )2 + df t (S t ) dr d 2 S t d 2 r d 2 S t+1 d 2 F t (S t ) d 2 σ 0 d 2 ( ds t ) S dσ 2 + df t (S t ) t 0 ds t d 2 S t d 2 + d2 F t (σ t ) σ 0 d 2 ( dσ t ) σ dσ 2 + t 0 d 2 S t+1 d 2 κ d 2 F t (S t ) d 2 S t d 2 S t df t (σ t ) dσ t d2 σ t d 2 σ 0 ( ds t dκ )2 + df t (S t ) ds t d 2 κ + d2 F t (σ t ) dσ t dκ d 2 F t (σ t ) d 2 σ t dσ t dκ + ( dσ t dκ )2 + df t (σ t ) dσ t d 2 σ t d 2 κ (a) Formulas for second-order Greeks of the(b) Formulas for second-order Greeks of the stock price The Greek Recursive equation (the second-order) d 2 σ t+1 d 2 G t (σ t ) d 2 σ 0 d 2 ( dσ t ) σ dσ 2 + dg t (σ t ) t 0 dσ t d 2 σ t d 2 σ 0 d 2 G t (σ t ) d 2 σ t+1 d 2 κ d 2 σ t+1 d 2 θ d 2 σ t+1 d 2 ξ volatility + 2 d2 G t (σ t ) dκdσ t d 2 κ dσ t dκ + d2 G t (σ t ) d 2 ( dσ t σ dκ )2 + t dg t (σ t ) dκ d2 σ t d 2 κ d 2 G t (σ t ) d d2 G t (σ t ) θ dθdσ t dσ t dθ + d2 G t (σ t ) d 2 ( dσ t σ dθ )2 + t d 2 G t (σ t ) d 2 ξ dg t (σ t ) dθ d2 σ t d 2 θ + 2 d2 G t (σ t ) dξdσ t dσ t dξ + d2 G t (σ t ) d 2 ( dσ t σ dξ )2 + t dg t (σ t ) dξ d2 σ t d 2 ξ 4.2. Architecture The parallel library for the rst and second-order Greeks (via Monte-Carlo simulation) was implemented in C++. The object-oriented software combines dierent HPC frameworks (CUDA, OpenCL, OpenMP, Xeon-Phi) and has a generic design to allow denition of various nancial models, various discretization methods and dierent data types. The below list presents the generic types for the components. MonteCarlo<Model denition<value>, Platform, Greeks<Value>, Option<Value Model : BlackScholes, Heston Platform : CPU, GPU, Xeon-Phi, OpenMP Value : single/double precision Greeks : FirstGreeks, SecondGreeks The Monte-Carlo simulation is parameterized by the model, the HPC platform and the rst and second-order Greeks specied for the Black-Scholes or Heston model. The instance of the MC object is further initialized with the model parameters. The next stage is to record symbolic model denition by using 16

19 overloaded operators and create DAG. Figure 1a presents data and processing ow for evaluation of the Greeks on HPC. The rst stage is an interpretation of discrete models by using operator overloading to transform the input discrete model (a recursive dependency of underlying commodity price and volatility - functions F and G) to a DAG in order to utilize the Adjoint for the Greeks. Further, the number of paths and steps per single simulation, and the input model arguments are passed to initialize the DAG. For GPUs, these structures are then transferred from the CPU to the GPU distributed across many shared memories. Further, the independent MC scenarios are processed by threads on GPU. The Automatic Dierentiation routines are executed and implemented on device side/ooad-regions to allow parallel dierentiation of MC scenarios and compute the rst/second-order Greeks. Xeon-Phi implementation includes regions that are processed in o-load mode by Xeon-Phi cores on the Intel Many Integrated Core Architecture. Codes fully utilize vectorization techniques to boost performance. Next, parallel Monte-Carlo simulation is performed with each path computed by a single core. In the rst step, the DAG is processed to evaluate an option price of a single path and store intermediate results of intrinsic functions. These values are further used by the Adjoint to calculate the gradient of F and G. Based on these derivatives, the Greeks are recursively updated in each time step until expiration time. Analogously, the procedure of the second-order Greeks evaluation is called. When the Monte-Carlo simulation is done, all option prices and the Greeks are collected to evaluate the estimated option price and the nal Greeks. In the nal stage, these results are returned to the CPU. 5. Experimentation 335 Studies have evaluated the Monte Carlo simulation for the rst and second- order Greeks and examined both sequential and parallel algorithm versions im- 17

20 plemented on CUDA, OpenCL, OpenMP and Xeon-Phi 2. Tests were performed for the Black-Scholes and Heston models with a range of paths and steps on the following congurations: CPU: Intel Xeon X Ghz with 48GB RAM; 2. Intel Xeon-Phi Coprocessor 7120p 3. CUDA and OpenCL: an NVIDIA Tesla M2070 with 448 cores and an NVIDIA Kepler K40 with 2880 cores; The sequential and OpenMP implementation utilizes the GNU Scientic Library [21] to generate random samples with a normal distribution, whereas, the parallel CUDA version uses the Mersenne-Twister algorithm included in the CURAND library [22]. For the OpenCL implementation, the MWC64X library for uniform random numbers and the Box-Muller transform have been combined [23], [4]. GPU algorithms use 1024 available threads per thread block. The Xeon-Phi implementation utilizes the Gaussian random sampling algorithm with the Box-Muler transform included in Intel Math Kernel Library. Xeon-Phi and OpenMP are tested with various number of threads Analysis CPU. Figure 1b compares speed-up using the Adjoint method with the pathwise and FD methods. As can be seen, performance is independent of the number of paths. The results were similar for a varied range of steps. Table 1a shows execution times on a CPU - as expected, they are proportional to the number of steps. GPU (CUDA and OpenCL). Figures 1e, 1f, 1g, 1h, 1a and 1b compare the speed-up for the rst/second-order Greeks on GPUs with dierent time steps to 2 optimization commands --use_fast_math,-o3, Black-Scholes: S 0 = 50, σ 0 = 0.2, K= 50, r=0.25, Heston: S 0 = , σ 0 = 0.1, K= , r=0.1, θ = 0.1 κ =

21 12 10 Legend Adjoint vs. Pathwise - Black Scholes Adjoint vs FD - Black Scholes Adjoint vs Pathwise - Heston Adjoint vs FD - Heston 8 Speed-up Number of paths (a) Data and processing ow on HPC(b) First-order Greeks Speedup for Adjoint vs. Pathwise and Adjoint vs. Finite Dierences (Black- Scholes, Heston) CPU execution. Performance increases with the number of discretization steps. Tables 1c and 1d compares the accuracy of the Greeks calculation ( dv ds 0 through the Adjoint with pathwise and FD methods. and dv dθ ) 365 Many-core architecture (Xeon-Phi). Table 3 compares the performance of the rst-order Greeks' calculation on a Xeon processor with a Xeon-Phi Coprocessor (both tested with a single thread). As expected, the results conrm that the Xeon-Phi core is slower than the Xeon core (around 14x for the tested implementation). Plots 1c, 1d present the speed-up of option price sensitivities 370 calculation achieved on an Xeon-Phi Coprocessor. The performance increases with the number of threads used on a Xeon-Phi. For the Black-Scholes model, the peak speed-up is reached (60x 300 steps) when paths are calculated by 64 threads. The best performance results (speed-up of 40x-50x) are observed when 64 threads are utilized. Speed-up of 60X has been obtained for the Heston 375 implementation on a Xeon-Phi for 300 steps and 128 threads. 19

22 (a) First-order Greeks - CPU execution times Number of paths Black- Scholes 100 steps Black- Scholes 300 steps Heston 100 steps (ms) Heston 300 steps (ms) (ms) (ms) Number of paths Black- Scholes 100 steps Heston 100 steps (ms) (ms) (b) Second-order Greeks - CPU execution times Number Option Adjoint Pathwise FD Relative Relative of Paths Price Error (%) Error (%) Number Option Adjoint Pathwise FD Relative Relative of Paths Price Error (%) Error (%) (c) First-order Greeks - Accuracy (Black-Scholes 300 (d) First-order Greeks - Accuracy (Heston 300 steps) steps) Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) CUDA (K40) Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) CUDA (K40) Speed-up 40 Speed-up Number of paths Number of paths (e) GPU: First-order Greeks Speedup (Black-(f) GPU: First-order Greeks Speedup (Black- Scholes 100) Scholes 300) Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) CUDA (K40) Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) CUDA (K40) Speed-up Speed-up Number of paths Number of paths (g) GPU: First-order Greeks Speedup (Heston(h) GPU: First-order Greeks Speedup (Heston 100 steps) 300 steps)

23 100 Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) 100 Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) 80 CUDA (K40) 80 CUDA (K40) Speed-up 40 Speed-up Number of paths Number of paths (a) GPU: Second-order Greeks Speedup (Black-(b) GPU: Second-order Greeks Speedup (Heston Scholes 100 steps) 100 steps) Legend 4 threads 8 threads 16 threads 32 threads 64 threads 128 threads 240 threads Legend 4 threads 8 threads 16 threads 32 threads 64 threads 128 threads 240 threads Speed-up 40 Speed-up Number of paths Number of paths (c) Xeon-Phi: First-order Greeks Speedup(d) Xeon-Phi: First-order Greeks Speedup (Heston 300 (Black-Scholes 300 steps) steps) Black-Scholes Black-Scholes Heston Heston Number of steps Intel Xeon vs. Xeon-Phi 7x 14x 14x 14x Table 3: Speedup Intel Xeon vs. Intel Xeon-Phi 21

24 6. Conclusion The experiments conrm that the performance improvement (towards three orders of magnitude) of the Adjoint combined with HPC oer signicant potential in risk management and both evaluate robust Greeks. Combination of the MC framework presented here with more complex discretization schemes such as quadratic/lognormal/double gamma and integrated double gamma [12] via overloaded operators should not signicantly aect performance results. Performance boost is roughly equivalent for more complicated discretization formulas consisting of more elementary operators, thus, this work produces faster Greeks than [12], [9], [7], [10] [11]. This work oers the potential for real-time risk analysis real trading and hedging of thousands of underlying assets per second. The work can be applied as a key module of replicating portolio systems, trading platforms and can be combined with Least-Squares Monte-Carlo methods to price American/exotic options [4]. The results also indicate that the option model calibration to market-quotes may be reduced from several hours to seconds. The secondorder Greeks can be combined with the least squares optimization algorithms to t the Black-Scholes/Basket/Heston models to market data. 395 Acknowledgments The authors wish to acknowledge support from the EU FP7/ under grant (Marie Curie Action HPCFinance). References 400 [1] Statistical release - otc derivatives statistics, Tech. rep., Bank for International Settlements (2013). [2] J. Hull, Options, Futures and other Derivatives, Pearson Prentice Hall,

25 [3] J. Gatheral, The volatility surface: The Pracitioner's Guide, Wiley Finance, [4] P. Glasserman, Monte Carlo Methods in Financial Engineering, Springer, [5] J. London, Modeling Derivatives in C++, Wiley Finance, [6] E. P. J. U. A. W. Shaun Forth, Paul Hovland, Recent Advances in Algorithmic Dierentiation, Springer, [7] L. C. Mike Giles, Algorithmic dierentiation: Adjoint greeks made easy, SSRN. [8] L. Capriotti, Reducing the variance of likelihood ratio greeks with monte carlo, Proceedings of the 2008 Winter Simulation Conference. [9] L. Capriotti, Fast greeks by algorithmic dierentiation, SSRN. 415 [10] C. Y. Mark Joshi, Ecient greek estimation in generic swap-rate market models. [11] C. Y. Mark Joshi, Algorithmic hessians and the fast computation of crossgamma risk. 420 [12] D. Z. Jiun Hong Chan, Mark S. Joshi, First and second order greeks in the heston model. [13] G. Kozikowski, Interval arithmetic and automatic dierentiation on gpu using opencl, in: PARA 2012 Proceedings. 425 [14] G. Kozikowski, Parallel approach to monte carlo simulation for option price sensitivities using the adjoint and interval analysis, in: PPAM Proceedings [15] C. W. O. Anthonie Willem Van der Stoep, Lech A. Grzelak, The heston stochastic-local volatility model: Ecient monte carlo simulation, SSRN. 23

26 [16] F. D. Rouhah, Euler and milsein discretization, Volopta. 430 [17] C. E. G. C. A. Floudas, A review of recent advances in global optimization, Journal of Global Optimization. [18] NVIDIA, NVIDIA CUDA SDK, cuda-gpus (2013). [19] Khronos, OpenCL SDK, (2013). 435 [20] Intel, Xeon Phi SDK, processors/xeon/xeon-phi-detail.html (2013). [21] F. S. Foundation, GNU Scientic Library, (2014). [22] NVIDIA, NVIDIA CUDA CURAND Library SDK, (2013). 440 [23] D. Thomas, The MWC64X Random Number Generator Documentation, (2014). 24

27 High Performance Risk Analysis: Evaluating the LIBOR Market Model Greeks using the Adjoint Grzegorz Kozikowski a, Erik Vynckier b,1, Xiao-Jun Zeng a, John Keane a a School of Computer Science, University of Manchester, Manchester, United Kingdom b Alliance Bernstein, London, United Kingdom Abstract To properly manage investment portfolios nancial institutions must frequently price interest rate derivatives instruments and measure their risks using stochastic dierential models. Computation of these interest rate models is highly challenging as they are multidimensional, have no analytical solutions and are solved through Monte-Carlo simulation. Further, risk analysis based on the Greeks sensitivities of option pricing models is far more dicult and time-consuming if nite dierences, likelihood or pathwise methods are utilized. These techniques yield inaccurate results with poor variance and are unsuitable for highly dimensional problems. This paper investigates the combination of the Adjoint numerical dierentiation method and HPC platforms that results in more accurate and faster Greeks' evaluation via Monte-Carlo simulation. Automatic Dierentiation produces unbiased Greeks and reduces the computational eort to evaluate the gradient of the multidimensional interest rate models when compared to the - nite dierences or pathwise methods. The computational experiments consider the LIBOR Market Model and show performance improvement up to three orders of magnitude on modern GPUs and many-core architectures in comparison to sequential implementations utilizing pathwise and nite dierence methods. addresses: kozikowg@cs.man.ac.uk (Grzegorz Kozikowski), erik.vynckier@alliancebernstein.com (Erik Vynckier ), x.zeng@manchester.ac.uk (Xiao-Jun Zeng), john.keane@manchester.ac.uk (John Keane) 1 SWIP (July August 2013), Alliance Bernstein (since September 2013)

28 Keywords: interest rate, interest rate derivatives, LIBOR, the Greeks, Pathwise, the Adjoint, CUDA, OpenCL, Xeon-Phi, OpenMP 1. Introduction Interest rate generally denes the amount of money which is paid to a borrower by the lender and is a key factor in pricing interest rate derivatives (contracts whose value depends on the interest rate uctuations) [1]. A critical interest rate is the London Interchange Bank Oer Rate (LIBOR), which is quoted once a day by the British Bankers' Association and measures the rate of interest which investment banks pay when making wholesale deposits with other banks. The LIBOR index is an underlying asset of options derivatives instruments that give the client the right to buy or sell underlying asset at a specied price in the future. Several stochastic dierential models have been derived to model dynamics of interest rate options based on LIBOR. Investment bank oering interest rates derivatives must repeatedly solve stochastic dierential (SDE) models which consider many alternative scenarios via Monte-Carlo (MC) simulation to predict their price [1]. These stochastic models are also used by investors/banking traders to mitigate nancial risk and interest rate movements [2]. Suppose that an investor bought an interest rate option and wants to secure his portfolio from interest rate movements by delta hedging/vega hedging. For this purpose, he calculates the rst/second-order derivatives with respect to model parameters called the Greeks, which measure how the change of the current interest rate aects the forward interest rate (predicted value) and option price. Relying on this value, the investor repeatedly adjusts his investment positions to cover further gain (or loss) on the interest rate market by loss (or gains) on derivatives market. This means that his overall portfolio value will remain constant for a short time (as option prices and interest rates prices permanently change). If we consider banking investment portfolios consisting of many thousands 2

29 of assets, the Greeks' calculation and frequent re-balancing process is often computationally expensive as this is solved via Monte-Carlo (MC) simulation. Most existing systems to obtain the Greeks are based on one of the pathwise forward, likelihood or nite dierences methods whose computational cost is at least proportional to the number of input parameters and time of a single MC simulation [2]. For high dimensional problems such as the LIBOR Market Model (LMM), the risk sensitivities must be computed with respect to all the interest rates at various periods. As a result, the pathwise forward, likelihood and nite dierences are extremely expensive [3]. This work proposes a more ecient approach to the Greeks' calculation by using the Adjoint reverse dierentiation on High Performance Computing (HPC) platforms such as CUDA, OpenCL, OpenMP and Xeon-Phi. The outcome of this work is performance and accuracy improvement of the LMM Greeks by three orders of magnitude on HPC when compared to the pathwise/likelihood/nite dierence methods on sequential machines. The rest of the paper is structured as follows: Section 2 overview the work related with the interest rate derivatives hedging using LMM. Section 3 discusses the LMM model and the utilized methodologies: Automatic Dierentiation and High Performance Computing with focus on dierent platforms. Section 4 explains principles of the parallel algorithms on CUDA, OpenCL and Xeon-Phi. Section 5 concerns experimentation results. Section 6 summarises the work and considers further extensions Related Work There are several strategies to evaluate the Greeks risk sensitivities of option pricing models. The intuitive nite dierence method relies on evaluation of two payos via two dierent MC simulations with small change in input parameters. This is often computationally demanding and requires as many doubled MC simulations as the number of inputs. The results can also be aected by large inaccuracy. The likelihood methods exploit standard normal probability 3

30 density functions and can be applied to non-smooth models. This method also features a large variance. Pathwise approaches are based on Automatic Dierentiation with the forward order. These dierentiate the discretization scheme and the payo step by step starting from independent variables through each elementary function. This requires as many MC sweeps as the number of model parameters used. Work [3], [4] investigates application of the Adjoint to the LMM for the rst-order Greeks using sequential machines ad compares this against nite differences and likelihood ratio methods. The experiments use a sequential FAD- BAD++ library to evaluate the Adjoint. The results show accuracy improvement of two orders of magnitude when compared to nite dierences (bumping) methods. The approach is particularly ecient in estimating sensitivities for a large set of initial rates on a forward curve or volatility. In the reported case studies, the LMM Greeks require twice as much computation as the model evaluation. The studies presented in [5] show the Greeks' computation on sequential machines using a predicator-corrected drift factor and log-euler approximation. The predicator-corrected drift approach improves the accuracy of the LMM Greeks as its relative error is less than 1% (the log-euler scheme has a relative error greater than 1%). Work [6] explores a GPU implementation of displaced diusion LMM. Experiments on an NVIDIA K20 card show a 100x speed-up compared to a sequential implementation. Unfortunately, this work does not include the Greeks' calculation module or the reduction techniques necessary to collect all the forward interest rates and evaluate the payo. Work [7] investigates GPU and multi-core implementations of the LIBOR Market Model via the control variate MC method. The multi-core version linearly improves performance with the number of threads. The routines only compute the payo and do not allow computation of risk sensitivities (the Greeks). This paper oers more ecient LMM Greeks than [3], [4], [8] and [6] by using an optimal dierentiation algorithm the Adjoint implemented on HPC 4

31 (CUDA, OpenCL, Xeon-Phi and OpenMP). This problem has not been previously addressed on HPC. The Adjoint method decreases the computational eort 90 of the Greeks as indicated in [3]. Additionally, HPC signicantly contributes to performance improvement (by two orders of magnitude vs. sequential version using the Adjoint). Combination of the Adjoint with HPC improves overall performance by three orders of magnitude when compared to pathwise/nite dierences/likelihood-based implementations Denitions 3.1. LIBOR market model Introduction. The LMM was introduced in [9] to capture the dynamics of the London Inter-Bank Oered Rate the average rate published by leading banks that would be charged if borrowing from other UK banks. The LIBOR is quoted 100 daily for dierent periods (maturities) as 1, 3, 6, 12 months and various curren- cies. The LMM is produced by multidimensional SDEs commonly solved by MC simulation as analytical solutions do not exist. To explain the LMM principles, assume that L denotes the current interest rate for the period of length T. This means that the prots over the period T are proportional to T L. The 105 forward interest rate works analogously and basically species the interest rates for a loan that will be oered for a xed period in the future. In practice, we need to consider and predict many forward interest rates at various times for dierent periods as illustrated in table 1. Columns denote time (tenor) at which T i T 1 T 2 T 3... T N 2 T N 1 T N T 1 L 1 (T 1 ) T 2 L 2 (T 1 ) L 2 (T 2 ) T 3 L 3 (T 1 ) L 3 (T 2 ) L 3 (T 3 ) T N 1 L N 1 (T 1 ) L N 1 (T 2 ) L N 1 (T 3 )... L N 1 (T N 2 ) L N 1 (T N 1 ) T N L N (T 1 ) L N (T 2 ) L N (T 3 )... L N (T N 2 ) L N (T N 1 ) L N (T N ) Table 1: Libor Market Model Matrix of forward interest rates 5

32 110 prediction is done. Rows present the periods for which the interest rates are modelled. In general, we consider a set of tenor dates T 1, T 2,..., T N 1, T N at which the forward interest rates are predicted: T 1 < T 2 <... < T N 1 < T N. Then let L i (T n ), i > n, n = 1, 2,..., N 1, N (1) be the forward interest rate at time T n for the period T i = T i+1 T i.: Model denition. The evolution of the LMM is described by a set of multidimensional SDEs as follows [9], [1]: dl i (t) L i (t) = µ i(t, L(t))dt + σ i (t)dw i (t) where 0 t T i and i = 1,..., N. Each Brownian motion is correlated according to the correlation matrix with the elements of the form dw i (t)dw j (t) = ρ j (t)dt. The parameter σ i (t) is the volatility of interest rate at time t. The drift term µ i (T, L(t)) is a function of the LIBOR vector L(t) as follows: i ρ ijdt iσ i(t)σ j(t)l j(t) j=k+1 1+dt jl j(t) if k < i µ i (t) = 0 if i=k i ρ ijdt iσ i(t)σ j(t)l j(t) j=k+1 1+dt jl j(t) if k > i Monte-Carlo simulation. In order to simulate the LMM, the Euler scheme is applied to the logarithms of the forward rates and this gives the evolution [3]: (2) L n+1 i = L n i exp((σ i n 1 S n i 1 2 σ2 i n 1)) T i + σ i n 1 Z n T i, i > n (3) 115 where L n i denotes the forward LIBOR rate for the interval time T i = T i+1 T i at time n = 0, 1, 2,..., N 1, N maturity 1. and (i > n), σ i n 1 is the interest rate volatility at time i, Z n is a random sample with normal distribution, and 6

33 S n i is 2 : if S n i i Si n = j=n+1 σ i n 1 T i L n j 1 + T i L n, i > n (4) j is as above, then Si+1 n for the (i+1)th period is equal to: S n i+1 = S n i + σ i n 1 T i L n j 1 + T i L n j, i > n (5) This recursive formula involves dependency between subsequent summations 1. Therefore, to calculate the forward interest rate L i+1 (T n ) at time T n for the (i+1)th period, the previously calculated S k i following factor: σ i n 1 T i L n j 1 + T i L n j The expected forward interest rate L N i is utilized (from L i (T n )) and the (6) for the period N at time i is estimated using the average of independent scenarios of L i j N with dierent Z n : L N i = E(L N i ) = Numpaths j=0 L N ij Num paths (7) where j is the index of j-th MC replication (j = 1,..., Num paths ). Options on interest rates (caplets). A caplet is a European call or option on the LIBOR interest rate in which the buyer receives payment when the forward interest rate at time T n exceeds the strike price K. Its payo is evaluated by the following formula [1]: caplet i = N period 1year max(l i K i, 0) (8) 120 where N denotes the notional of money the buyer invests. This contract is widely traded on the market and are underlying assets of caps collections of caplets [1]. 2 Assuming that volatility is a function of time to maturity 7

34 Figure 1: Monte-Carlo scheme for the LIBOR Market Model The Greeks. Let p i (T N ) denotes the payo of caplet for the LIBOR interest rate at time T N. In order to calculate the sensitivity (the Greek delta) with respect to the i-th forward rate L i (T 0 ) predicted at time T 0, we dierentiate the evolution of a single MC scenario as below [3]: dp i (T N ) dl i (T 0 ) = p i(t N ) L i (T N ) L i (T N 1 ) L i (T N ) L i (T N 1 ) L i (T N 2 )... L i(t 2 ) L i (T 1 ) L i (T 1 ) L i (T 0 ) (9) During dierentiation of Li(Tn+1) L i(t n) we need to take the inner subsequent summation factors and dierentiate these with respect to L n i ( Si(Tn) L i(t n) ). Analogously in the case of the Greeks with respect to the volatility, we need: S i(t n+1) σ. Considering the recursive MC formula: i n L i(t n+1) σ n i L n+1 i = L n i exp((σ i n 1 S n i 1 2 σ2 i n 1) T i + σ i n 1 Z n T i ), i > n (10) we can dierentiate a single path to obtain symbolic derivations required for Automatic Dierentiation (the Adjoint). The above formula can be expressed as dependency of L n+1 on L n i and S n i (Ln i )): and L n+1 i = F n+1 (L n i, S n i (L n i )) = exp((σ i n 1 S n i 1 2 σ2 i n 1) T i +σ i n 1 Z n T i ), i > n Dierentiating this by L 0 i we have: (11) 8

35 T i T 1 T 2 T 3... T N 2 T N 1 T N T 1 L 1 (T 1 ) L 1 (T 0 ) T 2. L 2 (T 2 ) L 2 (T 0 ) T 3.. L 3 (T 3 ) L 3 (T 0 ) T N L N 1 (T N 1 ) L N 1 (T 0 ) T N L N (T 1 ) L N (T 0 ) L N (T N ) L N (T 0 ) Table 2: Matrix of the Greeks for the LIBOR Market Model dl n+1 i dl 0 i = ( df n+1(l n i, Sn i (Ln i )) dl n i + df n+1(l n i, Sn i (Ln i )) dsi n(ln i ) dsn i (Ln i ) dl n i ) dln i dl 0 i (12) 125 Throughout the MC evolution, all the partial Greeks are recursively updated according to the above derivations to produce the nal Greeks. As result, we obtain a triangular matrix of the delta Greeks for the forward-interest rates predicted at time n for the various periods i Automatic Dierentiation Introduction. Automatic Dierentiation (AD) is a computational technique to precisely and eciently evaluate derivatives of a composite function [10]. The key concept of AD is that each composite function is a set of elementary (intrinsic) operators such as addition or multiplication. These functions are processed in a sequence (chain-rule) to calculate the nal value of the composite function. AD has two modes: Forward known as Pathwise and Reverse (the Adjoint) [11]. In order to explain AD, consider a single time-step calculation of the forward interest rate from the period n to n This composite function can be represented as a chain-rule of the elementary functions: f 1, f 9. Intuitively, to calculate the nal value we need to process it in forward order. By applying elementary dierentiation routines to these formulas (Column 1, Table 3) 3 Note, this also requires evaluated partial derivatives of S n i+1 with respect to Ln j and σ i n 1. These values are evaluated in a similar way as presented here 9

36 f 1 = σ S n df 1 = S n + dsn dσ f 9 = df9 df 9 = 1 f 2 = σ 2 df 2 = 2σ L n = df9 dl n f 9 = f 8 f 9 f 3 = 1 2 f 2 df 3 = 1 2 df 2 f 8 = df9 df 8 f 9 = L n f 9 f 4 = f 1 f 3 df 4 = df 1 df 3 f 7 = df8 df 7 f 8 = exp(f 7 ) f 8 f 5 = f 4 T df 5 = df 4 T f 5 = df7 df 5 f 7 = f 7 f 6 = Z n T σ df 6 = Z n T f 6 = df7 df 6 f 7 = f 7 f 7 = f 5 + f 6 df 7 = df 5 + df 6 dσ = df6 dσ f 6 = Z n T f 6 f 8 = exp(f 7 ) df 8 = exp(f 7 ) df 7 df 4 = df5 df 4 f 5 = T f 5 f 9 = L n f 8 df 9 = L n df 8 df 1 = df4 df 1 f 4 = f 4 f 3 = df4 df 3 f 4 = f 4 f 2 = df3 df 2 f 3 = 1 2 f 3 σ+ = 2 σ f 2 σ+ = df1 dσ f 1 = S n f 1 S n = df1 ds n = σ f 1 Table 3: Calculation and dierentiation of a single path for the forward-interest rate and processing forward, the rst-order derivative with respect to volatility (the 140 Greek) can be obtained as follows (Column 2, Table 3). The results are subject to rounding and not discretization error. The second mode the Adjoint is based on backward processing starting from the nal value of the composite function (Column 3, Table 3). This approach requires storage of the function values and all intermediate results. 145 Supposing we evaluate the nal function value, we begin dierentiating the - nal function df 9 with respect to df 9. Going back and dierentiating f 9 with respect to L n and f 8 parameters of f 9 in a reverse manner, we calculate partial derivatives. Analogously, further reverse processing of the chain-rule gives the rst-order derivatives of f 9 with respect to f 7 and f 6. As can be observed, 150 this procedure requires one iteration to obtain the gradient. 10

37 3.3. High Performance Computing (HPC) This work investigates the implementation of LMM on the following HPC platforms: CUDA is an NVIDIA platform, oering parallel computation by processing tasks as threads on multiple cores residing on a GPU [12]. CUDA allows access to several storage spaces (global/shared and constant memory) to optimize performance and data-transfer. The framework includes synchronization barrier methods to execute sequential parts requiring cooperation within threads in thread-block and thread-grid. CUDA architecture is suitable for embarrassingly parallel problems such as MC simulation. 2. OpenCL is a software framework providing a common low-level API for various HPC architectures with many-core processors, GPUs and FPGAs (eld parallel programmable gateways arrays) [13]. 3. OpenMP (OMP) is an API designed for multi-core architectures with shared memory. OMP denes a set of programming instructions to fork a highly parallel problem into many independent sub-problems separately processed on dierent cores [14]. 4. Xeon-Phi is an Intel architecture designed for highly dimensional parallel tasks. These are processed on an Xeon-Phi Coprocessor with 61 cores processing up to 244 threads. The Intel Xeon-Phi Cooprocessor computes up to 1.2 oating point operations per second [15]. 4. Library for the LMM Greeks using HPC 175 Introduction. As the library has been implemented with a focus on performance and optimization of available memory resources, vectorization, multi-threading, and cached/shared memory have been utilised [15], [12]. The key concept of the multi and many-core implementations is parallelization of MC simulation with respect to the number of scenarios. Each MC path is calculated and dierentiated by either a single core on Xeon-Phi or a thread on GPU [12]. 11

38 CUDA & OpenCL. The MC simulation on GPU is run by calling the kernel routine [12]: each-thread block is responsible for calculation of a single MC scenario for dierent tenor dates. Each thread within the thread-block evaluates and differentiates the forward interest rate for the xed period. Dierent MC scenarios are independently processed by various thread-blocks [12]. In brief, each row of Table 1 is processed by an independent thread and threads within the threadblock are synchronized to calculate the subsequent S. This function takes several input parameters: the chain-rule (as directed acyclic graphs (DAGSs)) represents forward interest rate and short rate, number of paths, number of steps, number of tenors and strike for pricing caplets. Firstly, the array representing S is dened in shared memory in order to avoid transfers between global and cached memory. This array stores subsequent values of S calculated in the rst inner for loop. Afterwards, the synchronization/barrier methods are employed to correctly update the array S according to (5). The second stage evaluates the forward rates (processing forwardratedag). In both cases, the partial results are stored in DAG nodes. The next step is reverse dierentiation through DAGs representing S and forward-rates. This process is done by the Adjoint functions. Having computed all the rst-derivatives of forward-rate at time T and S, the nal Greeks can be updated by using the formulas 12. Algorithm1 presents the principles of a kernel function for the LMM Greeks via MC simulation. Xeon-Phi & OpenMP. The implementation using OpenMP (OMP) and Xeon- Phi instructions is parallelized using the number of scenarios. According to Tables 1 and 1, each thread calculates all the forward interest rates for dierent periods forecast at various times. This additionally exploits vectorization techniques to process 16 or 8 paths represented by single-precision or doubleprecision numbers per single instruction. The outer loop iterating scenarios uses OpenMP pragmas [14] with the various number of threads. Two inner for instructions price the caplet and compute the Greeks of a single LMM scenario. Further, each function for evaluation and dierentiation DAGs takes 8/16 numerical values to process these by vectorization. This combination maximizes 12

39 input : forwardratedag, sdag forwardratedagsize, sdagsize, numpaths, numsteps, Strike output: payo, FirstGreeks, SecondGreeks kernelfunction( shared SArray[LMMBLOCKSIZE] pathid getnumberofthread() for step 1 to numsteps do end // Initialize DAGs for graphid 1 to sdagsize do EvaluateDAG (sdag, graphid, vecid) end // Initialize SArray with synchronization and reduction for graphid 1 to forwardratedagsize do EvaluateDAG (forwardratedag, graphid, vecid) end for graphid sdagsize to 1 do AdjointGradDAG (sdag, graphid, vecid) end for graphid forwardratedagsize to 1 do AdjointGradDAG (forwardratedag, graphid, vecid) end // Update FirstGreeks using recursive formulas // Collect partial results and calculate the estimated payoff and FirstGreeks ) Algorithm 1: GPU implementation 13

40 210 performance on Xeon-Phi [15]. 5. Experimentation 215 Sequential and parallel versions of the MCsimulation for the LMM have been implemented on CUDA, OpenCL, OMP, Xeon-Phi 4. Tests have been performed with 1024 tenors and various ranges of number of paths on the following machine congurations: CPU: Intel Xeon X Ghz with 48GB RAM memory Intel Xeon-Phi Coprocessor 7120p CUDA and OpenCL: NVIDIA Tesla M2070 with 448 cores and NVIDIA Kepler K40 with 2880 cores Parallel implementations use various random number generators. OMP the GNU Scientic Library [16] for random samples with normal distribution. CUDA the Mersenne-Twister algorithm implemented in the CURAND library [17]. OpenCL he MWC64X library [18], with the Box-Muller transform [2]. Xeon- Phi the Gaussian random sampling with the Box-Muller transform in the Intel Math Kernel library [15] Analysis GPU (CUDA and OpenCL) 230 Figure 2 compares speedup of CUDA and OpenCL implementations of the rst-order Adjoint Greeks for the LMM on NVIDIA Kepler and Tesla cards respectively. Performance on Tesla achives 44x speedup for 300 paths; for Kepler, performance uctuates at 200 paths, and then remains stable at 48x speedup; OpenCL shows slightly better performance than CUDA for Kepler; nally, the performance dierence between CUDA and OpenCL (on Tesla) is negligible. 4 Optimization commands --use_fast_math,-o3 14

41 Number CPU CUDA OpenCL CUDA OpenCL of time (2050) (2050) (K40) (K40) Legend OpenCL (M2070) OpenCL (K40) CUDA (M2070) CUDA (K40) paths (seconds) Speedup Number of paths Figure 2: GPU: First-order Greeks Speedup (LMM 1024 tenors) x 41x 49x 47x x 43x 48x 48x x 43x 48x 49x x 44x 48x 49x x 44x 48x 49x x 44x 48x 49x Figure 3: GPU: First-order Greeks Speedup (LMM 1024 tenors) Multi/many-core architectures (OpenMP and Xeon-Phi) Plot 4a compares speedup of the rst-order Greeks on a Xeon machine supported by OMP instructions diering the number of threads from 2 to 16. The speed-up is computed based on execution on a single thread. As expected, performance improves with the number of threads used in all cases; the best speed-up achieved was 4x on 16 threads. Figure 4b shows the performance results of Xeon-Phi routines in ooad-mode for a range of threads from 4 to 240 (scenario tasks are assigned to subsequent threads compact mode). The best speedup achieved is 11x on 240 threads. 15

42 6 5 4 Legend OMP (2 threads) OMP (4 threads) OMP (8 threads) OMP (16 threads) Legend 4 threads 8 threads 16 threads 32 threads 64 threads 128 threads 240 threads Speed-up 3 Speed-up Number of paths Number of paths (a) OMP: First-order Greeks Speedup (LMM 1024 tenors) (b) Xeon-Phi: First-order Greeks Speedup (LMM 50 tenors) 6. Conclusion This work evaluates the LIBOR interest rates and their Greeks in a more ecient way than reported in [3], [6], [4] and [8]. The results show that the combination of the Adjoint and HPC is attractive as it reduces the computational eort as well as improves accuracy. Overall performance is improved by up to three orders of magnitude when compared to pathwise forward, likelihood or nite dierences methods. The exibility of the framework oers application of other LIBOR derivations such as LIBOR-SABR [19], etc to capture the volatility smile [19]. This has great potential in real-time risk analysis/hedging interest rate derivatives and can be applied in more complicated systems such as CVA, replicating portfolios or PFE [20]. Acknowledgments 255 The authors wish to acknowledge support from the EU FP/ frame- work under grant (Marie Curie Programme HPCFinance). [1] J. Hull, Options, Futures and other Derivatives, Pearson Prentice Hall,

43 260 [2] P. Glasserman, Monte Carlo Methods in Financial Engineering, Springer, [3] L. C. Mike Giles, Algorithmic dierentiation: Adjoint greeks made easy, SSRN. [4] L. Capriotti, Fast greeks by algorithmic dierentiation, SSRN. [5] M. J. Nick Denson, Fast and accurate greeks for the libor market model. 265 [6] M. Joshi, Kooderive: Multi-core graphics cards, the libor market model, least-squares monte carlo and the pricing of cancellable swapstle. [7] X. C.-l. Liang Yi-Juan, Accelerating monte carlo simulation of libor market model via control variates and parallelization. 270 [8] D. Z. Jiun Hong Chan, Mark S. Joshi, First and second order greeks in the heston model. [9] M. M. Alan Brace, Dariusz Gatarek, The market model of interest rate dynamics. [10] E. P. J. U. A. W. Shaun Forth, Paul Hovland, Recent Advances in Algorithmic Dierentiation, Springer, [11] G. Kozikowski, Interval arithmetic and automatic dierentiation on gpu using opencl, in: PARA 2012 Proceedings. [12] NVIDIA, NVIDIA CUDA SDK, cuda-gpus (2013). [13] Khronos, OpenCL SDK, (2013). 280 [14] OpenMP, OpenMP, 0.pdf (July 2013). [15] Intel, Xeon Phi SDK, processors/xeon/xeon-phi-detail.html (2013). 17

44 285 [16] F. S. Foundation, GNU Scientic Library, (2014). [17] NVIDIA, NVIDIA CUDA CURAND Library SDK, (2013). 290 [18] D. Thomas, The MWC64X Random Number Generator Documentation, (2014). [19] J. L. S. C. V. Ana Ferreiro, Jose Garcia-Rodrigues, Sabr/libor market models: Pricing and calibration for some interest rate derivatives. [20] J. Gregory, Counterparty Credit Risk and Credit Value Adjustment:A Continuing Challenge for Global Financial Markets, 2nd Edition. 18

45 Monte Carlo simulation for the rst-order Greeks for the Black-Scholes and Heston models using Automatic Dierentiation on FPGA Grzegorz Kozikowski a, Erik Vynckier b,1, Xiao-Jun Zeng a, John Keane a a School of Computer Science, University of Manchester, Manchester, United Kingdom b Alliance Bernstein, London, United Kingdom Abstract Pricing options and calculating their risk sensitivities are computationally challenging problems in nancial and insurance sector. To manage their investment portfolios, nancial institutions must price and hedge derivatives by using and solving complex stochastic dierential models via Monte-Carlo simulations. Performing Monte-Carlo simulation for the Greeks (the rst-order derivatives of the option pricing model with respect to inputs) on a sequential machine is slow, furthermore, this is burdened by high discretization/bias and numerical error if nite dierences, likelihood or pathwise methods are used. This paper evaluates a dataow FPGA implementation of the Adjoint (AD) an ecient and accurate dierentiation method to calculate the Greeks for option pricing models. The Adjoint requires only one Monte-Carlo sweep to obtain all the Greeks partial model derivatives, hence, providing improvement when compared to the nite dierences, likelihood and pathwise methods. The computational experiments consider the Black-Scholes and Heston models for option pricing. The implementation improves performance by up to three orders of magnitude, hence oering signicant potential within hedging/replicating portfolio platforms and investment portfolio management. addresses: kozikowg@cs.man.ac.uk (Grzegorz Kozikowski), erik.vynckier@alliancebernstein.com (Erik Vynckier), x.zeng@manchester.ac.uk (Xiao-Jun Zeng), john.keane@manchester.ac.uk (John Keane) 1 SWIP (July August 2013), Alliance Bernstein (since September 2013)

46 Keywords: FPGA, Black-Scholes, Heston, the Greeks, the Adjoint 1. Introduction In order to reduce nancial risk and mitigate further market movements, investment banks invest money in derivatives market whose total amount of outstanding positions in the US is worth $700 trillion [1] in The principal traded instrument is an option that allows the holder the right to buy or sell the given commodity in the future (maturity time) for a xed price negotiated at the present time (strike price) [7]. Several models have been derived to describe dynamics of options [7]. The most common option pricing models are the Black-Scholes and the Heston model driven by stochastic probability [5]. Black-Scholes assumes that the volatility of commodity price is constant over time [7]. This is an underlying model used in pricing exotic/basket options [6]. Hestom model proposes various volatility of the commodity price [5]. This more accurately describes options with dierent expiration time (the time up to when the investor can exercise options buy or sell an underlying commodity at the previous contracted price). Both models are usually solved via Monte-Carlo simulation as their solution exists for European options (the investor can only exercise options at expiration time and not before) [7]. These models are utilized to price options as well as to calculate the rst-order derivatives (risk sensitivities) of option price with respect to model parameters. These numerical values, known as the Greeks, are used in trading to manage and hedge (compensate loss on the market by gain made on the derivatives market and secure overall portfolio value) investment portfolios [7]. Suppose that investor wants to neutralize his portfolio from commodity price movements. For this purpose he calculates the Greeks that measure how the change of input model parameters aects the option price. Relying on this value, he constantly adjusts his investment positions to cover further gains (or loss) on the stock market by loss (or gains) on derivatives market. This means 2

47 his overall portfolio remains constant for a short time (as commodity prices constantly uctuate). To avoid risk of movements on his portfolio, he must permanently re-calculate the Greeks and re-balance investment positions. For portfolios consisting of many thousands of assets, the re-balancing technique is computationally expensive as the option pricing are solved via Monte-Carlo simulations using many dierent scenarios. This paper oers a more ecient Greeks by a combination of high technology and numerical methods. The risk sensitivities calculation process uses Automatic Dierentiation algorithms on dataow architectures supported by a FPGA card. The Adjoint-AD method reduces overall computational eort of the Greeks and improves the accuracy compared to the pathwise/nite/likelihood methods used in commercial products. Further, FPGA greatly contributes to performance improvement (by three orders of magnitude vs. sequential version using the Adjoint). Overall the combination of the Adjoint with FPGA improves performance by four orders of magnitude when compared to pathwise/nite dierences/likelihood-based implementations on sequential machines. Dataow version is implemented by using Maxeler technology [9]. The paper is structures as below: Section 2 overview the work related with FPGA implementations of Monte-Carlo simulations for nancial option pricing. Section 3 discusses option pricing/hedging via the Black-Scholes and the Heston models, the utilized methodologies: Automatic Dierentiation, the Adjoint and Maxeler FPGA technology. Section 4 explains principles of the dataow algorithms on FPGA. Section 5 concerns experimentation results. Section 6 concludes the work. 2. Related Work 55 Several methodologies to evaluate the Greeks - option price sensitivities - have been derived. The simplest is the nite dierence method based on evaluation of two dierent MC simulations with small change in input model parameters. The sensitivity is calculated as a ratio of the output change to the small 3

48 change in the inputs. This is computationally expensive (the rst-order Greeks require 2 x model parameter x the cost of a single Monte-Carlo simulation) for all the Greeks. This is inaccurate if the change is too large. The likelihood methods utilize standard normal probability density functions and can be utilized in non-smooth functions. Nevertheless, this strategy often gives a large variance. Pathwise methods are based on Automatic Dierentiation (AD) with the forward order processing (starting from independent variables through each elementary arithmetic operator). This needs as many MC simulations as the number of model parameters to evaluate the Greeks. Work [3] presents a hardware implementation of Monte-Carlo simulation for the Heston model for European options. The approach developed in VisualHDL on Xilinx Virtex-5 saves around 89 % of energy and doubles performance vs. CPU equipped with an Intel Xeon CPU W Ghz and 8 GB RAM. Unfortunately, the implementation does not support the Greeks calculation required for hedging investment portfolios. Work [4] explores an FPGA implementation of a Monte-Carlo method to price Asian options via Black-Scholes model on a FPGA Altera Stratix-V card. The approach has been developed in the Impulse C environment supporting oating-point arithmetic on FPGA. For generating normal distribution random samples, the Mersenne Twister with Box-Muller transform has been utilized. The results show the speedup improvement by 504x over an execution on a single CPU (Intel i Ghz) and 149x per 4-core implementation supported by OpenMP. The implementation only investigates the Black-Scholes model and do not allow risk sensitivities calculation. Work [16] explores an FPGA engine for solving the Black-Scholes model via Monte-Carlo simulation. This is benchmarked on a Maxwell Supercomputer equipped with 64 FPGA Xilinx Viertex-4 XC4VSX55 cards and compared against a 32 core CPU cluster. The speed-up achieved was around 750x vs. CPU cluster. This work only presents the Black-Scholes model and does not evaluate the Greeks. Work of [12] addresses a FPGA framework to solve option pricing formulas on 4

49 FPGA via dierent methodologies: Monte-Carlo, nite dierences, quadrature method and binomial trees. As case-studies European and American options are considered. The Monte-Carlo simulation European options on FPGA is 41x times faster than on 8 core Intel Xeon CPU processor. This work does not include the module for the risk sensitivities of option pricing models. Studies presented in [11] explore a Quasi Monte-Carlo method for option pricing using Brownian paths. The performance results in 50x speed-up results on FPGA (Altera Stratix III EP3SE260-3) over an Intel Xeon 3 GHz CPU. This work does not support more complicated option pricing models nor the Greeks' calculation for hedging. This paper proposes a more ecient Greeks' calculation using automatic differentiation algorithms on dataow architectures supported by an FPGA card. The Adjoint (AD) method reduces overall computational eort of the Greeks and improves the accuracy compared to the pathwise, nite and likelihood methods used in commercial products. Further, FPGA greatly contributes to performance improvement (by three orders of magnitude vs. sequential version using the Adjoint). Overall, combination of the Adjoint with FPGA improves performance by four orders of magnitude when compared to pathwise, nite dierences and likelihood-based implementations on sequential machines. 3. Denitions 3.1. Option Pricing and the Greeks Options are contracts that allow trading underlying assets at the future time expiration time for the specied price (known as the strike price). Therefore, to price options, we forecast the future commodity price to see how it diers from the strike price. Option pricing models take several factors into account that aect the future commodity price [7]: current commodity; volatility of the commodity price; time to expiration; interest rate. 5

50 Black-Scholes Model Black-Scholes is the simplest model describing dynamics of the commodity price is expressed as a stochastic process (its changes over time are dependent on the probability). According, to this model, the commodity price holds the following stochastic dierential equation [7]: ds(t) = αs(t)dt + σs(t)dw (t) (1) Where α is the expected rate of return σ is a volatility and W (t) is a Wiener process (Brownian motion â a stochastic process whose increments over time have normal distribution and are independent of the previous process evolution) Heston Model The Heston model is an extension of Black-Scholes and proposes the various volatility factor over time (σ is a stochastic process). This model satises the below stochastic dierential equation [5]: ds t+1 = µs t dt + σ t S t Wt 1. (2) The non-constant volatility is expressed as a mean reverting stochastic process of the form [2]: dσ t+1 = κ (θ σ t ) dt + ξ σ t dwt 2. (3) where κ is a mean reversion of volatility, θ is the long-term variance, ξ denotes the volatility of volatility and σ 0 is the initial volatility, dwt 1 and dwt 2 are correlated random variables with normal distribution according to the factor ρ Discretization schemes The above option pricing models express the dynamics by continuous stochastic process. Numerical methods only operate on discrete models, hence, a continuous process must be transformed to a discrete approximated form. To approximate the solution of, we use Euler-Maruyama [13] for the Black-Scholes 6

51 formula: S t+dt = S t + αs(t)dt + σs(t) dtz (4) The discrete form of the Heston model is as follows: S t+1 = S t e (r 1 2 σt)dt+ σ tdt Z 1 (5) σ t+1 = σ t + κ (θ σ t )dt + ξ σ t dtz 2 (6) Monte-Carlo simulation Monte-Carlo simulation is the most ecient method to determine the results of multidimensional stochastic dierential functions that are too complicated to solve in an analytical way. Its computational eort increases approximately linearly with the number of paths, when the analytical form solution tends to increase exponentially [7]. MC simulation is based on generating many random dierent scenarios (paths) and evaluation of the expected value. This converges to the correct result when the number of paths increases. For option pricing models, the MC approach assumes that each scenario is a sample payo evaluated and discounted at the interest rate: C i = e rt max(s T K, 0) (7) where C i is the payo the option price along the i-th path S T is a commodity price at time T according to the i-th scenario, K denotes the strike price (the previously contracted price at which the commodity is traded at time T ). The expected value of the option price is the average of all discounted payos (M denotes the number of dierent payo scenarios) as below [6]: M i=0 v M = E(Φ(C i )) Φ(C i) (8) M Dierentiating the above with respect to the input, we must take into account all sample paths [6]: dv M dθ = E(dΦ(C i) ) dθ M i=0 dφ(c i) dθ M (9) 7

52 The Greek Recursive equation ds t+1 df t (S t ) ds t ds 0 ds t ds 0 ds t+1 df t (S t ) ds t dr ds t dr + df t (r) dr ds t+1 df t (S t ) ds t + df t (σ t ) σ t dσ 0 ds t dσ t dσ t σ 0 ds t+1 df t (σ t )) dσ t dκ dσ t dκ + df t (S t ) ds t ds t dκ Table 1: Recursive formulas for the Greeks of the underlying commodity 130 These known as the Greeks measure the impact of model parameters on the option price and are fundamental in hedging The Greeks For risk management and hedging investment portfolios, traders evaluate the Greeks to measure how the model parameters aect future commodity price and, the option price [7]. The impact on the option value is evaluated to quantify the dierent aspects of risk. When risk is acceptable, no adjustment is made to the investment portfolio. If it is unacceptable, an appropriate position for either the underlying asset or option contract is taken. The underlying concept of the Greeks' calculation combines symbolic dierentiation and the Adjoint. Considering the rst-order Greeks, the recursive formulas (along each scenario via Monte-Carlo simulation) must be derived. S t+1 = F t (S t, µ, σ t ) = S t e (r 1 2 σt)dt+ σ tdt Z 1 (10) σ t+1 = G t (κ, θ, ξ, σ t ) = σ t + κ (θ σ t )dt + ξ σ t dtz 2 (11) 145 Dierentiating the above formulas with respect to the inputs we have: As can be observed, the Greeks at time t+1 are dependent on the previously evaluated Greeks at time t and the partial derivatives of G and F. Based on this recursive dependencies of underlying commodity price and volatility, the Adjoint can be utilized to calculate the partial derivatives of F and G. Further, these results are used to update the nal rst-order Greeks. 8

53 Evaluation Pathwise Adjoint = = 20 Table 2: Number of arithmetic operations required to evaluate the gradient of the Black- Scholes model 3.2. Automatic Dierentiation Automatic Dierentiation (AD) is an algorithmic approach to precisely and eciently computed derivatives of a composite function [14]. The underlying concept of AD is that each composite function consists of elementary (intrinsic) operators as addition or multiplication. These arithmetic functions are processed in a sequence (chain-rule) to calculate the nal value of the composite function. AD has two modes: Forward known as Pathwise and Reverse (the Adjoint) [8]. To explain AD, consider a single time-step calculation of the Black-Scholes model from time t to t + 1. S t+1 = S t (1 + r dt + σ dt Z) (12) By applying elementary dierentiation routines to these formulas (Table 1, Column 1) and dierentiating with the forward order, the rst-order derivative with respect to the previous stock price (the Greek delta) can be calculated as follows (Table 1, Column 2). The Adjoint method is based on reverse processing starting from the nal value of the composite function ( Table 1, Column 3). This approach needs the function values and all intermediate results to be stored. Supposing we evaluate the nal function value, we begin dierentiation of the nal function df 7 with respect to S t. By going back and dierentiating f 7 with respect to S t and f 6 parameters of f 7 in a reverse manner we calculate partial derivatives. Analogously, further reverse processing of the chain-rule gives the rst-order derivatives of f 6 with respect to f 2 and f 5. As can be noted, this techniques requires one iteration to obtain the gradient. Table 2 gives the number of necessary arithmetic operations for the Pathwise and Adjoint methods (including the cost of function evaluation). 9

54 f 1 = r dt df 1 = 0 f 7 = df 7 df 7 = 1 f 2 = 1 + f 1 df 2 = df 1 f 6 = df 7 df 6 f 7 = S t f 7 f 3 = dt df 3 = 0 S t = df 7 f ds 7 = t f 4 = σ f 3 df 4 = dσ f 3 + σ df 3 f 2 = f 6 f 7 df 6 f df 6 = 2 f 6 f 5 = f 4 Z df 5 = df 4 Z f 5 = df 6 f df 6 = 5 f 6 f 6 = f 2 + f 5 df 6 = df 2 + df 5 f 4 = df 5 f df 5 = 4 Z f 5 f 7 = S t f 6 df 7 = S t df 6 Z = df 5 dz f 5 = f 5 f 3 = df 4 df 3 f 4 = σ f 4 σ = df 4 dσ f 4 = f 3 f 4 dt = df 3 ddt f 3 = 1 2 dt f 3 f 1 = df 2 df 1 f 2 = f 2 dt+ = df 1 ddt f 1 = r f 1 r = df 1 dr f 1 = dt f 1 Figure 1: Chain-rule of Black-Scholes (Pathwise and Adjoint methods 3.3. Maxeler FPGA technology Introduction. Maxeler technology is a computational framework supporting processing on FPGA cards [9]. This proposes an alternative approach to parallelism known as multi-scale dataow processing. In this paradigm, the application is considered as a dataow graph that consists of elementary operands processed from the independent nodes through intermediate nodes ending up at the nal nodes. Unlike a traditional CPU, FPGA do not perform load/transfer operations to/from memory, rather every operation and its result is stored in a node. The dataow graph is computed by a recongurable DataFlow Engine (DFE) consisting of thousands cores [9]. Each node is processed by a single dataow core. Maxeler multi-scale (pipeline) extensions allow processing arrays/vectors of nodes to optimally utilize FPGA capacity and its transistors. The pipelining strategy supports loop level parallelism of vectors with a spatial and pipelined processing. Flexible denition of primitive data types as oating-point or xed- 10

55 number types [9] is supported. The oating-point data can be dened by using various range of bits for exponent and mantissa (8 bits for mantissa and 24 exponent correspond to the oat type, 11 bits for mantissa and 53 for the exponent represent double numbers). These bits are further mapped to CPU application written in C. Fixed-point types are specied by the number of bits to represent integer numbers. Development. Accelerated Maxeler applications consist of three components: Kernel describes computation ow structurally by unidirectional acylic graphs. These graphs consist of several types of node, such as arithmetic operations, values, stream osets, multilplexers (condition instructions, counters, input/output operations). The kernel graphs are directly mapped to hardware and computed by dataow engines. 2. Manager conguration interconnects CPU via PCI (Peripheral Component Interconnect) Express bus to FPGA card. This includes input output stream information, memory accessing. The Manager usually calls the Kernel instantiation. 3. Host application the CPU application that calls the FPGA conguration and transfers data from RAM to FPGA memory. The host application is also responsible for data streams between CPU and FPGA. This overall application is managed by the Maxeler Operating System. Additionally, multiple dataow engines are connected via MaxRing to linearly scale the problem with the number of available DFEs [9]. 4. FPGA implementations for risk sensitivities calculation Overview Dataow implementations consist of a host application written in C++ and specied Maxeler applications (Kernel and Manager congurations) written in the MaxJ language [9]. Further, a separate sequential CPU version has been implemented for benchmarking purposes. 11

56 4.2. Architecture The underlying idea of the Kernel function is a denition of single scenario evolution from time t to time t+1. This part is translated and transformed to a DFE graph and further replicated with the number of pipelines and clocks. The computation of a scenario from t to t+1 requires multiple clock ticks to produce the intermediate output at (t+1) step. This number of clock ticks is termed the pipeline depth. The pipeline depth needs to be set as an oset in order to refer to the next/previous data in a stream. The algorithm 1 shows the concepts of the kernel implementation for the Black-Scholes model: 1. Declare scalars (volatility, dt, rate, numpaths, numsteps, Strike) and streamed outputs (payo, dstgreek, drategreek, dvolgreek); 2. The object Control counts every elements of input/output stream and manages its processing (splitting data tape in terms time evolution and dierent paths); 3. The random numbers with normal distribution are generated; for this purpose, the Mersenne Twister algorithm with the Box-Muller transform is implemented as a separate MaxJ class. 4. Instances of the DFEVector are created to store the last values calculated during the previous pipeline iteration; 5. Depending on the number of paths and steps, the previously evaluated values are initialized to 0 (if we start the rst scenario) or the stored values (in the process of a single MC scenario). 6. Having set previous data values, we start evaluation and dierentiation from time t to time t+1 by calling the functions: nextstockprice and AdjointStockPrice. These routines perform computations described in 1 and store their intermediate results; 7. Calculate the payo (if the nal step of the MC scenario) and update the Greeks using derivations 1. 12

57 The above process takes a xed number of ticks known as pipelining or latency depth (bsdelooplength). This number must be taken into account to refer to the previously computed partial results (the Greeks and the payo). 9. The stored objects are updated to refer to the previously computed data at the next clock tick (negative stream oset equal to -bsdelooplength). Following this the data can be send to the host application. The additions and dierences between the Black-Scholes and Heston models are: additional functions for calculation and dierentiation of volatility function; computation of additional recursive volatility sensitivities; and more intermediate variables with Black-Scholes as the model is more complicated. The generated DFE graph for Black-Scholes is presented in 4. 13

58 input : volatility, dt, rate, St, numpaths, numsteps, Strike output: payo, dstgreek, drategreek, dvolgreek kernelfunction( // Fetching the input data Control control DFEVector Zvar mersennetwister () boxmuller (Zvar) DFEVector carriedst newinstance () DFEVector carrieddstgreek DFEVector carriedaccst newinstance () DFEVector previousst init (control, carriedst) DFEVector previousdstgreek init (control, carrieddstgreek) DFEVector previousaccst init (control, carriedaccst) // Evaluation and differnentiation of the chain-rule inputs.st previousst DFECVector newst nextstockprice (inputs, chainrule, Zvar) AdjointStockPrice (inputs, chainrule, derchainrule, greeks) DFEVector newprice payoff (control, newst, Strike) DFEVector newgreekst greeks.st * previousdstgreek DFEVector newaccst previousaccst + (control.lastpoint? carriedaccst : 0) carriedst stream.oset(newst, -bsdelooplength) carrieddstgreek stream.oset(newgreekst, -bsdelooplength) carriedaccst stream.oset(newaccst, -bsdelooplength) // Transfer output data ) Algorithm 1: FPGA Kernel implementation 14

59 Experiments Sequential and dataow FPGA versions of the MC simulations for Black- Scholes and Heston have been tested with a xed number of steps 100 and a various range of scenarios from to The benchmarks have been performed on the following hardware congurations: 260 CPU: Intel Xeon X Ghz Maxeler FPGA Xilinx Virtex-5 (SX240T) Sequential and dataow versions use implemented Mersenne-Twister random generators with the Box-Muller transform Black-Scholes For the Black-Scholes implementation, the FPGA logic usage is around 62 %. DSP (digital signal processing) blocks exploit 68 %, LUTs (look-up tables) 51.01%, Primary FFs (ip-ops) 56.94% of available resources. Figure 4 presents a Kernel dataow graph generated after the compilation process on an FPGA. Table 3 presents CPU timings and performance results of the Black- Scholes Greeks with dierent numbers of pipes. In the case of the 1-pipe implementation, speed-up is around 25x vs. CPU; speed-up rises to 400x (4 pipes) and 1223x (8 pipes) for the maximum number of paths (200000) (2). 15

60 Legend FPGA (1 pipes) FPGA (4 pipes) FPGA (8 pipes) 1000 Speed-up Number of paths Figure 2: Black-Scholes Greeks Speedup FPGA vs. CPU Legend FPGA (2 pipes) FPGA (4 pipes) Speed-up Number of paths Figure 3: Heston Greeks Speedup FPGA vs. CPU 16

61 5.2. Heston results 275 The logic utilization rises 73%. DSP blocks utilize 66%, LUTs 59.07%, Primary FFs 67.70% of available resources.. The speed-up results for the Heston model Greeks using 2 and 4 pipes are also given in 4. For 2 pipes implementation the speed-up remains at 200x; for 4 pipes, performance reaches a peak of 585x for the maximum number of paths The plot 3 illustrates the performance results. 17

62 Number CPU FPGA FPGA FPGA of time (1 (4 (8 paths (sec- pipes) pipes) pipes) onds) x 153x 230x x 303x 698x x 353x 998x x 378x 1152x x 389x 1223x Table 3: First-order Greeks CPU timings and the speedup of the Black-Scholes model on FPGA vs. CPU Number CPU FPGA FPGA of paths time (2 (4 (sec- pipes) pipes) onds) x 239x x 456x x 530x x 565x x 585x Table 4: First-order Greeks CPU timings and the speedup of the Heston model on FPGA vs. CPU 18

63 Figure 4: DFE graph for Black-Scholes model 19