Back- and Side Testing of Price Simulation Models

Transcription

1 Back- and Side Testing of Price Simulation Models Universität Duisburg Essen - Seminarreihe Energy & Finance 23. Juni 2010 Henrik Specht, Vattenfall Europe AG

2 The starting point Question: How do I know my price simulation model is a good one? Expert answer: When only a few trajectories look bad! growing amount of research/publication on price modeling hardly literature on price model testing Back- and Side Testing of Price Simulation Models 2

3 Content Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary Back- and Side Testing of Price Simulation Models 3

4 Content Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary Back- and Side Testing of Price Simulation Models 4

5 Why testing price simulations? market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model normal distributed relative price changes µ,σ Black76 is a nice example for an integral connection of price model and valuation model How to test the underlying price model? C( t) = F N( d where d d 1 2 and 1 N(x) = σ 2π ) Se ( t µ ) r( T t) F 1 2 ln( ) + ( r + σ )( T t) = S 2 σ T t = d σ T t 1 1 x e 2σ 2 2 N( d 2 ) Back- and Side Testing of Price Simulation Models 5

6 Why testing price simulations? no possibility to test the underlying price model directly only indirect test via pricing of options and comparing to market if model option premium does not match observed market premium is the underlying price model wrong? is the valuation concept wrong? is the market wrong? Back- and Side Testing of Price Simulation Models 6

7 Why testing price simulations? when we deal with closed form models: all we can do is to check whether the parameters we use are numerically O.K. e.g. non linearity of maximum likelihood estimations is a proper parameter a guarantee for a good model performance? models do not always have stand alone parameters for all of their behavioral features. sometimes model behavior can not be matched in a one-toone sense to a certain parameter. Rather a combination of parameters shapes the simulations pure inspection of parameters does not deliver an inspection of model performance/behavior Back- and Side Testing of Price Simulation Models 7

8 Why testing price simulations? by parameter inspection we first assume the model is fitting to reality secondly guarantee the best parameters leverage of risk-factor inputs is usually larger than leverage of detailed pay-off function features (e.g. Power Plant valuation) full control over price- (risk factor) model is required to be able to judge model behavior, direct observation of model output (trajectories) is required Back- and Side Testing of Price Simulation Models 8

9 Why testing price simulations? Therefore best solution: isolate price- (risk factor-) model conduct explicit simulation runs requires full understanding of underlying price model assumptions Trivial in the case of well known classical GBM as used with e.g. Black Back- and Side Testing of Price Simulation Models 9

10 Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary Back- and Side Testing of Price Simulation Models 10

11 Setting up a simple simulation model EXAMPLE assuming we try to calculate Earnings-at-Risk for a portfolio of forwards and a base-peak spread option we need one common price model for forwards and spread option (joint Base-Peak model needed) otherwise risk aggregation would not be possible Back- and Side Testing of Price Simulation Models 11

12 Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factor) valuation- / risk - Model 40 cal08peak cal08 Base cal08 Peak cal08offpeak Back- and Side Testing of Price Simulation Models 12

13 Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 40 cal08peak cal08offpeak normal distributed price returns linear correlation Back- and Side Testing of Price Simulation Models 13

14 Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 40 cal08peak σ = 13.1% cal08offpeak ρ = σ = 16.5% Back- and Side Testing of Price Simulation Models 14

15 Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 95 now true behavior of price-model becomes observable testing market history 2006 Simulated Back- and Side Testing of Price Simulation Models 15

16 Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary Back- and Side Testing of Price Simulation Models 16

17 A concept for testing: general idea market data simulation data transformation transformation result compare results identical mathematical transformation for historical- and simulation data Back- and Side Testing of Price Simulation Models 17

18 A concept for testing: general idea There is theoretically a wide range of different transformations possible. The main work is to decide for the enlightening transformation functions transformations on one dimensional information examples mean price std.dev skewness kurtosis (partial) autocorrelation transformations on two-dimensional information price spread price ratios volatilities of spreads/ratios correlations of prices inner commodity correlations intra commodity correlations correlations on returns transformations on multi-dimensional information sum of squared correlation matrix entries copula parameters Back- and Side Testing of Price Simulation Models 18

19 A concept for testing: general idea sum of squared correlation matrix entries example does not tell us stand alone anything does not tell us whether simulation and history behave the same but tells us when simulation and history behave different The concept has the power to reject models The concept has not the power to validate model but when using enough different transformations? Back- and Side Testing of Price Simulation Models 19

20 A concept for testing: general idea σ hist example 16.6% example σ i sim 15% 16% 17% 18% 14.4% -18.3% a transformation procedure we could use: the classical volatility Back- and Side Testing of Price Simulation Models 20

21 A concept for testing: general idea The volatility (standard deviation of returns) is one possible but arbitrary transformation that condenses the historically observed price behavior into one number condenses the simulated data into n numbers enables comparison might not be interpretable as volatility example (e.g in a switching-regime world we do not have one volatility but at least two volatilities plus a transition probability matrix) behind the scenes this transformation is independent from model assumptions like normal distribution, switching-regime, mean-reversion, Back- and Side Testing of Price Simulation Models 21

22 A concept for testing: general idea if price behavior is not normal and we apply standard deviation of returns as transformation function, we can not state the simulated volatility is x%, while the history showed y% (because std.dev as volatility measure is misleading here) but we can conclude whether the simulated standard deviations contain the historically observed standard deviations or not where in the distribution of std.dev. from the simulations we can find the historical standard deviation Back- and Side Testing of Price Simulation Models 22

23 A concept for testing: general idea but this approach gives not always clear signs: transformation measure Model is bad! historical behavior not reproduced Depends transformation measure simulation behavior not centered around historical measure but historical behavior reproduced O.K. Depends transformation measure simulation behavior centered around historical measure but was history typical or exceptional.? Back- and Side Testing of Price Simulation Models 23

24 A concept for testing: general idea if we can make available more transformation results from history, we could get more information to compare against we need more measured categories (transformations) and more results (apply identical transformations more often) We jump back to our example market history 2006 Simulated Back- and Side Testing of Price Simulation Models 24

25 A concept for testing: example 65 [ /MWh] transform historical data into standard deviation of daily returns cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base Back- and Side Testing of Price Simulation Models 25

26 A concept for testing: example annualized standard deviations of daily price returns cal05 in 2004 cal04 in 2003 cal06 in 2005 cal07 in 2006 our simple example model seems to not to be flexible enough simulated for cal08 in % 10% 12% 14% 16% 18% 20% 22% 24% Back- and Side Testing of Price Simulation Models 26

27 A concept for testing: example time horizon of historical market data usually has impact on transformation results therefore possibility to increase number of results for comparison further by using moving window transformations /MWh rolling window σ cal08 (T-t) σ different time horizons become comparable σ cal07 (T-t) t T cal07 T cal08 trading day time to maturity Back- and Side Testing of Price Simulation Models 27

28 A concept for testing: example std.dev of daily returns one year frequency! cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base time to maturity Back- and Side Testing of Price Simulation Models 28

29 A concept for testing: example [ /MWh] cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base Back- and Side Testing of Price Simulation Models 29

30 A concept for testing: example moving window transformation results market dynamics become visible enables better comparison of market and model std.dev of daily returns profile cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base sim# time to maturity Back- and Side Testing of Price Simulation Models 30

31 A concept for testing: example moving window transformation results market dynamics become visible enables better comparison of market and model 40 annualized volatility [%] range annualized volatility [%] 250 time to maturity time to maturity Back- and Side Testing of Price Simulation Models 31

32 Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary Back- and Side Testing of Price Simulation Models 32

33 Summary 1/2 How do I know my price simulation model is a good one? testing of closed form (analytical) price-models only possible on the level of model parameter accuracy no direct testing of model output possible, only testing on the level of price-model application (e.g. VaR numbers, option prices etc.) thereby one has to rely that the model itself is appropriate generation of explicit simulations required for proper testing Back- and Side Testing of Price Simulation Models 33

34 Summary 2/2 The testing concept find enlightening transformation measures that can be used to compare historical data with simulations no need to have measures that are directly related to model philosophy or model parameters apply transformation measure individually in a rolling window approach on historical data ( one historically observed trajectory) apply transformation measure individually in a rolling window approach to each simulated price trajectory ( multiple trajectories) compare over time-to-maturity compare history with spreading of simulated values Back- and Side Testing of Price Simulation Models 34

35 Contact Henrik Specht Vattenfall AB Group Risk Management Models & Methodology Office: Chausseestrasse 23 D Berlin henrik.specht@vattenfall.de Back- and Side Testing of Price Simulation Models 35