Back- and Side Testing of Price Simulation Models Universität Duisburg Essen - Seminarreihe Energy & Finance 23. Juni 2010 Henrik Specht, Vattenfall Europe AG
The starting point Question: How do I know my price simulation model is a good one? Expert answer: When only a few trajectories look bad! growing amount of research/publication on price modeling hardly literature on price model testing 23.06.2010 Back- and Side Testing of Price Simulation Models 2
Content Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary 23.06.2010 Back- and Side Testing of Price Simulation Models 3
Content Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary 23.06.2010 Back- and Side Testing of Price Simulation Models 4
Why testing price simulations? market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model normal distributed relative price changes µ,σ Black76 is a nice example for an integral connection of price model and valuation model How to test the underlying price model? C( t) = F N( d where d d 1 2 and 1 N(x) = σ 2π ) Se ( t µ ) r( T t) F 1 2 ln( ) + ( r + σ )( T t) = S 2 σ T t = d σ T t 1 1 x e 2σ 2 2 N( d 2 ) 23.06.2010 Back- and Side Testing of Price Simulation Models 5
Why testing price simulations? no possibility to test the underlying price model directly only indirect test via pricing of options and comparing to market if model option premium does not match observed market premium is the underlying price model wrong? is the valuation concept wrong? is the market wrong? 23.06.2010 Back- and Side Testing of Price Simulation Models 6
Why testing price simulations? when we deal with closed form models: all we can do is to check whether the parameters we use are numerically O.K. e.g. non linearity of maximum likelihood estimations is a proper parameter a guarantee for a good model performance? models do not always have stand alone parameters for all of their behavioral features. sometimes model behavior can not be matched in a one-toone sense to a certain parameter. Rather a combination of parameters shapes the simulations pure inspection of parameters does not deliver an inspection of model performance/behavior 23.06.2010 Back- and Side Testing of Price Simulation Models 7
Why testing price simulations? by parameter inspection we first assume the model is fitting to reality secondly guarantee the best parameters leverage of risk-factor inputs is usually larger than leverage of detailed pay-off function features (e.g. Power Plant valuation) full control over price- (risk factor) model is required to be able to judge model behavior, direct observation of model output (trajectories) is required 23.06.2010 Back- and Side Testing of Price Simulation Models 8
Why testing price simulations? Therefore best solution: isolate price- (risk factor-) model conduct explicit simulation runs requires full understanding of underlying price model assumptions Trivial in the case of well known classical GBM as used with e.g. Black76. 23.06.2010 Back- and Side Testing of Price Simulation Models 9
Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary 23.06.2010 Back- and Side Testing of Price Simulation Models 10
Setting up a simple simulation model EXAMPLE assuming we try to calculate Earnings-at-Risk for a portfolio of forwards and a base-peak spread option we need one common price model for forwards and spread option (joint Base-Peak model needed) otherwise risk aggregation would not be possible 23.06.2010 Back- and Side Testing of Price Simulation Models 11
Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factor) valuation- / risk - Model 40 cal08peak 90 85 80 75 70 65 60 55 50 45 40 03.01.06 03.02.06 03.03.06 03.04.06 cal08 Base 03.05.06 03.06.06 03.07.06 cal08 Peak 03.08.06 03.09.06 03.10.06 03.11.06 03.12.06 30 20 10 0-0.04-0.02 0 0.02 0.04 0.06 0.08 40 30 20 10 cal08offpeak 0-0.04-0.02 0 0.02 0.04 0.06 0.08 23.06.2010 Back- and Side Testing of Price Simulation Models 12
Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 40 cal08peak 30 20 10 0-0.04-0.02 0 0.02 0.04 0.06 0.08 40 30 cal08offpeak normal distributed price returns linear correlation 20 10 0-0.04-0.02 0 0.02 0.04 0.06 0.08 23.06.2010 Back- and Side Testing of Price Simulation Models 13
Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 40 cal08peak 30 20 σ = 13.1% 10 0-0.04-0.02 0 0.02 0.04 0.06 0.08 40 cal08offpeak ρ = 0.88 30 20 σ = 16.5% 10 0-0.04-0.02 0 0.02 0.04 0.06 0.08 23.06.2010 Back- and Side Testing of Price Simulation Models 14
Setting up a simple simulation model The general modeling concept market data (history of quotes) model definition parameter estimation parameters generate simulations price simulations (risk factors) valuation- / risk - Model 95 now true behavior of price-model becomes observable testing 90 85 80 75 70 65 60 market history 2006 Simulated 2007 55 50 45 0 50 100 150 200 250 300 350 400 450 500 23.06.2010 Back- and Side Testing of Price Simulation Models 15
Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary 23.06.2010 Back- and Side Testing of Price Simulation Models 16
A concept for testing: general idea market data simulation data transformation transformation result compare results identical mathematical transformation for historical- and simulation data 23.06.2010 Back- and Side Testing of Price Simulation Models 17
A concept for testing: general idea There is theoretically a wide range of different transformations possible. The main work is to decide for the enlightening transformation functions transformations on one dimensional information examples mean price std.dev skewness kurtosis (partial) autocorrelation transformations on two-dimensional information price spread price ratios volatilities of spreads/ratios correlations of prices inner commodity correlations intra commodity correlations correlations on returns transformations on multi-dimensional information sum of squared correlation matrix entries copula parameters 23.06.2010 Back- and Side Testing of Price Simulation Models 18
A concept for testing: general idea sum of squared correlation matrix entries example does not tell us stand alone anything does not tell us whether simulation and history behave the same but tells us when simulation and history behave different The concept has the power to reject models The concept has not the power to validate model but when using enough different transformations? 23.06.2010 Back- and Side Testing of Price Simulation Models 19
A concept for testing: general idea 62 60 58 56 54 52 50 σ hist example 16.6% 48 46 0 50 100 150 200 250 80 75 70 65 60 55 50 45 example σ i sim 15% 16% 17% 18% 14.4% -18.3% 40 35 30 0 50 100 150 200 250 a transformation procedure we could use: the classical volatility 23.06.2010 Back- and Side Testing of Price Simulation Models 20
A concept for testing: general idea The volatility (standard deviation of returns) is one possible but arbitrary transformation that condenses the historically observed price behavior into one number condenses the simulated data into n numbers enables comparison might not be interpretable as volatility example (e.g in a switching-regime world we do not have one volatility but at least two volatilities plus a transition probability matrix) behind the scenes this transformation is independent from model assumptions like normal distribution, switching-regime, mean-reversion, 23.06.2010 Back- and Side Testing of Price Simulation Models 21
A concept for testing: general idea if price behavior is not normal and we apply standard deviation of returns as transformation function, we can not state the simulated volatility is x%, while the history showed y% (because std.dev as volatility measure is misleading here) but we can conclude whether the simulated standard deviations contain the historically observed standard deviations or not where in the distribution of std.dev. from the simulations we can find the historical standard deviation 23.06.2010 Back- and Side Testing of Price Simulation Models 22
A concept for testing: general idea but this approach gives not always clear signs: transformation measure Model is bad! historical behavior not reproduced Depends transformation measure simulation behavior not centered around historical measure but historical behavior reproduced O.K. Depends transformation measure simulation behavior centered around historical measure but was history typical or exceptional.? 23.06.2010 Back- and Side Testing of Price Simulation Models 23
A concept for testing: general idea if we can make available more transformation results from history, we could get more information to compare against we need more measured categories (transformations) and more results (apply identical transformations more often) 95 90 We jump back to our example 85 80 75 70 65 60 market history 2006 Simulated 2007 55 50 45 0 50 100 150 200 250 300 350 400 450 500 23.06.2010 Back- and Side Testing of Price Simulation Models 24
A concept for testing: example 65 [ /MWh] 60 55 50 45 40 35 30 25 transform historical data into standard deviation of daily returns cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base 20 01.01.2003 01.01.2004 01.01.2005 01.01.2006 01.01.2007 23.06.2010 Back- and Side Testing of Price Simulation Models 25
A concept for testing: example annualized standard deviations of daily price returns cal05 in 2004 cal04 in 2003 cal06 in 2005 cal07 in 2006 our simple example model seems to not to be flexible enough simulated for cal08 in 2007 8% 10% 12% 14% 16% 18% 20% 22% 24% 23.06.2010 Back- and Side Testing of Price Simulation Models 26
A concept for testing: example time horizon of historical market data usually has impact on transformation results therefore possibility to increase number of results for comparison further by using moving window transformations /MWh rolling window σ cal08 (T-t) σ different time horizons become comparable σ cal07 (T-t) t T cal07 T cal08 trading day time to maturity 23.06.2010 Back- and Side Testing of Price Simulation Models 27
A concept for testing: example 0.025 std.dev of daily returns 0.02 0.015 0.01 0.005 one year frequency! cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base 0 1 113 225 337 449 561 673 785 897 1009 1121 1233 1345 1457 time to maturity 23.06.2010 Back- and Side Testing of Price Simulation Models 28
A concept for testing: example 65 60 55 [ /MWh] 50 45 40 35 30 25 20 01.01.2003 01.01.2004 01.01.2005 01.01.2006 01.01.2007 cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base 23.06.2010 Back- and Side Testing of Price Simulation Models 29
A concept for testing: example moving window transformation results market dynamics become visible enables better comparison of market and model 0.025 std.dev of daily returns 0.02 0.015 0.01 0.005 profile cal03base cal04base cal05base cal06base cal07base cal08base cal09base cal10base cal11base cal12base cal13base sim#3 0 1 113 225 337 449 561 673 785 897 1009 1121 1233 1345 1457 time to maturity 23.06.2010 Back- and Side Testing of Price Simulation Models 30
A concept for testing: example moving window transformation results market dynamics become visible enables better comparison of market and model 40 annualized volatility [%] 30 20 range annualized volatility [%] 250 time to maturity 10 0 250 500 750 1000 1250 time to maturity 23.06.2010 Back- and Side Testing of Price Simulation Models 31
Why testing price simulations? - some general thoughts - Setting up a simple simulation model - prerequisite for having an example - A concept for testing - general idea - example Summary 23.06.2010 Back- and Side Testing of Price Simulation Models 32
Summary 1/2 How do I know my price simulation model is a good one? testing of closed form (analytical) price-models only possible on the level of model parameter accuracy no direct testing of model output possible, only testing on the level of price-model application (e.g. VaR numbers, option prices etc.) thereby one has to rely that the model itself is appropriate generation of explicit simulations required for proper testing 23.06.2010 Back- and Side Testing of Price Simulation Models 33
Summary 2/2 The testing concept find enlightening transformation measures that can be used to compare historical data with simulations no need to have measures that are directly related to model philosophy or model parameters apply transformation measure individually in a rolling window approach on historical data ( one historically observed trajectory) apply transformation measure individually in a rolling window approach to each simulated price trajectory ( multiple trajectories) compare over time-to-maturity compare history with spreading of simulated values 23.06.2010 Back- and Side Testing of Price Simulation Models 34
Contact Henrik Specht Vattenfall AB Group Risk Management Models & Methodology Office: Chausseestrasse 23 D 10115 Berlin 030 81 82 2572 henrik.specht@vattenfall.de 23.06.2010 Back- and Side Testing of Price Simulation Models 35