Welcome to Redefining Perspectives

Transcription

1 Welcome to Redefining Perspectives November 2012

2 Capital Markets Risk Management And Hadoop Kevin Samborn and Nitin Agrawal 2

3 Agenda Risk Management Hadoop Monte Carlo VaR Implementation Q & A 4

4 Risk Management 5

5 What is Risk Management Risk is a tool the goal is to optimize and understand risk o Too much risk is locally and systemically dangerous o Too little risk means the firm may be leaving profit on the table Portfolio exposure o Modern portfolios contain many different types of assets o Simple instruments, Complex instruments and derivatives Many types of risk measures o Defined scenario-based stress testing o Value at Risk (VaR) o Sensitivities Key is valuation under different scenarios VaR is used in banking regulations, margin calculations and risk management 6

6 Value at Risk (VaR) VaR is a statistical measure of risk expressed as amount of loss given probability %. E.g. 97.5% chance that the firm will not lose more than 1mill USD over the next 5 days Computing VaR is a challenging data sourcing and compute intensive process VaR calculation: o Generate statistical scenarios of market behavior o Revalue the portfolio for each scenario, compare returns to today s value o Sort results and select the desired percentage return: VALUE AT RISK Different VaR techniques: o Parametric analytic approximation o Historical captures real (historical) market dynamics o Monte Carlo many scenarios, depends on statistical distributions 7

7 VaR Graphically Source: An Introduction To Value at Risk (VAR), Investopedia, May

8 Complexities For modern financial firms, VaR is complex. Calculation requirements: o Different types of assets require different valuation models Risk-based approach Full revaluation o With large numbers of scenarios, many thousands of calculations are required o Monte Carlo simulations require significant calibration, depending on large historical data Many different reporting dimensions o VaR is not additive across dimensions. Product/asset class, Currency o Portfolio including what-if and intraday activity Intraday market changes requiring new simulations Incremental VaR how does a single (new) trade contribute to the total 9

9 Backtesting VaR 10

10 11

11 Hadoop Core Data stored with REDUNDANCY on a Distributed File System Abstracts H/W FAILURES delivering a highly-available service on COMMODITY H/W SCALES-UP from single to thousands of nodes Data stored WITHOUT A SCHEMA Tuned for SEQUENTIAL DATA ACCESS Provides an EASY ABSTRACTION for processing large data sets Infrastructure for PARALLEL DATA PROCESSING across huge Commodity cluster Infrastructure for TASK and LOAD MANAGEMENT Framework achieves DATA-PROCESS LOCALITY Makes two critical assumptions though: Data doesn t need to be updated Data doesn t need to be accessed randomly 12

12 A Simple Map Reduce Job Problem Statement: From historical price data, create frequency distribution of 1-day %age change for various stocks Stock Date Open Close BP 23-Nov NXT 23-Nov MKS 23-Nov BP 22-Nov NXT 22-Nov MKS 22-Nov BP 21-Nov NXT 21-Nov MKS 21-Nov BP 20-Nov NXT 20-Nov MKS 20-Nov BP 19-Nov NXT 19-Nov MKS 19-Nov BP 16-Nov NXT 16-Nov MKS 16-Nov BP 15-Nov NXT 15-Nov MKS 15-Nov Map 1 Map 2 Map M S O R T / S H U F F L E Reduce 1 Reduce 2 Reduce 3 Reduce N BP 1, 33 BP 2, 64 NXT 81, 2 NXT -20, 5 Output3 Output N public void reduce(text key, Iterable<IntWritable> values, public Context void context) map(longwritable throws IOException, key, Text value, InterruptedException Context { context) Map<Integer, throws Long> IOException, freqdist InterruptedException = buildfreqdistribution(values); { Set<Integer> SecurityAttributes percentchanges sa = = freqdist.keyset(); for (Integer RecordsReadHelper.readAttribs(value.toString()); percentchange : percentchanges) { context.write(new context.write(new Text(sa.getTicker()), Text(key.toString() + " " + percentchange.tostring()), new new LongWritable(freqDist.get(percentChange))); IntWritable(sa.getPercentChange())); } } 13

13 Hadoop Ecosystem How/Where These Fit VISUALIZATION TOOLS USERS DATA WAREHOUSE Sqoop hiho Scribe Flume LOAD PROCESSING STORAGE Zoo Keeper HUE SUPPORT 14

14 Monte-Carlo VaR Implementation 15

15 Monte Carlo VaR 2 Steps IBM MSFT IBM.CO V 1 V 2 V 3 V 10,000 SIMULATION HLV 1 = ( A i V i ) 1 HLV 2 = ( A i V i ) 2 HLV 10k = ( A i V i ) 10k Aggregation AGGREGATION Aggregation Challenges Daily trade data could be massive Valuations are Compute intensive VaR is not a simple arithmetic sum across hierarchies 16

16 Simulation Step - MapReduce MAP - Read-through portfolio data - Emit (K,V) as (Underlyer,InstrumentDetails) e.g. (IBM, IBM.CO.DEC14.225) IBM MSFT IBM.CO V 1 V 2 V 3 SIMULATION REDUCE - For the Underlyer, perform 10k random walks in parallel - For each random walk output, simulate derivative prices - Emit 10k sets of simulated prices of the stock and associated derivatives i.e. IBM, [V 1, V 2,..V ] IBM.CO.DEC14.225, [V 1, V 2,..V ] Job job = new Job(getConf()); job.setjobname("randomvaluationgenerator"); SecurityAttributes stockattrib = (SecurityAttributes) iter.next(); job.setmapperclass(securityattributemapper.class); simpricesstock = getsimpricesforstock(stockattrib); job.setreducerclass(pricesimulationsreducer.class); writereduceroutput(stockattrib, simpricesstock, context); public bsmp = void new BlackScholesMertonPricingOption(); map(longwritable key, Text value, Context context) throws IOException, InterruptedException while (iter.hasnext()) { { SecurityAttributes sa secattribs = RecordsReadHelper.readAttribs(value.toString()); = iter.next(); writereduceroutput(secattribs,getsimpricesforoptions( context.write(new Text(sa.getUnderlyer()), sa); } simpricesstock, bsmp, secattribs), context); } 17

17 Aggregation Step MapReduce HLV 1 ( A HLV i i = 2 1 ( A i V i ) 2 Aggregation MAP - Read-through de-normalized portfolio data - Emit (K,V) as (Hierarchy-level, Position Details) US, [IBM, 225, ] US Tech, [IBM, 400, ] US Tech Eric, [IBM, 400, ] REDUCE For the hierarchy level (e.g. US ERIC), perform A i V i for each simulation and get simulated portfolio values - HLV i Sort HLV i, find 1%, 5% and 10% values and emit position and VaR data Map<String, Double> portfoliopositiondata = combineinputforpfpositiondata(rows); Map<String, Double[]> simulatedprices= protected void map(longwritable key, HoldingWritable value, Context context) loadsimulatedprices(portfoliopositiondata.keyset()); throws java.io.ioexception,interruptedexception { for(long SecurityAttributes i=0; i<no_of_simulations-1; sa = RecordsReadHelper.readAttribs(value.toString()); i++) { Set<String> simulatedpfvalues.add(getpfsimulatedvalue(i, hierarchylevels = sa.gethierarchylevels(); portfoliopositiondata, for (String hierarchylevel simulatedprices)); : hierarchylevels) } { Collections.sort(simulatedPFValues); context.write(new Text(hierarchyLevel), new Text(sa.getPositionDtls())); emitresults(portfoliopositiondata, simulatedpfvalues); } 18

18 DEMO RUN 21

19 Observations As expected, processing time of Map jobs increased marginally when input data volume was increased Process was IO-bound on Simulation s Reduce job as intermediate data emitted was huge Data replication factor needs to be chosen carefully MapReduce jobs should be designed such that Map/Reduce output is not huge 22

20 Questions? 23

21 Thank You! 24

22 Appendix 25

23 26

24 27

25 Let s build a Simple Map Reduce Job Problem Statement: Across a huge set of documents, we need to find all locations (i.e. document, page, line) for all words having more than 10 characters. D A T A N O D E 2 STORAGE D A T A N O D E 1 Store Map 28

26 29