Trade and Manage Wealth with Deep RL and Memory

Size: px

Start display at page:

Download "Trade and Manage Wealth with Deep RL and Memory"

Gary Curtis
5 years ago
Views:

1 Trade and Manage Wealth with Deep RL and Memory NVIDIA GTC 2018 March 26, 2018 Daniel Egloff, Founder, CEO, Head R&D

2 Problem Retail investor customer demands Manage portfolio more actively Get additional return from smart investment decisions Lack of products Market barriers Costs Current digital platforms focus on passively managed products Regulations limits access to hedge fund style active products Existing actively managed products charge high fees

3 Question Can modern AI technology replace a PM or trader?

4 Solution AI Agents Smart data-aware AI agents for active investment decisions AI powered alternative to smart beta ETFs AI supervised and automated trading strategies Smart returns Save costs Save time Extra return from smart data driven decisions Fully automated without human interaction Don t miss market opportunities Delegate work to smart agents

5 Market Validation 90% of the robo-advisors today are ETF-based. ETFs alone have run out of steam to fuel the next growth of robo advisors. Robo advisors need more than 6 years to make a profit of a customer, post acquisition. Source: Burnmark report April Intergeneration wealth transfer Investor behavior change Reduced margins Digital channels Robo advisors Growth of ETF market Growth of Smart Beta ETFs Growth of online brokers 12tn wealth transferring from 1920/30 to 1946/64 generation. Source: Burnmark report April Smart beta 30% growth in Source: EY EFT report 2017.

6 AI Foundations Several recent innovations in AI and Big Data Deep Reinforcement Learning Differentiable Neural Computer Large scale data streaming infrastructures from ecommerce

7 Classical Portfolio Construction Information bottleneck Signal based, not end-to-end Partial and staged data usage Many data sources cannot be integrated with current factor based portfolio construction Retrospective Design, fit, deploy, re-engineer offline Missing feedback link and on-line learning Difficult to account for nonlinear dependencies

8 Signal Based Market state S t Next Market state S t+1 Supervised learning - Error Forecasting system θ 1 Signals Trade rule system θ 2 Weights Trades P&L/Utility U(θ 1, θ 2 ) Information bottleneck Transaction costs

9 Reinforcement Learning Based Full information in portfolio weights and trades Feedback loop to improve on good decisions and avoid unsuccessful decisions Allows for more realistic modeling of intelligence of a successful PM or trader Much easier process

10 Reinforcement Learning Based Market state S t Next Market state S t+1 Trade system θ 1 Reinforcement learning U(θ 1 ) Weights/Trades P&L/Utility U(θ 1 ) Delay Transaction costs

11 AI Trading Agents PaaS to design, train, test, deploy and run agents 4 5 Initial Training 2 Frequency 1h Universe AAPL GOOG Objective Risk Strategy & Style AMZN WMT BAC JMP AI Agent Excess return Medium 1 4 Online Learning 5 2 I learn from Price News VIX 5 Batch architecture My performance Live LOB News Historic Online architecture 2 Training Data 3 Scenarios Stats Stats 5 5 Brokers & Exchanges Bull Stats Bear Stats Crash Stats Streaming architecture

12 Challenges and Insights

13 Reinforcement Learning Setup Learning a behavioral strategy which maximizes long term sum of rewards by a direct interaction with an unknown and uncertain environment Action Environment Reward State While not terminal do: Agent perceives state s t Agent performs action a t Agent receives reward r t Environment evolves to state s t+1 Agent

14 Environment RL - State Environment state What is the market state? Which data required Price data top of the book LOB L1, L2 LOB messages Secondary and non standard data Event data versus time clocked data How to combine agent and market state to environment state? Agent

15 Environment RL - Policy Agent policy specification What is the agent action? Continuous action for percentages of wealth Discrete units of lots to buy/sell Order implementation using market/limit orders Long only vs. long/short Long only vs. long/short? Long only agent do not face bankruptcy Short position can lead to bankruptcy Agent

16 Environment RL - Policy Agent Distributions on the simplex Commonly known distributions (Dirichlet, ) not appropriate Exploiting less known Hilbert space structure on (open) simplex leading to isometry to Euclidian space (Aitchison) Isometry Pull back normal distribution, Student-t, etc.

17 Environment RL - Interaction Interaction of agent and environment Market evolution, LOB resilience Temporal and permanent market impact Position change Market prices Agent positions State Order cancellations Partial fills Policy Target trades Market liquidity Market evolution and impact Executed trades Filled trades Market prices Agent positions New state Time Agent

18 Our 6 Main Challenges Sparse trading, learn to use cash and wait for opportunities Robustness of RL Scaling up RL training Handling high resolution event time series data Adapting agents to changing markets while not forgetting Explaining agent decisions and behavior

19 Sparse Trading Reward modelling, including realistic transaction cost modelling Adding risk to give cash a value Properly balance risk and reward Combining tree search and RL or option framework to learn to postpone trading

20 Robustness Reward modelling Very long history Looking at different scales of time series Training on synthesized data, e.g. reconstruct prices from skewed sampling from empirical return distribution

21 Scaling up RL Breaking long episodes into partial episodes with differential memory Partial roll out p Partial states DNC Estimated initial state Action Policy s a s a s a s Environment states Environment reaction

22 High Resolution Event TS New hybrid RNN CNN network topology Properly apply convolution over time and cross section Cross section should be permutation invariant! Convolution at different time frequencies Residual NN OHLC OHLC too simplistic

23 Adapting while not Forgetting New attention mechanism relative to prior attention with penalty Prior attention reflects agent style Prioritization of new data History New Important time marked in prior Important time marked in prior

24 Explaining Agent Decisions Learning supervised model to explain agent returns Compare to different ETF and investment products following a specific investment style Value Growth Momentum Mean reversion

25 Contact Info

Disclaimer. 2 Disclaimer

Disclaimer. 2 Disclaimer Whitepaper v1.0 Disclaimer THIS WHITEPAPER DOES NOT CONSTITUTE LEGAL, FINANCIAL, BUSINESS OR TAX ADVICE AND YOU SHOULD ALWAYS CONSULT YOUR OWN LEGAL, FINANCIAL, TAX OR OTHER PROFESSIONAL ADVISER BEFORE