Stock Trading with Reinforcement Learning

Transcription

1 Stock Trading with Reinforcement Learning Jonah Varon and Anthony Soroka December 12, Introduction Considering the interest, there is surprisingly limited available research on reinforcement learning applied to stock trading. Hoping to either further public research or find a profitable strategy, we applied Q-Learning to stock trading. Specifically we attempted to place agent in states to find profitable strategies through technical analysis and mean reversion. Quantopian was used to to test the strategies. Quantopian is a crowd-sourced investment firm which provides market data and a python-based development platform for algorithmic investors. 2 Background and Related Work Many economists adhere to the Efficient-Market Hypothesis, advocating that it is impossible to beat the market on a risk-adjusted basis. Esteemed economist Burton Malkiel s famous book A Random Walk Down Wall Street [3] further argued the stock market follows a random walk. Yet investors continue to seek outperformance and place their money in actively managed funds. The repeated and relative consistent success of specific hedge funds, particularly the quantitatively inclined funds, encourage us to see if an agent can learn profitable strategies through reinforcement. As previously mentioned, research on Q-Learning based stock trading agents is relatively limited. One previous approach [2] used four agents, each entrusted with unique responsibilities (when to buy, at what price to buy, when to sell, at what price to sell). Specifically, buy and sell time-decision agents used states based on the long-term technical analysis structures of support and resistance bands, while buy and sell price-decision agents used states based off of shorter-term technical analysis structures of Japanese Candle Sticks. The study was conducted over Korea s KOSPI Index over the period of In another study conducted at Stanford University over the S&P 500 [1], an agent s reward was defined by risk metrics such as sharp ratio and derivative sharp ratio. Such research certainly has been valuable but more should be done. Specifically, using Q-Learning on single stocks rather than major indexes, on more recent data, and using different state representations would all advance the field. 1

2 3 Problem Specification Compared to previous research, we will be applying Q-Learning to trading in a different and perhaps more straightforward manner. Agent rewards will be defined by stock performance. Actions will be limited to Buy and Sell. States will be represented to position agents to learn either mean reversion or technical analysis based strategies - two known and widely used trading techniques. Quantopian will be our data source and development platform. Quantopian provides a wealth of technical, fundamental, and event information from thousands of stocks. 4 Approach The foundation of our research is based on the standard Q-Learning Algorithm as per below: Algorithm 1 Q-Learning Algorithm Initialize Q(s, a) arbitrarily repeat for each episode Choose a from s using ɛ-greedy Take action a, observe r,s Q(s, a) (1 α)q(s, a) + α[r + γ max a Q(s, a )] s s until s is terminal With ɛ-greedy defined as: a = { arg max a Q(s, a), with prob 1-ɛ. random, with prob ɛ. (1) Adapting Q-Learning for financial trading, we first limited possible actions to be: a : {Buy, Sell} We considered making the action space broader, specifically incorporating a hold action. However we decided against this alternative for two reasons: 1) limiting the action space will allow for quicker convergence to true Q(s,a) values 2) hold will never be more profitable than correctly deciding to buy or sell, hence the advantage of incorporating this action is limited. The rewards function is defined to be: r = percent return = (curr price - prev price)/(prev price) r(buy) = r r(sell) = r Next, we highlight a few interesting characteristics of stock trading that slightly alter Q-Learning: 2

3 a) Future State: Due to the ability to rebalance, max Q(s, a ) is the same regardless of Q(s, a) (presuming you are a small enough investor with minimal market impact when exiting positions). b) Fully Observable: Limiting the agent to actions of Buy and Sell, the reward from Q(s,a) is fully observable regardless of the actual action taken. Both approaches to updating Q(s,a) were attempted: i) only for action taken ii) for both taken and not taken actions. We find the second option preferable (the more information/learning the better). Given these two attributes, the Q(s, a) update operation becomes: Q(s, buy) (1 α)q(s, buy) + α[r] Q(s, sell) (1 α)q(s, sell) + α[ r] Next and perhaps most importantly, the state space must be defined. The possible stock market state space is vast, but limiting the space to allow the agent to learn is crucial. Given such, we represented states in hopes of positioning our agent to find known trading strategies through mean reversion and technical analysis. Mean Reversion: Previous week s stock performance s = { 1, if stock s previous week return was in the bottom 5 percent. 1, if stock s previous week return was in the top 5 percent. (2) Each week the 500 stocks in S&P 500 in the U.S. are filtered for only the over- and underperformers. Hence the agent makes 2,600 actions per year (52 weeks * 500 stocks * 2 * 5%). Technical Analysis: last 3 minutes of price and volume data s = [p t 3, p t 2, p t 1, v t 3, v t 2, v t 1 ] 1, if price decreased over the t-i minute. p t i = 0, if price did not change over the t-i minute. 1, if price increased over the t-i minute. 1, if volume decreased over the t-i minute. v t i = 0, if volume did not change over the t-i minute. 1, if volume increased over the t-i minute. For example, the state [1,1,1,-1,-1,-1] would be the state where price increased over each of the last three minutes, and volume decreased over the same last three minutes. There are 729 total states (3 6 ), and approximately 98,280 actions per year (252 days* 6.5 hours*60 minutes). Agents begin with $10,000. The agent is always fully invested. For the Mean Reversion strategy, on a given week the agent would make a decision on each of the 25 recent underperformers and 25 outperformers. Given the agent takes an action on each of these stocks, the agent might decide (3) (4) 3

4 to Buy 22 underperformers and 4 outperformers, and Sell 3 underperformers and 21 outperformers. The agent then constructs a delta neutral long-short portfolio. The long and short sides of the total portfolio are both equally weighted. For example, if the agent s wealth had risen to $12,480, it would invest $6,240 in the 26 Buys ($480 per name) and $6,240 in the 24 Sells ($520 per name). For the technical analysis agent, the agent only managed one stock, hence decides to either be fully long or fully short the single stock. For example, if the agent is currently long $13,000 of the stock, and decides to buy again, the position would be unchanged. If the agent then decided to sell, the position would become short $13,000. Lastly, regarding parameter selection, α (learning rate) of.3 and an ɛ (random action / explore rate) of.05 were used. Different parameter values were attempted, and these were found most appropriate. Originally fees and execution slippage costs were applied, but were later removed to reduce complexity for this initial study. 5 Experiments Both Mean Reversion and Technical Analysis Q-Learning agents show promise in producing profitable strategies. The Mean Reversion agent (a long-short delta neutral strategy) returned 68% from January 2011 to July In general, the Technical Analysis approach did not outperform the benchmark of the overall stock performance the agent trades. However deploying multiple Technical Analysis agents effectively scanned the market, and located specific stocks and specific periods where such an approach was profitable, such as during March July 2012 where the agent trading the semiconductor company AMTL returned 113% better than the stock. 5.1 Results Strategy Annualized Return Time Period Mean Reversion 9.7% Jan 11 - Jul 16 Long S&P % Jan 11 - Jul 16 Results: Mean Reversion Agent vs. Baseline While the Mean Reversion agent did not outperform its benchmark (Long S&P 500), it still generated very respectable returns of nearly 10% a year. The strategy is delta neutral and hence has less market exposure, highlighted by a strategy β of.21 versus a benchmark β of 1 (in finance β is a measure of the volatility, or systematic risk, of a portfolio in comparison to the market as a whole). Considering the time period investigated was a bull market, we should not necessarily expect a delta-neutral strategy to outperform. Advantages of the Mean Reversion agent is that given the agent has week-long holding periods, transaction/execution costs would be relatively minimal, making the strategy deployable as is. A potential weakness of the strategy is the simplicity of the state space. Given our state space, the agent is effectively only able to learn four strategies: 1) Mean Reversion (sell outperformers, buy underperformers) 2) Momentum (buy outperformers, sell underperformers) 3) Bull (buy all) 4) Bear (sell all). There is value in such learning (especially considering the learned strategy can 4

5 Figure 1: Mean Reversion Performance and Risk Metrics Figure 2: Benchmark (Long S&P) Risk Metrics change over time), but the learning is clearly not the most sophisticated. Regarding extensions of Q-Learning similar to the Mean Reversion strategy, news sentiment and/or fundamental-analysis metrics were considered. For example, rather than passing the agent overand underperforming stock, the agent could receive stock with positive or negative news. Alternatively, the agent could receive stock with low or high Price-to-Earnings ratios. Quantopian offers such data, however the states change infrequently (Quantopian news sentiment data is limited and generally P/E ratios only significantly change after multiple months). Hence the agent would not be provided with ample train data to learn from. Strategy Annualized Return Time Period Tech Analysis AMTL 35.4% Mar 11 - Jul 12 Long AMTL -52.3% Mar 11 - Jul 12 Tech Analysis YELP 18.9% Jan 12 - Dec 16 Long YELP +12.0% Jan 12 - Dec 16 Results: Tech-Analysis Agent vs. Baseline In general, applying our Technical Analysis agent to a given stock did not outperform the baseline of just being long the stock. However, utilizing a multi-agent approach effectively scans the market and locates stocks and time periods where a specific agent was profitable. Such examples are listed in the above table and below charts, where agents returned 35.4% and 18.9% annualized returns on ATML and YELP respectively. 5

6 The strengths and weaknesses of the Technical Analysis approach is effectively the opposite of those of the Mean Reversion agent. Specifically, the state space is relatively complex; the agent is learning patterns in the 3-minute price and volume data for a given stock. Given the 729 possible states, there is clear value in having an agent handle this compared to a human. A weakness however is the fees and execution slippage would need to be minimized and hence this strategy is not immediately deployable. Figure 3: Tech-Analysis ATML Performance and Risk Metrics Figure 4: Tech-Analysis YELP Performance and Risk Metrics How do we make sense of the the fact that the Technical-Analysis agent only proved successful for specific stocks and periods? One could argue the positve performers are simply illustrations of randomness/luck. Disproving such is notoriously difficult in finance. However, it is quite possible/reasonable that the Technical Analysis agent might work better on some stocks than others. Perhaps less-watched stocks are more receptive to such strategies. In terms of making use of this approach in practice, we propose two methods: 1) simply deploy agents on the stocks that have historically been receptive to the strategy 2) use the multi-agent approach to alert when a partic- 6

7 ular stock is receptive to the strategy, and an additional agent decides when to formally activate and deactivate the strategy on the receptive stock. 6 Discussion We applied reinforcement learning - specifically Q-Learning - to stock trading. Mean reversion and technical analysis based trading strategies were the foundation of our two attempted state representations. Both Mean Reversion and Technical Analysis agents showed promise in producing profitable strategies. The Mean Reversion agent did not outperform the S&P 500 benchmark, but led to a respectable 9.7% annualized returns with preferable risk metrics (β of.21). A multiagent Technical Analysis approach highlighted specific names which were receptive to the Technical Analysis Q-Learning strategy. The possible methods to define a state space are limitless and should still be investigated. Specifically, the Mean Reversion agent could be amended to account for news sentiment and/or fundamental stock data. The Technical Analysis agent could be adjusted in multiple ways, for example by simply using day rather than minute price and volume data. Lastly, approximate Q-Learning using features and feature weights might help manage the difficulty of balancing complex state representation and limited train data. While our explored methods show potential, our study additionally highlights just how competitive and efficient markets are. Additionally, we point out that stock trading is quite different from problems where Reinforcement Learning (and specifically Q-Learning) has proven effective. Stock trading differs from such problems in the following ways: Stochasticity of the environment Limited amount of train data Agent can observe rewards without taking actions Agent has little impact on the environment Given such, Deep Learning techniques such as LSTM might be better suited for this problem. LSTM s ability to learn from experience and predict time series with long time lags of unknown length seems apt for taking on the stock market. 7

8 A System Description Appendix 1 1. Sign up for a free account at quantopian.com 2. Go to My Algorithms : 3. For each algorithm, click the Create New Algorithm button and enter a name for the algorithm. 4. Delete all the sample algorithm text and paste in the code from the algorithm s python file (i.e. quantopian q learning technical analysis.py). 5. In the shaded yellow box in the upper right corner, choose a date range and amount of money to test the algorithm on (we used $10,000 for the single agent algorithms and $1,000,000 for the multi-agent). 6. Click Run Full Backtest and wait for the results of running the algorithm. B Group Makeup Appendix 2 We effectively followed our original contributions proposal. We pair programmed our algorithms, with Jonah leading this component of the project given his proficiency in Python. Anthony led much of the experimentation strategy such as deciding how to represent states, and how to fit Q-Learning to the problem. Both Jonah and Anthony worked on the poster and formal write-up, with Anthony leading this component of the project. References [1] Zhai Jinjian Du, Xin and Koupin Lv. Algorithm Trading using Q-Learning and Recurrent Reinforcement Learning, [2] Park Jonghun O Jangmin Lee Jongwoo Lee, Jae Won and Euyseok Hong. A Multiagent Approach to Q-Learning for Daily Stock Trading. IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, 37(6): , [3] Burton Malkiel. A Random Walk Down Wall Street. W. W. Norton Company,