The Behaviour of an Artificial Market with Adaptive Agents Under Different Conditions. A.H.Cruickshank

Transcription

1 The Behaviour of an Artificial Market with Adaptive Agents Under Different Conditions A.H.Cruickshank Master of Science Cognitive Science and Natural Language Processing School of Informatics University of Edinburgh 2011

2

3 Abstract Market simulations are an alternative approach to standard market theory that attempt to determine the dynamics underlying the observed behaviour of real markets. In this study we create a framework for the simple implementation of a wide variety of market simulation types. To prove the performance of this framework we perform a number of simulations involving learning traders operating in a market with a realistic clearance mechanism in order to determine the key factors that produce a realistic price series. We provide evidence that required elements are information on the fundamental stock value, a sufficiently large number of trading agents and a method that indicates when offers should be made. Using these elements we perform a number of further simulations to investigate market behaviour under different conditions, which may form the basis of further research. 1) Using agents that evaluate their performance based on other traders we find that the generated price series exhibits fewer statistical properties of real markets. We provide evidence that this is due to the reduced stochastic behaviour of the market. 2) We induce market shocks by causing sudden changes in stock s fundamental value. Agents are able to cope with such changes when they are within expected bounds, but the price series becomes unstable if the change is unprecedented. 3) Finally, we perform a simulation with biased (optimistic & pessimistic) agents that over-value and under-value the stock. In this case the generated price series is a closer match to that of our comparison real market data. In addition, there are signs optimistic traders are being driven from the market, which matches the results of the standard economic theory, but also that individual biased traders are consistently able to make a profit, which is in contradiction to this theory. iii

4 Acknowledgements Firstly I would like to thank my supervisor, Dr. Subramanian Ramamoorthy, for his support on this project and for many interesting discussions on a range of subjects. I was also aided along the way by the Angels of Appleton Tower and extend my gratitude for their company, cookies and proof-reading skills. Where errors occur in this thesis it is despite the best efforts of all of the above. I take full responsibility. It s all my fault. iv

5 Declaration I declare that this thesis was composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, and that this work has not been submitted for any other degree or professional qualification except as specified. (A.H.Cruickshank) v

6

7 Table of Contents 1 Introduction Overview Motivation Organisation Background The Efficient Market Hypothesis Market Simulations Common Market Simulation Components Market Simulation Basis Asset Pricing Under Endogenous Expectations in an Artificial Stock Market ( [Arthur et al., 1997]) An Heterogeneous, Endogenous and Co-Evolutionary GP-Based Financial Market ( [Martinez-Jaramillo and Tsang, 2009]) Herd Behaviour in Financial Markets ( [Bikhchandani and Sharma, 2001]) Market Simulation Overview Market Clearance Price Update Distribution of Shares Traders Market Predictors Action Selection Learning Market Simulation Overview Market Simulation Implementation Framework Implementation Overview/General Features TimeServer AMarket ATrader ValuationServer vii

8 4.2 Simulation Implementation Overview Market AGATrader MarketActionSelectorSet MarketWatcher Market Simulations Evaluation Criteria Simulations Overview Simulation 0: Initial Investigations Simulation 1: Introduction of the Learning Mechanism Simulation 2: Extension of Learning Mechanism to Sarsa(λ) Simulation 3: Increase Market Clearance Interval Simulation 4: Increase Number of Agents Simulation 5: Confirmation Evaluation Simulation 6: Relative Asset Trading Agents Simulation 7: Market Shocks Simulation 8: Inclusion of Biased Traders Conclusions Framework Simulations Further Work Summary A Equivalence of Market Price Setting 71 B Sample Configuration 73 Bibliography 81 viii

9 Chapter 1 Introduction 1.1 Overview In this work we describe a generalised framework that supports the creation of market simulations, including mechanisms for Asynchronous market clearance. Multiple stocks. Dynamic floatation and removal of stocks from the market. We then verify the operation of this framework by performing a number of market simulations. Our initial, exploratory simulations are used to determine the key requirements for our system to exhibit behaviour similar to real markets. We find that we need The presence of information that details the fundamental value of the stock. A sufficiently large population of agents. A method that provides an indication to the agents when to sell their shares. Using this baseline configuration further simulations were performed in which we modified the parameters to determine the effect on the market and trader behaviour: Relative Performance In this simulation agents adapted their strategies based on the performance relative to other traders in the market. In our simulation the generated price series did not match that of real markets. A tentative conclusion for this result is the reduction in the stochastic nature of the market. Market Shocks Two simulations were performed in which the fundamental value made sudden jumps in value. Our results show that, when these jumps are within the expected bounds learnt by our agents, they can successfully adapt. However, when the jumps are sufficiently large so as to become unexpected, agents are forced to re-learn strategies and the resultant price series becomes unstable. 1

10 2 Chapter 1. Introduction Biased/Noise Traders A final simulation was performed in which biased agents were included. A biased agent either over-estimates or under-estimates the fundamental value. The generated price series showed similar statistical properties to that of real markets. In addition, the distribution of wealth showed that over-estimating traders were being driven from the market, which is consistent with standard economic theory. However, individual biased traders were consistently able to make a profit, which is not consistent. 1.2 Motivation Market theory as proposed by economists does not match the conditions as experienced empirically by traders and analysts. Standard financial theory is based on the assumption of identical investors with shared expectations of an asset s future price and that discount this information into the price instantaneously and rationally. Under such a regime temporary price over-reactions, as evinced by bubbles and crashes, reflect rational changes in valuations. In addition, technical trading using historic patterns in market data to forecast the current price cannot be profitable except by chance. The market in this standard model is rational, mechanistic and efficient. Traders and market analysts take a contrasting view and see markets as offering many speculative opportunities. In particular, many believe there is a correlation between historic data and the current price that enables the use of profitable technical trading strategies. Many also believe that effects unrelated to market news can cause market bubbles and crashes. The market in this view is psychological, organic and inefficient. The advantage of market theory is that it provides a mathematically tractable method for modelling. However, there are occasions when its predictions do not match the behaviour of real markets. Further, when predictions are accurate the theory does not provide an insight into the underlying processes involved. A common approach to addressing these problems is to create simulations in which trading agents interact in a market and observing the emergent behaviour of the system, such as the generated price series and trading volume. 1.3 Organisation The rest of our study is organised as follows: In Chapter 2 we cover related research into markets, including an overview of standard economic theory. We then introduce previous studies on artificial markets (both simulated and mathematically modelled) that form the basis for our work. In Chapter 3 we provide details on the market and trading agents used within our model. In Chapter 4 we describe the implementation of the simulation framework and the specific components for our experiments. In Chapter 5 we give the results of our simulations, covering the exploratory experiments that led to our baseline configuration and the additional experiments described above. In Chapter 6 we discuss our conclusions and possible future work.

11 Chapter 2 Background In this chapter we provide a brief overview of the efficient market hypothesis that forms the basis of standard economic theory. We then introduce market simulations, covering some of the advantages they provide and their common components. Finally we describe the three previous studies that form the basis for our simulations ( [Arthur et al., 1997], [Martinez-Jaramillo and Tsang, 2009] and [Bikhchandani and Sharma, 2001]). 2.1 The Efficient Market Hypothesis The standard economic model for price formation depends on the efficient market hypothesis, which was introduced by [Fama, 1965]. This is based on work by [Samuelson, 1965] who showed that, assuming rational investors, the price series for a stock follows a random walk. This approach assumes that investors act as perfectly rational agents that have access to all available information and react instantly to discount this information into the current price. This leads to random walk behaviour as the current price captures all the available information and thus at any point in time the price only depends on the preceding price and any new information that may arise. In particular, price changes across time periods are independent of each other. Under this model no advantage can be gained from technical trading using patterns in past prices as no such patterns exist. Similarly, no profits can be made from using publicly available information as this has already been incorporated into the current price. This model allows for tractable mathematical analysis of prices, portfolio management and other economic data. There has been significant research into whether prices observed in real world markets are consistent with the operation of the efficient, rational market described by the standard economic model. Many studies have shown that prices can diverge from a fundamental asset value that is based on a random walk. For example, prices can show short term correlations possibly caused by investors reacting slowly to new information or alternatively by investors jumping on the bandwagon on seeing consecutive price increases or decreases. However, such correlations do not continue indefinitely and markets show evidence of mean reversion in which the price returns to a fundamental value. This effect is most clearly demonstrated when prices crash after a bubble. Thus, though the efficient market hypothesis, with price changes modelled as a random walk, is a good starting point it does not capture all the effects seen within real world markets. One potential source for the differences is in the simplifying assumptions concerning the agents in 3

12 4 Chapter 2. Background the market. These agents are assumed to have access to all public information and are able to deduce the correct action using perfectly rational computation. A further assumption is that all agents follow the same process and that agents are aware that the others are also following this process. However, these simplifying assumptions may not hold. Rational Computation Within the standard economic theory agents are assumed to have sufficient computational power to deduce the appropriate action independent of the problem complexity. However, in real markets traders rarely have sufficient computational resources and in situations where they do they may still resort to other, less rational decision making processes. Information The theory assumes that all agents have access to all public information. However, traders will not necessarily have full access and when they do, may deliberately ignore some part of the information. Common Reasoning Finally, agents are assumed to all follow the same reasoning process (based on the same information) and thus each will arrive at the same conclusion. However, different traders may well use different approaches and have access to (or place different emphasis on) other sources of information. A theory of bounded rationality that places limits on some part of the problem (such as computational complexity, memory or knowledge) have been proposed. However, it is unclear where these bounds should be applied and to what magnitude, though [Tsang, 2008] argues that it is an agent s computational power that determines its effective rationality. 2.2 Market Simulations Market simulations provide a tool for examining the dynamics of market behaviour. Such simulations provide a number of advantages over the mathematical models used in standard economic theory: The simplifying assumptions for agents are not required. Agents can be constructed that use different strategies (simple rules, based on heuristics or acquired using machine learning techniques). Similarly, simulations can be performed using agents with limited and/or different information sources. Even when the standard economic theory provides a sufficiently accurate prediction it does not provide any insight into the dynamic processes that lead to the observed behaviour. Simulations can be used to investigate situations where the standard economic theory cannot be applied, such as when there is no equilibrium solution. Beyond the standard economic theory, simulation environments can also be used to perform a number of experiments: [Darley and Outkin, 2007] created a simulation of the Nasdaq market to evaluate the impact of changing market rules. In 1990 the Santa Fe Institute created a double auction market in which computerised agents competed. The results, detailed in [Rust et al., 1994], determined that simple rules-of-thumb outperformed more complex strategies.

13 2.2. Market Simulations 5 [Kearns and Ortiz, 2003] also created a simulation, which incorporated real-time, real-world data where possible, in which agents competed. [Greenwald and Stone, 2001] use a simulation in which computerised agents competed to create holiday packages using resources bought and sold on three separate markets. Simple simulations are also used to perform behavioural experiments involving live subjects (e.g.: [Bossaerts and Plott, 2004]) or live subjects and computerised agents (e.g.: [Das et al., 2001], [Luca and Cliff, 2011]) Common Market Simulation Components In principle a market simulation contains only a few components. The market accepts trades (bids and offers) and periodically updates the value of the stocks and clears the trades (i.e. determining how shares are distributed for each bid and offer). Traders are typically given some initial funds and shares and make decisions in each trading period as to whether to bid, offer or hold according to some goal. Complexity arises in how the market clearance is performed and in the types of traders included in the simulation Market Markets in the context of simulations typically act as both a market broker, that accepts trades on behalf of others, and a market specialist, that sets the new price based on the received trades. Early work using the Santa Fe Artificial Stock Market ( [Arthur et al., 1997], [LeBaron et al., 1999]) determined the price update using a fairly sophisticated mathematical model based on a CARA (Constant Absolute Risk Aversion) utility measure. A different approach uses a simple price update based on the demand, which approximates the observed behaviour of market specialists ( [Martinez-Jaramillo and Tsang, 2009], [Cont and Polytechnique, 2000], [Giardina and Bouchaud, 2003], [Farmer, 1999], [Jefferies et al., 2001] and others). How trades are cleared is also an important consideration that impacts the results of the simulation. In the Santa Fe Artificial Stock Market bids and offers are abstracted out of the model and thus trades are always successful. In the simulations performed by [Martinez-Jaramillo and Tsang, 2009], [Giardina and Bouchaud, 2003] and others, market clearance was performed based on the trades received, so that bids had to be matched by offers and short falls were assigned equally amongst traders Traders Traders are typically modelled as one of three distinct types: Fundamental Traders Fundamental traders use rational information to assess the fundamental value of a stock. For example, profit & loss reports, sales figures and similar hard financial data. Fundamental traders issue bids when the current price is below the fundamental value and offers when it is above. These traders tend to follow a buy low, sell high strategy. In market simulations, such as [Arthur et al., 1997], [Martinez-Jaramillo and Tsang, 2009], [LeBaron et al., 1999] and others, the fundamental value is modelled as a relatively stable autoregression (AR(1)) process.

14 6 Chapter 2. Background Technical Traders Technical traders examine the historic values of the stock prices, using a variety of measurements to discover trends. These can be relatively simple, such as comparing the current price to the average over a selection of time periods ( [Arthur et al., 1997]) or can be extended to more complex measurements, such as breakout indicators and momentum indicators ( [Martinez- Jaramillo and Tsang, 2009]). Technical traders issue bids when they believe the price will rise and offers when they believe the price will fall. Noise Traders Noise traders are traders that act irrationally. Such traders can be viewed as using irrational information to assess the value of a stock. For example, expert reports, advice columns or even the news headlines. In [De Long et al., 1990] noise traders follow an irrational valuation that diverges from the fundamental value by the addition of gaussian noise. In more recent simulations, such as [Martinez-Jaramillo and Tsang, 2009], noise traders are modelled as truly random actors, selecting to bid, offer or hold using a uniform probability distribution. Fundamental and noise traders are generally implemented with simple strategies, with technical traders usually being the place where more sophisticated techniques are implemented. Simulations have used neural networks, reinforcement learning methods and genetic algorithm techniques, with the latter seeming to be the most common. A study by [Dempster et al., 2001] comparing two learning approaches (genetic programming and reinforcement learning) and two simpler methods (a Markov Decision Process solution and a heuristic) determined that the genetic programming approach was superior. Though some studies do not explicitly contain noise traders a degree of randomness (as shown in [De Long et al., 1990]), is required in order for the market simulation to produce the behaviour observed in real markets. 2.3 Market Simulation Basis The market simulations we present are based on the work of [Arthur et al., 1997], [Martinez-Jaramillo and Tsang, 2009] and [Bikhchandani and Sharma, 2001]. [Arthur et al., 1997] describe a simulation using relatively simple agents operating in a market with a clearance mechanism based on simplifying assumptions that lead to a mathematically tractable price update. [Martinez-Jaramillo and Tsang, 2009] also describe a market simulation using a more realistic market clearance mechanism and the full range of trading agent types described above. [Bikhchandani and Sharma, 2001] (and [Cont and Polytechnique, 2000]) describe a mathematical model that describes herding in markets. This previous work forms the basis for our own simulations, which incorporate: A price update mechanism based on demand ( [Martinez-Jaramillo and Tsang, 2009]). A market clearance that matches bids to offers ( [Martinez-Jaramillo and Tsang, 2009]). A set of trading agents that all attempt to learn effective strategies ( [Arthur et al., 1997]). Trading agents that have assets in the form of funds and shares ( [Martinez-Jaramillo and Tsang, 2009]). Trading agents that have only a limited set of market predictors ( [Arthur et al., 1997]).

15 2.3. Market Simulation Basis 7 Trading agents that are formed into clusters ( [Cont and Polytechnique, 2000]) to determine whether herding effects ( [Bikhchandani and Sharma, 2001]) arise Asset Pricing Under Endogenous Expectations in an Artificial Stock Market ( [Arthur et al., 1997]) In this early work market simulations were performed using automated trading agents that attempted to optimise the allocation of a high risk asset and risk-free asset in each trading period. The model contained N agents and N units of the high risk asset, which paid a stochastic dividend modelled as an exogenous AR(1) process. The risk-free asset was assumed to be in infinite supply and paid a fixed amount. In this simulation each agent uses the same constant absolute risk aversion (CARA) utility function (U(c) = exp(λc)). An assumption of the model is that agent predictions at time t for the next period s price and dividend are normally distributed with µ = E i,t [p t+1 + d t+1 ] (2.1) σ 2 = σ 2 i,t,p+d (2.2) where p t+1 is the price and d t+1 the dividend at t + 1. The gaussian distribution assumption on predictions and the CARA utility function forms the basis for each agent s demand d i,t = E i,t[p t+1 + d t+1 ] p (1 r) λσ 2 i,t,p+d (2.3) where r is the amount paid out by the risk-free asset and p is the clearing price. The model is closed by assuming that the total demand equals the number of issued shares, N, so that N i=1 x i,t = N (2.4) which determines the clearing price p. To learn values for the mean and variance agents use a set of market predictors, each of which contains a condition and a forecast. A condition is represented as a set of indicator flags, each with a value that represents the condition being true, false or irrelevant. For example, the first flag in the condition may represent the market condition current price interestrate(r) dividend > 0.25 and the setting for the predictor can take values 1 (the condition is true), 0 (the condition is not true), (the condition is not important to the forecast). The forecast of each market predictor contains values that represent the mean and variance of the normal distribution. (I.e the agents try to learn appropriate values for E i,t [p t+1 + d t+1 ] and σ 2 i,t,p+d that are used to determine their demand in Equation (2.3).) In the experiments, agents are provided with fundamental predictors that are based on the current price, interest rate and dividend (such as the example above) and technical predictors that compare the price with a moving average of past prices over different period lengths. In each trading period the state of the market is compared against each of the agent s predictors. From the matching predictors, the most accurate forecast values are used to determine the demand in

16 8 Chapter 2. Background Equation (2.3). After the results of the trading period are determined the accuracies of the forecast values for all matching predictors are updated using an inverse of the moving average of the squared error. Periodically the predictors are updated using a genetic algorithm approach. The predictors are ordered using a fitness function and then crossover and mutation are performed to create a new set of predictors. The study finds that, when predictor evolution occurs infrequently ( 1000 trading periods) a market regime that is similar to the rational expectations regime emerges. However, when predictor evolution occurs frequently ( 250 trading periods) a more complex regime emerges with evidence of technical trading being performed by the agents Notes A key feature of this study is that only one type of agent is used. In particular, the model does not contain the separate noise, fundamental and technical traders used in other simulations. The authors point out that, under their model, explicit noise traders are not required for the emergence of complex behaviour. However, the genetic algorithm approach results in a degree of non-determinism, especially following an evolution when an agent will be performing more exploratory actions. Thus noise trading, in the sense of agents making decisions based on irrational (or incomplete) information, is inherent in the simulation. In particular, when the evolution period is short the market will almost always contain what are effectively noise traders. Thus this study does not contradict the findings of [De Long et al., 1990]. The model is based on simplifying assumptions that enable the use of tractable rational market calculations to determine the demand and market clearance. Under this model, agents do not have to manage assets and the exchange of funds for stocks is assumed to always be possible. The market clearance and price update method is based on the assumption that all N shares are being traded in each period. Despite the use of a small number of market predictors, complex behaviour and technical trading patterns are shown to emerge. The study also indicates that agents learnt to prefer fundamental indicators when the evolution period was long and technical indicators when this was short An Heterogeneous, Endogenous and Co-Evolutionary GP-Based Financial Market ( [Martinez-Jaramillo and Tsang, 2009]) The more recent study performed by [Martinez-Jaramillo and Tsang, 2009] attempted to determine circumstances under which statistical properties associated with the price series of real markets emerged. The simulation environment included the three different types of trading agents: Noise Traders acted purely randomly, with parameters specifying the probability of bidding, offering or holding. Fundamental Traders used an AR(1) process to select an action. The same AR(1) process was used by all the fundamental traders, though each had randomly generated thresholds that governed their exact behaviour.

17 2.3. Market Simulation Basis 9 Technical Traders used a variety of sophisticated technical indicators and determined their actions based on a set of decision trees. Unlike the other two types of traders, technical traders were able to learn by using a genetic programming approach to periodically evolve their decision trees. Behaviour was also determined by two heuristic methods, thus the technical traders were not purely learning agents. The first heuristic was the inclusion of lower and upper thresholds that when crossed caused the technical trader to revert to fundamental behaviour, forcing the price back into a well defined range when it deviated too far. The second heuristic was the issuing of two limit offers after a successful bid, one above and one below the bidding price. These limit offers meant that the agent automatically sold the bought shares either to ensure a profit or to reduce loss. The price update was based on a simplified version of that used by market specialists, which uses the difference between the offered and bid trades. For a trading period t with a total number of bids B(t) and offers O(t) the updated price was determined using D(t) = B(t) O(t) (2.5) P(t + 1) = P(t) + D(t) λ where λ represents market sensitivity. To clear the market the bids and offers were matched, with any short fall divided equally. For example, when there were more bids than offers, the lower number of offers were divided equally amongst the bidding traders. as The analysis used to evaluate the performance of the simulation used the log return, which is defined r(t) = log P(t) = logp(t) logp(t 1) (2.7) P(t 1) The statistical properties included: (2.6) lack of auto-correlations in the returns. slow decay of auto-correlations of the absolute log returns. heavy tails, evidenced by a large, positive kurtosis. As in [Arthur et al., 1997] simulations were performed with long and short time periods between the technical traders performing evolution of their decision trees. In addition an approach based on traders evaluating their performance against that of other traders and modifying the decision trees when this fell below average was also used. The study found that the statistical properties of the simulation matched that of a sample from the FTSE 100 when technical traders were allowed to evolve their decision trees relatively frequently, a similar result to that found in [Arthur et al., 1997]. A closer match was found when the criteria for evolving the decision trees was based on a comparison of the agent s performance against other trading agents.

18 10 Chapter 2. Background Notes The simulation includes a large number of stochastic elements, many more than in [Arthur et al., 1997]. As well as the explicitly random noise traders, fundamental traders also have different, randomly generated thresholds that govern their behaviour and technical traders use a genetic programming approach, that involves inherent non-determinism, for learning a good set of decision trees. The technical traders in this study contain heuristic elements and are not purely learning agents. In [Arthur et al., 1997] agents are able to learn whether fundamental or technical indicators (or both) are more appropriate, but in this study this decision is hard coded and there is a definitive switch in the behaviour triggered by the price. The technical traders also do not have to learn when to sell as this decision is also heuristically encoded using limit offers. The study does incorporate a more realistic market clearance mechanism, with price updates based on a linear function of the demand and the bids and offers being matched. Unlike [Arthur et al., 1997], the market clearance does not assume that all shares are being traded in each trading period, thus agents implicitly perform a degree of resource management Herd Behaviour in Financial Markets ( [Bikhchandani and Sharma, 2001]) A feature common in markets is that of herding, which is described in [Bikhchandani and Sharma, 2001] as comprising two types: Spurious Herding occurs when traders behave similarly due to external sources of information. Intentional Herding occurs when traders behave similarly despite their sources of information. In [Bikhchandani and Sharma, 2001], which is based on an earlier work [Bikhchandani et al., 1992], a probabilistic model is used to show that intentional herding is rational under certain circumstances. The model used contains a set of traders, each of which has a private stochastic signal that indicates whether a stock should be bought or sold and which cannot be viewed by other traders. When traders act sequentially such that the previous actions are observed, if the two previous traders take the same action then the study shows that, using Bayesian reasoning, a trader should take the same action even if his private signal indicates otherwise. Note that this view of herding can be seen as a form of noise trading, in which the actions of other traders form (part of) the irrational signal used to determine the value of a stock. In [Cont and Polytechnique, 2000] a mathematical model of a simple stock market in which agents form coalitions or clusters is used to show how the heavy tailed distribution common to real world price series can arise. In this model, agents are assumed to group in clusters and then form the same decision. The created price series then has the high kurtosis that is evidence of a heavy tailed distribution. This study in a sense is the reverse of that by [Bikhchandani and Sharma, 2001] in that it assumes herding has already occurred and then shows that the result matches that of real markets.

19 Chapter 3 Market Simulation Overview In this chapter we describe the operation of the market and trading agents within our simulations, which is based on the previous work described in Chapter 2: A price update mechanism based on demand ( [Martinez-Jaramillo and Tsang, 2009]). A market clearance that matches bids to offers ( [Martinez-Jaramillo and Tsang, 2009]). A set of trading agents that all attempt to learn effective strategies ( [Arthur et al., 1997]). Trading agents that have assets in the form of funds and shares ( [Martinez-Jaramillo and Tsang, 2009]). Trading agents that have only a limited set of market predictors ( [Arthur et al., 1997]). Trading agents that are formed into clusters ( [Cont and Polytechnique, 2000]) to determine whether herding effects ( [Bikhchandani and Sharma, 2001]) arise. 3.1 Market Clearance The market clearance comprises two operations. First the price is updated based on the received bids and offers. The bids and offers are then matched, with any shortfall divided evenly amongst all traders. The costs of buying and the profits on selling are based on the updated price Price Update The market clearance mechanism by which a new price is set is based on a simple linear function of the accumulated demand, which is equivalent to that used in [Martinez-Jaramillo and Tsang, 2009]. Under this scheme the price increases when bid requests outnumber offer requests and decreases in the opposite case. The linear function is defined by four parameters: Minimum and maximum values of the accumulated demand (D, D + ) Minimum and maximum values of the price (P, P + ) 11

20 12 Chapter 3. Market Simulation Overview which define the gradient m = (P+ P ) (D + D ) (3.1) A further parameter, which can be viewed as equivalent to the market sensitivity used in [Martinez- Jaramillo and Tsang, 2009] (or the market stiffness used in [Giardina and Bouchaud, 2003]), adjusts the effect of the demand in a particular trading period on the accumulated demand. The minimum and maximum values for the accumulated demand form bounds, ensuring the price does not take on unreasonable values (less than zero, for example). Thus the procedure for setting the new price at trading period t, given the bids B(t) and offers O(t), is D(t) = B(t) O(t) (3.2) D(t) = D(t 1) + D(t) S D(t) = min(max(d(t),d ),D + ) (3.4) P(t) = m [ D(t) D ] + P (3.5) (3.3) Notes Incremental Price Setting mechanism is specified P(t) = P(t 1) + D(t) λ In [Martinez-Jaramillo and Tsang, 2009] an incremental price update (3.6) The mechanism used in our simulations is equivalent (see Appendix A) and is preferred due to the removal of the adverse effects of rounding errors. In early simulations the incremental approach was used but rounding errors quickly became significant, causing the price to either rise or fall sharply. In particular, the effect of rounding errors meant that the price did not return to its original value on successive trading periods with opposite demands. Upper Bound the performance of the system. In later simulations the enforcement of the upper bound was removed as this improved Distribution of Shares Once the updated price is determined the trades are cleared. As in [Martinez-Jaramillo and Tsang, 2009] bids are matched to offers, with trades being divided evenly amongst the agents when there is an imbalance. For example, in the case where there are more bids than offers (such that more agents want to buy shares than those that want to sell) all agents bidding receive only a fraction of the requested amount. B (t) = O(t) B(t) Notes Closed System (3.7) The simulation environment is thus a closed system in which funds and shares are exchanged, but no new funds or shares are generated. In early simulations funds were periodically

21 3.2. Traders 13 incremented via an interest rate mechanism. However, this resulted in an unstable system due to an exponential increase in the funds. 3.2 Traders The task for traders in the simulations is to learn when to bid, offer or hold based on market conditions and, optionally, the actions of other traders in order to achieve a specific goal. All traders are equipped with a set of simple market predictors, which they use to determine the current state of the market and then select an action. Market predictors comprise two parts: A set of market indicators that are used to identify the current market state. A set of action selectors that contain the results of applying each of the actions (bid, offer or hold) in previous matching market states. In each trading period agents determine which of their market predictors match the current market state. From this set of predictors the best action is selected and performed. The agent then waits for market clearance so that the effectiveness of the action can be determined. After market clearance the agent s scores its new state (based on assets) and updates the selected action in all previous matching predictors. Periodically the set of market predictors is updated, with poorly performing ones being replaced by new ones generated using genetic algorithm techniques. Thus agents learn on two time scales. The action information for matching predictors is updated on each trading period so that the agent quickly learns the effectiveness of each action. Market indicators are updated after a number of trading periods have elapsed. The two mechanisms are linked using the action results as the fitness of a market predictor is based on how often it matched market conditions and the rewards generated by its actions. This leads to trading agents being defined by two features: Scoring Method This is used to determine the effectiveness of an action by generating a reward. A typical method is to use the agent s assets after the action, either using only the funds or including the current value of the owned stock. In this case an action is deemed to be successful if the agent s assets increase. Performance Evaluation Method This is used to decide when to modify the agent s market indicators. A simple approach is to modify the indicators after an elapsed number of trading periods Notes Decision Trees [Martinez-Jaramillo and Tsang, 2009] use a more sophisticated set of market predictors using a larger number of technical measures (such as momentum and breakout measures) and based on decision trees, though genetic algorithm approaches are still used for learning. Heterogeneous Predictors [Martinez-Jaramillo and Tsang, 2009] also use traders that have different sets of predictors. In particular, their learning agents have access to more information than other technical traders in their simulations.

22 14 Chapter 3. Market Simulation Overview Market Predictors Traders are equipped with a number of market predictors (100 in our simulations, matching the number used in [Arthur et al., 1997]) that are used to match against the current market state and then select the best action Indicators The implementation of the indicators is based on [Arthur et al., 1997], in which the state of the market is encoded as a string of characters, each of which is based on a single condition. For example, a particular character might encode the condition that the moving average (MA) of the price over the last 5 trading periods is greater than the current price: I(x) = { 1 MA(5) > price 0 otherwise An example of an encoding for a particular market state is given below: }{{} f undamental : }{{} technical : bhoo }{{} trader with fundamental and technical conditions either true ( 1 ) or false ( 0 ) and the trader conditions indicating the last action, bid ( b ), hold ( h ) or offer ( o ). The indicators used by agents are similar to the encoding of the market but include a wildcard character that always matches against a condition ( ). The market indicators are divided into three sub-sets, the fundamental and technical predictors used in [Arthur et al., 1997] and an optional subset of trader predictors based on the actions of other traders in the cluster. An example indicator that matches the market condition (above) is given below: 100 }{{} : }{{} f undamental technical : ho }{{} trader Fundamental Indicators Within the market a simulation of the fundamental value of the stock is generated using an AR(1) process. The fundamental indicators are used to compare the current price with the fundamental value as follows: { 1 current price f undamentalvalue I(x) = > T i 0 otherwise for T i {0.25,0.5,0.75,0.9,1,1.1,1.25,1.5,1.75,2.0}. Technical Indicators The technical indicators are based on the moving average of the price series over a given period and the current price: { 1 MA(n) > price I(x) = 0 otherwise for n 5,10,20,50,100,200,500,1000

23 3.2. Traders 15 Trading Indicators The trading predictors extend the set used in [Arthur et al., 1997] and are based on the last action of other traders: b action = bid o action = offer I(x) = h action = hold u 1 action = unknown 1 The unknown indicator was implemented in later simulations Action Information Each market predictor contains the action information: Accumulated score (based on the effectiveness of the action) Usage (how often the action was used) Eligibility trace (used in the learning method described below) for each of the bid, offer and hold actions. An example set of action information is given below. BID( , 10, 0.49), OFFER( , 2, 0.001), HOLD(0, 0, 0) }{{} action(score,usage,eligibilitytrace) Action Selection To select an action traders use an ε-greedy approach (detailed in [Sutton and Barto, 1998] Section 2.2), which ensures continuous exploration of potential strategies. The ε-greedy action selection also means that all traders act as noise traders with a certain probability, ensuring the conditions required by [De Long et al., 1990]. First the agent identifies all the predictors that match the current market state using the indicators. (This set will be used in the action selection improvement 2 described later.) With a probability of p < ε traders choose randomly from the available actions (bid, offer or hold). Otherwise the agent selects the action from the matching predictors that has the highest score. Pseudo-code for action selection is given in Algorithm 1.

24 16 Chapter 3. Market Simulation Overview Algorithm 1 Pseudo-code for action selection. For market state M and market predictors P {Select all predictors that match the current market state} P {} for p P do if p.indicators matches M then P P p end if end for action unde f score unde f {ε-greedy action selection} r U[0,1] if r < ε then else action random( bid, o f f er, hold ) for p P do s max score p.action.score a argmax action p.action.score if s > score OR score == unde f then action a score s end if end for end if

25 3.2. Traders Learning Agents learn in two distinct ways, which are based on the two separate sections of the market predictors. Actions The scores associated with actions in matching market predictors are updated immediately after the result of the action becomes known. Indicators Indicators are updated periodically using a genetic algorithm approach that creates new indicators based on the most successful market predictors. Over time indicators that rarely (or never) match market state or which always lead to poor action selection are discarded Action Selection Improvement After an action has been performed the reward is determined, usually based on the updated assets owned by the agent, and the action score associated with all matching market predictors is updated. For example, if there are 10 matching market predictors from which the bid action is selected, the bid score for all of the market predictors will be updated. Note that this includes market predictors that potentially would have selected a different action and resulted in a different reward. In the early simulations only the score for the most recent action was updated, which led to unrealistic behaviour. In particular, for a scoring method based on funds the agent generally learnt that bidding for shares was to be avoided as it resulted in a reduced score. To create more realistic behaviour a Sarsa(λ) ( [Sutton and Barto, 1998] Section 7.5) approach was implemented, using eligibility traces to distribute the reward from the latest action backwards to previous actions. Thus agents can learn that a bid followed by an offer can result in an improved position. As market state is represented by a set of market predictors, rather than the more usual single state representation used in standard reinforcement learning approaches, the Sarsa(λ) implementation was modified. The standard Sarsa(λ) update rule, using replacing traces, for action a in state s is: Q t+1 (s,a) = Q t (s,a) + αδ t e t (s,a) s,a (3.8) where δ t = r t+1 + γq t (s t+1,a t+1 ) Q t (s t,a t ) (3.9) and { 1 if s = st and a = a t e t (s,a) = γλe t 1 (s,a) otherwise (3.10) As the market state is represented by a set of market predictors, the calculation of δ was modified where δ t = r t+1 + γ Q t (S t+1,a t+1 ) Q t (S t,a t ) (3.11) Q(S,a) = 1 S S i=1 Q(s i,a) s i S (3.12) That is, the Q-value of a market state is the average of all market predictors that match that state. The pseudo-code for updating the action scores is given in Algorithm 2.

26 18 Chapter 3. Market Simulation Overview Algorithm 2 Pseudo-code for action update. Take action (selected above in 1). Determine score based on the action. {P is the set of matching predictors from the previous trading period} {P is the set of matching predictors found in 1} {Determine the current Q-value for the market state} Q sum 0 n 0 for p P do Q sum Q sum + p.action.score n n + 1 end for Q Q sum n {Determine the update value, δ, given the previous average Q-value Q} δ r + γq Q {Update the eligibility traces (using replacing traces)} {Also update the usage for this action, which is used in indicator improvement Algorithm 3} for p P do p.action.eligibility = 1 p.action.usage = p.action.usage + 1 end for {Update the score for all actions in all predictors} for p P do for a p.action do a.score = a.score + αδa.eligibility a.eligibility = γλa.eligibility end for end for {Save the latest Q value and match predictors} Q Q P P

27 3.2. Traders 19 Note that this credit assignment problem does not occur in [Arthur et al., 1997] and other studies based on the Santa Fe Artificial Stock Market. In these simulations agents do not perform explicit bids and offers and also do not have to manage funds. Predictors are used to determine the ideal division between cash and a stock based on the assumption that all shares are traded in each period. The problem also does not occur in [Martinez-Jaramillo and Tsang, 2009] as agent learning was avoided completely. On a successful bid agents immediately placed two limit offers, one below and one above the buying price, thus ensuring that the agent eventually sold the stock. The lower limit offer also ensured that any loss due to the price dropping was mitigated, whilst the upper limit offer provided a mechanism for the amount of profit to be controlled Indicators Improvement Improvement of the indicators is performed using a Genetic Algorithm approach (as used in [Arthur et al., 1997]). First any market predictor that has a zero usage (i.e. it has never matched a market state) is removed. Note that usage is maintained individually for each action as this allows for more detailed reporting. The remaining predictors are then ordered using the fitness function f (p) = max score (p.action.score)) N I K (3.13) where N I is the number of non-wildcard predictors and K is a constant. The latter term encourages the use of more general predictors. The topmost N market predictors are kept and the remainder are generated using either mutation or crossover and added to the new set of market predictors. Mutation A single market indicator is chosen using tournament selection. A new market indicator is created by copying the market indicator and randomly mutating some of its predictors. (Each of the predictors has a chance of changing its value.) Crossover Two parent market predictors are chosen using tournament selection. A new market indicator is created by randomly selecting an indicator from one of the parents for each market condition. In tournament selection two predictors are picked at random, with a uniform chance across all predictors, and the one with the highest fitness is selected. This matches the selection process in [Arthur et al., 1997]. After each evolution of the market predictors the information on all actions is cleared so that the N kept predictors and the new market predictors can be compared equally. This also provides an additional source of noise trading as trading agents will essentially be selecting actions at random following evolution. The pseudo-code for updating the indicators is given in Algorithm 3.