Operational Planning of Thermal Generators with Factored Markov Decision Process Models

Transcription

1 MITSUBISHI ELECTRIC RESEARCH LABORATORIES Operational Planning of Thermal Generators with Factored Markov Decision Process Models Nikovski, D. TR Jne 03 Abstract We describe a method for creating conditional plans for controllable thermal power generators operating together with ncontrollable renewable power generators, nder significant ncertainty in demand and otpt. The reslting stochastic seqential decision problem has mixed discrete and continos state variables and dynamics, and we propose a discretization method for the continos part of the model that nifies all variables into a large discrete Markov decision process model. Althogh this model is way too large to be solved directly, its state transition probabilities can be factored efficiently, and a redction of all continos variables to one net demand variable makes it tractable by dynamic programming over a sitably constrcted AND/OR tree. The proposed algorithm otperformed existing non-stochastic solvers on several problem instances, reslting in both lower risks and operational costs. International Conference on Atomated Planning and Schedling (ICAPS) This work may not be copied or reprodced in whole or in part for any commercial prpose. Permission to copy in whole or in part withot payment of fee is granted for nonprofit edcational and research prposes provided that all sch whole or partial copies inclde the following: a notice that sch copying is by permission of Mitsbishi Electric Research Laboratories, Inc.; an acknowledgment of the athors and individal contribtions to the work; and all applicable portions of the copyright notice. Copying, reprodction, or repblishing for any other prpose shall reqire a license with payment of fee to Mitsbishi Electric Research Laboratories, Inc. All rights reserved. Copyright c Mitsbishi Electric Research Laboratories, Inc., 03 0 Broadway, Cambridge, Massachsetts 039

2 MERLCoverPageSide

3 Operational Planning of Thermal Generators with Factored Markov Decision Process Models Mitsbishi Electric Research Laboratories, 0 Broadway, Cambridge, MA 039, USA Abstract We describe a method for creating conditional plans for controllable thermal power generators operating together with ncontrollable renewable power generators, nder significant ncertainty in demand and otpt. The reslting stochastic seqential decision problem has mixed discrete and continos state variables and dynamics, and we propose a discretization method for the continos part of the model that nifies all variables into a large discrete Markov decision process model. Althogh this model is way too large to be solved directly, its state transition probabilities can be factored efficiently, and a redction of all continos variables to one net demand variable makes it tractable by dynamic programming over a sitably constrcted AND/OR tree. The proposed algorithm otperformed existing non-stochastic solvers on several problem instances, reslting in both lower risks and operational costs. Introdction The operational planning of thermal generators is a difficlt seqential optimization problem that electrical power tilities mst solve continosly to ensre that they meet power demand with maximal reliability and at a minimal cost. Fossil-brned thermal generators (sing coal, natral gas, or oil) consme vast amonts of expensive fel and contribte significantly to global warming, so minimizing the amont of consmed fel is of primary importance in the electrical power indstry. Given a set of generators with their cost strctre and fel consmption rates, the objective of optimal operational planning is to find the best seqence of commands to trn individal generators on or off, and the optimal amont of power prodced by each of them over an extended period of time, sbject to the operational constraints that these generators might have. Typical planning periods range between one day or one week, and the state of the generators can typically be changes once every hor. The predicted demand over the entire planning horizon is also assmed to be known, either exactly, or with some qantifiable ncertainty. There are several reasons why this problem is very comptationally challenging. The first reason lies in the temporal Copyright c 03, Association for the Advancement of Artificial Intelligence ( All rights reserved. constraints on the operational drations of individal generators that arise from their mechanical constrction and reqirements for reliable operation. It is generally not desirable (or freqently even possible) to trn the brners of the generators on and off at arbitrary moments, becase freqent switching wold case damage de to excessive thermal expansion and contraction. For this reason, once a generator is trned on or off, it mst remain in that state for several hors, and conversely, if it has been trned off, it mst be kept off for several hors. In other words, when a command is given to trn a generator on or off, it is committed to that state for mltiple decision periods, and that is why this problem is also known as the nit commitment problem. De to these temporal constraints, the planning problem mst be solved over the entire planning horizon, making it a seqential decision problem. Althogh the states of the generators are Boolean variables (on or off), the state variables of the seqential decision problem mst inclde information abot how many decision periods the generator has been on or off, and that increases the cardinality of the state space enormosly. The second reason for the high comptational complexity of this problem is its mixed continos/discrete natre. Some of the decision variables are discrete (e.g. the commitment stats of generation nits), and others are continos (e.g. the amont of power prodced by each nit). Moreover, the dynamics that govern the evoltion of the system are also mixed: the total demand for electricity is a continos scalar variable, whereas the main components (generation nits) switch between discrete modes (on or off). This significantly limits the nmber of soltion methods that can be applied to this problem, becase there are relatively few planning and optimization methods that can solve mixed continos/discrete problems efficiently. The third reason this problem is very difficlt is that at least part of the system dynamics are random, for most practical sitations. In all cases, for any ftre moment dring the planning period, the total demand for power will not be a completely known, deterministic vale, bt a random variable predicted from the information available at the time of planning. The prediction error can often be qantified (typical vales are arond % of total demand), and althogh most crrently deployed planning systems have chosen to ignore this ncertainty or deal with it in a heristic man-

4 ner, sch ncertainty information can argably be sed to improve the performance of the planning algorithm. Moreover, the increased penetration of renewable power sorces sch as photovoltaic panels and wind trbines, whose otpt depends strongly on ncontrollable atmospheric conditions sch as solar radiation and wind speed, has effectively introdced mch higher levels of ncertainty in the net demand for power to the controllable thermal generators. For example, if 0% of the power generator by a tility is spplied by wind trbines, in case the wind dies down sddenly, the net demand to the thermal generators might increase sddenly by 0%. The operational plan mst allow for sch contingencies, if forced otages are to be avoided. As a reslt, ignoring ncertainty in the system is becoming increasingly impossible for electrical power tilities. And, in addition, other sorces of ncertainty are possible falts in the generators, which are natrally random events, bt can be characterized probabilistically. De to its primary economic significance, the operational planning problem for thermal generators has been addressed by a very large nmber of soltion methods, inclding ones based on dynamic programming, Lagrangian relaxation, interior point methods, and mixed integer programming, as well as heristic methods sch as genetic algorithms, simlated annealing, evoltionary programming, differential evoltion, particle swarm optimization, Hopfield neral networks, etc. (Wood and Wollenberg 996; Xia and Elaiw 00). Formlations as a model predictive control or optimal control problem are possible, too (Xia, Zhang, and Elaiw 0). Dynamic programming methods can leverage sccessflly the seqential natre of the decision process in order to compte sitable plans efficiently, bt sffer from the well known crse of dimensionality de to the large size of the state space of the problem. Mixed integer programming methods can handle sccessflly the mixed continos/discrete natre of the planning task, bt again do not scale p very well becase of the sheer combinatorial complexity of the discrete optimization part. Lagrangian relaxation cold also be a very effective soltion to the mixed continos/discrete optimization problem, and has been shown to perform well on large problems. Global optimization methods sch as genetic algorithms and simlated annealing can be very effective on problems with disjoint feasibility regions, bt cannot garantee that global optima wold always be reached, in general. However, the majority of these methods either ignore ncertainty completely at the planning state, or deal with it heristically, or consider only a small nmber of possible ftre realizations of the ncertain variables, and sally compte a fixed operational plan for the entire period that is exected seqentially. As is well known in AI, sch plans can only scceed if the problem domain is static, completely observable, deterministic, and the action descriptions available to the planner are correct and complete. One common heristic is to inclde a safety margin of extra capacity (for example, 3%) to be committed for prodction. This reslts in operating more and/or larger nits than are necessary to meet expected demand. This approach is largely heristic, and is not likely to work in the ftre, when renewable energy sorces become even more widespread. And, in general, whereas there might be some vale in algorithms that can find fixed plans that are maximally robst with respect to ftre ncertain otcomes, a mch more natral approach wold be to se algorithms that can compte conditional plans that can select actions depending on ftre states (also known as contingency plans in AI, feedback controllers in control theory, and decision policies in operations research). This paper describes one sch approach based on factored Markov decision processes (fmdp), where continos dynamics are discretized by means of a barycentric approximation and added to the discrete dynamics, the state of the reslting completely discrete fmdp is prned by means of problem domain knowledge, and the optimal decision policy is fond by means of dynamic programming over AND/OR trees. Formlation of the Planning Problem Formally, the operational planning problem for generators can be described as follows. Given N available controllable generator nits, and a planning horizon of length T nits of sitable dration, for example one hor, the overall goal is to minimize the total operating cost for these nits, sbject to operating constraints and at an acceptable risk of a forced otage. The demands D t, t T, over the entire planning horizon are random variables coming from a stochastic process with known strctre and parameters. There are also K ncontrollable generators, and we assme that the realizations yt k of their random otpt amonts Yt k, t T, k K, also come from known stochastic processes. At all times, the sm of the spply from all generators, controllable and ncontrollable, mst match the total demand at that time. In order to formlate a seqential decision problem, we introdce the decision variables i t {0, } for all time periods t, t T, and controllable nits i, k K, which represent the intended commitment stats of the generators dring the next operational period. Similarly, we introdce the state variables x i t { l, l +,...,,,..., L, L}, where l is the minimm allowed time for keeping a generator off, and L is the minimm allowed time for keeping a generator on. Negative vales correspond to off condition, and positive vales correspond to on condition. For the state variables of the controllable part of the process, if we have an existing commitment stats i t for generator i, operation time x i t, and new commitment stats i t, the new operational time x i t can calclated by Eqation () where Ti cl is the cold start time of nit i, l i is the minimm down time of nit i, and L i is the minimm p time of nit i (Li, Johnson, and Svoboda 997).

5 x i t = if Tt cl x i t l i and i t = (start p) x i t + if x i t L i (p and mst stay p) L i if x i t = L i and i t = (p and available to sht down) if x i t = L i and i t = 0 (shtting down) x i t T cl i if l + x i t (down and mst stay down) or Ti cl + x i t l i and i t = 0 (down and available to start p) if x i t = Ti cl and i t = 0 Additional constraints, sch as maximal p/down times, can be accommodated by sitable modifications to Eq. (). For the demand variable, we assme that we have a stochastic dynamic model that specifies the probability P r(d t = d t D t = d t, D t = d t,..., D 0 = d 0 ) that vale (power demand) d t will be observed at time t if a series of demands d 0, d,..., d t, d t has been observed ntil then. Similarly, for each ncontrollable generator k we assme that we can estimate the probability P r(yt k = yt k Yt k = yt, k Yt k = yt, k..., Y0 k = y0 k ) that vale (power otpt) yt k will be observed at time t if a series of otpts y0 k, y k,..., yt, k yt k has been observed ntil then. Varios predictive models can be sed, sch as ato-regressive (AR), neral nets, spport vector machines, etc., that map past observations onto ftre vales. The planner mst observe several constraints in minimizing the total cost. The load balance constraint states that the total generation mst be eqal to the demand d t at any time step. If p i t is the generation of nit i at hor t, then N K p i t i t + yt k d t = 0, for t =,,..., T. () i= k= The objective fnction is presented in Eqation 3, where E 0,x 0,y 0,d 0 denotes the expectation operator with regard to the initial configration 0, operational time x 0, the initial demand d 0, and the initial otpt y 0. For notational simplicity, the decision variables at time t are represented as. the vector t = [ t, t,.., N t ], the state variables are. denoted by the vector x t = [x t, x t,..., x N t ], and the realizations of all ncontrollable generators are denoted as. y t = [y t, yt,..., yt K ]. J = min,,..., T E 0,x 0,y t,d t { T t=0 [ N i= f i(x i t, i t, y t, d t ) + N i= h i(x i t, i t, i t+) + g t ( t, y t, d t )]} (3) Here f i (x i t, i t, y t, d t ) denotes the operating cost of operating nit i in configration i t and state x i t for one time step in order to meet demand d t when the ncontrollable generators otpt electricity amont y t. The fnction () h i (x i t, i t, i t+) denotes the cost of switching to configration i t+ at the end of the step. The third cost component, g t ( t, y t, d t ), denotes the eqivalent cost of the risk of not being able to meet demand d t nder otpt of ncontrollable generators y t with the chosen configration of all nits t. This cost is proportional to the probability that the total capacity of the committed nits in t pls what the ncontrollable generators prodce (y t ) is less than the demand d t : g t ( t, y t, d t ) = αp r( N i = i tcap i + K k = y k t < d t ), where cap i is the maximal generation capacity of nit i. A sitably chosen proportionality coefficient α specifies the relative preference between minimizing operating cost and risk of failre to meet demand. By adding the operating cost and risk compensation cost together, the objective fnction represents a trade-off between fel costs and risk. At any given time, if we can find the optimal seqence,,..., T that minimizes the cost in Eqation 3 by whatever comptational means, we will have an operational plan that can be exected over the entire planning horizon. However, as arged above, sch an open-loop, nconditional plan is not tailored to the concrete sitation that will be encontered in the ftre. An alternative approach is to recognize that the ncertainty in power demand and generator spply makes the decision problem a stochastic one, and its optimal soltion is not an nconditional plan (seqence of commitment decisions), bt an entire decision policy. A conditional operational planner cold compte conditional plans that are robst to ftre variations of spply and demand, and cold provide a safety margin implicitly, by planning for all possible contingencies. One significant difficlty associated with this approach has been how to represent all sch possible contingencies, and how to plan for them. One proposal organizes all ftre possible realizations of the system (called scenarios) as a tree of scenario bndles (Takriti, Birge, and Long 996). However, this model for representing stochasticity is limited to only the few scenarios inclded in it, whereas in a practical system the ftre evoltion can be realized in an infinite nmber of ways. Or work aims to expand this approach by improving the probabilistic modeling of system evoltion. We propose a method for finding the optimal conditional operational plan of a set of power generators nder stochastic demand for electrical power and stochastic otpt of some generators. Unlike traditional operational plans, which are fixed in advance, a conditional operational plan depends on the ftre state of the observable random variables (demand and otpt), and can reslt in different actal seqences of decisions depending on the observed otcomes for these variables. The planner explicitly balances the operational cost of electricity generation with the risk of not being able to meet ftre electricity demand. We represent the stochastic dynamics of the components of the system as a factored Markov decision process (MDP) model, and propose efficient approximate algorithms for compting sitable conditional operational plans.

6 Demand Solar Oil Coal Gas Time t Time t+ a a a 3 Demand' Solar' Oil' Coal' Gas' a a a 3 Time t+ Demand'' Solar'' Oil'' Coal'' Gas'' Figre : DBN for a power generation problem with three controllable and one ncontrollable power generators. Factored Markov Decision Processes for Conditional Operational Planning We propose to represent a power generation system consisting of mltiple generators of the type described above by means of a factored Markov decision process (fmdp), and find the optimal conditional operational plan by means of approximate dynamic programming (Botilier, Dearden, and Goldszmidt 000). The fmdp is sally expressed graphically as a dynamic Bayesian network (DBN). A DBN consists of circles that represent random variables, diamonds that represent decision variables, and directed edges connecting the circles and diamonds that represent the statistical dependence between the corresponding variables. When dealing with a time-dependent system, each time period (e.g. one hor) is represented by its own set of random variables. Three time slices of the DBN for an example stochastic nit commitment problem with for generators, one of which ncontrollable (solar), are shown in Fig.. In Fig., one random, continos, and ncontrollable variable represents the power demand. Another random, continos, and ncontrollable variable represents the otpt of a photovoltaic generator. (In this case, these two model components are first-order Markovian, that is, the next state depends only on the crrent state, for example by means of an AT() model. However, this is not a fndamental limitation: for higher-order models, edges from previos time slices can be added, too.) In addition, three conventional controllable generators are shown, too; their discrete variables x i t take on l + L possible different vales, and represent the operational time of the respective generator. Three decision variables (shown as diamonds) represent the individal decisions a ti = i t to trn on/off the corresponding generators, and ths commit them for power prodction. These models components are necessarily first-order Markovian, bt do not need to be deterministic certain probability of failre to change the state of a generator as desired cold be modeled in them. The probabilistic evoltion of the system is described by local conditional probability tables for each variable, where the conditional dependence is defined only on the parents of that variable in the graph of the DBN. Ths, the DBN serves as a compact representation of a large Markov decision process whose state space is exponentially large in the nmber of states of the individal variables over which it is factored. In order to specify a factored MDP, the state, action, and transition model for each individal variable mst be defined, along with the reward/cost strctre. This is done differently for the thermal generators which are natrally represented by means of discrete variables, and for the demand and ncontrollable generators which are natrally represented by means of continos variables. For the fmdp part corresponding to thermal generators, the definitions of state and action variables coincide with those in the original seqential decision problem described in Section. For the continos variables, we se a discretization method based on barycentric coordinates that we have already applied to other seqential decision problems sch as train rncrve optimization and set-point schedling for air conditioners (Nikovski and Esenther 0; Nikovski et al. 0; Nikovski, X, and Nonaka 03). The main idea of the method is to replace the continos state variables with a discrete set of states in a way that approximates well the original continos dynamics. Let the dynamics of a continos component of the model be represented by the fnction z t+ = f z (z t, a t ), where z t is a vector variable that cold inclde one or more of the demand d t, the otpt of ncontrollable generators yt k, or some of their time-lagged vales d t, yt, k etc. Let the dimensionality of this vector be b. The objective of the conversion method is to represent the dynamics of the continos system z t+ = f z (z t, a t ) by a conditional probability transfer fnction P r(s t+ = s (j) s t = s (i), a t = a (k) ), defined over sitably chosen set S of N discrete states s (i), k K. The algorithm selects N states s (), s (),..., s (N) sch that each corresponds to a state z (i) R b, and their Delanay trianglation is compted (Fig. ), (Preparata and Shamos 990). Then, each available action a (l) is exected in each of them in trn, according the continos dynamics fnction z = f z (z (i), a (l) ), and the barycentric coordinates p, p,..., p b+ of the end state z are compted with respect to the simplex that encloses it. These barycentric coordinates are then sed as transition probabilities of the discrete MDP. The detailed comptational procedre, along with discssion of its comptational complexity, is available in (Nikovski and Esenther 0). Conceptally, we can think of this algorithm as a way of converting the system dynamics represented by the fnction f z to an eqivalent probabilistic representation involving only a small set of points s (i) embedded into the original continos state space of the system. If the system starts in one of these few points, the sccessor state z, in general, will

7 z p p 3 z' p large joint state space, its transition strctre is very sparse. The next step is to determine the transition cost, which, nlike transition probabilities that can be specified separately for each individal variable, mst be specified for the entire MDP. Given a joint MDP state ( t, x t, y t, d t ) and an action t+, the immediate one-step cost c( t, x t, t+, y t, d t ) is compted as z (i) Figre : A Delanay trianglation on a set of vertices sampled from the embedding two dimensional space. The dashed line shows the transition from some starting state z (i) nder action a reslting in end state z = f(z (i), a). The simplex (here, triangle) containing the end state z is shown with a dotted backgrond, and the barycentric coordinates p, p, and p 3 of z are compted with respect to the vertices of that simplex. These coordinates are also the transition probabilities from z (i) nder action a to the states corresponding to these vertices in the reslting MDP. not coincide with another one of these points. However, we can identify the b + points that define a simplex that completely encloses the sccessor state z, and can think that the system has transitioned not to point z itself, bt to the vertices of this simplex with varios probabilities, instead. The probabilities are eqal to the convex decomposition of point z with respect to the vertices of the simplex, also known as the barycentric coordinates of that point within the simplex. The similarities between convex combinations (barycentric coordinates) and probability mass fnctions reqired by the MDP formalism make this conversion possible. This procedre is applied in trn for every grop of variables in the DBN that have temporal dependence. For the demand variable D, we cold assme that the next demand D t+ depends only on the crrent demand D t (Markovian property of the nderlying stochastic process) with transition probability P r(d t+ = d t+ D t = d t ), and if a higherorder model is necessary, time-lagged vales of demand D t, D t, etc. cold be inclded. For the ncontrollable generators, we make similar assmptions that Yt+ k depends only on Yt k, with probability P r(yt+ k = yt+ Y k t k = yt k ). These transition probabilities can be estimated either from statistical data, or by means of discretizing a sitable continos stochastic Markov process, sch as the ato-regressive process of order (AR() process). Once the transition probabilities for all variables in the DBN have been determined, joint transition probability for the entire system P r( t+, x t+, y t+, d t+ t, x t, y t, d t ) can be compted from the transition probabilities of the individal random variables, as is cstomary for Bayesian networks. It can be observed that althogh the MDP has a very z c( t, x t, t+, y t, d t ) = N i= f i(x i t, i t, y t, d t ) + N i= h i(x i t, i t, i t+) + g t ( t, y t, d t ) (4) where the switching costs h i (x i t, i t, i t+) and risk cost g t ( t, y t, d t ) are compted as described above, and the fel costs f i (x i t, i t, y t, d t ) are compted by solving the following economic dispatch problem: minimize i F i(p i t) sbject to the generation limits for all generators and the load balance constraint for this particlar realization of the ncontrollable variables y t and demand d t : N i = i tp i t + K yt k d t = 0 k= where F i (p i t) is the cost of prodcing p i t nits of electricity by generator i; typically, this fnction is qadratic in p i t, and the economic dispatch problem can be solved by means of qadratic programming. The objective of economic dispatch is to find the optimal generation amonts p i t of the committed nits so that the cost of generation is minimized for a specific realization of the random variables. After the optimal generation amonts [p t, p t,..., p N t ] are fond, the individal generation costs can be calclated as f i (x i t, i t, y t, d t ) = F i (p i t), i N. Given sch an MDP, we can define its cost-to-go fnctions J t for each step t and each joint state of the MDP. For the terminal step T, when no frther decisions will be made, J T ( T, x T, y T, d T ) = 0. For all other steps, the cost-to-go fnction J t ( t, x t, y t, d t ) is defined iteratively by means of a Bellman eqation, as follows (Pterman 994): J t ( t, x t, y t, d t ) = min t+ {c( t, x t, t+, y t, d t ) + d t+,y t+ P r(d t+, y t+ d t, y t )J t+ ( t+, x t+, y t+, d t+ )} (5) Note that the transition probabilities P r(d t+, y t+ d t, y t ) are factored conveniently, de to the conditional independence relations in the DBN of the MDP. The cost-to-go fnction J 0 ( 0, x 0, y 0, d 0 ) of the initial state of the generators and demand wold then correspond to the minimal operating cost nder the optimal policy for the entire planning problem. In principle, this cost can be fond by compting the costs-to-go of all states in the MDP. However, when some of the variables are continos, the cost-to-go (vale fnction) of the MDP cannot be compted and represented efficiently. The discretization method described above addresses precisely this problem, by replacing the continos variables D and Y k with sets of discrete states S, making the entire

8 MDP discrete, and standard MDP soltion methods sch as dynamic programming, vale iteration, and policy iteration can be applied (Pterman 994). A soltion method based on dynamic programming over AND/OR trees is described in the next section. Frthermore, if these costs are compted and stored, the optimal decision t+ = π t ( t, x t, y t, d t ) for time step t and state ( t, x t, y t, d t ) can be identified as the one that minimizes the right-hand side of the Bellman eqation 5: π t ( t, x t, y t, d t ) = argmin t+ {c( t, x t, t+, y t, d t ) + d t+,y t+ P r(d t+, y t+ d t, y t )J t+ ( t+, x t+, y t+, d t+ )} that the target demand is ( + β) D. Sitable schedles S β (6) This policy is conditioned pon the crrent realizations of the random variables y t and d t, so it represents a conditional planner. By observing the otcomes y t and d t for each consective time step, different actal operating schedles will be obtained. Solving fmdp Models with Aggregated Net Demand The objective of solving the stochastic nit commitment problem represented by the fmdp is to find the optimal policy that maps the states of the fmdp onto the decision variables that signify which generators will be trned on/off in the next period, where optimality is defined in terms of jointly minimizing prodction cost and risk of failre. The straightforward method of solving fmdps is to expand the factored state and solve the reslting flat MDP by means of dynamic programming, applying eqation 5 repeatedly, starting from the terminal step and proceeding backwards to the first step (Pterman 994). However, for most practical problems, e.g. when L = l = 5, the nmber of generators N = 0, the nmber of one-hor time periods T = 4, the expanded MDP will have X = T (L + l) N = distinct states for the controllable generators only, and wold be impossible to solve. One practical simplification of the problem is to aggregate the otpt of the ncontrollable generators Y t into the demand variable, by sbtracting these otpts from the total demand D t to arrive at the net demand D t. If all ncontrollable random variables are Gassian processes, then D t is a Gassian process, too, with expected vale (mean) D t and variance σ t for each time period t. Henceforth, we will assme that D t denotes the net demand. For planning prposes, the net demand D t can be compted by sbtracting the expected vales Ȳt at the time of planning (t = 0). When execting the policy, the actally observed realizations y t at time t can be sed to estimate the distribtion of the random variable D t+, so that the estimates of the transition probabilities P r(d t+ d t, y t ) will in fact be based on y t, when determining the optimal configration t+ by means of Eqation 6. Another comptational simplification of the problem is to redce the size of the MDP in a reasonable manner. Intitively, if forecasts for the vales of the continos random variables D t and Y t are known in advance, and the assmption that these are Gassian processes holds tre, most of the configrations of the generators t at time t wold be irrelevant to satisfying demand at that time. Some of them will have capacities too low to meet demand, and others will se nnecessarily many generators to meet demand economically. By considering only configrations t of the controllable part of the MDP whose maximal committed capacity (MCC) is close to the expected net demand D t, we can drastically redce the size of the space of the MDP. A practical way of identifying sch sitable configrations is to rn a fast deterministic algorithm for nit commitment for several possible vales of target reserve β sch are identified for each β, and the generator configrations t present in S β are inclded in the redced state space of an approximate solver, which essentially switches between individal segments from mltiple schedles S β, depending on the time evoltion of power demand and ncontrollable generators. Hence, the fndamental idea of the soltion algorithm is to identify sitable configrations for representative demands, and then se them to prodce schedles for any possible realization of demand. We se an AND/OR tree (Martelli and Montanari 973) to represent all selected configrations of the generators and possible realizations of ftre demand. The AND/OR tree is then sed for planning for any demand instances. The algorithm is otlined in Table. Before discssing its details, we remark that commitment schedle specifies whether a generation nit is on or off. A commitment schedle may specify the on/off stats of the nits over all time steps. When restricted to a particlar step, it specifies the nit stats at that step. For convenience, in the rest of the paper, a schedle refers to a commitment schedle nless stated otherwise.. Pick a set of demand instances and solve UCs to obtain schedles. Use the schedles to bild an AND/OR tree 3. Retrn the AND/OR tree for planning Table : The algorithm to solve the factored MDP Generating Candidate Schedles This step identifies a finite set of representative commitment schedles so that they can be resed in the remaining steps. To solicit schedles, we first select a set of demand samples in hope that they are representative ones. For each slected demand, a deterministic UC problem associated with the demand is solved to obtain its schedle. We start identifying demand samples by finding the overall pper and lower demands of interest. Let the mean of the demand D be D = [ D, D,..., DT ]. Starting from a large positive nmber β and decreasing it gradally, we find a demand ( + β) D whose UC is feasible. This demand is the pper demand. Similar procedre can find the lower demand. The two demands determine a demand interval. The schedle generation procedre performs search in the interval and finds demands and their soltion schedles.

9 An iterative search procedre is as follows. Given a lower demand d and pper demand d, a new demand d(= (d + d)/) is created. (The demands d and d are called parent demands.) Its asscoiated UC problem U C(d) is solved. If UC(d) and its parent UC(d) have different schedles, their average demand (d + d)/ is added to consideration and the interval [d, d] is added to ftre search; otherwise, the interval [d, d] is discarded. By bookkeeping a demand and their parents, the procedre knows what interval it is searching. The advantage of this search strategy is that it focses on the regions that lead to distinct schedles. This is contrast to an evenly split approach that searches an interval [d, d] in a niform manner. The niformity in demand interval search does not necessarily mean schedle distinction. The demands selected in iteratvie interval search preserve the pward/downward trend in the demand vector over time steps. A pward or downward trend is that the demand mean at the next step is larger or smaller than that at the crrent step. Since the lower and pper demands are proportional to the mean iterative interval search and the newly created demands are the arithmetic average of their parents demands, the trends are preserved. To diversify the set of the selected demands, we add some randomized schedles to the schedle set. Specifically, we randomly change the commitment schedles for a small portion of the identified schedles. The psedo-codes that implement those ideas are presented in Table. The constant MinWidth determinines the size/width of the intervals to be discarded. The other constant MaxNm is the maximm nmber of schedles to find. They are initialzied at Line. A qee data strctre Q holds the processed and to-be-processed UCs. The pointer of the qee is Q_ptr. For a demand d, we se UC(d) to refer to its associated UC problem and (d) to refer to the schedle of UC(d). Line initializes the qee and the schedle set U to be empty. Line 3 generates the maximm demand and adds the UC to the qee. Line 4 generates the minimm demand and adds the UC to the qee. Line 5 prodces the first child demand and adds its UC to qee. It is pointed by the qee pointer at Line 6. Line 8 solves the UC. Line 9 loads the parent demands d and d. Line 0 examines the schedle of UC(d) and its pper parent UC(d). If they are different, or they are the same bt the interval is larger than the preset MinWidth, a new demand and its UC are created and added to the tail of the qee at Line. Lines 3-5 check the other parent likewise. Line 6 moves the pointer forward. Sch a process terminates if the pointer points to nll (no more UCs in the qee) or the maximm nmber of distinct schedles have been achieved (Line 7). Line 8 copies the set U with the identified schedles stored in the examined UCs of the qee. Line 9 pertrbs the schedles by adding some randomization. Finally, Line 0 retrns the schedles in U. Bilding the AND/OR tree An AND/OR tree has two types of nodes AND nodes and OR nodes. An AND/OR tree is a tree where () its root is an AND node, () it has alternating levels of AND and OR nodes, and (3) its terminal nodes are AND nodes. MinWidth constant, and MaxNm constant. Q, U 3. Find pper demand d; Solve UC(d); Add UC(d) to Q 4. Find ower demand d; Solve UC(d); Add UC(d) to Q 5. Add UC(d)(d = (d + d)/) to Q 6. Q_ptr 3 7. While Q_ptr nll && Q_ptr MaxNm 8. Solve UC(d) pointed by Q_ptr 9. (d, d) the parent demands of UC(d) 0. If differ((d), (d)) (same((d), (d))&& d d MinWidth). Add UC(d)(d = (d + d)/) to Q. End If 3. If differ((d), (d)) (same((d), (d))&& d d MinWidth) 4. Add UC(d)(d = (d + d)/)) to Q 5. End If 6. Q_ptr End of While 8. U UC schedles from Q to Q_(ptr ) 9. Pertrb U 0. Retrn U Table : Generating schedles to be resed in the remaining algorithmic steps (, x, d ) (, x, d ) (, x, d ) (, x, d ) (, x, d ) (, x, d ) (, x, d ) (, x, d ) (, x, d )... (, x 7, d ) (, x 7, d ) (, x 8, d ) (, x 8, d ) Figre 3: An AND/OR tree example (Martelli and Montanari 973). An AND/OR tree is shown in Fig. where the AND/OR nodes are respectively in rectanglar/circlar shapes. Note that in this case the otpts of the ncontrollable generators Y t have been aggregated into the net demand variable D t, and are not inclded in the AND/OR tree. An AND node for the UC problem is associated with a system state ( t, x t, d t ) at time step t, whereas an OR node is associated with the action t at that time. The root node corresponds to the initial state of the UC system. The vales of the nodes are evalated bottom-p. For an OR node t+, if its parent AND node is ( t, x t, d t ) and its children (AND) nodes are {( t+, x t+, d t+ ) d t+ }, then the vale of the OR node is evalated as

10 V t ( t+ t, x t, d t ) = c( t, x t, t+, d t ) + d t+ p(d t+ d t )V t+ ( t+, x t+, d t+ ) (7) Note that the notation V t ( t+ t, x t, d t ) means that the vale of OR node is conditional on its parent AND node. For an AND node ( t, x t, d t ), its vale V t ( t, x t, d t ) is evalated as follows: V t ( t, x t, d t ) = { c(t, x T, T, d T ), if t = T min t+ V t ( t+ t, x t, d t ) otherwise Note that the minimization in min t is over all children OR nodes t+, and that no configration switching cost is incrred at the last step, since the contination of the schedle at that time is yet nknown. Now we are ready to explore the AND/OR tree for planning prpose. It takes two steps bilding the AND/OR tree and constrcting the schedles for planning.. The root node ( 0, x 0, d 0 ) is the initial state of the system. It is the first node of the tree. It is the node at Level 0. Since the levels correspond to the time steps in a UC, we se steps to refer to levels. The rest of the tree is bilt over time steps: the AND nodes at Step t and an OR node are sed to bild AND nodes at Step t +. The OR nodes are the schedles generated at Step of the entire algorithm (Table ). For a node ( t, x t, d t ), we se every generated schedle to prodce AND nodes at the next time step. Let the OR node be t+. The stats of the nits at next time step is ( t+, x t+ ). Since the demand at t + is ncertain, we generate an AND node for every possible demand. So the set of next AND nodes is {( t+, x t+, d t+ d t+ )}. The process repeats ntil completion at Step T. The psedo-codes for this process are presented in Table 3. Let the notation N A t and be the AND nodes at Step t, and N O t be the schedles obtained from schedle generation bt restricted to Step t.. Initialize the root node N A 0 to the initial system state. For t =,..., T 3. N A t 4. N O t schedles from schedle generation 5. For each node ( t, x t, d t ) in N A t 6. For each schedle t in N O t 7. For each demand d t 8. Node ( t, x t, d t ) is generated 9. If the UC is feasible 0. N A t N A t {( t, x t, d t )}. End if feasible. End for each demand 3. End for each schedle 4. End for each node n 5. End for t 6. Retrn the tree represented by {N A 0, N A,..., N A T }. Table 3: Bilding an AND/OR tree. The AND and OR nodes in the tree are evalated by Eqations (7) and (8). In evalating the non-terminal AND nodes, there mst be an OR node that achieves the minimm in Eqation (8). The action represented by that OR node is the best action of the system state represented by the parent AND node. Evalating MDP Policies Once a policy has been compted and stored in the AND/OR tree, we adopt a sampling approach to evalate its operational cost and risk nder ftre random demand D. (8) For this prpose, we draw a sitable nmber of samples d = [d, d,..., d T ] from the demand variable D (e.g., 000 samples). For each sample, we start from the root of the tree and execte the actions specified by the tree. Sch an exection reslts in a path in the tree. Specifically, an exection path is a seqence of system states and actions {( 0, x 0, d 0 ),, (, x, d ),..., T, ( T, x T, d T )} that are prescribed by the initial system state, the AND/OR tree, and the demand realization d = [d, d,..., d T ]. The cost of a path can be accessed by solving the economic dispatch problem for each step, given the prescribed configrations t, while its risk can be calclated sing the committed capacity t and the realization of demand d t. The overall risks and costs are the average across the paths associated with the demand samples. These costs and risks show how the risks can be compromised by the additionally paid cost. Psedo-code of the simlation procedre is presented in Table 4.. For each demand sample. Determine the exection path from the AND/OR tree 3. Solve the UC based on the path to get the cost 4. Calclate the risk based on the path 5. Sm p the cost 6. Sm p the risk 7. End For each demand 8. Calclate the average cost and average risk 9. Retrn the average costs and risks Table 4: Simlating the cost and risk of an MDP policy Complexity Analysis The most expensive part of the algorithm is the bilding and evalation of the AND/OR tree, becase it is exponential in the planning horizon T, where the base of the exponent is the nmber of discrete levels of discretization for the demand variable D t. However, when an AND node is added to the tree, a feasibility check is performed first: a node is added only when it meets all temporal constraints pls the demand and load constraints, and the economic dispatch associated with the node has a feasible soltion. Experimental Reslts We experimented with the proposed method on a test problem adopted from (Li, Johnson, and Svoboda 997), extended with the introdction of ncertainty in the demand.

11 The standard deviation of demand was assmed to be % of expected demand: σ t = 0.0 D t. No ncontrollable generators were sed, so the net demand is eqal to the total demand. The approximate algorithm from the previos section was implemented and compared against two existing algorithms: one of them was based on a priority list ((Wood and Wollenberg 996)), and the other one was the decommitment algorithm proposed in (Li, Johnson, and Svoboda 997). Or reslts showed that the approximate soltion method provides a good balance between generation cost and risk of failre to meet demand. We performed experiments on two UC examples: one with 4 nits, and another one with 0 nits. We were able to calclate the trly optimal MDP soltion for the 4-nit UC example, so we were able to investigate the accracy of or approximation scheme on that problem, too. The experiments were performed on a compter with Intel Core Do E6600 CPU (.40GHz). The algorithm was implemented in MATLAB. Experimental Conditions The generation cost of a committed nit i at time t is compted as a qadratic fnction of the prodced amont of power by the nit: f i (x i t, i t, d t ) = c i 0 + c i p t i + c i (p i t). The nit switching and start-p cost is expressed as h(x i t, i t, i t+) = tcst i + bcst i ( exp( γx i t)), if i t = 0 and i t+ =, and zero otherwise. In the start-p cost, the fixed component tcst i represents the cost of starting generator i, while the second term bcst i represents the cost of starting the boiler and varies exponentially with the length of the time that the nit has been off. Under a Gassian assmption for demand (D t N( D t, σt )), the risk compensation cost g t ( t, d t ) is given by ˆ α C F SO exp( (D D t ) i i t πσ capi t σt ) dd where α is the proportionality constant, C F SO is the fll system operating costs (the cost of the system in which all nits are trned on and generate according to their maximm capacity), and the integral is the failre probability (risk). Failre happens when the actal demand D is greater than the Maximm Committed Capacity (MCC) i cap i i t of all operating nits. By increasing the constant α, the weight of the risk component in the objective fnction is increased, ths favoring configrations with higher MCC, at the expense of a higher operational cost for rnning sch configrations. A 4-nit example The decision horizon of the 4-nit UC problem was 4 hors. The coefficients tcst i and bcst i of the start-p costs for the for nits were [00,000;500,0000;00,700;44,00]. The fel cost coefficients [c 0, c, c ] for the for nits were [0.00,6.5,0.7; ,.05,33.6; 0.007,.6,37.0; ,5.9,660.8], in chosen cost nits. The minimm p and minimm down Risk Conditional exact Conditional approximate Decommitment Priority list Effective extra cost (Percentage) Figre 4: Performances of the algorithms on a 4-nit problem times were [3, 3,, ] and [4, 4, 3, 3]. The minimm and maximm capacities were [0, 0, 0, 0] and [00, 90, 80, 60], here and henceforward, in chosen power nits. The expected demand vector was D =[05,85,65,40,00,05,5,45,65,85,05,45,65,85, 00,40,00,05,5,45,65,85,05,5]. The initial operational times were x 0 = [5, 5, 5, 5]. The risk verss cost crves for varios methods are presented in Fig. 4. Conditional exact refers to the algorithm that solves the MDP exactly, i.e., all Bellman backps (Eqation 6) were performed. Conditional approximate refers to the algorithm proposed in the previos section. In the figre, the horizontal axis is the percentage of the extra operational cost with respect to a reference operational cost, taken to be the lowest experimentally obtained operational cost for any schedler on this problem. For this problem instance, it can be seen that the soltion of the proposed algorithm is very close to optimality (the conditional exact soltion), and the algorithm otperforms significantly both the priority list and the decommitment algorithms in balancing operational costs and risks. For the lowest levels of risk, which are probably close to the desired cost/risk trade-off point of an actal generation system, the loss of optimality is less than %, whereas the gain in costs with respect to deterministic schedlers is greater than 9%. 0-nit example In this experiment we sed all 0 generators described in (Li, Johnson, and Svoboda 997). The expected demand vector was [33.3, 33.3, 066.7, 066.7, 33.3, 33.3, 66.7, 400.0, 400.0, 400.0, 333.3, 00.0, 33.3, 33.3,00.0, 66.7, 400.0, 400.0, 400.0, 400.0, 333.3, 00.0, 00.0, 066.7]. It was no longer possible to find the trly optimal conditional schedles, bt it is possible to compare the performance of the conditional approx-

12 Risk Conditional approximate Decommitment Priority list Effective extra cost (Percentage) Figre 5: Performances of the algorithms on a 0-nit problem imate, priority list, and decommitment algorithms (Fig. 5). Again, the reslts show that the proposed novel algorithm niformly achieved a mch better risk/cost balance than the priority list and the decommitment approaches, with operational cost savings arond 4% for the lowest levels of risk. Conclsion and Ftre Work We have described a general method for representing the mixed continos/discrete dynamics of power generation systems nder mltiple sorces of ncertainty sch as variable power demand and intermittent renewable energy sorces, and have introdced a class of conditional operational plans where the nit commitment decisions are conditioned pon the state of observable random variables. The proposed factored Markov decision process models represented in the form of dynamic Bayesian networks are compact and are also easy to specify, maintain, and extend with new power sorces. We have also proposed one concrete algorithm for finding sch conditional operational schedles for power generation that depend on a single random variable the net demand that aggregates in itself all sorces of randomness. The algorithm focses on small sbsets of all possible configrations of generators in order to compte the schedle efficiently. Experimental reslts sggest that the reslting conditional plans are close to the trly optimal ones, and provide a mch better trade-off between generation cost and risk of failre to meet demand than two known nonstochastic nit commitment algorithms that compte fixed schedles. In the proposed soltion algorithm, we se AND/OR trees to represent, find, and evalate the optimal conditional plan. However, this algorithm is by no means the only possible way to solve stochastic generation problems represented by means of fmdps and DBNs. In ftre work, we plan to investigate other soltion methods based on approximate dynamic programming that cold reslt in mch better comptational complexity. Frthermore, the crrent soltion aggregates the variability of all stochastic variables into the net demand to the controllable power generators, for the sake of comptational efficiency. This simplifies the planning problem, becase the branching in the AND-OR tree is based only on that single variable. However, even higher efficiency might be possible if the conditional schedle is conditioned on the vales of each individal stochastic component. This wold significantly increase the complexity of the planning process, and wold depend critically on finding more comptationally efficient soltion methods for the ndrelying fmdp models. For example, the method proposed in (Feng et al. 004) represents the vale fnction of the dynamic programming problem over continos domains by adaptively discretizing sch continos variables. This approach might reslt in more accrate and compact representations than are possible with or method, where the tesselation of the continos domains is performed apriori, before vale fnctions are evalated. Adaptive discretization is indeed compatible with or discretization scheme, too, for example by sb-dividing a simplex where the vale fnction varies a lot (measred on its vertices), into mltiple smaller simplices. The application of symbolic dynamic programming (SDP, (Sanner, Delgado, and de Barros 0)) to the factored MDP-based formlation of the operational planning problem might be possible, too. The formlation of the fmdp described in the paper assmes that all generators assme their intended configration t i withot fail. This allows s to se the decision variables t i as components of the state of the system, ths simplifying the planning process. If the possibility of eqipment failre mst be taken into accont, the actal configration Ui t of the generators shold be inclded as a random state variable in the DBN, and its probabilistic dependence on the intended configration t i can be modeled according to the failre probabilities of individal generators. Sch an extension is completely compatible with the proposed modeling formalism of factored Markov decision processes. References Botilier, C.; Dearden, R.; and Goldszmidt, M Stochastic dynamic programming with factored representations. Artificial Intelligence Feng, Z.; Dearden, R.; Melea, N.; and Washington, R Dynamic programming for strctred continos markov decision problems. In Proceedings of the 0th conference on Uncertainty in Artificial Intelligence, UAI 04, Arlington, Virginia, United States: AUAI Press. Li, C.; Johnson, R. B.; and Svoboda, A. J A new nit commitment method. IEEE Transactions on Power Systems 3 9. Martelli, A., and Montanari, U Additive AND/OR graphs. In Proceedings of the Third International Joint Conference on Artificial intelligence,. Nikovski, D., and Esenther, A. 0. Constrction of embedded Markov decision processes for optimal control of