Stochastic Manufacturing & Service Systems. Discrete-time Markov Chain

Transcription

1 ISYE 33 B, Fall Week #7, September 9-October 3, Introduction Stochastic Manufacturing & Service Systems Xinchang Wang H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Discrete-time Markov Chain Newsvendor-type models, including profit maximization and cost minimization models, are specifically developed for controlling inventory levels of perishable products, for which you get marginal (or negative) salvage values for leftovers. One immediate question is: So, what about non-perishable products, such as cars, steal, and furniture? If you are making order/production plans based on months, you do not want to just simply get rid of leftovers at the end of a month; the leftovers are still valuable. For sure, the leftovers will be put on market for the next month if any just like brand new products. We can use leftovers to fill the demand for the next month. It is often the case that we also need to order new products which together with the leftovers to fill the demand for the next month. Yes, we need a new method to model this problem. Notice that we only care about one period for newsvendor problems and our order/production decision (e.g., order or not) is pretty much a one-time deal. For non-perishable products, a product could be put on market for more than one periods until when it is sold. We could have leftovers for every period and we need to make order/production decisions at the end of each period. The decisions are simply to order or not, and if order, how many to order? We have multiple periods to take care of and we are interested in comping up with the best order policy that make our business most profitable for long time. Here is another question: In what sense we can judge a decision policy to be the best? In other words, what policy/policies should be thought of as the best? We need a evaluation criterion. The criterion is the Long-run Average Profit. If an order policy gives us the maximum long-run average profit, that policy is the one that we are seeking for and that we call the best order policy. Let Π be the set of possible order policies. Let R π denote the long-run average profit under policy π Π. We are looking for some policy π such that R π R π () for all other π s. As you can see, the long-run average is only defined after we specify a policy to be used, i.e., we can only make sense of R π after we know π. How to solve ()? A intuitive idea is that for each possible policy π, we compute R π and we obtain the best π that gives us the biggest R π. For example, (a) Policy π gives R π = $.

2 (b) Policy π gives R π = $. (c) Policy π 3 gives R π3 = $5. So, choose policy π!. Long-run Average Profit Consider a business for selling non-perishable products. Our objective is to define what the long-run average profit is under a policy π. We suppress π for ease of notation. Start with a bunch of symbols: (a) n: periods: n =,,, 3,.... A period can be a day, week, month and year (light years too?). (b) X n : the # of products you have on hand at the end of period n =,,,...; X n is also called the inventory level at the end of period n. X is the initial inventory level before you start doing your business and is assumed to be known. (c) D n : the demand for period n =,,.... {D n } n= is assumed to be an i.i.d. sequence of rv s. First, I wanna reason that {X n } n= the following story given by steps. is a sequence of rv s, dependent on policy π. Consider Step. You have x = products at the end of period n =, i.e., X =. Step. You decide not to order. Step 3. Then, how many products you have on hand at the beginning of period n + = 3?! Step. Now, the inventory level, X n+, at the end of period n + = 3 is dependent on two things: () products on hand () the demand D n+. You can verify that X n+ = ( D n+ ) +, which is exactly the # of leftovers we calculated in the newsvendor problem. Step 5. A theory tells us that consider any rv X and a (strictly speaking, measurable function) function g. Then g(x) is a rv too. Thus, X n+ = ( D n+ ) + is a rv because D n+ is a rv. Step 6. We can conclude that {X n } n= is a sequence of rv s. Step 7. It follows from Step that X n is dependent on making the order or not. Yes, it is dependent on the order policy π. Remark.. {X n } n= is dependent on policy π and it becomes well defined only after π is given. Remark.. {X n } n= is a sequence of random rv s. We have a terminological name for this sequence of rv s, {X n } n= : Stochastic Process. Definition.. A sequence of random variables with a discrete set of indices is said to be a discrete-time stochastic process.

3 Second, let s compute the profit for each period n. Since we need to find a way to define the long-run average, we have to start with the profit for each period. Question: What is the profit for period n? Let R n denote the profit for period n. At the end of last period n, assume that we have X n = x leftovers and our policy π tells us not to order (for the ease of illustration). We ll keep x to the beginning of period n. The demand in period n is D n. (a) What we can sell in period n: (x D n ) (b) Profit we get: R n = c p (x D n ). Remember that the leftovers will be put on market for the next period and will become part of the inventory at the beginning of the next period. So, we do not have salvage value contributed to the profit for this period. You see, we have R n for period n. Do this for each period. Now, we are ready to define the long-run average profit R is represented by R = lim n R + R R n. n One last thing: we can remove the assumption that X n = x, i.e., we can allow the inventory level to be random (and it is random) and we can still get the above expression for the long-run average profit. In that case, R n = (X n D n ) +. Now, as you can observe here, the long-run average profit R is computed as the average of the profits for all periods in the long-run, and more importantly, it is dependent on X, X,..., X n,.... Remark.3. The long-run average profit R is dependent on {X n } n= Go back to our aim: to solve ()! It follows from Remark. and.3 that, we have the following chain: (a): understand (b): Compute a policy π {X, X,..., X n..., } R Thus, we have steps to approach our goal. A month of lectures will be given just for step (a): understand X n given a policy π. But I will be back to step (b) and solve () after we understand X n.. A Popular (s, S) policy A popular type of inventory control policies is as follows: Remember X n is the inventory level at the end of period n. Say if X n > do not order. If X n, order up to S = items. This type of policies is called (s, S) policy. In general, we can define it as. If X n s, order S X n items.. Otherwise, do not order. This is very popular policy. Virtually every company has some version of this policy. We only restrict our attention on this type of policies, called (s, S) policies. In other worlds, we will understand X n under a (s, S) policy. 3

4 An Inventory Model for Non-perishable Products Example. (Inventory Model for Non-Perishable Item). D n is the demand in the nth period. Note that inventory that is left at the end of a week can be used to satisfy the demand in the following week. For example, {D n } is an iid sequence. d 3 P(D n = d) How do you analyze the performance of an (s, S) policy? Every Friday 5pm, let s say we decide how much to order for the following week so that the ordered items will arrive at am (or arrive immediately by assumption) the following Monday.. Matrix Representation of a Bunch of Transition Probabilities We begin analysis with building a table. Assume i.i.d. demand d 3 P(D = d) / / / / In the real world, demand is usually not i.i.d. There could be some seasonality. But, in some business like WalMart selling items as cheap as possible, demand can be modeled as i.i.d.. Let X n be inventory level at the end of week n. Note that values that X n can take is in {,,, 3, } (why?). Question.. What is the value for P(X n+ = 3 X n = ) =?,! Question.. How about the following probability? P(X n+ = 3 X n = ) = P(D n+ = ) = Note that we will order according to our inventory policy over the nth weekend, so the demand for (n + )th week should be. Matrix is a good way to denote these probability in a neat form. 3 5 Some properties that I want you to notice: 3 := P (a) Row index: from what state the system transitions (b) Column index: to what state the system transitions (c) P = (p ij ) i,j S satisfies that p i,j [, ]. (d) Each row sums to unity. (e) Each column may not sum to unity.

5 3 Formal Definition of a Discrete-time Markov Chain (DTMC) Let us formally define discrete-time Markov Chain. A discrete-time Markov Chain has the following elements. (a) State space S, e.g. S = {,,, 3, }: You will see that S does not have to be finite. A state presents a possible value that X n can take. The state space is the set of all such values. (b) Transition probability matrix P = (p ij ) such that p ij and j S P ij = : This is the matrix you just saw above. (The sum of row should be but the sum of column does not have to be.) (c) Initial state (distribution): This is how much inventory you are given at the starting point. It is the information about X. Definition 3. (Discrete Time Markov Chain). A discrete time stochastic process X = {X n : n =,,, } is said to be a DTMC on state space S with transition matrix P if for each n for i, i, i,, i, j S P(X n+ = j X = i, X = i, X = i,, X n = i n, X n = i) = P ij. () The most important part of this definition is (). At this point, let us recall the definition of conditional probability. P(A B) = P(A B) P(B) = P(A, B) P(B) This () is called the Markov property. In plain English, it says that once you know today s state, tomorrow s state has nothing to do with past information. No matter how you reached the current state, your tomorrow will only depend on the current state. In mathematical notation, P(X n+ = j X = i, X = i, X = i,, X n = i n, X n = i) = P(X n+ = j X n = i).. Past states: X = i, X = i, X = i,, X n = i n. Current state: X n = i 3. Future state: X n+ = j 5