CS885 Reinforcement Learning Lecture 3b: May 9, 2018
|
|
|
- Joanna Nichols
- 5 years ago
- Views:
Transcription
1 CS885 Reinforcement Learning Lecture 3b: May 9, 2018 Intro to Reinforcement Learning [SutBar] Sec , , 6.5, [Sze] Sec. 3.1, 4.3, [SigBuf] Sec , [RusNor] Sec , CS885 Spring 2018 Pascal Poupart 1
2 Markov Decision Process Definition States:! # Actions: $ % Rewards: & R Transition model: Pr(! +! +-., $ +-. ) Reward model: Pr(& +! +, $ + ) Discount factor: discounted: 3 < 1 undiscounted: 3 = 1 Horizon (i.e., # of time steps): h Finite horizon: h N infinite horizon: h = Goal: find optimal policy : such that : = $&<=$>? C +AB 3 + D? [& + ] CS885 Spring 2018 Pascal Poupart 2
3 Reinforcement Learning Problem Agent State Reward Action Environment Goal: Learn to choose actions that maximize rewards CS885 Spring 2018 Pascal Poupart 3
4 Reinforcement Learning Definition States:! # Actions: $ % Rewards: & R Transition model: Pr(! +! +-., $ +-. ) Reward model: Pr(& +! +, $ + ) Discount factor: discounted: 3 < 1 undiscounted: 3 = 1 Horizon (i.e., # of time steps): h Finite horizon: h N infinite horizon: h = Goal: find optimal policy : such that : = $&<=$>? C +AB 3 + D? [& + ] unknown model CS885 Spring 2018 Pascal Poupart 4
5 Policy optimization Markov Decision Process: Find optimal policy given transition and reward model Execute policy found Reinforcement learning: Learn an optimal policy while interacting with the environment CS885 Spring 2018 Pascal Poupart 5
6 Example: Inverted Pendulum State:! ",! $ ", % ", % (") Action: Force ) Reward: 1 for any step where pole balanced Problem: Find *:,. that maximizes rewards CS885 Spring 2018 Pascal Poupart 6
7 Important Components in RL RL agents may or may not include the following components: Model: Pr # $ #, &, Pr(( #, &) Environment dynamics and rewards Policy: +(#) Agent action choices Value function:,(#) Expected total rewards of the agent policy CS885 Spring 2018 Pascal Poupart 7
8 Categorizing RL agents Value based No policy (implicit) Value function Policy based Policy No value function Actor critic Policy Value function Model based Transition and reward model Model free No transition and no reward model (implicit) CS885 Spring 2018 Pascal Poupart 8
9 Toy Maze Example r r r +1 u u -1 u l l l Start state: (1,1) Terminal states: (4,2), (4,3) No discount: % = 1 Reward is for non-terminal states Four actions: up (u), left (l), right (r), down (d) Do notknow the transition probabilities What is the value!(#) of being in state #? CS885 Spring 2018 Pascal Poupart 9
10 Model free evaluation Given a policy!, estimate " # $ without any transition or reward model Monte Carlo evaluation " # $ = & # [ ) * ) + ) ]. /(1) /(1) 34. ) * ) 3 + ) (sample approximation) Temporal difference (TD) evaluation " # $ = & + $,!($) + * 1 7 Pr $ : $,!($) " # ($ : ) + + *" # ($ : ) (one sample approximation) CS885 Spring 2018 Pascal Poupart 10
11 Monte Carlo Evaluation Let! " be a one-trajectory Monte Carlo target! " = % & % (") ' % Approximate value function +(0)!" * +, - / +(0) "1/ = / +(0) = / +(0)! +(0) + "1/ +(0)3/!",! +(0) + 4(-) 1 * +3/ -, = * +3/ - + / +(0),! +(0) * +3/ - Incremental update *,,, + - * +3/ ! + * +3/ - learning rate 1/4(-) CS885 Spring 2018 Pascal Poupart 11
12 Temporal Difference Evaluation Approximate value function:! " ($) ' + )! " ($ * ) Incremental update! " " + $! +-. $ + / + ' + )! " +-. ($ * " )! +-. $ Theorem: If / + is appropriately decreased with number of times a state is visited then! + " ($) converges to correct value Sufficient conditions for / + : (1) + / + (2) + / + 5 < Often / + $ = 1/:($) Where :($) = # of times $ is visited CS885 Spring 2018 Pascal Poupart 12
13 Temporal Difference (TD) evaluation TDevaluation(!, # $ ) Repeat Execute!(&) Observe & and ) Update counts: * & * & + 1 Learning rate:. 1/*(&) Update value: # $ & # $ & +.() + 0# $ & 1 # $ & ) & & Until convergence of # $ Return # $ CS885 Spring 2018 Pascal Poupart 13
14 Comparison Monte Carlo evaluation: Unbiased estimate High variance Needs many trajectories Temporal difference evaluation: Biased estimate Lower variance Needs less trajectories CS885 Spring 2018 Pascal Poupart 14
15 Model Free Control Instead of evaluating the state value fn,! " ($), evaluate the state-action value fn, & " ($, () & " $, ( : value of executing ( followed by * & " $, ( =, - $, ( + / $ 4 $, (! " ($ ) Greedy policy * 4 : * 4 $ = (-67(8 9 & " ($, () CS885 Spring 2018 Pascal Poupart 15
16 Bellman s Equation Optimal state value function! ($)! $ = max * +, $, , $ 5 $,.! ($ ) Optimal state-action value function 7 ($,.) 7 $,. = +, $, , $ 5 $,. max * 3 7 ($ 5,. 5 ) where! $ = 8.9 * 7 $,. : $ =.,;8.9 * 7 ($,.) CS885 Spring 2018 Pascal Poupart 16
17 Monte Carlo Control Let! " # be a one-trajectory Monte Carlo target Alternate between Policy evaluation! " # = % & (") + +,-. + % + (") a 9 / , 4 / 06-2, ! # 1 0 / 06-2, 4 Policy improvement 9 (2) 4%;<4= # / 1 (2, 4) CS885 Spring 2018 Pascal Poupart 17
18 Temporal Difference Control Approximate Q-function:! #, % = ' ( #, % + * +, - Pr # 0 #, % max 4 -! (# 0, % ) ( + * max 4 -! (# 0, % 0 ) Incremental update! 9 #, %! 9;< #, % + = 9 ( + * max 4 -! 9;< (# 0, % )! 9;< #, % CS885 Spring 2018 Pascal Poupart 18
19 Q-Learning Qlearning(!, # ) Repeat Select and execute % Observe! and ' Update counts: (!, % (!, % + 1 Learning rate:, 1/((!, %) Update Q-value: #!, % #!, % +, ' + 0 max 4 5 #! 6, % 6 #!, %!! Until convergence of # Return # CS885 Spring 2018 Pascal Poupart 19
20 Q-learning example s 1 73 s s g = 0.9, a = 0.5, & = 0 for non-terminal states ' ( ), &+,h. = ' ( ), &+,h. + 0 & + 1 max ' ( 5 6 7, 8 9 ' ( ), &+,h. = max 66,81, = (17) = 81.5 CS885 Spring 2018 Pascal Poupart 20
21 Q-Learning Qlearning(!, # ) Repeat Select and execute % Observe! and ' Update counts: (!, % (!, % + 1 Learning rate:, 1/((!, %) Update Q-value: #!, % #!, % +, ' + 0 max 4 5 #! 6, % 6 #!, %!! Until convergence of # Return # CS885 Spring 2018 Pascal Poupart 21
22 Exploration vs Exploitation If an agent always chooses the action with the highest value then it is exploiting The learned model is not the real model Leads to suboptimal results By taking random actions (pure exploration) an agent may learn the model But what is the use of learning a complete model if parts of it are never used? Need a balance between exploitation and exploration CS885 Spring 2018 Pascal Poupart 22
23 Common exploration methods e-greedy: With probability! execute random action Otherwise execute best action " " = "%&'"( ) *(,, ") Boltzmann exploration 3,) Pr " = 12 4 ) 1 2 3,) 4 CS885 Spring 2018 Pascal Poupart 23
24 Exploration and Q-learning Q-learning converges to optimal Q-values if Every state is visited infinitely often (due to exploration) The action selection becomes greedy as time approaches infinity The learning rate! is decreased fast enough, but not too fast (sufficient conditions for!): (1) #! # (2) #! # ) < CS885 Spring 2018 Pascal Poupart 24
25 Summary We can optimize a policy by RL when the transition and reward functions are unknown Model free, value based agent: Monte Carlo learning (unbiased, but lots of data) Temporal difference learning (low variance, less data) Active learning: Exploration/exploitation dilemma CS885 Spring 2018 Pascal Poupart 25
Introduction to Reinforcement Learning. MAL Seminar
Introduction to Reinforcement Learning MAL Seminar 2014-2015 RL Background Learning by interacting with the environment Reward good behavior, punish bad behavior Trial & Error Combines ideas from psychology
10703 Deep Reinforcement Learning and Control
10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department rsalakhu@cs.cmu.edu Temporal Difference Learning Used Materials Disclaimer: Much of the material and slides
Lecture 17: More on Markov Decision Processes. Reinforcement learning
Lecture 17: More on Markov Decision Processes. Reinforcement learning Learning a model: maximum likelihood Learning a value function directly Monte Carlo Temporal-difference (TD) learning COMP-424, Lecture
Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration
Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration Piyush Rai CS5350/6350: Machine Learning November 29, 2011 Reinforcement Learning Supervised Learning: Uses explicit supervision
Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration
Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration Piyush Rai CS5350/6350: Machine Learning November 29, 2011 Reinforcement Learning Supervised Learning: Uses explicit supervision
Reinforcement Learning. Monte Carlo and Temporal Difference Learning
Reinforcement Learning Monte Carlo and Temporal Difference Learning Manfred Huber 2014 1 Monte Carlo Methods Dynamic Programming Requires complete knowledge of the MDP Spends equal time on each part of
Reinforcement Learning and Simulation-Based Search
Reinforcement Learning and Simulation-Based Search David Silver Outline 1 Reinforcement Learning 2 3 Planning Under Uncertainty Reinforcement Learning Markov Decision Process Definition A Markov Decision
Markov Decision Processes
Markov Decision Processes Robert Platt Northeastern University Some images and slides are used from: 1. CS188 UC Berkeley 2. AIMA 3. Chris Amato Stochastic domains So far, we have studied search Can use
4 Reinforcement Learning Basic Algorithms
Learning in Complex Systems Spring 2011 Lecture Notes Nahum Shimkin 4 Reinforcement Learning Basic Algorithms 4.1 Introduction RL methods essentially deal with the solution of (optimal) control problems
COMP417 Introduction to Robotics and Intelligent Systems. Reinforcement Learning - 2
COMP417 Introduction to Robotics and Intelligent Systems Reinforcement Learning - 2 Speaker: Sandeep Manjanna Acklowledgement: These slides use material from Pieter Abbeel s, Dan Klein s and John Schulman
Intro to Reinforcement Learning. Part 3: Core Theory
Intro to Reinforcement Learning Part 3: Core Theory Interactive Example: You are the algorithm! Finite Markov decision processes (finite MDPs) dynamics p p p Experience: S 0 A 0 R 1 S 1 A 1 R 2 S 2 A 2
CS 461: Machine Learning Lecture 8
CS 461: Machine Learning Lecture 8 Dr. Kiri Wagstaff kiri.wagstaff@calstatela.edu 2/23/08 CS 461, Winter 2008 1 Plan for Today Review Clustering Reinforcement Learning How different from supervised, unsupervised?
Reinforcement Learning
Reinforcement Learning Basic idea: Receive feedback in the form of rewards Agent s utility is defined by the reward function Must (learn to) act so as to maximize expected rewards Grid World The agent
Lecture 4: Model-Free Prediction
Lecture 4: Model-Free Prediction David Silver Outline 1 Introduction 2 Monte-Carlo Learning 3 Temporal-Difference Learning 4 TD(λ) Introduction Model-Free Reinforcement Learning Last lecture: Planning
Reinforcement Learning
Reinforcement Learning MDP March May, 2013 MDP MDP: S, A, P, R, γ, µ State can be partially observable: Partially Observable MDPs () Actions can be temporally extended: Semi MDPs (SMDPs) and Hierarchical
Reinforcement Learning 04 - Monte Carlo. Elena, Xi
Reinforcement Learning 04 - Monte Carlo Elena, Xi Previous lecture 2 Markov Decision Processes Markov decision processes formally describe an environment for reinforcement learning where the environment
2D5362 Machine Learning
2D5362 Machine Learning Reinforcement Learning MIT GALib Available at http://lancet.mit.edu/ga/ download galib245.tar.gz gunzip galib245.tar.gz tar xvf galib245.tar cd galib245 make or access my files
Reinforcement Learning. Slides based on those used in Berkeley's AI class taught by Dan Klein
Reinforcement Learning Slides based on those used in Berkeley's AI class taught by Dan Klein Reinforcement Learning Basic idea: Receive feedback in the form of rewards Agent s utility is defined by the
Complex Decisions. Sequential Decision Making
Sequential Decision Making Outline Sequential decision problems Value iteration Policy iteration POMDPs (basic concepts) Slides partially based on the Book "Reinforcement Learning: an introduction" by
The Agent-Environment Interface Goals, Rewards, Returns The Markov Property The Markov Decision Process Value Functions Optimal Value Functions
The Agent-Environment Interface Goals, Rewards, Returns The Markov Property The Markov Decision Process Value Functions Optimal Value Functions Optimality and Approximation Finite MDP: {S, A, R, p, γ}
Monte Carlo Methods (Estimators, On-policy/Off-policy Learning)
1 / 24 Monte Carlo Methods (Estimators, On-policy/Off-policy Learning) Julie Nutini MLRG - Winter Term 2 January 24 th, 2017 2 / 24 Monte Carlo Methods Monte Carlo (MC) methods are learning methods, used
Sequential Decision Making
Sequential Decision Making Dynamic programming Christos Dimitrakakis Intelligent Autonomous Systems, IvI, University of Amsterdam, The Netherlands March 18, 2008 Introduction Some examples Dynamic programming
CS 188: Artificial Intelligence
CS 188: Artificial Intelligence Markov Decision Processes Dan Klein, Pieter Abbeel University of California, Berkeley Non-Deterministic Search 1 Example: Grid World A maze-like problem The agent lives
91.420/543: Artificial Intelligence UMass Lowell CS Fall 2010
91.420/543: Artificial Intelligence UMass Lowell CS Fall 2010 Lecture 17 & 18: Markov Decision Processes Oct 12 13, 2010 A subset of Lecture 9 slides from Dan Klein UC Berkeley Many slides over the course
Reinforcement Learning
Reinforcement Learning Monte Carlo Methods Heiko Zimmermann 15.05.2017 1 Monte Carlo Monte Carlo policy evaluation First visit policy evaluation Estimating q values On policy methods Off policy methods
Lecture 2: Making Good Sequences of Decisions Given a Model of World. CS234: RL Emma Brunskill Winter 2018
Lecture 2: Making Good Sequences of Decisions Given a Model of World CS234: RL Emma Brunskill Winter 218 Human in the loop exoskeleton work from Steve Collins lab Class Structure Last Time: Introduction
Basic Framework. About this class. Rewards Over Time. [This lecture adapted from Sutton & Barto and Russell & Norvig]
![Basic Framework. About this class. Rewards Over Time. [This lecture adapted from Sutton & Barto and Russell & Norvig]](/thumbs/82/85647934.jpg "Basic Framework. About this class. Rewards Over Time. [This lecture adapted from Sutton & Barto and Russell & Norvig]")
Basic Framework [This lecture adapted from Sutton & Barto and Russell & Norvig] About this class Markov Decision Processes The Bellman Equation Dynamic Programming for finding value functions and optimal
CS 343: Artificial Intelligence
CS 343: Artificial Intelligence Markov Decision Processes II Prof. Scott Niekum The University of Texas at Austin [These slides based on those of Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC
Making Complex Decisions
Ch. 17 p.1/29 Making Complex Decisions Chapter 17 Ch. 17 p.2/29 Outline Sequential decision problems Value iteration algorithm Policy iteration algorithm Ch. 17 p.3/29 A simple environment 3 +1 p=0.8 2
CS 188: Artificial Intelligence Spring Announcements
CS 188: Artificial Intelligence Spring 2011 Lecture 9: MDPs 2/16/2011 Pieter Abbeel UC Berkeley Many slides over the course adapted from either Dan Klein, Stuart Russell or Andrew Moore 1 Announcements
Topics in Computational Sustainability CS 325 Spring 2016
Topics in Computational Sustainability CS 325 Spring 2016 Note to other teachers and users of these slides. Andrew would be delighted if you found this source material useful in giving your own lectures.
Motivation: disadvantages of MC methods MC does not work for scenarios without termination It updates only at the end of the episode (sometimes - it i
Temporal-Di erence Learning Taras Kucherenko, Joonatan Manttari KTH tarask@kth.se manttari@kth.se March 7, 2017 Taras Kucherenko, Joonatan Manttari (KTH) TD-Learning March 7, 2017 1 / 68 Motivation: disadvantages
Reinforcement Learning Lectures 4 and 5
Reinforcement Learning Lectures 4 and 5 Gillian Hayes 18th January 2007 Reinforcement Learning 1 Framework Rewards, Returns Environment Dynamics Components of a Problem Values and Action Values, V and
Monte-Carlo Planning: Introduction and Bandit Basics. Alan Fern
Monte-Carlo Planning: Introduction and Bandit Basics Alan Fern 1 Large Worlds We have considered basic model-based planning algorithms Model-based planning: assumes MDP model is available Methods we learned
Temporal Abstraction in RL. Outline. Example. Markov Decision Processes (MDPs) ! Options
Temporal Abstraction in RL Outline How can an agent represent stochastic, closed-loop, temporally-extended courses of action? How can it act, learn, and plan using such representations?! HAMs (Parr & Russell
Monte-Carlo Planning: Introduction and Bandit Basics. Alan Fern
Monte-Carlo Planning: Introduction and Bandit Basics Alan Fern 1 Large Worlds We have considered basic model-based planning algorithms Model-based planning: assumes MDP model is available Methods we learned
CSEP 573: Artificial Intelligence
CSEP 573: Artificial Intelligence Markov Decision Processes (MDP)! Ali Farhadi Many slides over the course adapted from Luke Zettlemoyer, Dan Klein, Pieter Abbeel, Stuart Russell or Andrew Moore 1 Outline
CS 360: Advanced Artificial Intelligence Class #16: Reinforcement Learning
CS 360: Advanced Artificial Intelligence Class #16: Reinforcement Learning Daniel M. Gaines Note: content for slides adapted from Sutton and Barto [1998] Introduction Animals learn through interaction
Temporal Abstraction in RL
Temporal Abstraction in RL How can an agent represent stochastic, closed-loop, temporally-extended courses of action? How can it act, learn, and plan using such representations? HAMs (Parr & Russell 1998;
Non-Deterministic Search
Non-Deterministic Search MDP s 1 Non-Deterministic Search How do you plan (search) when your actions might fail? In general case, how do you plan, when the actions have multiple possible outcomes? 2 Example:
The Problem of Temporal Abstraction
The Problem of Temporal Abstraction How do we connect the high level to the low-level? " the human level to the physical level? " the decide level to the action level? MDPs are great, search is great,
EE266 Homework 5 Solutions
EE, Spring 15-1 Professor S. Lall EE Homework 5 Solutions 1. A refined inventory model. In this problem we consider an inventory model that is more refined than the one you ve seen in the lectures. The
CS 188: Artificial Intelligence Fall 2011
CS 188: Artificial Intelligence Fall 2011 Lecture 9: MDPs 9/22/2011 Dan Klein UC Berkeley Many slides over the course adapted from either Stuart Russell or Andrew Moore 2 Grid World The agent lives in
CS 188: Artificial Intelligence
CS 188: Artificial Intelligence Markov Decision Processes Dan Klein, Pieter Abbeel University of California, Berkeley Non Deterministic Search Example: Grid World A maze like problem The agent lives in
CSE 473: Artificial Intelligence
CSE 473: Artificial Intelligence Markov Decision Processes (MDPs) Luke Zettlemoyer Many slides over the course adapted from Dan Klein, Stuart Russell or Andrew Moore 1 Announcements PS2 online now Due
Lecture 12: MDP1. Victor R. Lesser. CMPSCI 683 Fall 2010
Lecture 12: MDP1 Victor R. Lesser CMPSCI 683 Fall 2010 Biased Random GSAT - WalkSat Notice no random restart 2 Today s lecture Search where there is Uncertainty in Operator Outcome --Sequential Decision
17 MAKING COMPLEX DECISIONS
267 17 MAKING COMPLEX DECISIONS The agent s utility now depends on a sequence of decisions In the following 4 3grid environment the agent makes a decision to move (U, R, D, L) at each time step When the
Importance Sampling for Fair Policy Selection
Importance Sampling for Fair Policy Selection Shayan Doroudi Carnegie Mellon University Pittsburgh, PA 15213 shayand@cs.cmu.edu Philip S. Thomas Carnegie Mellon University Pittsburgh, PA 15213 philipt@cs.cmu.edu
CPS 270: Artificial Intelligence Markov decision processes, POMDPs
CPS 270: Artificial Intelligence http://www.cs.duke.edu/courses/fall08/cps270/ Markov decision processes, POMDPs Instructor: Vincent Conitzer Warmup: a Markov process with rewards We derive some reward
Decision Theory: Value Iteration
Decision Theory: Value Iteration CPSC 322 Decision Theory 4 Textbook 9.5 Decision Theory: Value Iteration CPSC 322 Decision Theory 4, Slide 1 Lecture Overview 1 Recap 2 Policies 3 Value Iteration Decision
TDT4171 Artificial Intelligence Methods
TDT47 Artificial Intelligence Methods Lecture 7 Making Complex Decisions Norwegian University of Science and Technology Helge Langseth IT-VEST 0 helgel@idi.ntnu.no TDT47 Artificial Intelligence Methods
Markov Decision Processes (MDPs) CS 486/686 Introduction to AI University of Waterloo
Markov Decision Processes (MDPs) CS 486/686 Introduction to AI University of Waterloo Outline Sequential Decision Processes Markov chains Highlight Markov property Discounted rewards Value iteration Markov
Markov Decision Processes: Making Decision in the Presence of Uncertainty. (some of) R&N R&N
Markov Decision Processes: Making Decision in the Presence of Uncertainty (some of) R&N 16.1-16.6 R&N 17.1-17.4 Different Aspects of Machine Learning Supervised learning Classification - concept learning
Markov Decision Process
Markov Decision Process Human-aware Robotics 2018/02/13 Chapter 17.3 in R&N 3rd Ø Announcement: q Slides for this lecture are here: http://www.public.asu.edu/~yzhan442/teaching/cse471/lectures/mdp-ii.pdf
Economics 2010c: Lecture 4 Precautionary Savings and Liquidity Constraints
Economics 2010c: Lecture 4 Precautionary Savings and Liquidity Constraints David Laibson 9/11/2014 Outline: 1. Precautionary savings motives 2. Liquidity constraints 3. Application: Numerical solution
Monte-Carlo Planning Look Ahead Trees. Alan Fern
Monte-Carlo Planning Look Ahead Trees Alan Fern 1 Monte-Carlo Planning Outline Single State Case (multi-armed bandits) A basic tool for other algorithms Monte-Carlo Policy Improvement Policy rollout Policy
Dynamic Programming and Reinforcement Learning
Dynamic Programming and Reinforcement Learning Daniel Russo Columbia Business School Decision Risk and Operations Division Fall, 2017 Daniel Russo (Columbia) Fall 2017 1 / 34 Supervised Machine Learning
Reasoning with Uncertainty
Reasoning with Uncertainty Markov Decision Models Manfred Huber 2015 1 Markov Decision Process Models Markov models represent the behavior of a random process, including its internal state and the externally
EC316a: Advanced Scientific Computation, Fall Discrete time, continuous state dynamic models: solution methods
EC316a: Advanced Scientific Computation, Fall 2003 Notes Section 4 Discrete time, continuous state dynamic models: solution methods We consider now solution methods for discrete time models in which decisions
CS221 / Spring 2018 / Sadigh. Lecture 8: MDPs II
CS221 / Spring 218 / Sadigh Lecture 8: MDPs II cs221.stanford.edu/q Question If you wanted to go from Orbisonia to Rockhill, how would you get there? ride bus 1 ride bus 17 ride the magic tram CS221 /
Computational Finance Least Squares Monte Carlo
Computational Finance Least Squares Monte Carlo School of Mathematics 2019 Monte Carlo and Binomial Methods In the last two lectures we discussed the binomial tree method and convergence problems. One
Lecture 1: Lucas Model and Asset Pricing
Lecture 1: Lucas Model and Asset Pricing Economics 714, Spring 2018 1 Asset Pricing 1.1 Lucas (1978) Asset Pricing Model We assume that there are a large number of identical agents, modeled as a representative
Making Decisions. CS 3793 Artificial Intelligence Making Decisions 1
Making Decisions CS 3793 Artificial Intelligence Making Decisions 1 Planning under uncertainty should address: The world is nondeterministic. Actions are not certain to succeed. Many events are outside
MDPs: Bellman Equations, Value Iteration
MDPs: Bellman Equations, Value Iteration Sutton & Barto Ch 4 (Cf. AIMA Ch 17, Section 2-3) Adapted from slides kindly shared by Stuart Russell Sutton & Barto Ch 4 (Cf. AIMA Ch 17, Section 2-3) 1 Appreciations
Lecture 7: Bayesian approach to MAB - Gittins index
Advanced Topics in Machine Learning and Algorithmic Game Theory Lecture 7: Bayesian approach to MAB - Gittins index Lecturer: Yishay Mansour Scribe: Mariano Schain 7.1 Introduction In the Bayesian approach
6.231 DYNAMIC PROGRAMMING LECTURE 10 LECTURE OUTLINE
6.231 DYNAMIC PROGRAMMING LECTURE 10 LECTURE OUTLINE Rollout algorithms Cost improvement property Discrete deterministic problems Approximations of rollout algorithms Discretization of continuous time
AM 121: Intro to Optimization Models and Methods
AM 121: Intro to Optimization Models and Methods Lecture 18: Markov Decision Processes Yiling Chen and David Parkes Lesson Plan Markov decision processes Policies and Value functions Solving: average reward,
Logistics. CS 473: Artificial Intelligence. Markov Decision Processes. PS 2 due today Midterm in one week
CS 473: Artificial Intelligence Markov Decision Processes Dan Weld University of Washington [Slides originally created by Dan Klein & Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials
CS221 / Autumn 2018 / Liang. Lecture 8: MDPs II
CS221 / Autumn 218 / Liang Lecture 8: MDPs II cs221.stanford.edu/q Question If you wanted to go from Orbisonia to Rockhill, how would you get there? ride bus 1 ride bus 17 ride the magic tram CS221 / Autumn
Introduction to Fall 2007 Artificial Intelligence Final Exam
NAME: SID#: Login: Sec: 1 CS 188 Introduction to Fall 2007 Artificial Intelligence Final Exam You have 180 minutes. The exam is closed book, closed notes except a two-page crib sheet, basic calculators
Lecture 2 Dynamic Equilibrium Models: Three and More (Finite) Periods
Lecture 2 Dynamic Equilibrium Models: Three and More (Finite) Periods. Introduction In ECON 50, we discussed the structure of two-period dynamic general equilibrium models, some solution methods, and their
Action Selection for MDPs: Anytime AO* vs. UCT
Action Selection for MDPs: Anytime AO* vs. UCT Blai Bonet 1 and Hector Geffner 2 1 Universidad Simón Boĺıvar 2 ICREA & Universitat Pompeu Fabra AAAI, Toronto, Canada, July 2012 Online MDP Planning and
CS 234 Winter 2019 Assignment 1 Due: January 23 at 11:59 pm
CS 234 Winter 2019 Assignment 1 Due: January 23 at 11:59 pm For submission instructions please refer to website 1 Optimal Policy for Simple MDP [20 pts] Consider the simple n-state MDP shown in Figure
Compound Reinforcement Learning: Theory and An Application to Finance
Compound Reinforcement Learning: Theory and An Application to Finance Tohgoroh Matsui 1, Takashi Goto 2, Kiyoshi Izumi 3,4, and Yu Chen 3 1 Chubu University, 1200 Matsumoto-cho, Kasugai, 487-8501 Aichi,
Duopoly models Multistage games with observed actions Subgame perfect equilibrium Extensive form of a game Two-stage prisoner s dilemma
Recap Last class (September 20, 2016) Duopoly models Multistage games with observed actions Subgame perfect equilibrium Extensive form of a game Two-stage prisoner s dilemma Today (October 13, 2016) Finitely
The Irrevocable Multi-Armed Bandit Problem
The Irrevocable Multi-Armed Bandit Problem Ritesh Madan Qualcomm-Flarion Technologies May 27, 2009 Joint work with Vivek Farias (MIT) 2 Multi-Armed Bandit Problem n arms, where each arm i is a Markov Decision
Game Theory. Wolfgang Frimmel. Repeated Games
Game Theory Wolfgang Frimmel Repeated Games 1 / 41 Recap: SPNE The solution concept for dynamic games with complete information is the subgame perfect Nash Equilibrium (SPNE) Selten (1965): A strategy
Comparing Direct and Indirect Temporal-Difference Methods for Estimating the Variance of the Return
Comparing Direct and Indirect Temporal-Difference Methods for Estimating the Variance of the Return Craig Sherstan 1, Dylan R. Ashley 2, Brendan Bennett 2, Kenny Young, Adam White, Martha White, Richard
Adaptive Experiments for Policy Choice. March 8, 2019
Adaptive Experiments for Policy Choice Maximilian Kasy Anja Sautmann March 8, 2019 Introduction The goal of many experiments is to inform policy choices: 1. Job search assistance for refugees: Treatments:
Competing Mechanisms with Limited Commitment
Competing Mechanisms with Limited Commitment Suehyun Kwon CESIFO WORKING PAPER NO. 6280 CATEGORY 12: EMPIRICAL AND THEORETICAL METHODS DECEMBER 2016 An electronic version of the paper may be downloaded
Pakes (1986): Patents as Options: Some Estimates of the Value of Holding European Patent Stocks
Pakes (1986): Patents as Options: Some Estimates of the Value of Holding European Patent Stocks Spring 2009 Main question: How much are patents worth? Answering this question is important, because it helps
COS402- Artificial Intelligence Fall Lecture 17: MDP: Value Iteration and Policy Iteration
COS402- Artificial Intelligence Fall 2015 Lecture 17: MDP: Value Iteration and Policy Iteration Outline The Bellman equation and Bellman update Contraction Value iteration Policy iteration The Bellman
G5212: Game Theory. Mark Dean. Spring 2017
G5212: Game Theory Mark Dean Spring 2017 Bargaining We will now apply the concept of SPNE to bargaining A bit of background Bargaining is hugely interesting but complicated to model It turns out that the
Final exam solutions
EE365 Stochastic Control / MS&E251 Stochastic Decision Models Profs. S. Lall, S. Boyd June 5 6 or June 6 7, 2013 Final exam solutions This is a 24 hour take-home final. Please turn it in to one of the
Example: Grid World. CS 188: Artificial Intelligence Markov Decision Processes II. Recap: MDPs. Optimal Quantities
CS 188: Artificial Intelligence Markov Deciion Procee II Intructor: Dan Klein and Pieter Abbeel --- Univerity of California, Berkeley [Thee lide were created by Dan Klein and Pieter Abbeel for CS188 Intro
Probabilistic Robotics: Probabilistic Planning and MDPs
Probabilistic Robotics: Probabilistic Planning and MDPs Slide credits: Wolfram Burgard, Dieter Fox, Cyrill Stachniss, Giorgio Grisetti, Maren Bennewitz, Christian Plagemann, Dirk Haehnel, Mike Montemerlo,
Dynamic Programming (DP) Massimo Paolucci University of Genova
Dynamic Programming (DP) Massimo Paolucci University of Genova DP cannot be applied to each kind of problem In particular, it is a solution method for problems defined over stages For each stage a subproblem
Intelligent Systems (AI-2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 9 Sep, 28, 2016 Slide 1 CPSC 422, Lecture 9 An MDP Approach to Multi-Category Patient Scheduling in a Diagnostic Facility Adapted from: Matthew
Reinforcement Learning
Reinforcement Learning Model-based RL and Integrated Learning-Planning Planning and Search, Model Learning, Dyna Architecture, Exploration-Exploitation (many slides from lectures of Marc Toussaint & David
Extending MCTS
Extending MCTS 2-17-16 Reading Quiz (from Monday) What is the relationship between Monte Carlo tree search and upper confidence bound applied to trees? a) MCTS is a type of UCT b) UCT is a type of MCTS
Machine Learning for Physicists Lecture 10. Summer 2017 University of Erlangen-Nuremberg Florian Marquardt
Machine Learning for Physicists Lecture 10 Summer 2017 University of Erlangen-Nuremberg Florian Marquardt Function/Image representation Image classification [Handwriting recognition] Convolutional nets
Multi-period mean variance asset allocation: Is it bad to win the lottery?
Multi-period mean variance asset allocation: Is it bad to win the lottery? Peter Forsyth 1 D.M. Dang 1 1 Cheriton School of Computer Science University of Waterloo Guangzhou, July 28, 2014 1 / 29 The Basic
A Non-Random Walk Down Wall Street
A Non-Random Walk Down Wall Street Andrew W. Lo A. Craig MacKinlay Princeton University Press Princeton, New Jersey list of Figures List of Tables Preface xiii xv xxi 1 Introduction 3 1.1 The Random Walk
Optimal Dam Management
Optimal Dam Management Michel De Lara et Vincent Leclère July 3, 2012 Contents 1 Problem statement 1 1.1 Dam dynamics.................................. 2 1.2 Intertemporal payoff criterion..........................
CS 188: Artificial Intelligence Fall Markov Decision Processes
CS 188: Artificial Intelligence Fall 2007 Lecture 10: MDP 9/27/2007 Dan Klein UC Berkeley Markov Deciion Procee An MDP i defined by: A et of tate S A et of action a A A tranition function T(,a, ) Prob
Multi-armed bandit problems
Multi-armed bandit problems Stochastic Decision Theory (2WB12) Arnoud den Boer 13 March 2013 Set-up 13 and 14 March: Lectures. 20 and 21 March: Paper presentations (Four groups, 45 min per group). Before
Reinforcement learning and Markov Decision Processes (MDPs) (B) Avrim Blum
Reinforcement learning and Markov Decision Processes (MDPs) 15-859(B) Avrim Blum RL and MDPs General scenario: We are an agent in some state. Have observations, perform actions, get rewards. (See lights,
Theory and practice of option pricing
Theory and practice of option pricing Juliusz Jabłecki Department of Quantitative Finance Faculty of Economic Sciences University of Warsaw jjablecki@wne.uw.edu.pl and Head of Monetary Policy Analysis
Q1. [?? pts] Search Traces
![Q1. [?? pts] Search Traces](/thumbs/80/81482567.jpg "Q1. [?? pts] Search Traces")
CS 188 Spring 2010 Introduction to Artificial Intelligence Midterm Exam Solutions Q1. [?? pts] Search Traces Each of the trees (G1 through G5) was generated by searching the graph (below, left) with a
INTRODUCTION TO THE ECONOMICS AND MATHEMATICS OF FINANCIAL MARKETS. Jakša Cvitanić and Fernando Zapatero
INTRODUCTION TO THE ECONOMICS AND MATHEMATICS OF FINANCIAL MARKETS Jakša Cvitanić and Fernando Zapatero INTRODUCTION TO THE ECONOMICS AND MATHEMATICS OF FINANCIAL MARKETS Table of Contents PREFACE...1