Eligibility Traces. Suggested reading: Contents: Chapter 7 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, 1998.


 Phoebe Wilkerson
 2 years ago
 Views:
Transcription
1 Eligibility Traces 0 Eligibility Traces Suggested reading: Chapter 7 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, Eligibility Traces Eligibility Traces 1 Contents: nstep TD Prediction The forward view of TD(!) The backward view of TD(!) Sarsa(!) Q(!) Actorcritic methods Replacing traces
2 Eligibility traces! Eligibility Traces 2 Eligibility traces are one of the basic mechanisms of reinforcement learning.! There are two ways to view eligibility traces:! The more theoretical view is that they are a bridge from TD to Monte Carlo methods (forward view).! According to the other view, an eligibility trace is a temporary record of the occurrence of an event, such as the visiting of a state or the taking of an action (backward view). The trace marks the memory parameters associated with the event as eligible for undergoing learning changes.! nstep TD Prediction! Eligibility Traces 3 Idea: Look farther into the future when you do TD backup (1, 2, 3,, n steps)! TD(0)
3 Mathematics of nstep TD Prediction! Eligibility Traces 4 Monte Carlo:! R t = r t +1 +"r t +2 +" 2 r t " T #t #1 r T TD:! R (1) t = r t +1 +"V t (s t +1 ) Use V! to estimate remaining return!! nstep TD:! 2 step return:! nstep return:!!! R t (2) = r t +1 +"r t +2 +" 2 V t (s t +2 ) R t (n ) = r t +1 +"r t +2 +" 2 r t " n #1 r t +n +" n V t (s t +n ) If the episode ends in less than n steps, then the truncation in a nstep return occurs at the episode's end, resulting in the conventional complete return: T " t # n $ R t (n ) = R t (T "t ) = R t nstep Backups!! Eligibility Traces 5 Backup (online or offline):! (n "V t (s t ) = #[ R ) t $V t (s t )] online:! offline:!! the updates are done during the episode, as soon as the increment is computed. V t +1 (s )! := V t (s ) + "V t (s ) the increments are accumulated "on the side" and are not used to change value estimates until the end of the episode. $ T #1 V (s ) := V (s ) + "V t (s ) t =0!
4 Eligibility Traces 6 Error reduction property of nstep returns! For any V, the expected value of the nstep return using V is guaranteed to be a better estimate of V! than V is: the worst error under the new estimate is guaranteed to be less than or equal to " n times the worst error under V. Maximum error using nstep return Maximum error using V Using this, you can show that TD prediction methods using nstep backups converge. nstep TD Prediction! Eligibility Traces 7 Random Walk Examples:! A onestep method would change only the estimate for the last state, V(E), which would be incremented toward 1, the observed return. How does 2step TD work here?! How about 3step TD?! A twostep method would increment the values of the two states preceding termination: V(E), and V(D).
5 nstep TD Prediction! Eligibility Traces 8 A larger example:! Task:! 19 state random walk! Averaging nstep Returns! Eligibility Traces 9 Backups can be done not just toward any nstep return but toward any average of nstep returns.! One backup e.g. backup half of 2step and half of 4step! Called a complex backup! Draw each component with a horizontal line above them! Label with the weights for that component! positive and sum to 1 weights
6 Eligibility Traces 10!return algorithm (forward view of TD(!))! TD(!) is a particular method for averaging all nstep backups.!!return:! weight by! n1 (time since visitation)! normalization factor Backup using!return:!!return weighting Function! Eligibility Traces 11 After a terminal state has been reached, all subsequent step returns are equal to R t.! T #t #1 R " t = (1 # ") $ " n #1 R (n) t + " T #t #1 R t n =1 Until termination After termination!
7 Relation to TD(0) and MC! Eligibility Traces 12!return can be rewritten as:! If! = 1, you get MC:! Until termination After termination If! = 0, you get TD(0)! Forward view of TD(!)! Eligibility Traces 13 For each state visited, we look forward in time to all the future rewards and states to determine its update.! Offline:!return algorithm TD(!)
8 !return on the Random Walk! Eligibility Traces 14 Same 19 state random walk as before! Backward View of TD(!)! Eligibility Traces 15 Shout! t backwards over time! The strength of your voice decreases with temporal distance by "#!
9 Backward View of TD(!)! Eligibility Traces 16 The forward view was for theory! The backward view is for mechanism! New variable called eligibility trace. The eligibility trace for state at time is denoted! On each step, decay all traces by "# and increment the trace for the current state by! discount rate tracedecay parameter Backward View of TD(!)! Eligibility Traces 17 The traces indicate the degree to which each state is eligible for undergoing learning changes should a reinforcing event occur.! The reinforcing events we are concerned with are the momentbymoment onestep TD errors. For example, statevalue prediction TD error:! onestep TD error The global TD error signal triggers proportional updates to all recently visited states, as signaled by their nonzero traces: As always, these increments could be done on each step to form an online algorithm, or saved until the end of the episode to produce an offline algorithm.!
10 Online Tabular TD(!)! Eligibility Traces 18 Backward View TD(0)! Eligibility Traces 19 The backward view of TD(!) is oriented backward in time. At each moment we look at the current TD error and assign it backward to each prior state according to the state's eligibility trace at that time. The TD(!) update reduces to the simple TD rule (TD(0)). Only the one state preceding the current one is changed by the TD error.
11 Backward View TD(1) MC! Eligibility Traces 20 Similar to MC. Similar to MC for an undiscounted episodic task. If you set # to 1, you get MC but in a better way! Can apply TD(1) to continuing tasks! Works incrementally and online (instead of waiting to the end of the episode)! Equivalence of Forward and Backward Views! Eligibility Traces 21 The forward (theoretical,!return algorithm) view of TD(!) is equivalent to the backward (mechanistic) view for offline updating! The book shows:! " I sst = s = s :1 t # $ else : 0 Backward updates Forward updates! algebra shown in book Online updating with small " is similar!
12 Eligibility Traces 22 Online versus Offline on Random Walk! Same 19 state random walk! Online performs better over a broader range of parameters! Sarsa(!)! Eligibility Traces 23 Eligibilitytraces for stateaction pairs:! % e t (s,a) = "#e t $1(s,a) +1 & ' "#e t $1 (s,a) if s = s t and a = a t otherwise Q t +1 (s,a) = Q t (s,a) + () t e t (s,a) ) t = r t +1 +"Q t (s t +1,a t +1 ) $ Q t (s t,a t )! Triggers an update of all recently visited stateaction values
13 Sarsa(!)! Eligibility Traces 24 Algorithm: Sarsa(!)! Eligibility Traces 25 Gridworld Example: With one trial, the agent has much more information about how to get to the goal! not necessarily the best way! Can considerably accelerate learning!
14 Q(!)! Eligibility Traces 26 A problem occurs for offpolicy methods such as Qlearning when exploratory actions occur, since you backup over a nongreedy policy. This would violate GPI.! Three Approaches to Q(!):! Watkins: Zero out eligibility trace after a nongreedy action. Do max when backing up at first nongreedy choice.! Peng: No distinction between exploratory and greedy actions.! Naïve: Similar to Watkins's method, except that the traces are not set to zero on exploratory actions.! Watkins s Q(!)! Eligibility Traces 27 Watkins: Zero out eligibility trace after a nongreedy action. Do max when backing up at first nongreedy choice.! Disadvantage to Watkins s method:! Early in learning, the eligibility trace will be cut frequently resulting in short traces!
15 Watkins s Q(!)! Eligibility Traces 28 Peng s Q(!)! Eligibility Traces 29 No distinction between exploratory and greedy actions! Backup max action except at end! Never cut traces! The earlier transitions of each are onpolicy, whereas the last (fictitious) transition uses the greedy policy! Disadvantage:! Complicated to implement! Theoretically, no guarantee to converge to the optimal value!
16 Peng s Q(!) Algorithm! Eligibility Traces 30 For a complete description of the needed implementation, see Peng & Williams (1994, 1996). Naive Q(!)! Eligibility Traces 31 Idea: is it really a problem to backup exploratory actions?! Never zero traces! Always backup max at current action (unlike Peng or Watkins s)! Works well in preliminary empirical studies!
17 Comparison Results! Eligibility Traces 32 Comparison of Watkins s, Peng s, and Naïve (called McGovern s here) Q(!)! Deterministic gridworld with obstacles! 10x10 gridworld! 25 randomly generated obstacles! 30 runs! " = 0.05, # = 0.9,! = 0.9, $ = 0.05, accumulating traces! From McGovern and Sutton (1997). Towards a better Q(!) Convergence of the Q(!) s! Eligibility Traces 33 None of the methods are proven to converge.! Watkins s is thought to converge to Q *! Peng s is thought to converge to a mixture of Q $ and Q *! Naive  Q *?!
18 Eligibility Traces 34 Eligibility Traces for ActorCritic Methods! Critic: Onpolicy learning of V $. Use TD(!) as described before.! Actor: Needs eligibility traces for each stateaction pair.! We change the update equation of the actor:! to Replacing Traces! Eligibility Traces 35 Using accumulating traces, frequently visited states can have eligibilities greater than 1! This can be a problem for convergence! Replacing traces: Instead of adding 1 when you visit a state, set that trace to 1!
19 Replacing Traces Example! Eligibility Traces 36 Same 19 state random walk task as before! Replacing traces perform better than accumulating traces over more values of!! Why Replacing Traces?! Eligibility Traces 37 Replacing traces can significantly speed learning! They can make the system perform well for a broader set of parameters! Accumulating traces can do poorly on certain types of tasks! Why is this task particularly onerous for accumulating traces?
20 More Replacing Traces! Eligibility Traces 38 There is an interesting relationship between replacetrace methods and Monte Carlo methods in the undiscounted case:!!just as conventional TD(1) is related to the everyvisit MC algorithm, offline replacetrace TD(1) is identical to firstvisit MC (Singh and Sutton, 1996).! Extension to actionvalues:! When you revisit a state, what should you do with the traces for the other actions?! Singh & Sutton (1996) proposed to set them to zero:! Variable!! Eligibility Traces 39 Can generalize to variable!% Here! is a function of time! Could define!
21 Conclusions! Eligibility Traces 40 Provides efficient, incremental way to combine MC and TD! Includes advantages of MC (can deal with lack of Markov property)! Includes advantages of TD (using TD error, bootstrapping)! Can significantly speed learning! Does have a cost in computation! References! Eligibility Traces 41 Singh, S. P. and Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22:
Lecture 5: ModelFree Control
Lecture 5: ModelFree Control David Silver Outline 1 Introduction 2 OnPolicy MonteCarlo Control 3 OnPolicy TemporalDifference Learning 4 OffPolicy Learning 5 Summary Introduction ModelFree Reinforcement
More informationUsing Markov Decision Processes to Solve a Portfolio Allocation Problem
Using Markov Decision Processes to Solve a Portfolio Allocation Problem Daniel Bookstaber April 26, 2005 Contents 1 Introduction 3 2 Defining the Model 4 2.1 The Stochastic Model for a Single Asset.........................
More informationA Sarsa based Autonomous Stock Trading Agent
A Sarsa based Autonomous Stock Trading Agent Achal Augustine The University of Texas at Austin Department of Computer Science Austin, TX 78712 USA achal@cs.utexas.edu Abstract This paper describes an autonomous
More informationMotivation. Motivation. Can a software agent learn to play Backgammon by itself? Machine Learning. Reinforcement Learning
Motivation Machine Learning Can a software agent learn to play Backgammon by itself? Reinforcement Learning Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut
More informationA Multiagent Qlearning Framework for Optimizing Stock Trading Systems
A Multiagent Qlearning Framework for Optimizing Stock Trading Systems Jae Won Lee 1 and Jangmin O 2 1 School of Computer Science and Engineering, Sungshin Women s University, Seoul, Korea 136742 jwlee@cs.sungshin.ac.kr
More informationLearning to Play Board Games using Temporal Difference Methods
Learning to Play Board Games using Temporal Difference Methods Marco A. Wiering Jan Peter Patist Henk Mannen institute of information and computing sciences, utrecht university technical report UUCS2005048
More informationJae Won idee. School of Computer Science and Engineering Sungshin Women's University Seoul, 136742, South Korea
STOCK PRICE PREDICTION USING REINFORCEMENT LEARNING Jae Won idee School of Computer Science and Engineering Sungshin Women's University Seoul, 136742, South Korea ABSTRACT Recently, numerous investigations
More informationTD(0) Leads to Better Policies than Approximate Value Iteration
TD(0) Leads to Better Policies than Approximate Value Iteration Benjamin Van Roy Management Science and Engineering and Electrical Engineering Stanford University Stanford, CA 94305 bvr@stanford.edu Abstract
More informationSYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis
SYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu October 17, 2015 Outline
More informationReinforcement Learning with Factored States and Actions
Journal of Machine Learning Research 5 (2004) 1063 1088 Submitted 3/02; Revised 1/04; Published 8/04 Reinforcement Learning with Factored States and Actions Brian Sallans Austrian Research Institute for
More informationReinforcement Learning of Task Plans for Real Robot Systems
Reinforcement Learning of Task Plans for Real Robot Systems Pedro Tomás Mendes Resende pedro.resende@ist.utl.pt Instituto Superior Técnico, Lisboa, Portugal October 2014 Abstract This paper is the extended
More informationProjective Simulation compared to reinforcement learning
COMPUTER SCIENCE Master thesis Projective Simulation compared to reinforcement learning By: Øystein Førsund Bjerland Supervisor: Matthew Geoffrey Parker June 1, 2015 Acknowledgements I would like to thank
More informationNEURAL NETWORKS AND REINFORCEMENT LEARNING. Abhijit Gosavi
NEURAL NETWORKS AND REINFORCEMENT LEARNING Abhijit Gosavi Department of Engineering Management and Systems Engineering Missouri University of Science and Technology Rolla, MO 65409 1 Outline A Quick Introduction
More informationAlgorithms for Reinforcement Learning
Algorithms for Reinforcement Learning Draft of the lecture published in the Synthesis Lectures on Artificial Intelligence and Machine Learning series by Morgan & Claypool Publishers Csaba Szepesvári June
More informationComputing Near Optimal Strategies for Stochastic Investment Planning Problems
Computing Near Optimal Strategies for Stochastic Investment Planning Problems Milos Hauskrecfat 1, Gopal Pandurangan 1,2 and Eli Upfal 1,2 Computer Science Department, Box 1910 Brown University Providence,
More informationBeyond Reward: The Problem of Knowledge and Data
Beyond Reward: The Problem of Knowledge and Data Richard S. Sutton University of Alberta Edmonton, Alberta, Canada Intelligence can be defined, informally, as knowing a lot and being able to use that knowledge
More informationSupply Chain Yield Management based on Reinforcement Learning
Supply Chain Yield Management based on Reinforcement Learning Tim Stockheim 1, Michael Schwind 1, Alexander Korth 2, and Burak Simsek 2 1 Chair of Economics, esp. Information Systems, Frankfurt University,
More informationLecture 1: Introduction to Reinforcement Learning
Lecture 1: Introduction to Reinforcement Learning David Silver Outline 1 Admin 2 About Reinforcement Learning 3 The Reinforcement Learning Problem 4 Inside An RL Agent 5 Problems within Reinforcement Learning
More informationTime Hopping Technique for Faster Reinforcement Learning in Simulations
BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 11, No 3 Sofia 2011 Time Hopping Technique for Faster Reinforcement Learning in Simulations Petar Kormushev 1, Kohei Nomoto
More informationNeuroDynamic Programming An Overview
1 NeuroDynamic Programming An Overview Dimitri Bertsekas Dept. of Electrical Engineering and Computer Science M.I.T. September 2006 2 BELLMAN AND THE DUAL CURSES Dynamic Programming (DP) is very broadly
More informationCreating Algorithmic Traders with Hierarchical Reinforcement Learning MSc Dissertation. Thomas Elder s0789506
Creating Algorithmic Traders with Hierarchical Reinforcement Learning MSc Dissertation Thomas Elder s78956 Master of Science School of Informatics University of Edinburgh 28 Abstract There has recently
More informationLearning the Structure of Factored Markov Decision Processes in Reinforcement Learning Problems
Learning the Structure of Factored Markov Decision Processes in Reinforcement Learning Problems Thomas Degris Thomas.Degris@lip6.fr Olivier Sigaud Olivier.Sigaud@lip6.fr PierreHenri Wuillemin PierreHenri.Wuillemin@lip6.fr
More informationA Reinforcement Learning Approach for Supply Chain Management
A Reinforcement Learning Approach for Supply Chain Management Tim Stockheim, Michael Schwind, and Wolfgang Koenig Chair of Economics, esp. Information Systems, Frankfurt University, D60054 Frankfurt,
More informationWhere Do Rewards Come From?
Where Do Rewards Come From? Satinder Singh baveja@umich.edu Computer Science & Engineering University of Michigan, Ann Arbor Richard L. Lewis rickl@umich.edu Department of Psychology University of Michigan,
More informationSENSITIVITY ANALYSIS AND INFERENCE. Lecture 12
This work is licensed under a Creative Commons AttributionNonCommercialShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this
More informationOptions with exceptions
Options with exceptions Munu Sairamesh and Balaraman Ravindran Indian Institute Of Technology Madras, India Abstract. An option is a policy fragment that represents a solution to a frequent subproblem
More informationIntelligent Agents Serving Based On The Society Information
Intelligent Agents Serving Based On The Society Information Sanem SARIEL Istanbul Technical University, Computer Engineering Department, Istanbul, TURKEY sariel@cs.itu.edu.tr B. Tevfik AKGUN Yildiz Technical
More informationGoalDirected Online Learning of Predictive Models
GoalDirected Online Learning of Predictive Models Sylvie C.W. Ong, Yuri Grinberg, and Joelle Pineau McGill University, Montreal, Canada Abstract. We present an algorithmic approach for integrated learning
More informationUrban Traffic Control Based on Learning Agents
Urban Traffic Control Based on Learning Agents PierreLuc Grégoire, Charles Desjardins, Julien Laumônier and Brahim Chaibdraa DAMAS Laboratory, Computer Science and Software Engineering Department, Laval
More informationIntelligent Tutoring Systems Using Reinforcement Learning
Intelligent Tutoring Systems Using Reinforcement Learning A THESIS submitted by SREENIVASA SARMA B. H. for the award of the degree of MASTER OF SCIENCE (by Research) DEPARTMENT OF COMPUTER SCIENCE AND
More informationAn Application of Inverse Reinforcement Learning to Medical Records of Diabetes Treatment
An Application of Inverse Reinforcement Learning to Medical Records of Diabetes Treatment Hideki Asoh 1, Masanori Shiro 1 Shotaro Akaho 1, Toshihiro Kamishima 1, Koiti Hasida 1, Eiji Aramaki 2, and Takahide
More informationHierarchical Reinforcement Learning in Computer Games
Hierarchical Reinforcement Learning in Computer Games Marc Ponsen, Pieter Spronck, Karl Tuyls Maastricht University / MICCIKAT. {m.ponsen,p.spronck,k.tuyls}@cs.unimaas.nl Abstract. Hierarchical reinforcement
More informationMachine Learning Applications in Algorithmic Trading
Machine Learning Applications in Algorithmic Trading Ryan Brosnahan, Ross Rothenstine Introduction: Computer controlled algorithmic trading is used by financial institutions around the world and accounted
More informationArtificially Intelligent Adaptive Tutoring System
Artificially Intelligent Adaptive Tutoring System William Whiteley Stanford School of Engineering Stanford, CA 94305 bill.whiteley@gmail.com Abstract This paper presents an adaptive tutoring system based
More informationSkill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining George Konidaris Computer Science Department University of Massachusetts Amherst Amherst MA 01003 USA gdk@cs.umass.edu
More informationBetween MDPs and SemiMDPs: A Framework for Temporal Abstraction in Reinforcement Learning
Between MDPs and SemiMDPs: A Framework for Temporal Abstraction in Reinforcement Learning Richard S. Sutton a, Doina Precup b, and Satinder Singh a a AT&T Labs Research, 180 Park Avenue, Florham Park,
More informationLearning Agents: Introduction
Learning Agents: Introduction S Luz luzs@cs.tcd.ie October 22, 2013 Learning in agent architectures Performance standard representation Critic Agent perception rewards/ instruction Perception Learner Goals
More informationPreliminaries: Problem Definition Agent model, POMDP, Bayesian RL
POMDP Tutorial Preliminaries: Problem Definition Agent model, POMDP, Bayesian RL Observation Belief ACTOR Transition Dynamics WORLD b Policy π Action Markov Decision Process X: set of states [x s,x r
More informationA linear algebraic method for pricing temporary life annuities
A linear algebraic method for pricing temporary life annuities P. Date (joint work with R. Mamon, L. Jalen and I.C. Wang) Department of Mathematical Sciences, Brunel University, London Outline Introduction
More informationInductive QoS Packet Scheduling for Adaptive Dynamic Networks
Inductive QoS Packet Scheduling for Adaptive Dynamic Networks Malika BOURENANE Dept of Computer Science University of EsSenia Algeria mb_regina@yahoo.fr Abdelhamid MELLOUK LISSI Laboratory University
More informationMachine Learning. 01  Introduction
Machine Learning 01  Introduction Machine learning course One lecture (Wednesday, 9:30, 346) and one exercise (Monday, 17:15, 203). Oral exam, 20 minutes, 5 credit points. Some basic mathematical knowledge
More informationChisquare Tests Driven Method for Learning the Structure of Factored MDPs
Chisquare Tests Driven Method for Learning the Structure of Factored MDPs Thomas Degris Thomas.Degris@lip6.fr Olivier Sigaud Olivier.Sigaud@lip6.fr PierreHenri Wuillemin PierreHenri.Wuillemin@lip6.fr
More informationChapter ML:IV. IV. Statistical Learning. Probability Basics Bayes Classification Maximum aposteriori Hypotheses
Chapter ML:IV IV. Statistical Learning Probability Basics Bayes Classification Maximum aposteriori Hypotheses ML:IV1 Statistical Learning STEIN 20052015 Area Overview Mathematics Statistics...... Stochastics
More informationAssignment #1. Example: tetris state: board configuration + shape of the falling piece ~2 200 states! Recap RL so far. Page 1
Generalization and function approximation CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Represent
More informationAP Physics 1 and 2 Lab Investigations
AP Physics 1 and 2 Lab Investigations Student Guide to Data Analysis New York, NY. College Board, Advanced Placement, Advanced Placement Program, AP, AP Central, and the acorn logo are registered trademarks
More informationReinforcement Learning: An Introduction
i Reinforcement Learning: An Introduction Second edition, in progress Richard S. Sutton and Andrew G. Barto c 2012 A Bradford Book The MIT Press Cambridge, Massachusetts London, England ii In memory of
More informationOPTIMAL DESIGN OF A MULTITIER REWARD SCHEME. Amir Gandomi *, Saeed Zolfaghari **
OPTIMAL DESIGN OF A MULTITIER REWARD SCHEME Amir Gandomi *, Saeed Zolfaghari ** Department of Mechanical and Industrial Engineering, Ryerson University, Toronto, Ontario * Tel.: + 46 979 5000x7702, Email:
More informationFigure 1: Cost and Speed of Access of different storage components. Page 30
Reinforcement Learning Approach for Data Migration in Hierarchical Storage Systems T.G. Lakshmi, R.R. Sedamkar, Harshali Patil Department of Computer Engineering, Thakur College of Engineering and Technology,
More informationA Markov Decision Process Model for Traffic Prioritisation Provisioning
Issues in Informing Science and Information Technology A Markov Decision Process Model for Traffic Prioritisation Provisioning Abdullah Gani The University of Sheffield, UK Omar Zakaria The Royal Holloway,
More informationApplication of Markov chain analysis to trend prediction of stock indices Milan Svoboda 1, Ladislav Lukáš 2
Proceedings of 3th International Conference Mathematical Methods in Economics 1 Introduction Application of Markov chain analysis to trend prediction of stock indices Milan Svoboda 1, Ladislav Lukáš 2
More informationLocal Graph Partitioning as a Basis for Generating TemporallyExtended Actions in Reinforcement Learning
Local Graph Partitioning as a Basis for Generating TemporallyExtended Actions in Reinforcement Learning Özgür Şimşek University of Massachusetts Amherst, MA 010039264 ozgur@umass.cs.edu Alicia P. Wolfe
More informationTraining Factor Graphs with Reinforcement Learning for Efficient MAP Inference
Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference Michael Wick, Khashayar Rohanimanesh, Sameer Singh, Andrew McCallum Department of Computer Science University of Massachusetts
More informationA SOFTWARE SYSTEM FOR ONLINE LEARNING APPLIED IN THE FIELD OF COMPUTER SCIENCE
The 1 st International Conference on Virtual Learning, ICVL 2006 223 A SOFTWARE SYSTEM FOR ONLINE LEARNING APPLIED IN THE FIELD OF COMPUTER SCIENCE Gabriela Moise PetroleumGas University of Ploieşti 39
More informationTetris: Experiments with the LP Approach to Approximate DP
Tetris: Experiments with the LP Approach to Approximate DP Vivek F. Farias Electrical Engineering Stanford University Stanford, CA 94403 vivekf@stanford.edu Benjamin Van Roy Management Science and Engineering
More informationAn efficient method for simulating interest rate curves
BANCO DE MÉXICO An efficient method for simulating interest rate curves Javier Márquez DiezCanedo Carlos E. Nogués Nivón Viviana Vélez Grajales February 2003 Summary The purpose of this paper is to present
More informationChapter 12 TimeDerivative Models of Pavlovian Reinforcement Richard S. Sutton Andrew G. Barto
Approximately as appeared in: Learning and Computational Neuroscience: Foundations of Adaptive Networks, M. Gabriel and J. Moore, Eds., pp. 497 537. MIT Press, 1990. Chapter 12 TimeDerivative Models of
More informationCHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER
93 CHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER 5.1 INTRODUCTION The development of an active trap based feeder for handling brakeliners was discussed
More informationAutomatic Selection of Loop Scheduling Algorithms Using Reinforcement Learning
Automatic Selection of Loop Scheduling Algorithms Using Reinforcement Learning Sumithra Dhandayuthapani [1,2], Ioana Banicescu [1,2], Ricolindo L. Cariño [2], Eric Hansen [1], Jaderick P. Pabico [1,2],
More informationGoalDirected Online Learning of Predictive Models
GoalDirected Online Learning of Predictive Models Sylvie C.W. Ong, Yuri Grinberg, and Joelle Pineau School of Computer Science McGill University, Montreal, Canada Abstract. We present an algorithmic approach
More informationReinforcement Learning on the Combinatorial Game of Nim ERIK JÄRLEBERG
Reinforcement Learning on the Combinatorial Game of Nim ERIK JÄRLEBERG Bachelor of Science Thesis Stockholm, Sweden 2011 Reinforcement Learning on the Combinatorial Game of Nim ERIK JÄRLEBERG Bachelor
More informationparent ROADMAP MATHEMATICS SUPPORTING YOUR CHILD IN HIGH SCHOOL
parent ROADMAP MATHEMATICS SUPPORTING YOUR CHILD IN HIGH SCHOOL HS America s schools are working to provide higher quality instruction than ever before. The way we taught students in the past simply does
More informationIE 680 Special Topics in Production Systems: Networks, Routing and Logistics*
IE 680 Special Topics in Production Systems: Networks, Routing and Logistics* Rakesh Nagi Department of Industrial Engineering University at Buffalo (SUNY) *Lecture notes from Network Flows by Ahuja, Magnanti
More informationAnalysis of a Production/Inventory System with Multiple Retailers
Analysis of a Production/Inventory System with Multiple Retailers Ann M. Noblesse 1, Robert N. Boute 1,2, Marc R. Lambrecht 1, Benny Van Houdt 3 1 Research Center for Operations Management, University
More informationADVANTAGE UPDATING. Leemon C. Baird III. Technical Report WLTR Wright Laboratory. WrightPatterson Air Force Base, OH
ADVANTAGE UPDATING Leemon C. Baird III Technical Report WLTR931146 Wright Laboratory WrightPatterson Air Force Base, OH 454337301 Address: WL/AAAT, Bldg 635 2185 Avionics Circle WrightPatterson Air
More informationMonte Carlo Methods in Finance
Author: Yiyang Yang Advisor: Pr. Xiaolin Li, Pr. Zari Rachev Department of Applied Mathematics and Statistics State University of New York at Stony Brook October 2, 2012 Outline Introduction 1 Introduction
More informationDistributed Reinforcement Learning for Network Intrusion Response
Distributed Reinforcement Learning for Network Intrusion Response KLEANTHIS MALIALIS Doctor of Philosophy UNIVERSITY OF YORK COMPUTER SCIENCE September 2014 For my late grandmother Iroulla Abstract The
More informationOptimizing admission control while ensuring quality of service in multimedia networks via reinforcement learning*
Optimizing admission control while ensuring quality of service in multimedia networks via reinforcement learning* Timothy X Brown t, Hui Tong t, Satinder Singh+ t Electrical and Computer Engineering +
More informationChannel Allocation in Cellular Telephone. Systems. Lab. for Info. and Decision Sciences. Cambridge, MA 02139. bertsekas@lids.mit.edu.
Reinforcement Learning for Dynamic Channel Allocation in Cellular Telephone Systems Satinder Singh Department of Computer Science University of Colorado Boulder, CO 803090430 baveja@cs.colorado.edu Dimitri
More informationCircuits 1 M H Miller
Introduction to Graph Theory Introduction These notes are primarily a digression to provide general background remarks. The subject is an efficient procedure for the determination of voltages and currents
More informationAmajor benefit of MonteCarlo schedule analysis is to
2005 AACE International Transactions RISK.10 The Benefits of Monte Carlo Schedule Analysis Mr. Jason Verschoor, P.Eng. Amajor benefit of MonteCarlo schedule analysis is to expose underlying risks to
More informationReinforcement Learning: A Tutorial Survey and Recent Advances
A. Gosavi Reinforcement Learning: A Tutorial Survey and Recent Advances Abhijit Gosavi Department of Engineering Management and Systems Engineering 219 Engineering Management Missouri University of Science
More informationNonLife Insurance Mathematics
Thomas Mikosch NonLife Insurance Mathematics An Introduction with the Poisson Process Second Edition 4y Springer Contents Part I Collective Risk Models 1 The Basic Model 3 2 Models for the Claim Number
More informationWhy? A central concept in Computer Science. Algorithms are ubiquitous.
Analysis of Algorithms: A Brief Introduction Why? A central concept in Computer Science. Algorithms are ubiquitous. Using the Internet (sending email, transferring files, use of search engines, online
More informationObjective. Materials. TI73 Calculator
0. Objective To explore subtraction of integers using a number line. Activity 2 To develop strategies for subtracting integers. Materials TI73 Calculator Integer Subtraction What s the Difference? Teacher
More informationJiří Matas. Hough Transform
Hough Transform Jiří Matas Center for Machine Perception Department of Cybernetics, Faculty of Electrical Engineering Czech Technical University, Prague Many slides thanks to Kristen Grauman and Bastian
More informationFlorida Department of Education/Office of Assessment January 2012. Grade 7 FCAT 2.0 Mathematics Achievement Level Descriptions
Florida Department of Education/Office of Assessment January 2012 Grade 7 FCAT 2.0 Mathematics Grade 7 FCAT 2.0 Mathematics Reporting Category Geometry and Measurement Students performing at the mastery
More informationRisk Analysis Using Monte Carlo Simulation
Risk Analysis Using Monte Carlo Simulation Here we present a simple hypothetical budgeting problem for a business startup to demonstrate the key elements of Monte Carlo simulation. This table shows the
More informationEVALUATING ACADEMIC READINESS FOR APPRENTICESHIP TRAINING Revised For ACCESS TO APPRENTICESHIP
EVALUATING ACADEMIC READINESS FOR APPRENTICESHIP TRAINING For ACCESS TO APPRENTICESHIP MATHEMATICS SKILL OPERATIONS WITH INTEGERS AN ACADEMIC SKILLS MANUAL for The Precision Machining And Tooling Trades
More informationSkillbased Routing Policies in Call Centers
BMI Paper Skillbased Routing Policies in Call Centers by Martijn Onderwater January 2010 BMI Paper Skillbased Routing Policies in Call Centers Author: Martijn Onderwater Supervisor: Drs. Dennis Roubos
More informationNeural Fitted Q Iteration  First Experiences with a Data Efficient Neural Reinforcement Learning Method
Neural Fitted Q Iteration  First Experiences with a Data Efficient Neural Reinforcement Learning Method Martin Riedmiller Neuroinformatics Group, University of Onsabrück, 49078 Osnabrück Abstract. This
More informationTracking Algorithms. Lecture17: Stochastic Tracking. Joint Probability and Graphical Model. Probabilistic Tracking
Tracking Algorithms (2015S) Lecture17: Stochastic Tracking Bohyung Han CSE, POSTECH bhhan@postech.ac.kr Deterministic methods Given input video and current state, tracking result is always same. Local
More informationInequality, Mobility and Income Distribution Comparisons
Fiscal Studies (1997) vol. 18, no. 3, pp. 93 30 Inequality, Mobility and Income Distribution Comparisons JOHN CREEDY * Abstract his paper examines the relationship between the crosssectional and lifetime
More informationCharacterization and Modeling of Packet Loss of a VoIP Communication
Characterization and Modeling of Packet Loss of a VoIP Communication L. Estrada, D. Torres, H. Toral Abstract In this work, a characterization and modeling of packet loss of a Voice over Internet Protocol
More informationElementary Number Theory We begin with a bit of elementary number theory, which is concerned
CONSTRUCTION OF THE FINITE FIELDS Z p S. R. DOTY Elementary Number Theory We begin with a bit of elementary number theory, which is concerned solely with questions about the set of integers Z = {0, ±1,
More informationAn ant colony optimization for singlemachine weighted tardiness scheduling with sequencedependent setups
Proceedings of the 6th WSEAS International Conference on Simulation, Modelling and Optimization, Lisbon, Portugal, September 2224, 2006 19 An ant colony optimization for singlemachine weighted tardiness
More informationLecture 6. Weight. Tension. Normal Force. Static Friction. Cutnell+Johnson: 4.84.12, second half of section 4.7
Lecture 6 Weight Tension Normal Force Static Friction Cutnell+Johnson: 4.84.12, second half of section 4.7 In this lecture, I m going to discuss four different kinds of forces: weight, tension, the normal
More informationLearning Tetris Using the Noisy CrossEntropy Method
NOTE Communicated by Andrew Barto Learning Tetris Using the Noisy CrossEntropy Method István Szita szityu@eotvos.elte.hu András Lo rincz andras.lorincz@elte.hu Department of Information Systems, Eötvös
More informationReinforcement Learning: An Introduction. ****Draft****
i Reinforcement Learning: An Introduction Second edition, in progress ****Draft**** Richard S. Sutton and Andrew G. Barto c 2014, 2015, 2016 A Bradford Book The MIT Press Cambridge, Massachusetts London,
More informationNODAL ANALYSIS. Circuits Nodal Analysis 1 M H Miller
NODAL ANALYSIS A branch of an electric circuit is a connection between two points in the circuit. In general a simple wire connection, i.e., a 'shortcircuit', is not considered a branch since it is known
More informationBig Graph Processing: Some Background
Big Graph Processing: Some Background Bo Wu Colorado School of Mines Part of slides from: Paul Burkhardt (National Security Agency) and Carlos Guestrin (Washington University) Mines CSCI580, Bo Wu Graphs
More informationA Behavior Based Kernel for Policy Search via Bayesian Optimization
via Bayesian Optimization Aaron Wilson WILSONAA@EECS.OREGONSTATE.EDU Alan Fern AFERN@EECS.OREGONSTATE.EDU Prasad Tadepalli TADEPALL@EECS.OREGONSTATE.EDU Oregon State University School of EECS, 1148 Kelley
More informationCOMPUTATIONAL MODELS OF CLASSICAL CONDITIONING: A COMPARATIVE STUDY
COMPUTATIONAL MODELS OF CLASSICAL CONDITIONING: A COMPARATIVE STUDY Christian Balkenius Jan Morén christian.balkenius@fil.lu.se jan.moren@fil.lu.se Lund University Cognitive Science Kungshuset, Lundagård
More informationTennessee Mathematics Standards 20092010 Implementation. Grade Six Mathematics. Standard 1 Mathematical Processes
Tennessee Mathematics Standards 20092010 Implementation Grade Six Mathematics Standard 1 Mathematical Processes GLE 0606.1.1 Use mathematical language, symbols, and definitions while developing mathematical
More informationFundamentals of Actuarial Mathematics
Fundamentals of Actuarial Mathematics S. David Promislow York University, Toronto, Canada John Wiley & Sons, Ltd Contents Preface Notation index xiii xvii PARTI THE DETERMINISTIC MODEL 1 1 Introduction
More informationSimilarity Search and Mining in Uncertain Spatial and Spatio Temporal Databases. Andreas Züfle
Similarity Search and Mining in Uncertain Spatial and Spatio Temporal Databases Andreas Züfle Geo Spatial Data Huge flood of geo spatial data Modern technology New user mentality Great research potential
More informationApplied mathematics and mathematical statistics
Applied mathematics and mathematical statistics The graduate school is organised within the Department of Mathematical Sciences.. Deputy head of department: Aila Särkkä Director of Graduate Studies: Marija
More informationDynamic Programming. Lecture 11. 11.1 Overview. 11.2 Introduction
Lecture 11 Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to solve many different types of problems in time O(n 2 ) or O(n 3 ) for which a naive approach
More informationAbsolute Value of Reasoning
About Illustrations: Illustrations of the Standards for Mathematical Practice (SMP) consist of several pieces, including a mathematics task, student dialogue, mathematical overview, teacher reflection
More informationEFFICIENT USE OF REINFORCEMENT LEARNING IN A COMPUTER GAME
EFFICIENT USE OF REINFORCEMENT LEARNING IN A COMPUTER GAME Yngvi Björnsson, Vignir Hafsteinsson, Ársæll Jóhannsson, and Einar Jónsson Reykjavík University, School of Computer Science Ofanleiti 2 Reykjavik,
More information3 More on Accumulation and Discount Functions
3 More on Accumulation and Discount Functions 3.1 Introduction In previous section, we used 1.03) # of years as the accumulation factor. This section looks at other accumulation factors, including various
More information