# NEURAL NETWORKS AND REINFORCEMENT LEARNING. Abhijit Gosavi

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 NEURAL NETWORKS AND REINFORCEMENT LEARNING Abhijit Gosavi Department of Engineering Management and Systems Engineering Missouri University of Science and Technology Rolla, MO

2 Outline A Quick Introduction to Reinforcement Learning The Role of Neural Networks in Reinforcement Learning Some Algorithms The Success Stories and the Failures Some Online Demos Future of Neural Networks and Reinforcement Learning 2

3 What is Reinforcement Learning? Reinforcement Learning (RL) is a technique useful in solving control optimization problems. By control optimization, we mean the problem of recognizing the best action in every state visited by the system so as to optimize some objective function, e.g., the average reward per unit time and the total discounted reward over a given time horizon. Typically, RL is used when the system has a very large number of states (>> 1000) and has complex stochastic structure, which is not amenable to closed form analysis. When problems have a relative small number of states and the underlying random structure is relatively simple, one can use dynamic programming. 3

4 Q-Learning The central idea in Q-Learning is to recognize or learn the optimal action in every state visited by the system (also called the optimal policy) via trial and error. The trial and error mechanism can be implemented within the real-world system (commonly seen in robotics) or within a simulator (commonly seen in management science / industrial engineering). 4

5 Q-Learning: Working Mechanism The agent chooses an action, obtains feedback for that action, and uses the feedback to update its database. In its database, the agent keeps a so-called Q-factor for every state-action pair. When the feedback for selecting an action in a state is positive, the associated Q-factor s value is increased, while if the feedback is negative, the value is decreased. The feedback consists of the immediate revenue or reward plus the value of the next state. 5

6 Simulator (environment) Feedback r(i,a,j) Action a RL Algorithm (Agent) Figure 1: Trial and error mechanism of RL. The action selected by the RL agent is fed into the simulator. The simulator simulates the action, and the resultant feedback obtained is fed back into the knowledge-base (Q-factors) of the agent. The agent uses the RL algorithm to update its knowledge-base, becomes smarter in the process, and then selects a better action. 5

7 Q-Learning: Feedback The immediate reward is denoted by r(i, a, j), where i is the current state, a the action chosen in the current state, and j the next state. The value of any state is given by the maximum Q-factor in that state. Thus, if there are two actions in each state, the value of a state is the maximum of the two Q-factors for that state. In mathematical terms: feedback = r(i, a, j) + λ max b Q(j, b), where λ is the discount factor, which discounts the values of future states. Usually, λ = 1/(1 + R) where R is the rate of discounting. 6

8 Q-Learning: Algorithm The core of the Q-Learning algorithm uses the following updating equation: Q(i, a) [1 α]q(i, a) + α [feedback], i.e., Q(i, a) [1 α]q(i, a) + α [ ] r(i, a, j) + λ max Q(j, b), b where α is the learning rate (or step size). 7

9 Q-Learning: Where Do We Need Neural Networks? When we have a very large number of state-action pairs, it is not feasible to store every Q-factor separately. Then, it makes sense to store the Q-factors for a given action within one neural network. When a Q-factor is needed, it is fetched from its neural network. When a Q-factor is to be updated, the new Q-factor is used to update the neural network itself. For any given action, Q(i, a) is a function of i, the state. Hence, we will call it a Q-function in what follows. 8

10 Incremental or Batch? Neural networks are generally of two types: batch updating or incremental updating. The batch updating neural networks require all the data at once, while the incremental neural networks take one data piece at a time. For reinforcement learning, we need incremental neural networks since every time the agent receives feedback, we obtain a new piece of data that must be used to update some neural network. 9

11 Neurons and Backpropagation Neurons are used for fitting linear forms, e.g., y = a + bi where i is the input (the state in our case). Also called adenaline rule or Widrow-Hoff rule. Backprop is used when the Q-factor is non-linear in i, which is usually the case. (Algorithm was invented by Paul Werbos in 1975). Backprop is a universal function approximator, and ideally should fit any Q-function! Neurons can also be used by fitting the Q-function in a piecewise manner, where a linear fit is introduced in every piece. 10

12 Algorithm for Incremental Neuron Step 1: Initialize the weights of the neural network. Step 2a: Compute the output o p using output = k w(j)x(j), where j=0 w(j) is the jth weight of neuron and x(j) is the jth input. Step 2b: Update each w(i) for i = 0, 1,..., k using: w(i) w(i) + µ[target output]x(i), where the target is the updated Q-factor. Step 3: Increment iter by 1. If iter < iter max, return to Step 2. 11

13 Q-Learning combined with Neuron We now discuss a simple example of Q-Learning coupled with a neuron using incremental updating on an MDP with two states and two actions. Step 1. Initialize the weights of the neuron for action 1, i.e., w(1, 1) and w(2, 1), to small random numbers, and set the corresponding weights for action 2 to the same values. Set k, the number of state transitions, to 0. Start system simulation at any arbitrary state. Set k max to a large number. Step 2. Let the state be i. Simulate action a with a probability of 1/ A(i). Let the next state be j. Step 3. Evaluate the Q-factor for state-action pair, (i, a), which we 12

14 will call Q old, using the following: Q old = w(1, a) + w(2, a)i. Now evaluate the Q-factor for state j associated to each action, i.e., Q next (1) = w(1, 1) + w(2, 1)j; Q next (2) = w(1, 2) + w(2, 2)j. Now set Q next = max {Q next (1), Q next (2)}. Step 3a. Update the relevant Q-factor as follows (via Q-Learning). Qnew (1 α)q old + α [r(i, a, j) + λq next ]. (1) Step 3b. The current step in turn may contain a number of steps and involves the neural network updating. Set m = 0, where m is the number of iterations used within the neural network. Set m max, the maximum number of iterations for neuronal updating, 13

15 to a suitable value (we will discuss this value below). Step 3b(i). Update the weights of the neuron associated to action a as follows: w(1, a) w(1, a)+µ(qnew Q old )1; w(2, a) w(2, a)+µ(qnew Q old )i. (2) Step 3b(ii). Increment m by 1. If m < m max, return to Step 3b(i); otherwise, go to Step 4. Step 4. Increment k by 1. If k < k max, set i j; then go to Step 2. Otherwise, go to Step 5. Step 5. The policy learned, ˆµ, is virtually stored in the weights. To determine the action prescribed in a state i where i S, compute the following: µ(i) arg max a A(i) [w(1, a) + w(2, a)i]. 14

16 Some important remarks need to be made in regards to the algorithm above. Remark 1. Note that the update in Equation (2) is the update used by an incremental neuron that seeks to store the Q-factors for a given action. Remark 2. The step-size µ is the step size of the neuron, and it can be also be decayed with every iteration m 15

17 Backpropagation The algorithm is more complicated, since it involves multiple layers and the threshold functions. The algorithm requires significant tuning. Incremental version of the algorithm must be used. 16

18 Integration of neural networks with Q-Learning In some sense resembles integration of hardware (neural network) and software (Q-Learning). Integration has to be tightly controlled; otherwise results can be disappointing. Mixed success with backprop; Crites and Barto (1996; elevator case study), Das et al (1999; preventive maintenance used backprop with great success), and Sui (supply chains) have obtained great success, but some other failures reported (Sutton and Barto s 1998 textbook). Piecewise fitting of the function using neurons has also shown robust and stable behavior: Gosavi (2004; airline revenue management). 17

19 Toy Problem: Two states and Two actions Table 1: The table shows Q-factors. Method Q(1, 1) Q(1, 2) Q(2, 1) Q(2, 2) Q-factor-value iteration Q-Learning with α = 150/(300 + n) Q-Learning with Neuron

20 Future Directions A great deal of current research in function approximation and RL is using regression, e.g., algorithms such as LSTD (least squares temporal differences) However, most exciting RL applications in robotics and neuro-science studies are still using neural networks. Support Vector Machines are yet another data mining tool that have seen some recent applications in RL. 19

21 References D. Bertsekas and J. Tsitsiklis. Neuro-dynamic programming, Athena, R. Crites and A. Barto. Improving elevator performance using reinforcement learning. In Neural Information Processing Systems (NIPS) T.K. Das,, S. Mahadevan, and N. Marchalleck. Solving semi-markov decision problems using average reward reinforcement learning. Management Science, 45(4):560574, Simulation-based optimization: Parametric optimization techniques and reinforcement learning, Kluwer Academic Publishers, 2009., N. Bandla, and T. K. Das. A reinforcement learning 20

22 approach to a single leg airline revenue management problem with multiple fare classes and overbooking. IIE Transactions (Special Issue on Large-Scale Optimization edited by Suvrajeet Sen), 34(9):729742, R. Sutton and A. Barto. Reinforcement Learning: An Introduction, MIT Press,

### Motivation. Motivation. Can a software agent learn to play Backgammon by itself? Machine Learning. Reinforcement Learning

Motivation Machine Learning Can a software agent learn to play Backgammon by itself? Reinforcement Learning Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut

### Semi-Markov Adaptive Critic Heuristics with Application to Airline Revenue Management

Semi-Markov Adaptive Critic Heuristics with Application to Airline Revenue Management Ketaki Kulkarni Abhijit Gosavi Email: gosavia@mst.edu Susan Murray Katie Grantham Department of Engineering Management

### Neuro-Dynamic Programming An Overview

1 Neuro-Dynamic Programming An Overview Dimitri Bertsekas Dept. of Electrical Engineering and Computer Science M.I.T. September 2006 2 BELLMAN AND THE DUAL CURSES Dynamic Programming (DP) is very broadly

### Inductive QoS Packet Scheduling for Adaptive Dynamic Networks

Inductive QoS Packet Scheduling for Adaptive Dynamic Networks Malika BOURENANE Dept of Computer Science University of Es-Senia Algeria mb_regina@yahoo.fr Abdelhamid MELLOUK LISSI Laboratory University

### Using Markov Decision Processes to Solve a Portfolio Allocation Problem

Using Markov Decision Processes to Solve a Portfolio Allocation Problem Daniel Bookstaber April 26, 2005 Contents 1 Introduction 3 2 Defining the Model 4 2.1 The Stochastic Model for a Single Asset.........................

### Temporal Difference Learning in the Tetris Game

Temporal Difference Learning in the Tetris Game Hans Pirnay, Slava Arabagi February 6, 2009 1 Introduction Learning to play the game Tetris has been a common challenge on a few past machine learning competitions.

### Reinforcement Learning

Reinforcement Learning LU 2 - Markov Decision Problems and Dynamic Programming Dr. Martin Lauer AG Maschinelles Lernen und Natürlichsprachliche Systeme Albert-Ludwigs-Universität Freiburg martin.lauer@kit.edu

### A Sarsa based Autonomous Stock Trading Agent

A Sarsa based Autonomous Stock Trading Agent Achal Augustine The University of Texas at Austin Department of Computer Science Austin, TX 78712 USA achal@cs.utexas.edu Abstract This paper describes an autonomous

### Learning Tetris Using the Noisy Cross-Entropy Method

NOTE Communicated by Andrew Barto Learning Tetris Using the Noisy Cross-Entropy Method István Szita szityu@eotvos.elte.hu András Lo rincz andras.lorincz@elte.hu Department of Information Systems, Eötvös

### Reinforcement Learning

Reinforcement Learning LU 2 - Markov Decision Problems and Dynamic Programming Dr. Joschka Bödecker AG Maschinelles Lernen und Natürlichsprachliche Systeme Albert-Ludwigs-Universität Freiburg jboedeck@informatik.uni-freiburg.de

### Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method

Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method Martin Riedmiller Neuroinformatics Group, University of Onsabrück, 49078 Osnabrück Abstract. This

### TD(0) Leads to Better Policies than Approximate Value Iteration

TD(0) Leads to Better Policies than Approximate Value Iteration Benjamin Van Roy Management Science and Engineering and Electrical Engineering Stanford University Stanford, CA 94305 bvr@stanford.edu Abstract

### Reinforcement Learning: A Tutorial Survey and Recent Advances

A. Gosavi Reinforcement Learning: A Tutorial Survey and Recent Advances Abhijit Gosavi Department of Engineering Management and Systems Engineering 219 Engineering Management Missouri University of Science

### Learning. Artificial Intelligence. Learning. Types of Learning. Inductive Learning Method. Inductive Learning. Learning.

Learning Learning is essential for unknown environments, i.e., when designer lacks omniscience Artificial Intelligence Learning Chapter 8 Learning is useful as a system construction method, i.e., expose

### Learning Agents: Introduction

Learning Agents: Introduction S Luz luzs@cs.tcd.ie October 22, 2013 Learning in agent architectures Performance standard representation Critic Agent perception rewards/ instruction Perception Learner Goals

### INTRODUCTION TO NEURAL NETWORKS

INTRODUCTION TO NEURAL NETWORKS Pictures are taken from http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html http://research.microsoft.com/~cmbishop/prml/index.htm By Nobel Khandaker Neural Networks An

### Eligibility Traces. Suggested reading: Contents: Chapter 7 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, 1998.

Eligibility Traces 0 Eligibility Traces Suggested reading: Chapter 7 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, 1998. Eligibility Traces Eligibility Traces 1 Contents:

### Recurrent Neural Networks

Recurrent Neural Networks Neural Computation : Lecture 12 John A. Bullinaria, 2015 1. Recurrent Neural Network Architectures 2. State Space Models and Dynamical Systems 3. Backpropagation Through Time

### Lecture 6. Artificial Neural Networks

Lecture 6 Artificial Neural Networks 1 1 Artificial Neural Networks In this note we provide an overview of the key concepts that have led to the emergence of Artificial Neural Networks as a major paradigm

### Learning to Play Board Games using Temporal Difference Methods

Learning to Play Board Games using Temporal Difference Methods Marco A. Wiering Jan Peter Patist Henk Mannen institute of information and computing sciences, utrecht university technical report UU-CS-2005-048

### Efficient online learning of a non-negative sparse autoencoder

and Machine Learning. Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-93030-10-2. Efficient online learning of a non-negative sparse autoencoder Andre Lemme, R. Felix Reinhart and Jochen J. Steil

### CHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER

93 CHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER 5.1 INTRODUCTION The development of an active trap based feeder for handling brakeliners was discussed

### Chapter 4: Artificial Neural Networks

Chapter 4: Artificial Neural Networks CS 536: Machine Learning Littman (Wu, TA) Administration icml-03: instructional Conference on Machine Learning http://www.cs.rutgers.edu/~mlittman/courses/ml03/icml03/

### Call Admission Control and Routing in Integrated Service Networks Using Reinforcement Learning

Call Admission Control and Routing in Integrated Service Networks Using Reinforcement Learning Peter Marbach LIDS MIT, Room 5-07 Cambridge, MA, 09 email: marbach@mit.edu Oliver Mihatsch Siemens AG Corporate

### Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector

### Assignment #1. Example: tetris state: board configuration + shape of the falling piece ~2 200 states! Recap RL so far. Page 1

Generalization and function approximation CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Represent

### ADVANTAGE UPDATING. Leemon C. Baird III. Technical Report WL-TR Wright Laboratory. Wright-Patterson Air Force Base, OH

ADVANTAGE UPDATING Leemon C. Baird III Technical Report WL-TR-93-1146 Wright Laboratory Wright-Patterson Air Force Base, OH 45433-7301 Address: WL/AAAT, Bldg 635 2185 Avionics Circle Wright-Patterson Air

### Lecture 5: Model-Free Control

Lecture 5: Model-Free Control David Silver Outline 1 Introduction 2 On-Policy Monte-Carlo Control 3 On-Policy Temporal-Difference Learning 4 Off-Policy Learning 5 Summary Introduction Model-Free Reinforcement

### In the past two decades, interest in supply chain management

A Reinforcement Learning Approach for Inventory Replenishment in Vendor-Managed Inventory Systems With Consignment Inventory Zheng Sui Terra Technology Abhijit Gosavi Missouri University of Science and

### OPTIMIZATION AND OPERATIONS RESEARCH Vol. IV - Markov Decision Processes - Ulrich Rieder

MARKOV DECISIO PROCESSES Ulrich Rieder University of Ulm, Germany Keywords: Markov decision problem, stochastic dynamic program, total reward criteria, average reward, optimal policy, optimality equation,

### Reinforcement Learning with Factored States and Actions

Journal of Machine Learning Research 5 (2004) 1063 1088 Submitted 3/02; Revised 1/04; Published 8/04 Reinforcement Learning with Factored States and Actions Brian Sallans Austrian Research Institute for

### Jae Won idee. School of Computer Science and Engineering Sungshin Women's University Seoul, 136-742, South Korea

STOCK PRICE PREDICTION USING REINFORCEMENT LEARNING Jae Won idee School of Computer Science and Engineering Sungshin Women's University Seoul, 136-742, South Korea ABSTRACT Recently, numerous investigations

### On-line Policy Improvement using Monte-Carlo Search

On-line Policy Improvement using Monte-Carlo Search Gerald Tesauro IBM T. J. Watson Research Center P. O. Box 704 Yorktown Heights, NY 10598 Gregory R. Galperin MIT AI Lab 545 Technology Square Cambridge,

### Algorithms for Reinforcement Learning

Algorithms for Reinforcement Learning Draft of the lecture published in the Synthesis Lectures on Artificial Intelligence and Machine Learning series by Morgan & Claypool Publishers Csaba Szepesvári June

### On-Line Learning Control by Association and Reinforcement

264 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 2, MARCH 2001 On-Line Learning Control by Association and Reinforcement Jennie Si, Senior Member, IEEE, and Yu-Tsung Wang, Member, IEEE Abstract This

### An Environment Model for N onstationary Reinforcement Learning

An Environment Model for N onstationary Reinforcement Learning Samuel P. M. Choi Dit-Yan Yeung Nevin L. Zhang pmchoi~cs.ust.hk dyyeung~cs.ust.hk lzhang~cs.ust.hk Department of Computer Science, Hong Kong

### Tetris: Experiments with the LP Approach to Approximate DP

Tetris: Experiments with the LP Approach to Approximate DP Vivek F. Farias Electrical Engineering Stanford University Stanford, CA 94403 vivekf@stanford.edu Benjamin Van Roy Management Science and Engineering

### Channel Allocation in Cellular Telephone. Systems. Lab. for Info. and Decision Sciences. Cambridge, MA 02139. bertsekas@lids.mit.edu.

Reinforcement Learning for Dynamic Channel Allocation in Cellular Telephone Systems Satinder Singh Department of Computer Science University of Colorado Boulder, CO 80309-0430 baveja@cs.colorado.edu Dimitri

### Tree-Based Batch Mode Reinforcement Learning

Journal of Machine Learning Research 6 (2005) 503 556 Submitted 11/03; Revised 4/04; Published 4/05 Tree-Based Batch Mode Reinforcement Learning Damien Ernst Pierre Geurts Louis Wehenkel Department of

### Introduction to Machine Learning and Data Mining. Prof. Dr. Igor Trajkovski trajkovski@nyus.edu.mk

Introduction to Machine Learning and Data Mining Prof. Dr. Igor Trakovski trakovski@nyus.edu.mk Neural Networks 2 Neural Networks Analogy to biological neural systems, the most robust learning systems

### Reinforcement learning and conditioning: an overview. Université Charles de Gaulle, Lille, France

Reinforcement learning and conditioning: an overview Jérémie Jozefowiez Université Charles de Gaulle, Lille, France jozefowiez@univ-lille3.fr October 8, 2002 Contents 1 Computational foundations of reinforcement

### 6.2.8 Neural networks for data mining

6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural

### Rate-based artificial neural networks and error backpropagation learning. Scott Murdison Machine learning journal club May 16, 2016

Rate-based artificial neural networks and error backpropagation learning Scott Murdison Machine learning journal club May 16, 2016 Murdison, Leclercq, Lefèvre and Blohm J Neurophys 2015 Neural networks???

### Solving Nonlinear Equations Using Recurrent Neural Networks

Solving Nonlinear Equations Using Recurrent Neural Networks Karl Mathia and Richard Saeks, Ph.D. Accurate Automation Corporation 71 Shallowford Road Chattanooga, Tennessee 37421 Abstract A class of recurrent

### Introduction to Machine Learning Using Python. Vikram Kamath

Introduction to Machine Learning Using Python Vikram Kamath Contents: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Introduction/Definition Where and Why ML is used Types of Learning Supervised Learning Linear Regression

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### Neural Networks and Support Vector Machines

INF5390 - Kunstig intelligens Neural Networks and Support Vector Machines Roar Fjellheim INF5390-13 Neural Networks and SVM 1 Outline Neural networks Perceptrons Neural networks Support vector machines

### Lecture 1: Introduction to Neural Networks Kevin Swingler / Bruce Graham

Lecture 1: Introduction to Neural Networks Kevin Swingler / Bruce Graham kms@cs.stir.ac.uk 1 What are Neural Networks? Neural Networks are networks of neurons, for example, as found in real (i.e. biological)

### Goal-Directed Online Learning of Predictive Models

Goal-Directed Online Learning of Predictive Models Sylvie C.W. Ong, Yuri Grinberg, and Joelle Pineau McGill University, Montreal, Canada Abstract. We present an algorithmic approach for integrated learning

### Chapter 2 The Research on Fault Diagnosis of Building Electrical System Based on RBF Neural Network

Chapter 2 The Research on Fault Diagnosis of Building Electrical System Based on RBF Neural Network Qian Wu, Yahui Wang, Long Zhang and Li Shen Abstract Building electrical system fault diagnosis is the

### The Neuro Slot Car Racer: Reinforcement Learning in a Real World Setting

The Neuro Slot Car Racer: Reinforcement Learning in a Real World Setting Tim C. Kietzmann Neuroinformatics Group Institute of Cognitive Science University of Osnabrück tkietzma@uni-osnabrueck.de Martin

### Hiroshi ISHIGURO, Ritsuko SATO & Toru ISHIDA. Kyoto University. Sakyo-ku, Kyoto , JAPAN.

Robot Oriented State Space Construction Hiroshi ISHIGURO, Ritsuko SATO & Toru ISHIDA Department of Information Science Kyoto University Sakyo-ku, Kyoto 606-01, JAPAN E-mail: ishiguro/ritsuko/ishida@kuis.kyoto-u.ac.jp

### Self Organizing Maps: Fundamentals

Self Organizing Maps: Fundamentals Introduction to Neural Networks : Lecture 16 John A. Bullinaria, 2004 1. What is a Self Organizing Map? 2. Topographic Maps 3. Setting up a Self Organizing Map 4. Kohonen

### Neural Networks. Introduction to Artificial Intelligence CSE 150 May 29, 2007

Neural Networks Introduction to Artificial Intelligence CSE 150 May 29, 2007 Administration Last programming assignment has been posted! Final Exam: Tuesday, June 12, 11:30-2:30 Last Lecture Naïve Bayes

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Data Mining Lab 5: Introduction to Neural Networks

Data Mining Lab 5: Introduction to Neural Networks 1 Introduction In this lab we are going to have a look at some very basic neural networks on a new data set which relates various covariates about cheese

### SIMULATING CANCELLATIONS AND OVERBOOKING IN YIELD MANAGEMENT

CHAPTER 8 SIMULATING CANCELLATIONS AND OVERBOOKING IN YIELD MANAGEMENT In YM, one of the major problems in maximizing revenue is the number of cancellations. In industries implementing YM this is very

### Neural Networks. Neural network is a network or circuit of neurons. Neurons can be. Biological neurons Artificial neurons

Neural Networks Neural network is a network or circuit of neurons Neurons can be Biological neurons Artificial neurons Biological neurons Building block of the brain Human brain contains over 10 billion

### PMR5406 Redes Neurais e Lógica Fuzzy Aula 3 Multilayer Percetrons

PMR5406 Redes Neurais e Aula 3 Multilayer Percetrons Baseado em: Neural Networks, Simon Haykin, Prentice-Hall, 2 nd edition Slides do curso por Elena Marchiori, Vrie Unviersity Multilayer Perceptrons Architecture

### A Reinforcement Learning Approach for Supply Chain Management

A Reinforcement Learning Approach for Supply Chain Management Tim Stockheim, Michael Schwind, and Wolfgang Koenig Chair of Economics, esp. Information Systems, Frankfurt University, D-60054 Frankfurt,

### ANN-DEA Integrated Approach for Sensitivity Analysis in Efficiency Models

Iranian Journal of Operations Research Vol. 4, No. 1, 2013, pp. 14-24 ANN-DEA Integrated Approach for Sensitivity Analysis in Efficiency Models L. Karamali *,1, A. Memariani 2, G.R. Jahanshahloo 3 Here,

### A Multi-level Artificial Neural Network for Residential and Commercial Energy Demand Forecast: Iran Case Study

211 3rd International Conference on Information and Financial Engineering IPEDR vol.12 (211) (211) IACSIT Press, Singapore A Multi-level Artificial Neural Network for Residential and Commercial Energy

### Reinforcement Learning for Resource Allocation and Time Series Tools for Mobility Prediction

Reinforcement Learning for Resource Allocation and Time Series Tools for Mobility Prediction Baptiste Lefebvre 1,2, Stephane Senecal 2 and Jean-Marc Kelif 2 1 École Normale Supérieure (ENS), Paris, France,

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Building MLP networks by construction

University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2000 Building MLP networks by construction Ah Chung Tsoi University of

### Planning in Artificial Intelligence

Logical Representations and Computational Methods for Marov Decision Processes Craig Boutilier Department of Computer Science University of Toronto Planning in Artificial Intelligence Planning has a long

### A Gentle Introduction to Machine Learning

A Gentle Introduction to Machine Learning Second Lecture Part I Olov Andersson, AIICS Linköpings Universitet Outline of Machine Learning Lectures First we will talk about Supervised Learning Definition

### Goal-Directed Online Learning of Predictive Models

Goal-Directed Online Learning of Predictive Models Sylvie C.W. Ong, Yuri Grinberg, and Joelle Pineau School of Computer Science McGill University, Montreal, Canada Abstract. We present an algorithmic approach

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### IN AVIATION it regularly occurs that an airplane encounters. Guaranteed globally optimal continuous reinforcement learning.

GUARANTEED GLOBALLY OPTIMAL CONTINUOUS REINFORCEMENT LEARNING 1 Guaranteed globally optimal continuous reinforcement learning Hildo Bijl Abstract Self-learning and adaptable autopilots have the potential

### Introduction to Neural Networks : Revision Lectures

Introduction to Neural Networks : Revision Lectures John A. Bullinaria, 2004 1. Module Aims and Learning Outcomes 2. Biological and Artificial Neural Networks 3. Training Methods for Multi Layer Perceptrons

### Tetris: A Study of Randomized Constraint Sampling

Tetris: A Study of Randomized Constraint Sampling Vivek F. Farias and Benjamin Van Roy Stanford University 1 Introduction Randomized constraint sampling has recently been proposed as an approach for approximating

### Introduction to Artificial Neural Networks. Introduction to Artificial Neural Networks

Introduction to Artificial Neural Networks v.3 August Michel Verleysen Introduction - Introduction to Artificial Neural Networks p Why ANNs? p Biological inspiration p Some examples of problems p Historical

### MSCA 31000 Introduction to Statistical Concepts

MSCA 31000 Introduction to Statistical Concepts This course provides general exposure to basic statistical concepts that are necessary for students to understand the content presented in more advanced

### IFT3395/6390. Machine Learning from linear regression to Neural Networks. Machine Learning. Training Set. t (3.5, -2,..., 127, 0,...

IFT3395/6390 Historical perspective: back to 1957 (Prof. Pascal Vincent) (Rosenblatt, Perceptron ) Machine Learning from linear regression to Neural Networks Computer Science Artificial Intelligence Symbolic

### Learning Tetris Using the Noisy Cross-Entropy Method

Learning Tetris Using the Noisy Cross-Entropy Method István Szita 1, András Lőrincz 2 Department of Information Systems, Eötvös Loránd University Pázmány Péter sétány 1/C, Budapest, Hungary H-1117 Abstract

### TIME SERIES FORECASTING WITH NEURAL NETWORK: A CASE STUDY OF STOCK PRICES OF INTERCONTINENTAL BANK NIGERIA

www.arpapress.com/volumes/vol9issue3/ijrras_9_3_16.pdf TIME SERIES FORECASTING WITH NEURAL NETWORK: A CASE STUDY OF STOCK PRICES OF INTERCONTINENTAL BANK NIGERIA 1 Akintola K.G., 2 Alese B.K. & 2 Thompson

### Optimizing admission control while ensuring quality of service in multimedia networks via reinforcement learning*

Optimizing admission control while ensuring quality of service in multimedia networks via reinforcement learning* Timothy X Brown t, Hui Tong t, Satinder Singh+ t Electrical and Computer Engineering +

### Lecture 1: Introduction to Reinforcement Learning

Lecture 1: Introduction to Reinforcement Learning David Silver Outline 1 Admin 2 About Reinforcement Learning 3 The Reinforcement Learning Problem 4 Inside An RL Agent 5 Problems within Reinforcement Learning

### A Multi-agent Q-learning Framework for Optimizing Stock Trading Systems

A Multi-agent Q-learning Framework for Optimizing Stock Trading Systems Jae Won Lee 1 and Jangmin O 2 1 School of Computer Science and Engineering, Sungshin Women s University, Seoul, Korea 136-742 jwlee@cs.sungshin.ac.kr

### NEURAL NETWORKS A Comprehensive Foundation

NEURAL NETWORKS A Comprehensive Foundation Second Edition Simon Haykin McMaster University Hamilton, Ontario, Canada Prentice Hall Prentice Hall Upper Saddle River; New Jersey 07458 Preface xii Acknowledgments

### Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh

Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem

### Learning Exercise Policies for American Options

Yuxi Li Dept. of Computing Science University of Alberta Edmonton, Alberta Canada T6G 2E8 Csaba Szepesvari Dept. of Computing Science University of Alberta Edmonton, Alberta Canada T6G 2E8 Dale Schuurmans

### Optimization under uncertainty: modeling and solution methods

Optimization under uncertainty: modeling and solution methods Paolo Brandimarte Dipartimento di Scienze Matematiche Politecnico di Torino e-mail: paolo.brandimarte@polito.it URL: http://staff.polito.it/paolo.brandimarte

### Combining Q-Learning with Artificial Neural Networks in an Adaptive Light Seeking Robot

Combining Q-Learning with Artificial Neural Networks in an Adaptive Light Seeking Robot Steve Dini and Mark Serrano May 6, 2012 Abstract Q-learning is a reinforcement learning technique that works by learning

### Lecture 8 Artificial neural networks: Unsupervised learning

Lecture 8 Artificial neural networks: Unsupervised learning Introduction Hebbian learning Generalised Hebbian learning algorithm Competitive learning Self-organising computational map: Kohonen network

### Introduction to Logistic Regression

OpenStax-CNX module: m42090 1 Introduction to Logistic Regression Dan Calderon This work is produced by OpenStax-CNX and licensed under the Creative Commons Attribution License 3.0 Abstract Gives introduction

### degrees of freedom and are able to adapt to the task they are supposed to do [Gupta].

1.3 Neural Networks 19 Neural Networks are large structured systems of equations. These systems have many degrees of freedom and are able to adapt to the task they are supposed to do [Gupta]. Two very

### Stochastic Models for Inventory Management at Service Facilities

Stochastic Models for Inventory Management at Service Facilities O. Berman, E. Kim Presented by F. Zoghalchi University of Toronto Rotman School of Management Dec, 2012 Agenda 1 Problem description Deterministic

### Chapter 12 Discovering New Knowledge Data Mining

Chapter 12 Discovering New Knowledge Data Mining Becerra-Fernandez, et al. -- Knowledge Management 1/e -- 2004 Prentice Hall Additional material 2007 Dekai Wu Chapter Objectives Introduce the student to

### Intelligent Agents Serving Based On The Society Information

Intelligent Agents Serving Based On The Society Information Sanem SARIEL Istanbul Technical University, Computer Engineering Department, Istanbul, TURKEY sariel@cs.itu.edu.tr B. Tevfik AKGUN Yildiz Technical

### Introduction to Artificial Neural Networks MAE-491/591

Introduction to Artificial Neural Networks MAE-491/591 Artificial Neural Networks: Biological Inspiration The brain has been extensively studied by scientists. Vast complexity prevents all but rudimentary

### Introduction to Machine Learning

Introduction to Machine Learning Prof. Alexander Ihler Prof. Max Welling icamp Tutorial July 22 What is machine learning? The ability of a machine to improve its performance based on previous results:

### Improving Generalization

Improving Generalization Introduction to Neural Networks : Lecture 10 John A. Bullinaria, 2004 1. Improving Generalization 2. Training, Validation and Testing Data Sets 3. Cross-Validation 4. Weight Restriction

### Scheduling Algorithms for Downlink Services in Wireless Networks: A Markov Decision Process Approach

Scheduling Algorithms for Downlink Services in Wireless Networks: A Markov Decision Process Approach William A. Massey ORFE Department Engineering Quadrangle, Princeton University Princeton, NJ 08544 K.

### Introduction to Online Learning Theory

Introduction to Online Learning Theory Wojciech Kot lowski Institute of Computing Science, Poznań University of Technology IDSS, 04.06.2013 1 / 53 Outline 1 Example: Online (Stochastic) Gradient Descent

### NEURAL NETWORKS IN DATA MINING

NEURAL NETWORKS IN DATA MINING 1 DR. YASHPAL SINGH, 2 ALOK SINGH CHAUHAN 1 Reader, Bundelkhand Institute of Engineering & Technology, Jhansi, India 2 Lecturer, United Institute of Management, Allahabad,

### Nine Common Types of Data Mining Techniques Used in Predictive Analytics

1 Nine Common Types of Data Mining Techniques Used in Predictive Analytics By Laura Patterson, President, VisionEdge Marketing Predictive analytics enable you to develop mathematical models to help better