THE LOGIC OF ADAPTIVE BEHAVIOR

Transcription

1 THE LOGIC OF ADAPTIVE BEHAVIOR Knowledge Representation and Algorithms for Adaptive Sequential Decision Making under Uncertainty in First-Order and Relational Domains Martijn van Otterlo Department of Computer Science Katholieke Universiteit Leuven, Belgium IOS Press Amsterdam Berlin Oxford Tokyo Washington, DC

2 Contents Preface vii CHAPTER 1 introduction l 1.1. Science and Engineering of Adaptive Behavior Artificial Intelligence Constructing Artificial Behavior The Reinforcement Learning Paradigm You Can Only Learn What You Can Represent Generalization, Abstraction and Representation Formation CANTOR: Representing the World in Snapshots BOOLE: Representing the World in Twenty Questions FREGE: Representing the World in Terms of Objects and Relations The World Might be Larger than We See About the Contents and Structure of this Book Main Theme of This Book A Road Map Other Main Themes and Contributions 28 I Learning Sequential Decision Making under Uncertainty 29 CHAPTER 2 Markov Decision Processes: Concepts and Algorithms Learning Sequential Decision Making A Formal Framework Markov Decision Processes " Policies Optimality Criteria and Discounting Value Functions and Bellman Equations Solving Markov decision processes Dynamic Programming: Model-Based Solution Techniques Fundamental DP Algorithms Efficient DP Algorithms Reinforcement Learning: Model-Free Solution Techniques Temporal Difference Learning 54

3 Monte Carlo Methods Efficient Exploration and Value Updating Beyond the Markov Assumption Partially Observable Markov Decision Processes Discussion 66 CHAPTER 3 Generalization and Abstraction in Markov Decision Processes From Algorithmic to Representational Algorithmic Aspects Fundamental Problems of Huge State Spaces Representational Aspects..".' The Essence of Abstraction Knowledge Representation Definitions and Theories of Abstraction Representation Change Abstraction in the MDP Setting Dimensions of MDP Abstractions The PIAGET-Principle Representations in MDP Abstractions ABSTRACTION TYPE I: State Spaces Model-Based State Abstractions Model-Free State Abstractions ABSTRACTION TYPE II: Factored Markov Decision Processes Structured Representation Structured Algorithms ABSTRACTION TYPE III: Value Function Approximation Fundamentals of Value Function Approximation Architectures for VFA ABSTRACTION TYPE IV: Searching in Policy Space ABSTRACTION TYPE V: Hierarchical and Temporal Abstraction Semi-Markov Decision Processes Fixed Hierarchical Abstractions Model-Minimization for SMDPs Dynamic Hierarchical Abstractions An Abstraction Case Study: Fingerprint Recognition Reinforcement Learning for Minutiae Detection Experimental Results Benefits of Various Abstractions Discusssion 144 II Sequential Decisions in the First-Order Setting 151 CHAPTER 4 Reasoning, Learning and Acting in Worlds with Objects The World Consists of Objects Objects are Omnipresent and Indispensable A Relational Domain: BLOCKS WORLD 163 XII

4 Representing a World of Objects and Relations Representation and Inference in First-Order Domains First-Order Logic Fragments and Extensions of FOL First-Order Abstraction and Generalization Learning in First-Order Domains Obtaining Logical Abstractions Inductive Logic Programming Statistical Relational Learning..' Acting in First-Order Domains S Formalizing and Modeling First-Order Domains Two Characteristic Systems Beyond Basic Action Theories Learning Sequential Decision Making in Relational Domains Lifting the MDP Framework to First-Order Domains The PIAGET-Principle in First-Order Domains Learning and Representation Tasks in Relational RL What is Relational RL?: Different Viewpoints Conclusions 243 CHAPTER 5 Model-Free Algorithms for Relational MDPs Model-Free Relational Reinforcement Learning Sampling and Structural Induction Representations, and Value Functions vs. Policies CARCASS: A Model-Free, Value-Based Approach Relational Abstractions over RMDPs Q-LearningforCARCASSs Indirect Value Learning for CARCASSs using Approximate Models Analysis and Experiments Discussion A Survey of Model-Free, Value-Based Approaches Value-Based Learning on Fixed Abstraction Levels Value-Based Learning using Dynamic Generalization Discussion of Model-Free, Value-Based Techniques GREY: Evolutionary Policy Search in Relational Domains Evolutionary Search and ILP GREY'S Anatomy Experimental Evaluation A Survey of Policy-Based Model-Free Relational RL Evolutionary Policy Search Policy Search as Classification Policy Gradient Approaches Discussion 301 CHAPTER 6 Model-Based Algorithms for Relational MDPs Intensional Dynamic Programming in Five Easy Steps STEP I: Classical Dynamic Programming 310 XIII

5 STEP II: Replacing Tables by Sets STEP III: Set-Based Value Functions STEP IV: Set-Based Dynamic Programming STEP V: Intensional Dynamic Programming A Relational State Description Language Abstract States Abstract Actions Rewards Domain Theory and Constraint Handling Markov Decision Programs, Value Functions and Policies REBEL: Value Iteration for Markov Decision Programs Overlaps, Regression and Weakest Preconditions Combination and First-Order Decision-Theoretic Regression Maximization: Computing Abstract State Values Relational Bellman Backup Operator Experiments Logic Programming meets Dynamic Programming Tabling Policy Induction in REBEL Other Extensions and Domain Theories A Survey of Model-Based Approaches Methods for exact IDP in First-Order Domains Approximate Model-Based Methods for First-Order MDPs Beyond the Markov Assumption Discussion 392 III Implications, Challenges and Conclusions 395 CHAPTER 7 Sapience, Models and Hierarchy Scaling Up... ; Extending Mental States Declarative versus Procedural Representations Learning vs. Reasoning Examples of Existing Formalisms Characterizing Sapient Agents Cognitive Agents Learning in Cognitive Agents The Social Environment Discussion of the Sapient Model of Agents A Survey of Hierarchies, Models, Guidance and Transfer Learning World Models Bias, Guidance and Heuristics Hierarchies Transfer Multi-Agent Approaches 419 XIV

6 7.4. Discussion 420 CHAPTER 8 Conclusions and Future Directions Conclusions and Reflections Main Argument Contributions Dimensions of First-Order MDPs and Solution Algorithms Future Challenges Upgrading the Complete Spectrum of RL Methods Techniques Developed in this Book i v.._._ Representational Aspects Algorithmic Aspects Theory Agents, Cognitive Architectures, Reasoning and Transfer Applications, Actions, Robots and Activities Benchmarks and Toolboxes Beyond the Markov Assumption Concluding Remarks 442 Bibliography 443 List of Acronyms 477 Author Index 479 XV