Statistical Reinforcement Learning: Modern Machine Learning Approaches (Hardcover)

Introduction to Reinforcement Learning
Reinforcement Learning
Mathematical Formulation
Structure of the Book
     Model-Free Policy Iteration
     Model-Free Policy Search
     Model-Based Reinforcement Learning

MODEL-FREE POLICY ITERATION

Policy Iteration with Value Function Approximation
Value Functions
     State Value Functions
     State-Action Value Functions
Least-Squares Policy Iteration
      Immediate-Reward Regression
     Algorithm
     Regularization
     Model Selection
Remarks

Basis Design for Value Function Approximation
Gaussian Kernels on Graphs
     MDP-Induced Graph
     Ordinary Gaussian Kernels
     Geodesic Gaussian Kernels
     Extension to Continuous State Spaces
Illustration
     Setup
     Geodesic Gaussian Kernels
     Ordinary Gaussian Kernels
     Graph-Laplacian Eigenbases
     Diffusion Wavelets
Numerical Examples
     Robot-Arm Control
     Robot-Agent Navigation
Remarks

Sample Reuse in Policy Iteration
Formulation
Off-Policy Value Function Approximation
     Episodic Importance Weighting
     Per-Decision Importance Weighting
     Adaptive Per-Decision Importance Weighting
     Illustration
Automatic Selection of Flattening Parameter
     Importance-Weighted Cross-Validation
     Illustration
Sample-Reuse Policy Iteration
     Algorithm
     Illustration
Numerical Examples
     Inverted Pendulum
     Mountain Car
Remarks

Active Learning in Policy Iteration
Efficient Exploration with Active Learning
     Problem Setup
     Decomposition of Generalization Error
     Estimation of Generalization Error
     Designing Sampling Policies
     Illustration
Active Policy Iteration
     Sample-Reuse Policy Iteration with Active Learning
     Illustration
Numerical Examples
Remarks

Robust Policy Iteration
Robustness and Reliability in Policy Iteration
     Robustness
     Reliability
Least Absolute Policy Iteration
     Algorithm
     Illustration
     Properties
Numerical Examples
Possible Extensions
     Huber Loss
     Pinball Loss
     Deadzone-Linear Loss
     Chebyshev Approximation
     Conditional Value-At-Risk
Remarks

MODEL-FREE POLICY SEARCH

Direct Policy Search by Gradient Ascent
Formulation
Gradient Approach
     Gradient Ascent
     Baseline Subtraction for Variance Reduction
     Variance Analysis of Gradient Estimators
Natural Gradient Approach 
     Natural Gradient Ascent
     Illustration
Application in Computer Graphics: Artist Agent
     Sumie Paining 
     Design of States, Actions, and Immediate Rewards
     Experimental Results
Remarks

Direct Policy Search by Expectation-Maximization
Expectation-Maximization Approach
Sample Reuse
     Episodic Importance Weighting
     Per-Decision Importance Weight
     Adaptive Per-Decision Importance Weighting
     Automatic Selection of Flattening Parameter
     Reward-Weighted Regression with Sample Reuse
Numerical Examples
Remarks

Policy-Prior Search
Formulation
Policy Gradients with Parameter-Based Exploration 
     Policy-Prior Gradient Ascent
     Baseline Subtraction for Variance Reduction
     Variance Analysis of Gradient Estimators
     Numerical Examples
Sample Reuse in Policy-Prior Search 
     Importance Weighting
     Variance Reduction by Baseline Subtraction
     Numerical Examples
Remarks

MODEL-BASED REINFORCEMENT LEARNING

Transition Model Estimation
Conditional Density Estimation
     Regression-Based Approach
     Q-Neighbor Kernel Density Estimation
     Least-Squares Conditional Density Estimation
Model-Based Reinforcement Learning
Numerical Examples
     Continuous Chain Walk
     Humanoid Robot Control
Remarks

Dimensionality Reduction for Transition Model Estimation
Sufficient Dimensionality Reduction
Squared-Loss Conditional Entropy
     Conditional Independence
     Dimensionality Reduction with SCE
     Relation to Squared-Loss Mutual Information
Numerical Examples
     Artificial and Benchmark Datasets 
     Humanoid Robot
Remarks

References
Index