Code for:

Reinforcement Learning: An Introduction, 2nd edition
by Richard S. Sutton and Andrew G. Barto

• Re-implementations in Python by Shangtong Zhang
• Re-implementations in julialang by Jun Tian
• Original code for the first edition
• Re-implementation of first edition code in Matlab by John Weatherwax
  

• Chapter 1: Introduction
  • Tic-Tac-Toe Example (Lisp). In C.
• Chapter 2: Multi-armed Bandits
  • 10-armed Testbed Example, Figure 2.12(Lisp)
  • Testbed with Softmax Action Selection, Exercise 2.2 (Lisp)
  • Optimistic Initial Values Example, Figure 2.3 (Lisp)
  • UCB results, Figure 2.4 (Lisp)
  • Gradient bandits, Figure 2.5 (Lisp)
    
  • Parameter study of multiple algorithms, Figure 2.6 (Lisp)
    
• Chapter 3: Finite Markov Decision Processes
  • Pole-Balancing Example, Example 3.4 (C)
  • Gridworld Example 3.5 and 3.8, Code for Figures 3.2 and 3.5 (Lisp)
• Chapter 4: Dynamic Programming
  • Policy Evaluation, Gridworld Example 4.1, Figure 4.1 (Lisp)
  • Policy Iteration, Jack's Car Rental Example, Figure 4.2 (Lisp)
  • Value Iteration, Gambler's Problem Example, Figure 4.3 (Lisp)
• Chapter 5: Monte Carlo Methods
  • Monte Carlo Policy Evaluation, Blackjack Example 5.1, Figure 5.1 (Lisp)
  • Monte Carlo ES, Blackjack Example 5.3, Figure 5.2 (Lisp)
  • Blackjack estimate one state, Figure 5.3 (Lisp)
  • Infinite variance Example 5.5, Figure 5.4 (Lisp)
• Chapter 6: Temporal-Difference Learning
  • TD Prediction in Random Walk, Example 6.2 (Lisp)
  • TD Prediction in Random Walk with Batch Training, Example 6.3, Figure 6.2 (Lisp)
  • TD Prediction in Random Walk (MatLab by Jim Stone)
  • Double Q-learning vs conventional Q-learning Example 6.7, Figure 6.5 (Lisp)
    
• Chapter 7: n-step Bootstrapping
  
  • N-step TD on the Random Walk, Example 7.1, Figure 7.2: online and offline (Lisp). In C.
• Chapter 8: Planning and Learning with Tabular Methods
  • Trajectory Sampling Experiment, Figure 8.8 (Lisp)
• Chapter 9: On-policy Prediction with Approximation
  • State Aggregation on the 1000-state Random Walk, Figures 9.1, 9.2, and 9.5 (Lisp)
  • Coarseness of Coarse Coding, Example 9.3, Figure 9.8 (Lisp)
  • Why we use coarse coding, Figure 9.15 (Lisp)
  • Tile Coding: tiles3.html
    
• Chapter 10: On-policy Control with Approximation
• Linear Semi-gradient Sarsa(lambda) on the Mountain-Car, Figure 10.1
• n-step Sarsa on Mountain Car, Figures 10.2-4 (Lisp) with tile coding
  
• R-learning on Access-Control Queuing Task, Example 10.2, Figure 10.5 (Lisp), (C version)
• Chapter 11: Off-policy Methods with Approximation
• Baird Counterexample Results, Figures 11.2, 11.5, and 11.6 (Lisp)
• Chapter 12: Eligibility Traces
• Offline lambda-return results, Figure 12.3 (Lisp)
• TD(lambda) and true online TD(lambda) results, Figures 12.6 and 12.8 (Lisp)
• Sarsa(lambda) on Mountain Car (Lisp) (Python: MC and Sarsa) with tile coding
• Chapter 13: Policy Gradient Methods (this Python code is available at github)
  
• Figure 13.1
• Figure 13.2
• Example 13.1