1 / 52

Reinforcement Learning & Apprenticeship Learning

Reinforcement Learning & Apprenticeship Learning. Chenyi Chen. Markov Decision Process (MDP). What’s MDP? A sequential decision problem Fully observable, stochastic environment Markovian transition model: the n th state is only determined by ( n-1 ) th state and ( n-1 ) th action

harlan
Download Presentation

Reinforcement Learning & Apprenticeship Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Reinforcement Learning & Apprenticeship Learning Chenyi Chen

  2. Markov Decision Process (MDP) • What’s MDP? • A sequential decision problem • Fully observable, stochastic environment • Markovian transition model: the nth state is only determined by (n-1)th state and (n-1)thaction • Each state has a reward, and the reward is additive

  3. Markov Decision Process (MDP) • State s: a representation of current environment;

  4. Markov Decision Process (MDP) • Example: Tom and Jerry, control Jerry (Jerry’s perspective) • State: the position of Tom and Jerry, 25*25=625 in total; One of the states One of the states

  5. Markov Decision Process (MDP) • State s: a representation of current environment; • Action a: the action can be taken by the agent in state s;

  6. Markov Decision Process (MDP) • Example: Tom and Jerry, control Jerry (Jerry’s perspective) • State: the position of Tom and Jerry, 25*25=625 in total; • Action: both can move to the neighboring 8 squares or stay; One of the states One of the states

  7. Markov Decision Process (MDP) • State s: a representation of current environment; • Action a: the action can be taken by the agent in state s; • Reward R(s): the reward of current state s (+,-,0); • Value (aka utility) of state s: different from reward, related with future optimal actions;

  8. An Straightforward Example • 100 bucks if you came to class • Reward of “come to class” is 100 • You can use the money to: • Eat food (you only have 50 bucks left) • Stock market (you earn 1000 bucks, including the invested 100 bucks) • The value (utility) of “come to class” is 1000

  9. Markov Decision Process (MDP) • Example: Tom and Jerry, control Jerry (Jerry’s perspective) • State: the position of Tom and Jerry, 25*25=625 in total; • Action: both can move to the neighboring 8 squares or stay; • Reward: 1) Jerry and cheese at the same square, +5; 2) Tom and Jerry at the same square, -20; 3) otherwise 0; One of the states One of the states

  10. Markov Decision Process (MDP) • State s: a representation of current environment; • Action a: the action can be taken by the agent in state s; • Reward R(s): the reward of current state s (+,-,0); • Value (aka utility) of state s: different from reward , related with future optimal actions; • Transition probability P(s’|s,a):given the agent is in state s and taking action a, the probability of reaching state s’ in the next step;

  11. Markov Decision Process (MDP) • Example: Tom and Jerry, control Jerry (Jerry’s perspective) • State: the position of Tom and Jerry, 25*25=625 in total; • Action: both can move to the neighboring 8 squares or stay; • Reward: 1) Jerry and cheese at the same square, +5; 2) Tom and Jerry at the same square, -20; 3) otherwise 0; • Transition probability: about Tom’s moving pattern. One of the states One of the states

  12. Markov Decision Process (MDP) • Example: Tom and Jerry, control Jerry (Jerry’s perspective) …

  13. Markov Decision Process (MDP) • State s: a representation of current environment; • Action a: the action can be taken by the agent in state s; • Reward R(s): the reward of current state s (+,-,0); • Value (aka utility) of state s: different from reward , related with future optimal actions; • Transition probability P(s’|s,a):given the agent is in state s and taking action a, the probability of reaching state s’ in the next step; • Policy π(s)->a: a table of state-action pairs, given state s, output action a that should be taken.

  14. Bellman Equation • The expected utility of state s obtained by executing π starting in s is given by ( is a discount factor): , where • The optimal policy is given by: • Denote as , the optimal policy chooses the action that maximizes the expected utility of the subsequent state:

  15. Bellman Equation • Bellman Equation: • The utility of a state is the immediate reward for that state plus the expected discounted utility of the next state, assuming that the agent choose the optimal action • with , is the unique solution to Bellman equation

  16. Value Iteration initialize , as a discount factor repeat for each state sinSdo ifthen until return Bellman Equation:

  17. Value Iteration • Naïve example: R(s)=3, R(s’)=5,=0.9 Initially U(s)=0, U(s’)=0 (1) U(s)=3+0.9*0=3, U(s’)=5+0.9*3=7.7 (2) U(s)=3+0.9*7.7=9.93, U(s’)=5+0.9*9.93=13.937 (3) U(s)=3+0.9*13.937=15.5433, U(s’)=5+0.9*15.5433= 18.989 … (29) U(s)=39.3738, U(s’)=40.4364 (30 ) U(s)=39.3928, U(s’)= 40.4535 • Solve the equation: U(s)=3+0.9*U(s’) U(s’)=5+0.9*U(s) the true value is: U(s)= 39.4737, U(s’)= 40.5263 Value iteration

  18. Reinforcement Learning • Similar to MDPs • But we assume the environment model (transition probability P(s’|s,a) ) is unknown

  19. Reinforcement Learning • How to solve it? • Solution #1: Use Monte Carlo method to sample the transition probability, then implement Value Iteration limitation: too slow for problems with many possible states because it ignores frequencies of states

  20. Monte Carlo Method • A broad class of computational algorithms that rely on repeated random sampling to obtain numerical results; • Typically one runs simulations many times in order to obtain the distribution of an unknown probabilistic entity. From Wikipedia

  21. Monte Carlo Example :the element is the probability P(s’|s,a) initialize tablewith all elements repeat at current state s, random choose a valid action a simulate for one step, get a new state s’ until sampled enough times return

  22. Reinforcement Learning • How to solve it? • Solution #1: Use Monte Carlo method to sample the transition probability, then implement Value Iteration limitation: too slow for problems with many possible states because it ignores frequencies of states • Solution #2: Q-learning the major algorithm for reinforcement learning

  23. Q-learning Bellman Equation: • Q-value is defined by: • The relationship between utility and Q-value is: • The optimal policy is given by: • Q-learning algorithm is used to learn this Q-value table

  24. Q-learning Q: a table of Q-values indexed by state and action, initially zero s, a,R(s): state, action, and reward. Initial state is given by the environment, and initial action is randomly picked up γ: discount factor α: learning rate f(.): greedy function, at the beginning, Q-table is bad, so we make some random choice While not coverage run one step to obtain s’ from s and a through the environment (e.g. the game engine) ) return Q-value is defined by:

  25. Playing Atari with Deep Reinforcement Learning • The Atari 2600 is a video game console released in September 1977 by Atari, Inc.  • Atari emulator: Arcade Learning Environment (ALE)

  26. What did they do? • Train a deep learning convolutional neural network • Input is current state (raw image sequence) • Output is all the legal action and corresponding Q(s,a) value • Let the CNN play Atari games

  27. What’s Special? • Input is raw image! • Output is the action! • Game independent, same convolutional neural network for all games • Outperform human expert players in some games

  28. Problem Definition • State: • Action: possible actions in the game • Reward: score won in the Atari games (output of the emulator) • Learn the optimal policy through training

  29. A Variant of Q-learning In the paper: Q-value is defined by:

  30. Deep Learning Approach Approach the Q-value with a convolutional neural network Q(s,a;θ) VS Straightforward structure The structure used in the paper

  31. How to Train the Convolutional Neural Network? Loss function: Where: Q-value is defined as: Do gradient descent:

  32. Some Details • The distribution of action a(ε-greedy policy): choose a “best” action with probability 1-ε, and selects a random action with probability ε, ε annealed linearly from 1 to 0.1 • Input image preprocessing function φ(st) • Build a huge database to store historical samples

  33. During Training… Add new data sample to database Do mini-batch gradient descent on parameter θ for one step with probability 1-ε or random action with probability ε Play the game for one step

  34. CNN Training Pipeline

  35. After Training… Play the game

  36. Results Screen shots from five Atari 2600 games: (Left-to-right) Beam Rider, Breakout, Pong, Seaquest, Space Invaders Comparison of average total reward for various learning methods by running an ε-greedy policy with ε = 0.05 for a fixed number of steps

  37. Results • The leftmost plot shows the predicted value function for a 30 frame segment of the game Seaquest. The three screenshots correspond to the frames labeled by A, B, and C respectively

  38. Apprenticeship Learning via Inverse Reinforcement Learning • Teach the computer to do something by demonstration, rather than by telling it the rules or reward • Reinforcement Learning: tell computer the reward, let it learn by itself using the reward • Apprenticeship Learning: demonstrate to the computer, let it mimic the performance

  39. Why Apprenticeship Learning? • For standard MDPs, a reward for each state needs to be specified • Specify a reward some time is not easy, what’s the reward for driving? • When teaching people to do something (e.g. driving), usually we prefer to demonstrate rather than tell them the reward function

  40. How Does It Work? • Reward is unknown, but we assume it’s a linear function of features, is a function mapping state s to features, so:

  41. Example of Feature • State st of the red car is defined as: st==1 left lane, st==2 middle lane, st==3 right lane • Feature φ(st) is defind as: [1 0 0] left lane, [0 1 0] middle lane, [0 0 1] right lane • w is defined as: w=[0.1 0.5 0.3] R(left lane)=0.1, R(middle lane)=0.5, R(right lane)=0.3 • So in this case staying in the middle lane is preferred

  42. How Does It Work? • Reward is unknown, but we assume it’s a linear function of features, is a function mapping state s to features, so: • The value (utility) of policy π is: The expected utility obtained by executing π starting in s is given by: , where

  43. How Does It Work? • Define feature expectation as: • Then: • Assume the expert’s demonstration defines the optimal policy: • We need to sample the expert’s feature expectation by (sample m times):

  44. What Does Feature Expectation Look Like? • State st of the red car is defined as: st==1 left lane, st==2 middle lane, st==3 right lane • Feature φ(st) is defind as: [1 0 0] left lane, [0 1 0] middle lane, [0 0 1] right lane • During sampling, assume γ=0.9 Step 1, red car in middle lane μ=0.9^0*[0 1 0]=[0 1 0] Step 2, red car still in middle lane μ= [0 1 0]+0.9^1*[0 1 0]=[0 1.9 0] Step 3, red car move to left lane μ= [0 1.9 0]+0.9^2*[1 0 0]=[0.81 1.9 0] …

  45. How Does It Work? • We want to mimic the expert’s performance by minimize the difference between and • If we have , and assuming Then

  46. Pipeline

  47. Supporting Vector Machine (SVM) • The 2nd step of the pipeline is a SVM problem Which can be rewritten as:

  48. Pipeline

  49. Their Testing System

  50. Demo Videos http://ai.stanford.edu/~pabbeel/irl/

More Related