Object Focused Q-learning for Autonomous Agents

1. Object Focused Q-learning for Autonomous Agents M. Onur canci

2. What is of-q learning • Object Focused Q-learning (OF-Q), a novel re-inforcement learning algorithm that can offer exponentialspeed-ups over classic Q-learning on domains composed ofindependent objects. • An OF-Q agent treats the state spaceas a collection of objects organized into different object classes. • Key part is a control policy that uses non-optimalQ-functions to estimate the risk of ignoring parts of the statespace.

3. reinforcementlearning • Why reinforcement learning isn’t enough? • Because of the curse of dimensionality,the time required for convergence in high-dimensional statespaces can make the use of RL impractical. • Solution? • There is noeasy solution for this problem • Observing human behaviours

4. reinforcementlearning • Adult humans are consciously aware of only one stimulus out of each 3 hundred thousand receivedand theycan only hold a maximum of 3 to 5 meaningful items orchunks in their short-term memory. • Humans deal with high-dimensionality simply by paying attention to a very small number of features at once, a capability known as selective attention.

5. OF-Q learning • OF-Q embraces the approach of paying attention to a small number of objects at any moment just like human • Instead of a high-dimensional policy, OF-Q simultaneously learn a collection of low-dimensional policiesalong with when to apply each one, i.e. : where to focus ateach moment.

6. Of-q learning • OF-Q learningsharessameconceptwith ; • OO-MDP OO-MDP solvers see the state space asa combination of objects of specific classes.Solversalso need a designer to define a set of domainspecicrelationsthatdefine how differentobjectsinteractwitheachother. • Modular reinforcementlearning (RMDP)Requires a full dynamic Bayesian networkwhichneeds either an initial policy that performs well or agood cost heuristic of the domain. • Differswith; • Learning theQ-values of non-optimal policies to measure the consequences of ignoring parts of the state space • OF-Q onlyrelies on online exploration of the domain. • For each object class, we learn the Q-values for optimaland non-optimal policies and use that information for theglobal control policy.

7. NOTATION • A Markov decision process (MDP) is a tuple • Theprobability of transitioning to state s0 when taking action ain state s • Immediate reward when takingaction a in state s, • Discount factor

8. NOTATION - 2 • Defines which action an agent takes ina particular state s. • the state-value of s when following policy , i.e., the expected sum of discounted rewards that the agent obtainswhen following policy from state s. • the discounted reward received when choosing action ain state s and then following policy .

9. Notation - 3 • is the optimalpolicy maximizing the value of each state. • and are then the optimal state valueand Q-values. • Q values are not always optimal respect to a given policy that is notnecessarily optimal.

10. Of-q learning’starget • OF-Q is designed to solve episodic MDPs with the following properties; • The state space S is defined by a variable number ofindependent objects. These objects are organized intoclasses of objects that behave alike. • The agent is seen as an object of a specific class, constrained to be instantiated exactly once in each state.Other classes can have none to many instances in anyparticular state. • Each object provides its own reward signal and theglobal reward is the sum of all object rewards. • Reward from objects can be positive or negative

11. C is the set of object classes in the domain. Using the same sample, the Q-valueestimate for the random policy of class o:class is updated with

12. Overview of of-q

13. Overview of of-q • Control Policy Control policy is simple.First decideA, the set of actions that are safe to take

14. Overview of of-q (Thresholdoperations) • Thresholdinitialization To avoid poor actions that result in low rewards,thresholdforeachclass is initializedwithQmin, worst Q value of domain. • ARBITRATION Modular RL approachesusetwosimplearbitrationpolicy: • Winnertakesall : modulewithhighest Q valuedecidesthenextaction • Greatestmass : modulewithhighestsum of the Q valuesdecidesthenextaction

15. Test domain • Experimental domain for testing OF-Q is space invaders game

16. Illusion of control • Winner takes all • Problem with winner takes all approach is that; it may take an action that is very positivefor one object but fatal for the overall reward. In the SpaceInvaders domain, this control policy would be completelyblind to the bombs that the enemies drop, because therewill always be an enemy to kill that offers a positive Q-value,while bomb Q-values are always negative.

17. Illusion of control Greatest Mass It does not make sense to sum Q-values from different policies, because Q-values from different modules are defined with respect to different policies, and in subsequentsteps we will not be able to follow several policies at once. With these two sources of reward, greatest- mass would choose the lower state, expecting a reward of 10. The optimal action is going to the upper state.

18. OF-Q ARBITRATION For the pessimal Q-values, both bombs are just as dangerous, because they both can possibly hit the ship. Random policy Q- Values willidentify the closest bomb as a bigger threat.

19. OF-Q ARBITRATION

20. OF-Q ARBITRATION

21. OF-Q LEARNING (spaceinvaders)

22. Of-q learning(normandy) Normandy. The agent starts in a random cell in thebottom row and must collect the two rewards randomlyplaced at the top, avoiding the cannon re.

23. Of-q learning(normandy)

24. END • Thanks