The Option-Critic Architecture

Link to the Research Paper

Policy Gradient theorems for options

Introduction

Existing work has focused on finding subgoals

• Can't scale up due to combinatorial flavour
• Learning policies associated with subgoals can be expensive - data and computation time

New method based on policy gradient theorem - enables simultaneous

• Learning process of the intra-option policies
• Termination functions
• Policy over them

Works with

• Linear/Non-Linear function approximators
• Discrete/Continous State space
• Discrete/Continous Action space

Learning Options

• At any time - distill all of the available experience intor every component of system
  • Value function
  • Policy over options
  • Intra-option policies
  • Termination polices

Call-and-return options execution model

• An agent picks option
  $\omega$ according to its policy over options
  ${\pi }_{\mathrm{\Omega }}$
• Then follows the intra-option policy
  ${\pi }_{\omega }$ until termination (
  ${\beta }_{\omega }$)

${\pi }_{\omega ,\theta }$ - Intra-option policy of option

$\omega$ parametrized by

$\theta$

${\beta }_{\omega ,\nu }$ - Termination function of

$\omega$ parameterized by

$\nu$

Option-Value function

$Q_{\mathrm{\Omega }}(s,\omega )=\sum _a{\pi }_{\omega ,\theta }(a|s)Q_U(s,\omega ,a)$

$Q_U(s,\omega ,a):\mathcal{S}\times \mathrm{\Omega }\times \mathcal{A}\to \mathbb{R}$ - Value of executing an action in the context of a state-option pair

$Q_U(s,\omega ,a)=r(s,a)+\gamma \sum _{s^{\prime }}P(s^{\prime }|s,a)U(\omega ,s^{\prime })$

$U:\mathrm{\Omega }\times \mathcal{S}\to \mathbb{R}$ - Option-Value function upon arrival

$U(\omega ,s^{\prime })=(1-{\beta }_{\omega ,\nu }(s^{\prime }))Q_{\mathrm{\Omega }}(s^{\prime },\omega )+{\beta }_{\omega ,\nu }(s^{\prime })V_{\mathrm{\Omega }}(s^{\prime })$

Theorem 1 - Intra-Option Policy Gradient Theorem

• Given a set of Markov options with stochastic intra-option policies differentiable in their parameters
  $\theta$
• The gradient of the expected discounted return with respect to
  $\theta$ and initial condition
  $(s_0,{\omega }_0)$ is
  
  ${\displaystyle \sum _{s,\omega }{\mu }_{\mathrm{\Omega }}(s,\omega |s_0,{\omega }_0)\sum _a{\displaystyle \frac{\partial {\pi }_{\omega ,\theta }(a|s)}{\partial \theta }}{Q}_U(s,\omega ,a)}$
• ${\mu }_{\mathrm{\Omega }}(s,\omega |s_0,{\omega }_0)$ - discounted weighting of state-option pairs along trajectories starting from
  $(s_0,{\omega }_0)$

This gradient describes to effect of a local change at the primitive level on the global expected return

Gradient for termination function

${\displaystyle \frac{\partial Q_{\mathrm{\Omega }}(s,\omega )}{\partial \nu }=\sum _a{\pi }_{\omega ,\theta }(a|s)\sum _{s^{\prime }}\gamma P(s^{\prime }|s,a)\frac{\partial U(\omega ,s^{\prime })}{\partial \nu }}$

Theorem 2 - Termination Gradient Theorem

• Given - set of Markov options with stochastic termination functions differentiable in their parameters
  $\nu$
• Gradient of expected discounted return objective is - inital condition
  $(s_1,{\omega }_0)$
  
  $-{\displaystyle \sum _{s^{\prime },w}{\mu }_{\mathrm{\Omega }}(s^{\prime },\omega |s_1,{\omega }_0)\frac{\partial {\beta }_{\omega ,\nu }(s^{\prime })}{\partial \nu }{A}_{\mathrm{\Omega }}(s^{\prime },\omega )}$

Advantage Function - forming a baseline reduces the variance in gradient estimates

Option choice suboptimal - expected value over all options - Advantage function is negative and it drives the gradient corrections up - increases the odds of terminating

Interrupting Execution model

• Termination is forced whenever the value of
  $Q_{\mathrm{\Omega }}(s^{\prime },\omega )$ for the current option
  $\omega$ is less than
  $V_{\mathrm{\Omega }}(s^{\prime })$

Algorithms and Architecture

• Learn at fast timescale while updating the intra-option policies and termination functions at a slower rate
• Actor - Intra-option policeis, termination functions and policy over options
• Critic -
  $Q_U$ and
  $A_{\mathrm{\Omega }}$

Experiments