Paper Contributions

• Non-Euclidean geometry with a constant negative Gaussian curvature
• Can be visualized as the forward sheet of a two-sheeted hyperboloid
• Four common equivalent models:
  1. Klein model
  2. Poincaré disk model
  3. Lorentz (hyperboloid) model
  4. Poincaré half-plane

The Lorentz Model

\[ \mathbb{H}^n = \left\{\mathbf{z} \in \mathbb{R}^{n+1}: \langle \mathbf{z}, \mathbf{z} \rangle_\mathcal{L} = -1, z_0 > 0 \right\} \]
where
\[ \langle \mathbf{z}, \mathbf{z'} \rangle_\mathcal{L} = -z_0z_0' + \sum_{i=1}^{n}z_iz_i' \]

The Lorentz Model

• The Lorentzian inner product also functions as metric tensor on hyperbolic space.
• The one-hot vector \(\mathbf{\mu}_0 = [1, 0, \ldots, 0] \in \mathbb{H}^n \subset \mathbb{R}^{n+1}\)
  will be referred as the origin of the hyperbolic space.
• The distance between two points \(\mathbf{z}, \mathbf{z'} \in \mathbb{H}^n\) is given by
  \[ d_\mathcal{l}(\mathbf{z}, \mathbf{z'}) = \text{arccosh}(-\langle \mathbf{z}, \mathbf{z'} \rangle_\mathcal{L}) \]

Overview of of Pseudo-hyperbolic Gaussian Construction

1. Sample a vector from \(\mathcal{N}(\mathbf{0}, \Sigma)\)
2. Transport the vector from \(\mathbf{\mu}_0\) to \(\mathbf{\mu}\) along the geodesic
3. Project the vector onto the surface

The result is the desired distribution \(\mathcal{G}(\mathbf{\mu}, \Sigma)\).

For an arbitrary pair of point \(\mathbf{\mu}, \mathbf{\nu} \in \mathbb{H}^n\):

• the parallel transport from \(\mathbf{\nu}\) to \(\mathbf{\mu}\) is defined as a map \(\text{PT}_{\mathbf{\nu}\to\mathbf{\mu}}\),
  • from \(T\mathbf{\nu}\mathbb{H}^n\) to \(T\mathbf{\mu}\mathbb{H}^n\),
  • by carrying a vector from \(\mathbf{\mu}\) to \(\mathbf{\nu}\) along the geodesic in a parallel manner without changing its metric tensor, i.e.:

\[ \left\langle \text{PT}_{\mathbf{\nu}\to\mathbf{\mu}}(\mathbf{v}), \text{PT}_{\mathbf{\nu}\to\mathbf{\mu}}(\mathbf{v'}) \right\rangle_\mathcal{L} = \langle \mathbf{v}, \mathbf{v'} \rangle_\mathcal{L} \]

The formula for the parallel transport on the Lorentz model is given by:

where \(\alpha = -\langle \mathbf{\nu}, \mathbf{\mu} \rangle_\mathcal{L}\).

Inverse parallel transport \(\text{PT}_{\mathbf{\nu}\to\mathbf{\mu}}^{-1}\) carries the vector in \(T\mathbf{\nu}\mathbb{H}^n\) back to \(T\mathbf{\mu}\mathbb{H}^n\) along the geodesic, that is,

• Recall that every \(\mathbf{u} \in T\mathbf{\mu}\mathbb{H}^n\) determines a unique maximal geodesic \(\gamma_\mathbf{\mu}: [0,1] \to \mathbb{H}^n\)
  • with \(\gamma_\mathbf{\mu}(0) = \mathbf{\mu}\)
  • and \(\gamma_{\mathbf{\mu}}'(0) = \mathbf{u}\)
• Exponential Map \(\exp_\mathbf{\mu}: T\mathbf{\mu}\mathbb{H}^n \to \mathbb{H}^n\) is defined by
  • \(\exp_\mathbf{\mu}(\mathbf{u}) = \gamma_{\mathbf{\mu}}(1)\)
  • projection from \(\mathbf{\mu}\) to destination coincides with \(\|\mathbf{v}\|_\mathcal{L}\)

For hyperbolic space, the map is given by

• It can be shown that this map is norm preserving in the sense that
  • \(d_\mathcal{l}(\mathbf{\mu}, \exp_\mathbf{\mu}(\mathbf{u})) = \text{arcosh}\left(-\langle \mathbf{\mu}, \exp_\mathbf{\mu}(\mathbf{u})\rangle_\mathcal{L}\right)\)

To evaluate the density of a point on hyperbolic space we need to compute the inverse of the exponential map, that is, the logarithmic map:

Putting all together…

• If \(\mathbf{X}\) is a random variable with PDF \(p(\mathbf{x})\), the \(\log\)-likelihood of \(Y = f(\mathbf{X})\) at \(\mathbf{y}\) can be expressed as
  \[ \log_\mathbf{y} = \log p(\mathbf{x}) - \log\det \left( \frac{\partial f} {\partial \mathbf{x}} \right) \]

• Hence we only need to evaluate the determinant of the gradient of the defined projection:

  

• Because the directional derivative in the direction of \(\vec{\mathbf{u}}\) has magnitude 1, and because each components are orthogonal to each other, the norm of the directional derivative is given by

  

• Finally, observe that parallel transport is itself a norm preserving map, thus, we have that
  \[ \partial \text{PT}_{\mathbf{\mu_0}\to\mathbf{\mu}}(\mathbf{v})/ {\partial \mathbf{v}} = 1. \]

• The authors extend work by Vilnis & McCallum (2015) by changing the destination of the defined map to the family of \(\mathcal{G}\).
  • Such map attempts to extract linguistic and contextual properties of words into a dictionary.

• Data was constructed from a binary tree of depth 8.

• Given a (initial) set of binary representations \(A_0\), a generated set \(A\) would be generated by randomly flipping each coordinate value with probability \(\epsilon = 0.1\).

• Finally, these probabilities are projected to a \(\mathbb{R}^d\) space using a MLP of depth 3, and 100 hidden variables at each layer for encoder and decoder of the VAE.

• A Hyperbolic VAE was applied to a binarized version of MNIST.

• An MLP of depth 3 and 500 hidden units at each layer was used for both, encoder and decoder.

• This method outperformed Vanilla VAE with small latent dimension.

• In Deep RL the number of possible state-action trajectories grows exponentially with the time horizon.

  • Such trajectories may be thought as having a tree-like structure

• Hyperbolic VAE was applied to a set of trajectories that were explored by a trained policy during multiple episodes of Breakout.

  • A Deep Q-Network was used with an epsilon-greedy (\(\epsilon = 0.1\)) strategy.

• Dataset consisted of 100K trajectories

  • 80K were used for training
  • 10K were used for validation
  • 10K were used for testing

• Samples from the dataset:
  

• Authors trained WordNet noun benchmarks and evaluated reconstruction performance following the procedure proposed with Poincaré embeddings (Nickel & Kiela, 2017).

• The authors present a comparison with word embeddings with Gaussian distribution on Euclidean space.

  • Hyperbolic model outperformed the Euclidean counterpart when the latent space was low dimensional