---
# System prepended metadata

title: Relational Transformer

---

Relational Transformer
=

## Introduction
The core idea of the self-attention model is to model relationships among input tokens, and it could be represented as $\alpha_{ij} = f(x_i,x_j,i,j)$, where $x_i, x_j$ means input tokens and $i,j$ are their positions in text. The common implementation of $f(\cdot)$ is $f(x_i,x_j,i,j) = \frac{W_Q x_i (W_k x_j)^T}{\sqrt d}$, and we can find this formula factorize the $f$ as the product of two terms, $W_Qx_i$ and $W_K x_j$. A rising problem is this factorization is an approximation, for example, the multi-word expressions problem. To understand the meaning of "kick the bucket" require the model treats it as a single token rather the combination of its components. In general, the question here is some relationships are hard to decompose. On the other hand, the factorization saves computation and storage cost. So, we provide a solution which preserves the factorization form and adds the relation embedding as complementary. 

Like the form of positional encoding and relative positional encoding, $$f(x_i,x_j,i,j)= \frac{(W_Q x_i+p_i) (W_k x_j+p_j)^T}{\sqrt d}$$ 
$$f(x_i,x_j,i,j)= \frac{W_Q x_i (W_k x_j+r_{i-j})^T}{\sqrt d}$$

We inject the relation embedding to the original form.
$$f(x_i,x_j,i,j)= \frac{W_Q x_i (W_k x_j+r(x_i,x_j,i,j))^T}{\sqrt d}$$


## Method
The new term $r(x_i,x_j,i,j)$ means the embedding of relationships and the relationship is determined by an external system which takes $x_i,x_j,i,j$ as inputs. 

Some possible edge types
* word co-occurrence $\frac{count(a,b)}{\sqrt{count(a)count(b)}}$
* approximate dependency 
* know graph (entities are two sparse in text)

## Experiment
Ongoing

WMT16 en-de
|model|newstest14|
|-|-|
|Transformer| 27.58|
|Ours| 27.89|


