---
# System prepended metadata

title: 'Part 5 : GRAPH ATTENTION NETWORKS (GAT)'

---

# **GRAPH ATTENTION NETWORKS (GAT)**
> [Deepak Yadav](https://hackmd.io/profile)

- **Description**
- **Can we do better than GCNs?**
- **What is Graph Attention Network (GAT)**
- **How does the GAT layer work?**

---

## **Description**

GAT (Graph Attention Network), is a novel neural network architecture that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods’ features, the method enables (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, GAT addresses several key challenges of spectral-based graph neural networks simultaneously, and make the model readily applicable to inductive as well as transductive problems.

Analyzing and Visualizing the learned attentional weights also lead to a more interpretable model in terms of importance of neighbors.

But before getting into the meat of this method, I want you to be familiar and thorough with the Attention Mechanism, because we’ll be building GATs on the concept of **Self Attention and Multi-Head Attention** introduced by Vaswani et al. If not, you may read this blog, The [Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/) by Jay Alamar.

---

## **Can we do better than GCNs?**
From Graph Convolutional Network (GCN), we learnt that combining local graph structure and node-level features yields good performance on node classification task. However, the way GCN aggregates messages is structure-dependent, which may hurt its generalizability.

The fundamental novelty that GAT brings to the table is how the information from the one-hop neighborhood is aggregated. For GCN, a graph convolution operation produces the normalized sum of neighbors' node features as follows:

![](https://i.imgur.com/AeF73WF.png)

where **N(i)** is the set of its one-hop neighbors (to include vi in the set, we simply added a self-loop to each node), **cij = √|N(i)|√|N(j)|** is a normalization constant based on graph structure, σ is an activation function (GCN uses ReLU), and **W<sup>i</sup>** is a shared weight matrix for node-wise feature transformation.


---
## **What is Graph Attention Network (GAT)**
A Graph Attention Network (GAT) is a neural network architecture that operates on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods’ features, a GAT enables (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront.

GAT introduces the attention mechanism as a substitute for the statically normalized convolution operation. The figure below clearly illustrates the key difference.

![](https://i.imgur.com/7iQPqAd.jpg)

---

## **How does the GAT layer work?**
The particular attentional setup utilized by GAT closely follows the work of Bahdanau et al. (2015) i.e Additive Attention, but the framework is agnostic to the particular choice of attention mechanism.

The input to the layer is a set of node features, h=h⃗1,h⃗2,…,h⃗N,h⃗i,R∈F, where N is the number of nodes, and F is the number of features in each node. The layer produces a new set of node features (of potentially different cardinality F′ ), h=h′⃗1,h′⃗2,…,h′⃗N,h′⃗i,R∈F′, as its output.

---
**The Attentional Layer broken into 4 separate parts**:

1) Simple linear transformation: In order to obtain sufficient expressive power to transform the input features into higher level features, atleast one learnable linear transformation is required. To that end, as an initial step, a shared linear transformation, parametrized by a weight matrix, W∈RF'×F , is applied to every node.
![](https://i.imgur.com/qQqSNDa.png)

---
2) Attention Coefficients: We then compute a pair-wise un-normalized attention score between two neighbors. Here, it first concatenates the z embeddings of the two nodes, where || denotes concatenation, then takes a dot product of it with a learnable weight vector a<sup>(l)</sup>, and applies a LeakyReLU in the end. This form of attention is usually called additive attention, in contrast with the dot-product attention used for the Transformer model. We then perform self-attention on the nodes, a shared attentional mechanism a : RF'×RF'→R to compute attention coefficients.
![](https://i.imgur.com/XEf1kQS.png)


![](https://i.imgur.com/C1m5Vip.jpg)

---
**Q. Is this step the most important step?**

**Ans.** Yes! This indicates the importance of node **j′s** features to node i. This step allows every node to attend on every other node, dropping all structural information.

**NOTE:** The graph structure is injected into the mechanism by performing **masked attention**, we only compute **eij** for nodes **j ∈ Ni**, where **Ni** is some neighborhood of node i in the graph. In all the experiments, these will be exactly the first-order neighbors of i (including i).

---
3) Softmax: This makes coefficients easily comparable across different nodes, we normalize them across all choices of j using the softmax function
![](https://i.imgur.com/FjLd2DI.png)

---
4) Aggregation: This step is similar to GCN. The embeddings from neighbors are aggregated together, scaled by the attention scores.
![](https://i.imgur.com/N0poutG.png)

![](https://i.imgur.com/hToVkCx.jpg)

---
Analogous to multiple channels in a Convolutional Net, GAT uses multi-head attention to enrich the model capacity and to stabilize the learning process. Specifically, K independent attention mechanisms execute the transformation of Equation 4, and then their outputs can be combined in 2 ways depending on the use:

![](https://i.imgur.com/NpBI9wl.png)

---
**1) Concatenation** As can be seen in this setting, the final returned output, h', will consist of KF' features (rather than F′) for each node.

---
**2) Averaging**
If we perform multi-head attention on the final (prediction) layer of the network, concatenation is no longer sensible and instead, averaging is employed, and delay applying the final nonlinearity (usually a softmax or logistic sigmoid for classification problems).

**Thus concatenation for intermediary layers and average for the final layer are used.**

---

## CODE
[GRAPH ATTENTION NETWORKS (GAT)](https://colab.research.google.com/drive/1huk1L9FclboFFHDU5TaapXdgqsyF1cve#scrollTo=_CzX7T8LHgDE)