# ABC-CNN:An Attention Based Convolutional Neural Network for Visual Question Answering

###### tags: `VQA`, `paper`

>source : https://arxiv.org/abs/1511.05960

- ABC-CNN finda the corresponding visual features in the visual feature maps with a "configurable convlolution" operation.
- In common VQA models, the visual features and dense question embeddings are integrated through a linear/non-linear transform which jointly projects features from image space and semantic space into answer space.
- According to the authors, this integration is not normally sufficient to fully exploit the relationship of vision part and question understanding part because it loses the opportunity to exploit the **intent of queries** to focus on different regions in an image.
- They reason that by treating image features as global visual features, the studies in VQA and image captioning fail to exploit the valuable information in questions to focus their attention on the corresponding regions in images.

### Question-Guided Attention:

- mechanism of finding informative region in the image based on input question's **intent**.

#### Question-guided attention map (QAM) :

- QGA information is provided by QAM which is modeled as a latent information.
- It is generated by searching for visual features that correspond to the input query's semantics in the spatial image feature map.
- This search is done through configurable convolutional kernel(CCK).

#### Configurable Convolutional Kernel (CCK):

- This kernel is convolved over the visual feature map of the image.
- It is generated by ttransforming question embeddings from semantic space into visual space.
- Convolving the CCK wwith image feature map adaptively reresents each region's imprtance for answering given qusetion as a QAM.
- These QAMs can be used to spatially weight the visual feature maps to filter out noise and unrelated information.

---

### ABC-CNN Model in detail:


ABC-CNN is composed of 4 components:
1. Image feature extraction
2. Question understanding
3. Attention extraction
4. Answer generation

---
1. Image Feature Extraction: 

- deep CNN used .
- VGG-19 pretrained on 1000-class ImageNet classification challenge 2012 dataset and fully conv, segmentation neural network pretained on PASCAL 2007 segmentation dataset used.(using segmentation model slightly boosted the performance)

2. Question Understanding: 

- LSTM used.

3. Attention extraction: 

- configures set of CCK according to dense embedding from question understanding part.
- CCK can be thought of as searching spatial image features maps for specifc image features that correspond to question's intent.
- CCK is obtained by projecting the dense question embedding s from semantic space to visual space (a linear layers with sigmoid non-linearity).
- CCK has the same number of channels as that of the image feature map.

$m = softmax(k * I)$

where, m is the QAM,
$k$ is the CCK,
$I$ are the image features and
$*$ is the convolution operator

4. Answer Generation:
- Here they have only considered single word answers.
- Therefore, it is a classification task
- It is a multiclass classifier based on image features map, the attention weighted image feature map and dense question embedding vector.

$I_i' = I_i \odot m$

where,
$I_i'$ is the ith channel of attention weighted feature,
I is the image feature map and
$\odot$ represent elementwise product

- To avoid overfitting, they applied a 1x1 convolition on the attention weighted feature map to reduce the number of channels.

$h = tanh(W_{ih}I + W_{rh}I_r+ W_{sh}s + b_h)$

where $h$ denotes final projection and $I_r$ denotes the reduced feature map after the 1x1 convolution.