# Attention models in Deep Learning

## Introduction

Deep learning models have achieved impressive results on a variety of tasks, such as image and speech recognition, machine translation, and natural language processing. However, these models have a significant limitation: they can only process a fixed-size input. For example, a convolutional neural network (CNN) can only process an image of a fixed size, and a recurrent neural network (RNN) can only process a sequence of a fixed length.

To overcome this limitation, researchers have developed attention mechanisms, which allow deep learning models to focus on specific parts of the input while processing it. Attention mechanisms have been particularly successful in natural language processing tasks, such as machine translation and text summarization.

There are two main types of attention mechanisms: additive attention and multiplicative attention. Additive attention, also known as "Bahdanau attention," weights the input by learning a set of weights that are summed to produce the output. Multiplicative attention, also known as "Luong attention," weights the input by learning a set of weights that are multiplied with the input to produce the output.

Additive attention was introduced in a 2014 paper by Bahdanau et al. and has been widely used in natural language processing tasks. It has the advantage of being able to handle alignment errors, which occur when the attention mechanism is not able to align the input and output correctly. However, additive attention is computationally expensive because it requires computing a weighted sum for each element in the input.

Multiplicative attention was introduced in a 2015 paper by Luong et al. and has the advantage of being faster and more memory efficient than additive attention. However, multiplicative attention is sensitive to alignment errors and can struggle with long input sequences.

## Seq2Seq models

Seq2Seq models, also known as encoder-decoder models, are a type of neural network architecture used for tasks that involve sequential input and output, such as machine translation, language modeling, and text summarization.

The architecture consists of two parts: an encoder and a decoder. The encoder processes the input sequence and encodes it into a fixed-length context vector. The decoder processes the context vector and produces the output sequence.

Here is a general outline of the architecture:

1. The input sequence is fed into the encoder one token (e.g., word or subword) at a time. The encoder processes the input using one or more recurrent neural network (RNN) layers, such as long short-term memory (LSTM) or gated recurrent unit (GRU) layers.

2. The output of the encoder at each time step is typically a hidden state vector, which is passed through a linear layer to produce the context vector. The context vector is a fixed-length representation of the input sequence and is used by the decoder to generate the output sequence.

3. The decoder processes the output sequence one token at a time. It takes in the previous output token and the context vector as input and produces the next output token using one or more RNN layers. The decoder also typically uses an attention mechanism, which allows it to focus on different parts of the input sequence at different times while generating the output.

4. The output sequence is produced token by token until the decoder generates an end-of-sequence token or reaches the maximum output length.

Seq2Seq models can be trained using supervised learning, where the input and output sequences are paired and the model is trained to predict the output sequence given the input sequence. They can also be trained using unsupervised learning, where the model is trained to reconstruct the input sequence from a corrupted version of it.

## Score functions
Bahdanau score function and Luong score function are types of attention mechanisms used in encoder-decoder models. These attention mechanisms allow the decoder to focus on different parts of the input sequence while generating the output sequence, which can improve the performance of the model on tasks such as machine translation.

The Bahdanau score function, also known as the additive attention mechanism, calculates the attention weights by comparing the current hidden state of the decoder with all the hidden states of the encoder. The attention weights are then used to compute a weighted sum of the encoder hidden states, which is used as additional input to the decoder at each time step.

The Luong score function, also known as the dot-product attention mechanism, calculates the attention weights by taking the dot product of the current hidden state of the decoder with all the hidden states of the encoder. The attention weights are then used to compute a weighted sum of the encoder hidden states, which is used as additional input to the decoder at each time step.

Both the Bahdanau score function and the Luong score function can be implemented using a single feedforward neural network with a single output, which is used to compute the attention weights for each encoder hidden state. The choice of score function can depend on the specific task and the performance of the model.

##

In summary, attention mechanisms allow deep learning models to focus on specific parts of the input, improving their performance on tasks such as machine translation and text summarization. Additive attention is more robust to alignment errors but is computationally expensive, while multiplicative attention is faster and more memory efficient but is sensitive to alignment errors.