HW4 Programming: Image Captioning

Getting the stencil

Please click here to get the stencil code. Reference this guide for more information about GitHub and GitHub Classroom.

Assignment Overview

In this assignment, we will be generating English language captions for images using the Flickr 8k dataset. This dataset contains over 8000 images with 5 captions apiece. Provided is a slightly-altered version of the dataset, with uncommon words removed and all sentences limited to their first 20 words to make training easier.

Follow the instructions in preprocessing.py to download and process the data. This should take around 10 minutes the first time and will generate a 100MB file. The dataset file itself is around 1.1GB and can be found here. Note: You may need to sign in with google and create an account for kaggle. Make sure to download these and put the Images and captions.txt in your data directory before you run preprocessing.py.

You will implement two different types of image captioning models that first encode an image and then decode the encoded image to generate its English language caption. The first model you will implement in this assignment will be based on Recurrent Neural Networks, while the second model is based on Transformers. Since both of these models are trying to solve the same problems, they share the same preprocessing, training, testing, and loss function.

To simplify the assignment and improve your models’ quality, all the images in the dataset have been passed through ResNet-50 to get 2048-D feature vectors. You will use these feature vectors in both your RNN and Transformer models to generate captions.

Roadmap

Step 1: Training the Model

In this assignment, both of the models you are implementing perform the same task, the only difference is their implementation. Because of this, there are some methods which can be shared between them in the ImageCaptionModel class, located in the model.py file. This class takes in a decoder layer that we will implement in the next step and gives us a fully functioning image captioning model. This class is partially filled out, except for the train method which is left to you. After steps 1 and 2, you will be able to run your RNN model.

ImageCaptionModel

• train and test: In contrast to most of your previous assignments which use the keras fit routines, a large amount of control is required over your scenarios (for experimentation purposes). In these cases, it may be better to specify the optimization process in all of its details. While doing this, you can also mimic the functionality of keras components (i.e. compile, call, etc.) to make your model architecture feel familiar to users who are already used to the standard practices employed in keras.

Step 2: RNN Image Captioning

You will now build an RNN decoder that takes in a sequence of word IDs (an incomplete sentence) and an image embedding, and outputs logits over the vocabulary for each timestep in the sequence, following a similar architecture to the one in Show and Tell: A Neural Image Caption Generator.

Running your RNN Model

Your assignment.py should now be able to run your RNN model.

Depending on your use cases, you may choose to structure your model in a variety of ways. In contrast to previous assignments, this one is intended to mimic a lot of modern research-oriented repositories you might find in the wild. Specifically: *Instead of providing easy-to-use APIs for experimenters, they rigidify their implementation to make tests replicable.* Specifically, they may provide a command-line interface and define testing/training procedures which log results.

First, try running

python assignment.py

You'll notice that you'll get back a nice usage error message if you are missing any required arguments. You can investigate assignment.py to find that main will try to parse command-line arguments and fill in a variety of defaults. Specifically, you'll find this:

def parse_args(args=None):
    """ 
    Perform command-line argument parsing (other otherwise parse arguments with defaults). 
    To parse in an interative context (i.e. in notebook), add required arguments.
    These will go into args and will generate a list that can be passed in.
    For example: 
        parse_args('--type', 'rnn', ...)
    """
    parser = argparse.ArgumentParser(...)
    parser.add_argument('--type',           required=True,              ...)
    parser.add_argument('--task',           required=True,              ...)
    parser.add_argument('--data',           required=True,              ...')
    parser.add_argument('--epochs',         type=int,   default=3,      ...)
    parser.add_argument('--lr',             type=float, default=1e-3,   ...)
    parser.add_argument('--optimizer',      type=str,   default='adam', ...)
    parser.add_argument('--batch_size',     type=int,   default=100,    ...)
    parser.add_argument('--hidden_size',    type=int,   default=256,    ...)
    parser.add_argument('--window_size',    type=int,   default=20,     ...)
    parser.add_argument('--chkpt_path',     default='',                 ...)
    parser.add_argument('--check_valid',    default=True,               ...)
    if args is None: 
        return parser.parse_args()      ## For calling through command line
    return parser.parse_args(args)      ## For calling through notebook.

Feel free to add extra CLI arguments to adjust your training process. The following command will therefore be sufficient to try what an author (or you) might consider to be a "default training run" of the model:

## TODO: Increase epochs to a larger size when ready (maybe 2 or 3 would be enough?)
python assignment.py --type rnn --task both --data ../data/data.p --epochs 1 --chkpt_path ../rnn_model

Since this command also saves the model, we should be able to load it back in and use it. Feel free to modify the saving utility as needed based on your modifications, but the default system should work fine for the initial requirements.

Hyperparameters for RNN

While not required, we’ve filled in the arguments in the notebook/defaults that we found to be the best for our implementation. Additionally we recommend choosing embeddings/hidden layer sizes between 32-256 and training for around 2-4 epochs.

Step 3: Attention and Transformer Blocks

RNNs are neat! However, since 2017, Transformers have been the standard. Transformer-based models have shown state-of-the-art results on a variety of NLP tasks like language modeling, translation, and image captioning.

In the first part of this assignment, we used built in keras RNN layers for our sequence decoder, but in this part of the assignment we will be implementing our own Transformer layer to use in our TransformerDecoder.

These architectures rely on stacked self-attention modules rather than recurrence, and we will specifically be implementing a simplified version of the Attention Is All You Need architecture.

Step 4: Transformers Image Captioning

Running your Transformer

Like your RNN, you can run your transformer with

python assignment.py --type transformer --task both --data ../data/data.p --epochs 1 --chkpt_path ../transformer_model

Mandatory Hyperparameters for Transformer

Grading

Code: You will be primarily graded on functionality. Your RNN model should have a perplexity of ≤ 20, and your Transformer model should have a perplexity that is ≤ 18. This specifically refers to the testing set.

README: Your README should contain your perplexity, accuracy, and any bugs.

Autograder

Our autograder will take your saved model and will run it, thereby alleviating the burden of having to wait for the autograder to train your model. Additionally, the Transformer components will be tested in pseudo-isolation to make sure they function as expected.