Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering

If you make use of this code, please cite the following paper (and give us a star ^_^):

@InProceedings{Nguyen_2018_CVPR,
author = {Nguyen, Duy-Kien and Okatani, Takayuki},
title = {Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}

If you have any suggestion to improve this code, please feel free to contact me at kien@vision.is.tohoku.ac.jp.

Overview

This repository contains Pytorch implementation of "Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering" paper. The network architecture is illustrated in Figure 1.

Figure 1: Overview of Dense Co-Attention Network architecture.

Figure 1: The Dense Co-Attention Network architecture.

Files

├──preprocess/ - Preprocessing code before training the network
├──dense_coattn/ - Dense Co-Attention code
├──demo/ - Demo imgs for pretrained Dense Co-Attention model
train.py - Train the model
answer.py - Generate the answer for test dataset
ensemble.py - Ensemble multiple results from different models

Dependencies

Tests are performed with following version of libraries:

Training from Scratch

The dataset can be downloaded from: http://visualqa.org/.

We provide the scripts for training our network from scratch by simply running the train.py script to train the model.

Evaluation

Run answer.py file to generate all of answers for the test set. You can use ensemble.py to ensemble multiple model's results for the evaluation

License

The source code is licensed under MIT License.