This is an implementation of Attention (only supports Bahdanau Attention right now)
data (Download data and place it here) |--- small_vocab_en.txt |--- small_vocab_fr.txt layers |--- attention.py (Attention implementation) examples |--- nmt |--- model.py (NMT model defined with Attention) |--- train.py ( Code for training/inferring/plotting attention with NMT model) |--- train_variable_length_seq.py ( Code for training/inferring with variable length sequences) |--- nmt_bidirectional |--- model.py (NMT birectional model defined with Attention) |--- train.py ( Code for training/inferring/plotting attention with NMT model) models (created by train_nmt.py to store model) results (created by train_nmt.py to store model)
Just like you would use any other
from attention_keras.layers.attention import AttentionLayer attn_layer = AttentionLayer(name='attention_layer') attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])
encoder_outputs- Sequence of encoder ouptputs returned by the RNN/LSTM/GRU (i.e. with
decoder_outputs- The above for the decoder
attn_out- Output context vector sequence for the decoder. This is to be concat with the output of decoder (refer
model/nmt.pyfor more details)
attn_states- Energy values if you like to generate the heat map of attention (refer
An example of attention weights can be seen in
After the model trained attention result should look like below.
small_vocab_fr.txtfrom Udacity deep learning repository and place them in the
run.shwill take you inside the docker container.
run.shappropriately depending on whether you need the GPU version of the CPU version
pip install -r requirements.txt -r requirements_tf_cpu.txt(For CPU)
pip install -r requirements.txt -r requirements_tf_gpu.txt(For GPU)
python3 src/examples/nmt/train.py. Set
degug=Trueif you need to run simple and faster.
If you have improvements (e.g. other attention mechanisms), contributions are welcome!