Code for the paper "Constructing a Natural Language Inference Dataset using Generative Neural Networks" (Starc and Mladenic, 2016)
Run setup.sh
to download and install the SNLI dataset and Glove word vectors
Run experiment.py
to train the models, generate datasets and evaluation
python experiment.py method version hidden_size latent size
Methods:
orig_class
- to train a classifier on the original datatrain_gen
- to train a generative modelaugment
- to construct a dataset using the trained generative modeltrain_class
- to train classifiers on the generated datasettrain_discriminator
- to train a discriminative model on the generated and original datasetevaluate
- to evaulate the model and generated dataset Generative model versions:
version |
Model | Accuracy |
---|---|---|
9 | BaseEmbedDecoder | 77.7 |
8 | AttEmbedDecoder | 78.5 |
7 | VaeEncoderDecoder | 77.9 |
6 | EncoderDecoder | 72.4 |
Example:
python experiment.py train_gen 8 50 16
You can generate hypotheses by specifying your own premises and labels in the interpreter.
In [1]: run -i experiment.py load_gen 8 150 4
Using Theano backend.
Loading training data
Loading dev data
Loading test data
Data loaded
Transforming finished
Word vec preparation finished
Dataset created
In [2]: import visualize
In [3]: premise = visualize.load_sentence('two children playing on the floor with toy trains .', wi, prem_len)
In [4]: visualize.print_hypos(premise, 2, gtest, 8, hypo_len, latent_size, wi)
Premise: two children playing on the floor with toy trains .
Label: entailment
Hypotheses:
two children playing on the floor .
there are two children outside .
kids playing at the park .
people are playing outside
two children are on a floor with toys .
children are playing with toys .
there are people outside playing .
two children are playing on the floor
In [5]: visualize.print_hypos(premise, 1, gtest, 8, hypo_len, latent_size, wi)
Premise: two children playing on the floor with toy trains .
Label: contradiction
Hypotheses:
a group of kids are playing with each other .
two children are playing in a park .
three little kids play on the floor .
two children are playing video games .
the floor is empty
two kids are outside .
two men are eating a sandwich outside .
the two children are playing with each other .