SemEval2019/Task4 Team Bertha-von-Suttner submission

This is the code for the SemEval 2019 Task 4, Hyperpartisan News Detection submitted by team Bertha von Suttner:

A blog article on the GATE blog briefly describes the approach taken.

If you wish to see the code as it was prepared for the SemEval 2019 task, then refer to the semval-2019 tag in the git repo.

Preparation / Requirements

Once spaCy is installed, you also need to install its en_core_web_sm model. Like this:

python -m spacy download en_core_web_sm

Once NLTK is installed, you also need to install its stopwords data:

python -m nltk.downloader stopwords

Preparation steps:

Run the following steps

Convert the XML file into a tsv file with one article per line:
- python Preprocessing/xml2line.py -A data/articles-training-byarticle-20181122.xml -T data/ground-truth-training-byarticle-20181122.xml -F article_sent,title_sent work/train.text.tsv
Convert the tsv file containing text into a tsv file containing elmo embeddings:
- If you have a GPU: python Preprocessing/line2elmo2.py -g -l 100 work/train.text.tsv work/train.elmo.tsv
- Otherwise: python Preprocessing/line2elmo2.py -l 100 work/train.text.tsv work/train.elmo.tsv If you get problems with the GPU memory or RAM, use the -b option to reduce the batch size
Make sure the directory saved_models does not contain any model files from previous runs:
- rm saved_models/*.hdf5
Train the actual model: KERAS_BACKEND=tensorflow python CNN_elmo.py work/train.elmo.tsv This will create a number of model files in the saved_models directory. The file names contain the validation accuracy.

Convert the XML to tsv: python Preprocessing/xml2line.py -A $TESTXMLFILE -F article_sent,title_sent work/test.text.tsv
Convert the text to elmo embeddings:
- If you have a GPU: python Preprocessing/line2elmo2.py -g -l 100 work/test.text.tsv work/test.elmo.tsv
- Otherwise: python Preprocessing/line2elmo2.py -l 100 work/test.text.tsv work/test.elmo.tsv
Run the actual application of the model ensemble
- ./ensemble_pred.sh work/test.elmo.tsv work/test.preds.txt