EELECTION at SemEval-2017 Task 10: Ensemble of nEural Learners for kEyphrase ClassificaTION

SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications.

This repository contains the code needed to reproduce our results for the shared task ScienceIE reported in Eger et al., EELECTION at SemEval-2017 Task 10: Ensemble of nEural Learners for kEyphrase ClassificaTION.

Please cite the paper as:

  author    = {Eger, Steffen and Do Dinh, Erik-Lân and Kutsnezov, Ilia and Kiaeeha, Masoud and Gurevych, Iryna},
  title     = {{EELECTION at SemEval-2017 Task 10: Ensemble of nEural Learners for kEyphrase ClassificaTION}},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017)},
  month     = {August},
  year      = {2017},
  address   = {Vancouver, Canada},
  publisher = {Association for Computational Linguistics},
  pages     = {942--946},
  url       = {}

Abstract: This paper describes our approach to the SemEval 2017 Task 10: "Extracting Keyphrases and Relations from Scientific Publications", specifically to Subtask (B): "Classification of identified keyphrases". We explored three different deep learning approaches: a character-level convolutional neural network (CNN), a stacked learner with an MLP meta-classifier, and an attention based Bi-LSTM. From these approaches, we created an ensemble of differently hyper-parameterized systems, achieving a micro-F1-score of 0.63 on the test data. Our approach ranks 2nd (score of 1st placed system: 0.64) out of four according to this official score. However, we erroneously trained 2 out of 3 neural nets (the stacker and the CNN) on only roughly 15% of the full data, namely, the original development set. When trained on the full data (training+development), our ensemble has a micro-F1-score of 0.69.

Contact persons:

Don't hesitate to contact us if something is broken (and it shouldn't be) or if you have further questions.

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.

Project structure


Running the experiments

To run the experiments described in our paper you have to aquire following resources.

Put the following embeddings into data/embeddings:

Put the training, dev and test data into data/train, data/dev and data/test, respectively. For running the experiment scripts below, also create data/combined, and copy the train and dev data into it.

Further, the keras version has a bug regarding unicode, which has to be fixed as e.g. described in keras-fix.

The scripts to start the experiments can be found in scripts_submission.