DeepLearning RS Evaluation

This is the repository of our article RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches". The full text of the article is freely available on ACM Digital Library, ResearchGate or ArXiv, source code of our experiments and full results are available here. The slides and poster are also available.

NEWS: The repository also includes the results of the extended version of the RecSys 2019 paper (still under review), now available on ArXiv "A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research". We welcome your feedback!

We are still actively pursuing this research direction in evaluation and reproducibility, we are open to collaboration with other reseachers. Follow our project on ResearchGate!

Please cite our article if you use this repository or our implementations of baseline algorithms, remember also to cite the original authors if you use our porting of the DL algorithms.

author={Ferrari Dacrema, Maurizio
and Cremonesi, Paolo
and Jannach, Dietmar},
title={Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches},
journal={Proceedings of the 13th ACM Conference on Recommender Systems (RecSys 2019)},
note={Source: \url{}},

Full results and hyperparameters

The full results and corresponding hyperparameters for all DL algorithms are accessible HERE. For information on the requirements and how to install this repository, see the following Installation section.

Code organization

This repository is organized in several subfolders.

Deep Learning Algorithms

The Deep Learning algorithms are all contained in the Conferences folder and further divided in the conferences they were published in. For each DL algorithm the repository contains two subfolders:

Note that in some cases the original repository contained also the data split used by the original authors, those are included as well.

Baseline algorithms

Folders like "KNN", "GraphBased", "MatrixFactorization", "SLIM_BPR", "SLIM_ElasticNet" and "EASE_R" contain all the baseline algorithms we used in our experiments. The complete list is as follows, details on all algorithms and references can be found HERE:

The following similarities are available for all KNN models: cosine, adjusted cosine, pearson correlation, dice, jaccard, asymmetric cosine, tversky, euclidean


The folder Base.Evaluation contains the two evaluator objects (EvaluatorHoldout, EvaluatorNegativeSample) which compute all the metrics we report.


The data to be used for each experiment is gathered from specific DataReader objects within each DL algoritm's folder. Those will load the original data split, if available. If not, automatically download the dataset and perform the split with the appropriate methodology. If the dataset cannot be downloaded automatically, a console message will display the link at which the dataset can be manually downloaded and instructions on where the user should save the compressed file. The data of ConvNCF cannot be automatically handled and should be manually downloaded HERE and decompressed in folder "Conferences/IJCAI/ConvNCF_github/Data".

The folder _Datamanager contains a number of DataReader objects each associated to a specific dataset, which are used to read datasets for which we did not have the original split.

Whenever a new dataset is downloaded and parsed, the preprocessed data is saved in a new folder called _Data_manager_splitdatasets, which contains a subfolder for each dataset. The data split used for the experimental evaluation is saved within the result folder for the relevant algorithm, in a subfolder data .

Hyperparameter optimization

Folder ParameterTuning contains all the code required to tune the hyperparameters of the baselines. The script _run_parametersearch contains the fixed hyperparameters search space used in all our experiments. The object SearchBayesianSkopt does the hyperparameter optimization for a given recommender instance and hyperparameter space, saving the explored configuration and corresponding recommendation quality.

Run the experiments

See see the following Installation section for information on how to install this repository. After the installation is complete you can run the experiments.

All experiments related to a DL algorithm reported in our paper can be executed by running the corresponding script, which is preceeded by _run__, the conference name and the year of publication. The scripts have the following boolean optional parameters (all default values are False except for the print-results flag):

For example, if you want to run all the experiments for SpectralCF, you should run this command:

python -b True -a True -p True

The script will:


Note that this repository requires Python 3.6

First we suggest you create an environment for this project using virtualenv (or another tool like conda)

First checkout this repository, then enter in the repository folder and run this commands to create and activate a new environment:

If you are using virtualenv:

virtualenv -p python3 DLevaluation
source DLevaluation/bin/activate

If you are using conda:

conda create -n DLevaluation python=3.6 anaconda
conda activate DLevaluation

Then if you want to run the experiments on CPU you should install all the requirements and dependencies using the following command. If you wish a GPU installation please install the dependencies as described in subsection Installation on GPU.

pip install -r requirements.txt

At this point, having installed all dependencies for either CPU or GPU usage, you have to compile all Cython algorithms.

In order to compile you must first have installed: gcc and python3 dev. Under Linux those can be installed with the following commands:

sudo apt install gcc 
sudo apt-get install python3-dev

If you are using Windows as operating system, the installation procedure is a bit more complex. You may refer to THIS guide.

Now you can compile all Cython algorithms by running the following command. The script will compile within the current active environment. The code has been developed for Linux and Windows platforms. During the compilation you may see some warnings.


Installation on GPU

In order to run the experiments on GPU you should install requirements and dependencies in the following way. This commands only work if you are using conda to manage your virtual environment. It is possible to install them using other tools or pip but it may prove to be a much more complex task.

conda install tensorflow-gpu
conda install -c anaconda keras-gpu
conda install -c hcc dm-sonnet-gpu

pip install -r requirements_gpu.txt

Matlab engine

In addition to the repository dependencies, KDD CollaborativeDL also requires the Matlab engine, due to the fact that the algorithm is developed in Matlab. To install the engine you can use a script provided directly with your Matlab distribution, as described in the Matlab Documentation. The algorithm requires also a GSL distribution, whose installation folder can be provided as a parameter in the fit function of our Python wrapper. Please refer to the original CollaborativeDL README for all installation details.