Introduction

This open-source project, referred to as PT-Ranking (Learning to Rank in PyTorch) aims to provide scalable and extendable implementations of typical learning-to-rank methods based on PyTorch. On one hand, this project enables a uniform comparison over several benchmark datasets leading to an in-depth understanding of previous learning-to-rank methods. On the other hand, this project makes it easy to develop and incorporate newly proposed models, so as to expand the territory of techniques on learning-to-rank.

Key Features:

How-to-Start and Learning more

Demo Scripts

To get a taste of learning-to-rank models without writing any code, you could try the following script. You just need to specify the model name, the dataset id, as well as the directories for input and output.

To get familiar with the process of data loading, you could try the following script, namely, get the statistics of a dataset.

Develop a new model based on PT-Ranking

PT-Ranking offers deep neural networks as the basis to construct a scoring function based on PyTorch and can thus fully leverage the advantages of PyTorch. NeuralRanker is a class that represents a general learning-to-rank model. A key component of NeuralRanker is the neural scoring function. The configurable hyper-parameters include activation function, number of layers, number of neurons per layer, etc. All specific learning-to-rank models inherit NeuralRanker and mainly differ in the way of computing the training loss. The following figure shows the main step in developing a new learning-to-rank model based on Empirical Risk Minimization, where batch_preds and batch_stds correspond to outputs of the scoring function and ground-truth lables, respectively. We can observe that the main work is to define the surrogate loss function.

NewLoss

To fully leverage PT-Ranking, one needs to be familiar with PyTorch.

For detailed introduction on learning-to-rank, please refer to the book: Learning to Rank for Information Retrieval.

List of implemented models & References

Model
Pointwise RankMSE
Pairwise RankNet
Listwise LambdaRank ・ ListNet ・ ListMLE ・ RankCosine ・ ApproxNDCG ・ WassRank ・ STListNet
Model
Pointwise IR_GAN_Point
Pairwise IR_GAN_Pair
Listwise IR_GAN_List
Model
Listwise LambdaMART(L) ・ LambdaMART(X)

References

Supported Datasets and Formats

The widely used benchmark datasets listed below can be directly used once downloaded

-- LETOR4.0 (including MQ2007 | MQ2008 | MQ2007-semi | MQ2008-semi | MQ2007-list | MQ2008-list )

-- MSLR-WEB10K (including MSLR-WEB10K | MSLR-WEB30K)

-- Yahoo! LETOR (including Set1 | Set2)

-- Istella (including Istella-S | Istella | Istella-X)

Some notes on the above datasets

PT-Ranking currently supports to ingest data with the LibSVM formats

\<ground-truth label int> qid\: : ... :

For example:

4 qid:105 2:0.4 8:0.7 50:0.5

1 qid:105 5:0.5 30:0.7 32:0.4 48:0.53

0 qid:210 4:0.9 38:0.01 39:0.5 45:0.7

1 qid:210 1:0.2 8:0.9 31:0.93 40:0.6

The above sample dataset includes two queries, the query "105" has 2 documents, the corresponding ground-truth labels are 4 and 1, respectively.

This functionality is required when using the implementation of LambdaMART provided in LightGBM and XGBoost.

Test Setting

PyTorch (1.3)

Python (3.7)

Ubuntu 16.04 LTS

Call for Contribution

We are adding more learning-to-rank models all the time. Please submit an issue if there is something you want to have implemented and included. Meanwhile, anyone who are interested in any kinds of contributions and/or collaborations are warmly welcomed.

Relevant Resources

Name Language Deep Learning
PT-Ranking Python PyTorch
TF-Ranking Python TensorFlow
MatchZoo Python Keras / PyTorch
Shoelace Python Chainer
LEROT Python x
Rank Lib Java x
Propensity SVM^Rank C x
QuickRank C++ x