Easily Extendable Basic Deep Metric Learning Pipeline

Karsten Roth (karsten.rh1@gmail.com), Biagio Brattoli (biagio.brattoli@gmail.com)

When using this repo in any academic work, please provide a reference to

@misc{roth2020revisiting,
    title={Revisiting Training Strategies and Generalization Performance in Deep Metric Learning},
    author={Karsten Roth and Timo Milbich and Samarth Sinha and Prateek Gupta and Björn Ommer and Joseph Paul Cohen},
    year={2020},
    eprint={2002.08473},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

Based on an extendend version of this repo, we have created a thorough comparison and evaluation of Deep Metric Learning:

https://arxiv.org/abs/2002.08473

The newly released code can be found here: https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch

It contains more criteria, miner, metrics and logging options!

For usage, go to section 3 - for results to section 4

1. Overview

This repository contains a full, easily extendable pipeline to test and implement current and new deep metric learning methods. For referencing and testing, this repo contains implementations/dataloaders for:

Loss Functions

Triplet Loss (https://arxiv.org/abs/1412.6622)
Margin Loss (https://arxiv.org/abs/1706.07567)
ProxyNCA (https://arxiv.org/abs/1703.07464)
N-Pair Loss (https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective.pdf)

Sampling Methods

Random Sampling
Softhard Sampling (Soft Version of hard tuple sampling)
Semihard Sampling (https://arxiv.org/abs/1503.03832)
Distance Sampling (https://arxiv.org/abs/1706.07567)
N-Pair Sampling (https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective.pdf)

Datasets

CUB200-2011 (http://www.vision.caltech.edu/visipedia/CUB-200.html)
CARS196 (https://ai.stanford.edu/~jkrause/cars/car_dataset.html)
Stanford Online Products (http://cvgl.stanford.edu/projects/lifted_struct/)
In-Shop Clothes (http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/InShopRetrieval.html, download from https://drive.google.com/drive/folders/0B7EVK8r0v71pVDZFQXRsMDZCX1E. Thanks to KunHe for providing the link!)
(optional) PKU Vehicle-ID (https://www.pkuml.org/resources/pku-vds.html)

Architectures

ResNet50 (https://arxiv.org/abs/1512.03385)
GoogLeNet (https://arxiv.org/abs/1409.4842)
[Note: This version follows the official torchvision implementation, which differs from the original variant.]

NOTE: PKU Vehicle-ID is (optional) because there is no direct way to download the dataset, as it requires special licensing. However, if this dataset becomes available (in the structure shown in part 2.2), it can be used directly.

1.1 Related Repos:

2. Repo & Dataset Structure

2.1 Repo Structure

Repository
│   ### General Files
│   README.md
│   requirements.txt    
│   installer.sh
|
|   ### Main Scripts
|   Standard_Training.py     (main training script)
|   losses.py   (collection of loss and sampling impl.)
│   datasets.py (dataloaders for all datasets)
│   
│   ### Utility scripts
|   auxiliaries.py  (set of useful utilities)
|   evaluate.py     (set of evaluation functions)
│   
│   ### Network Scripts
|   netlib.py       (contains impl. for ResNet50)
|   googlenet.py    (contains impl. for GoogLeNet)
│   
│   
└───Training Results (generated during Training)
|    │   e.g. cub200/Training_Run_Name
|    │   e.g. cars196/Training_Run_Name
|
│   
└───Datasets (should be added, if one does not want to set paths)
|    │   cub200
|    │   cars196
|    │   online_products
|    │   in-shop
|    │   vehicle_id

2.2 Dataset Structures

CUB200-2011/CARS196

cub200/cars196
└───images
|    └───001.Black_footed_Albatross
|           │   Black_Footed_Albatross_0001_796111
|           │   ...
|    ...

Online Products

online_products
└───images
|    └───bicycle_final
|           │   111085122871_0.jpg
|    ...
|
└───Info_Files
|    │   bicycle.txt
|    │   ...

In-Shop Clothes

in-shop
└─img
|    └─MEN
|         └─Denim
|               └─id_00000080
|                  │   01_1_front.jpg
|                  │   ...
|               ...
|         ...
|    ...
|
└─Eval
|  │   list_eval_partition.txt

PKU Vehicle ID

vehicle_id
└───image
|     │   <img>.jpg
|     |   ...
|     
└───train_test_split
|     |   test_list_800.txt
|     |   ...

3. Using the Pipeline

[1.] Requirements

The pipeline is build around Python3 (i.e. by installing Miniconda https://conda.io/miniconda.html') and Pytorch 1.0.0/1. It has been tested around cuda 8 and cuda 9.

To install the required libraries, either directly check requirements.txt or create a conda environment:

conda create -n <Env_Name> python=3.6

Activate it

conda activate <Env_Name>

and run

bash installer.sh

Note that for kMeans- and Nearest Neighbour Computation, the library faiss is used, which can allow to move these computations to GPU if speed is desired. However, in most cases, faiss is fast enough s.t. the computation of evaluation metrics is no bottleneck.
NOTE: If one wishes not to use faiss but standard sklearn, simply use auxiliaries_nofaiss.py to replace auxiliaries.py when importing the libraries.

[2.] Exemplary Runs

The main script is Standard_Training.py. If running without input arguments, training of ResNet50 on CUB200-2011 with Marginloss and Distance-sampling is performed.
Otherwise, the following flags suffice to train with different losses, sampling methods, architectures and datasets:

python Standard_Training.py --dataset <dataset> --loss <loss> --sampling <sampling> --arch <arch> --k_vals <k_vals> --embed_dim <embed_dim>

The following flags are available:

<dataset> <- cub200, cars196, online_products, in-shop, vehicle_id
<loss> <- marginloss, triplet, npair, proxynca
<sampling> <- distance, semihard, random, npair
<arch> <- resnet50, googlenet
<k_vals> <- List of Recall @ k values to evaluate on, e.g. 1 2 4 8
<embed_dim> <- Network embedding dimension. Default: 128 for ResNet50, 512 for GoogLeNet.

For all other training-specific arguments (e.g. batch-size, num. training epochs., ...), simply refer to the input arguments in Standard_Training.py.

NOTE: If one wishes to use a different learning rate for the final linear embedding layer, the flag --fc_lr_mul needs to be set to a value other than zero (i.e. 10 as is done in various implementations).

Finally, to decide the GPU to use and the name of the training folder in which network weights, sample recoveries and metrics are stored, set:

python Standard_Training.py --gpu <gpu_id> --savename <name_of_training_run>

If --savename is not set, a default name based on the starting date will be chosen.

If one wishes to simply use standard parameters and wants to get close to literature results (more or less, depends on seeds and overall training scheduling), refer to sample_training_runs.sh, which contains a list of executable one-liners.

[3.] Implementation Notes regarding Extendability:

To extend or test other sampling or loss methods, simply do:

For Batch-based Sampling:
In losses.py, add the sampling method, which should act on a batch (and the resp. set of labels), e.g.:

def new_sampling(self, batch, label, **additional_parameters): ...

This function should, if it needs to run with existing losses, a list of tuples containing indexes with respect to the batch, e.g. for sampling methods returning triplets:

return [(anchor_idx, positive_idx, negative_idx) for anchor_idx, positive_idx, negative_idx in zip(anchor_idxs, positive_idxs, negative_idxs)]

Also, don't forget to add a handle in Sampler.__init__().

For Data-specific Sampling:
To influence the data samples used to generate the batches, in datasets.py edit BaseTripletDataset.

For New Loss Functions:
Simply add a new class inheriting from torch.nn.Module. Refer to other loss variants to see how to do so. In general, include an instance of the Sampler-class, which will provide sampled data tuples during a forward()-pass, by calling self.sampler_instance.give(batch, labels, **additional_parameters).
Finally, include the loss function in the loss_select()-function. Parameters can be passed through the dictionary-notation (see other examples) and if learnable parameters are added, include them in the to_optim-list.

[4.] Stored Data:

By default, the following files are saved:

Name_of_Training_Run
|  checkpoint.pth.tar   -> Contains network state-dict.
|  hypa.pkl             -> Contains all network parameters as pickle.
|                          Can be used directly to recreate the network.
| log_train_Base.csv    -> Logged training data as CSV.                      
| log_val_Base.csv      -> Logged test metrics as CSV.                    
| Parameter_Info.txt    -> All Parameters stored as readable text-file.
| InfoPlot_Base.svg     -> Graphical summary of training/testing metrics progression.
| sample_recoveries.png -> Sample recoveries for best validation weights.
|                          Acts as a sanity test.

Sample Recoveries Note: Red denotes query images, while green show the resp. nearest neighbours.

Sample Recoveries Note: The header in the summary plot shows the best testing metrics over the whole run.

[5.] Additional Notes:

To finalize, several flags might be of interest when examining the respective runs:

--dist_measure: If set, the ratio of mean intraclass-distances over mean interclass distances
                (by measure of center-of-mass distances) is computed after each epoch and stored/plotted.
--grad_measure: If set, the average (absolute) gradients from the embedding layer to the last
                conv. layer are stored in a Pickle-File. This can be used to examine the change of features during each iteration.

For more details, refer to the respective classes in auxiliaries.py.

4. Results

These results are supposed to be performance estimates achieved by running the respective commands in sample_training_runs.sh. Note that the learning rate scheduling might not be fully optimised, so these values should only serve as reference/expectation, not what can be ultimately achieved with more tweaking.

Note also that there is a not insignificant dependency on the used seed.

CUB200

Architecture	Loss/Sampling	NMI	F1	Recall @ 1 -- 2 -- 4 -- 8
ResNet50	Margin/Distance	68.2	38.7	63.4 -- 74.9 -- 86.0 -- 90.4
ResNet50	Triplet/Softhard	66.2	35.5	61.2 -- 73.2 -- 82.4 -- 89.5
ResNet50	NPair/None	65.4	33.8	59.0 -- 71.3 -- 81.1 -- 88.8
ResNet50	ProxyNCA/None	68.1	38.1	64.0 -- 75.4 -- 84.2 -- 90.5

Cars196

Architecture	Loss/Sampling	NMI	F1	Recall @ 1 -- 2 -- 4 -- 8
ResNet50	Margin/Distance	67.2	37.6	79.3 -- 87.1 -- 92.1 -- 95.4
ResNet50	Triplet/Softhard	64.4	32.4	75.4 -- 84.2 -- 90.1 -- 94.1
ResNet50	NPair/None	62.3	30.1	69.5 -- 80.2 -- 87.3 -- 92.1
ResNet50	ProxyNCA/None	66.3	35.8	80.0 -- 87.2 -- 91.8 -- 95.1