Introduction

This repository implements Lee et al., Hierarchical Novelty Detection for Visual Object Recognition, CVPR 2018 in PyTorch.

@inproceedings{lee2018hierarchical,
  title={Hierarchical Novelty Detection for Visual Object Recognition},
  author={Lee, Kibok and Lee, Kimin and Min, Kyle and Zhang, Yuting and Shin, Jinwoo and Lee, Honglak},
  booktitle={CVPR},
  year={2018}
}

Dependencies

Python 3.6.2 (or https://www.anaconda.com/download/)
PyTorch 0.3.0.post4 (since Pytorch frequently updates their libraries, our code would not work if you have a different version.)
torchvision 0.2.0
NumPy 1.14.0
SciPy 0.19.1 (to load features in mat format for AWA and CUB experiments)
h5py 2.7.0 (to save features and random numbers)
matplotlib 2.0.2 (to plot known-novel class accuracy curve)
NLTK 3.2.4 (to convert WordNet synsets to offset ids for CUB experiments)

Data

You may download either raw images or ResNet-101 features. If you download ResNet-101 features, place them in datasets/{dataset}/. ({dataset} = ImageNet, AWA2, CUB)

ImageNet

Raw images
- Move ILSVRC 2012 train to datasets/ImageNet/train/.
- e.g., an image should be found in datasets/ImageNet/train/n01440764/n01440764_18.JPEG.
- Move ILSVRC 2012 val to datasets/ImageNet/known/.
- ILSVRC 2012 validation dataset is not sorted. You can move validation images to labeled subfolders using [this script].
- Move ImageNet Fall 2011 to datasets/ImageNet/novel/.
ResNet-101 features [train] [val] [known] [novel]

AWA, CUB

ResNet-101 features

WordNet

You do not have to download the files, but we provide the source of them for your reference.

Preparation

Taxonomy

Run sh scripts/preparation.sh {d}. ({d} = imagenet_full, imagenet, awa2, cub)

{d} = imagenet_full if you have raw images
{d} = imagenet if you have ResNet-101 features

Output files are in taxonomy/{dataset}/.

You can download pre-built taxonomies [here].

Feature extraction (ImageNet) or conversion (AWA, CUB)

Run sh scripts/feature.sh {d}. ({d} = imagenet, awa2, cub)

Output files are in datasets/{dataset}/.

If you have ResNet-101 features for ImageNet, skip this.

Train, test

Run sh scripts/train.sh {d} {m}. ({d} = imagenet, awa2, cub, {m} = relabel, td, loo, td+loo)

Output files are in train/.

You can download models reported in the paper [here].

Note

The code keeps all random numbers and final models. For new experiment, you may remove the following if exist:
```
datasets/{dataset}/balanced_order_{:d}.h5
datasets/{dataset}/relabels_{:d}.h5
train/
```
The code can produce results in hierarchical measures. To see them, build the taxonomy with additional argument, e.g., python build_taxonomy.py ImageNet detailed