MNIST-baselines

This repository contains various baseline models for comparison on the polluted MNIST dataset

Requirements

Usage

Create a model

  1. Prepare Model_Name.json in config/
  2. Prepare Model_Name.py in models/
  3. Prepare Trainer_Name.py in trainers/ (optional)

    Train a model

    python3 main.py --method Model_Name

    Test a model

    python3 main.py --method Model_Name --test

    Trained models

    Available at https://drive.google.com/open?id=1jXe2kJJLkhGH8T30juGbyUgsaDQZXISu

Polluted MNIST

Feature Extraction

A visualization of the first 15 principle components:

Model Accuracy / % Model Accuracy / %
Plain DNN 90.72 DNN + Dropout 90.74
DNN + Batch Normalization 92.39 DNN + PCA (0.99 variance) 21.40
DNN + PCA (0.95 variance) 21.34 DNN + PCA (0.90 variance) 20.55
DNN + ICA 11.93 DNN + NMF 11.30
DNN + VAE 95.24

Conventional Classification Approaches

Model Accuracy / % Model Accuracy / %
Empty Model 11.04 Naive Bayes 19.14
Logistic Regression 24.37 Decision Tree (gini) 51.24
Decision Tree (entropy) 50.91 KNN (3 neighbours) 78.82
KNN (5 neighbours) 78.46 KNN (10 neighbours) 76.40
SGD 22.30 SVM (sigmoid kernel) 10.84
SVM (rbf kernel) 85.94 SVM (polynomial kernel) 87.11

The training curves of the above DNN models:

Deep Convolutional Neural Networks

Model Accuracy / % Model Accuracy / %
LeNet-5 98.35 MobileNetV2 99.63
VGG-19 99.64 ShuffleNetG3 99.65
ResNet-101 99.71 GoogLeNet 99.82
PreAct ResNet-152 99.68 PNASNet 99.75
DenseNet-161 99.75 ResNeXt-29-8x64d 99.71
DPN-92 99.76 SENet-18 99.69
MobileNet 99.44 CapsNet 98.84

Error Analysis

Team Members