Perceptron is a robustness benchmark for computer vision DNN models. It
supports both image classification and object detection models on PyTorch,
Tensorflow, Keras, PaddlePaddle (in progress), as well as cloud APIs. Perceptron inherits the
foolbox, and is designed to be agnostic to
the deep learning frameworks the models are built on.
Documentation is available on
Currently, you can use Perceptron either through its python API or its command line tool.
Tensorflow packages are required if you want to test their models, we requir user to manually install them on demand. Otherwise, run the following command to install Perceptron Benchmark
pip install -e .
In the docker shell, run the test through Perceptron command line interface
python perceptron/launcher.py \ --framework keras \ --model resnet50 \ --metric carlini_wagner_l2 \ --image example.png
In the example, user specifies framework as
keras, the model as
resnet50, the metric as
carlini_wagner_l2, the input image as
example.png. The output would be as follows.
To visualize the adversary, we also provide the plot of the original image, adversary image, and their difference as follows.
You can try different combinations of frameworks, models, criteria, and
metrics. To see more options using
-h for help message.
python perceptron/launcher.py -h
Build the docker image and all dependencies will be installed automatically.
nvidia-docker build -t perceptron:env .
The following example serves the same purpose as the command line example.
This example benchmarks the robustness of Keras ResNet50 model against
metric by measuring the minimal required :math:
L2 perturbation for a
success. The minimum Mean Squred Distance (MSE) will be logged.
import numpy as np import keras.applications as models from perceptron.models.classification.keras import KerasModel from perceptron.utils.image import imagenet_example from perceptron.benchmarks.carlini_wagner import CarliniWagnerL2Metric from perceptron.utils.criteria.classification import Misclassification # instantiate the model from keras applications resnet50 = models.ResNet50(weights='imagenet') # initialize the KerasModel # keras resnet50 has input bound (0, 255) preprocessing = (np.array([104, 116, 123]), 1) # the mean and stv of the whole dataset kmodel = KerasModel(resnet50, bounds=(0, 255), preprocessing=preprocessing) # get source image and label # the model expects values in [0, 255], and channles_last image, _ = imagenet_example(data_format='channels_last') label = np.argmax(kmodel.predictions(image)) metric = CarliniWagnerL2Metric(kmodel, criterion=Misclassification()) adversary = metric(image, label, unpack=False)
Running the example will give you the minimal MSE required for
C&W2 to fool
resnet50 model (i.e., changing the predicted label).
You are welcome to send pull requests and report issues on GitHub or iCode. Note that the Perceptron Benchmark project follows the Git flow development model.
Perceptron Robustness Benchmark is provided under Apache-2.0 license. For a copy, see LICENSE file.