The Problem

Deep neural networks are not robust.

image of a cat classified as cat
Tiger Cat

+

attack image that can be merged with original image
Adversarial Perturbation

=

resulting image of a cat that classifies as ostrich
» Ostrich «

Even todays most advanced machine learning models are easily fooled by almost imperceptual perturbations of their inputs. These so-called adversarial perturbations demarcate a striking difference between human and machine perception. Closing this gap is one of the next major frontiers of Deep Learning.

Robust Vision Benchmark

A benchmark for the robustness of your model and the effectiveness of your attack.

The Robust Vision Benchmark is a platform for researchers to test the robustness of their models or the effectiveness of new adversarial attacks. The goal is to use this co-evolution to find truly robust models that resist even the most effective adversarial attacks. We will start with three independent challenges with different levels of difficulty, one for each of the common datasets and tasks: MNIST, CIFAR, and ImageNet.

Three independent challenges.

The MNIST challenge requires the robust classification of handwritten digits with a resolution of 28 by 28 pixels.

The CIFAR challenge requires the robust classification of 32 by 32 pixel natural images from ten different classes.

The ImageNet challenge requires the robust classification of natural images into 1000 classes.

Trained a robust model?

The best way to prove your model’s robustness.

Being robust against a specific adversarial attack is easy, but it does not guarantee robustness against adversarial perturbations in general. By submitting your model to the Robust Vision Benchmark, it is automatically tested against all existing as well as future adversarial attacks. To be truly robust a model has to score well in this benchmark.

Created a new adversarial attack?

Running your attack against every model has never been easier.

Assessing the effectiveness of a new adversarial attack can be difficult, because it can heavily depend on the dataset and the machine learning model that is attacked. Running an attack on many models, possibly implemented in different languages and frameworks, requires a lot of work. Moreover, new machine learning models are developed all the time. By submitting your attack to the Robust Vision Benchmark, it will automatically be run against a huge collection of existing models as well as against new models submitted in the future.

Organizers

Contact Us

The Robust Vision Benchmark is organized by the Bethge lab and supported through several grants and initiatives such as the Collaborative Research Center 1233 Robust Vision and NINAI. If you have any questions or proposals, don't hesitate to contact us.

Learn more at bethgelab.org