Recent progress in machine-learning technologies has reported impressive performances in computer vision and security-sensitive tasks.

Understanding the security properties of learning algorithms, as well as designing suitable countermeasures, has thus become a timely, relevant and challenging research field.

A recent WIRED article has reported that AI has a hallucination problem that’s proving tough to fix.

Read the interview to our team members here

Our research team has been among the first to:

  • show that machine-learning algorithms are vulnerable to gradient-based adversarial manipulations of the input data, both at test time (evasion attacks) and at training time (poisoning attacks)
  • derive a systematic framework for security evaluation of learning algorithms
  • develop suitable countermeasures for improving their security

Evasion attacks (also recently referred to as adversarial examples) consist of manipulating input data to evade a trained classifier at test time. These include, for instance, manipulation of malware code to have the corresponding sample undetected (i.e., misclassified as legitimate), or manipulation of images to mislead object recognition.

We have been the first to demonstrate these attacks against nonlinear classifiers, including Support Vector Machines and Neural Networks, in [Biggio et al., ECML-PKDD 2013 ]. Notably, these classifiers were believed to be more secure than linear classifiers at that time, due to their complex input-output mapping relationships [Šrndić & Laskov, NDSS 2013 ]. We demonstrated how to overcome this difficulty with a straightforward gradient-based evasion attack, and highlighted the vulnerability of such classifiers to evasion in different application settings, including handwritten digit recognition and PDF malware detection.

Evasion attacks have been independently derived in the area of deep learning and computer vision later in [C. Szegedy et al., ICLR 2014 ], under the name of adversarial examples, namely, images that can be misclassified by deep-learning algorithms while being only imperceptibly distorted.

Adversarial example from ECML-PKDD 2013

The very first adversarial example from our ECML-PKDD 2013 paper

We have also recently developed a secure-learning algorithm to counter adversarial examples in Android malware detection [A. Demontis et al., IEEE TDSC 2017 ]. We have derived a robust version of Drebin, a popular malware detection tool based on static code analysis [D. Arp et al., NDSS 2014 ]. A similar attempt has been also recently reported in [K. Grosse et al., ESORICS 2017 ].

Create adversarial examples with our evasion attack here:

Web Demo

Poisoning attacks are subtler. Their goal is to mislead the learning algorithm during the training phase by manipulating only a small fraction of the training data, in order to significantly increase the number of misclassified samples at test time, causing a denial of service. These attacks require access to the training data used to learn the classification algorithm, which is possible in some application-specific contexts.

We demonstrated poisoning attacks against Support Vector Machines in [B. Biggio et al., ICML 2012 ], then against LASSO, Ridge and Elastic-net Regression in [H. Xiao et al., ICML 2015 ], and more recently against Neural Networks and Deep Learning algorithms [L. Muñoz-González et al., AISec 2017 ].

Poisoning attack against SVMs (Fig 1.) Poisoning attack against SVMs (Fig 2.)

Poisoning attack against SVMs [Biggio et al., ICML 2012 ]

  • (2012) Poisoning Attacks against Support Vector Machines

    B. Biggio, B. Nelson, P. Laskov. In ICML 2012.

  • (2013) Evasion Attacks against Machine Learning at Test Time

    B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, F. Roli. In ECML-PKDD 2013.

  • (2014) Security Evaluation of Pattern Classifiers under Attack

    B. Biggio, G. Fumera, F. Roli. In IEEE TKDE 2014.

  • (2015) Is Feature Selection Secure against Training Data Poisoning?

    H. Xiao, B. Biggio, G. Brown, G. Fumera, C. Eckert, F. Roli. In ICML 2015.

  • (2017) Is Deep Learning Safe for Robot Vision? Adversarial Examples against the iCub Humanoid

    M. Melis, A. Demontis, B. Biggio, G. Brown, G. Fumera, F. Roli. In 2017 ICCV Workshop ViPAR.

  • (2017) Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization

    L. Muñoz-González, B. Biggio, A. Demontis, A. Paudice, V. Wongrassamee, E. C. Lupu, F. Roli. In AISec 2017.

  • (2017) Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection

    A. Demontis, M. Melis, B. Biggio, D. Maiorca, D. Arp, K. Rieck, I. Corona, G. Giacinto, F. Roli. In IEEE TDSC 2017.

  • (2017) Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

    B. Biggio, F. Roli. In Pattern Recognition (under review).

Battista Biggio

Assistant Professor at PRA Lab, and Co-Founder of Pluribus One

Marco Melis

PhD student at University of Cagliari

Ambra Demontis

PhD fellow at University of Cagliari

Fabio Roli

Professor of Computer Engineering and Director of the PRA Lab

The sad thing about artificial intelligence is that it lacks artifice and therefore intelligence.