Benutzer:ShirkahnW/Adversariales maschinelles Lernen

Dieser Artikel (Adversariales maschinelles Lernen) ist im Entstehen begriffen und noch nicht Bestandteil der freien Enzyklopädie Wikipedia.

Wenn du dies liest: Der Text kann teilweise in einer Fremdsprache verfasst, unvollständig sein oder noch ungeprüfte Aussagen enthalten. Wenn du Fragen zum Thema hast, nimm am besten Kontakt mit dem Autor ShirkahnW auf.

Wenn du diesen Artikel überarbeitest: Bitte denke daran, die Angaben im Artikel durch geeignete Quellen zu belegen und zu prüfen, ob er auch anderweitig den Richtlinien der Wikipedia entspricht (siehe Wikipedia:Artikel). Nach erfolgter Übersetzung kannst du diese Vorlage entfernen und den Artikel in den Artikelnamensraum verschieben. Die entstehende Weiterleitung kannst du schnelllöschen lassen. Importe inaktiver Accounts, die länger als drei Monate völlig unbearbeitet sind, werden gelöscht.

Vorlage:Importartikel/Wartung-2022-06

Feindliches machinelles Lernen (engl. adversarial machine learning) ist die Untersuchung der Angriffe auf Algorithmen des maschinellen Lernens und der Verteidigungsmaßnahmen gegen solche Angriffe. Eine 2020 durchgeführte Umfrage hat ergeben, dass in der Praxis ein dringender Bedarf an einem besseren Schutz für maschinelle Lernsysteme in industriellen Anwendungen besteht.

Zum Verständnis sei angemerkt, dass die meisten Verfahren des maschinellen Lernens für die Bearbeitung bestimmter Problemstellungen konzipiert sind, wobei davon ausgegangen wird, dass die Trainings- und Testdaten aus derselben statistischen Verteilung stammen. Diese Annahme wird jedoch in praktischen Anwendungen mit hohem Risiko oft auf gefährliche Weise verletzt, da die Benutzer absichtlich gefälschte Daten liefern können, die die statistische Annahme verletzen.

Zu den am meisten bedrohten Modellen im Bereich des feindlichen maschinellen Lernens gehören Umgehungsangriffe, Data-Poisoning-Angriffe, Byzantine-Gradienten-Angriffe und Modellextraktion.

History

Im Jahr 2004 stellten Nilesh Dalvi und andere fest, dass die in Spam-Filtern verwendeten linearen Klassifizierer durch einfache "Umgehungsangriffe" überwunden werden können, indem Spammer "gute Wörter" in ihre Spam-E-Mails einfügen. (Um 2007 fügten einige Spammer zufälliges Rauschen hinzu, um Wörter in "Bild-Spam" zu verwischen, um OCR-basierte Filter zu umgehen). Im Jahr 2006 veröffentlichten Marco Barreno und andere die Publikation "Can Machine Learning Be Secure?", in der sie eine umfassende Taxonomie von Angriffen aufstellten. Noch 2013 hofften viele Forscher, dass nichtlineare Klassifizierer (wie Support-Vektor-Maschinen und neuronale Netze) gegen Angreifer resistent sein könnten, bis Battista Biggio und andere die ersten gradientenbasierten Angriffe auf solche Machine-Learning-Modelle demonstrierten (2012-2013). Im Jahr 2012 begannen tiefe neuronale Netze Computer-Vision Probleme zu dominieren; ab 2014 zeigten Christian Szegedy und andere, dass tiefe neuronale Netze von Angreifern überlistet werden können, wobei sie wiederum einen gradientenbasierten Angriff verwendeten, um gegnerische Störungen zu erzeugen.

Kürzlich wurde festgestellt, dass Angriffe in der Praxis schwieriger zu realisieren sind, da die verschiedenen Umgebungsbedingungen die Wirkung von Rauschen aufheben. So kann zum Beispiel jede kleine Drehung oder leichte Beleuchtung eines ungünstigen Bildes die Unglaubwürdigkeit zerstören. Darüber hinaus weisen Forscher wie Nicholas Frosst von Google Brain darauf hin, dass es viel einfacher ist, selbstfahrende Autos dazu zu bringen, Stoppschilder zu übersehen, indem man das Schild selbst physisch entfernt, anstatt Gegenbeispiele zu schaffen. Frosst ist auch der Meinung, dass die Gemeinschaft des feindlichen maschinellen Lernens fälschlicherweise davon ausgeht, dass Modelle, die für eine bestimmte Datenverteilung trainiert wurden, auch bei einer völlig anderen Datenverteilung gut funktionieren. Er schlägt vor, einen neuen Ansatz für das maschinelle Lernen zu erforschen, und arbeitet derzeit an einem einzigartigen neuronalen Netzwerk, das der menschlichen Wahrnehmung ähnlicher ist als herkömmliche Ansätze.

Während das feindliche maschinelle Lernen nach wie vor stark im akademischen Bereich verwurzelt ist, haben große Technologieunternehmen wie Google, Microsoft und IBM damit begonnen, Dokumentationen und Open-Source-Codebasen zu erstellen, um anderen die Möglichkeit zu geben, die Robustheit von maschinellen Lernmodellen konkret zu bewerten und das Risiko feindlicher Angriffe zu minimieren.

Examples

Examples include attacks in spam filtering, where spam messages are obfuscated through the misspelling of "bad" words or the insertion of "good" words; attacks in computer security, such as obfuscating malware code within network packets or modifying the characteristics of a network flow to mislead intrusion detection; attacks in biometric recognition where fake biometric traits may be exploited to impersonate a legitimate user; or to compromise users' template galleries that adapt to updated traits over time.

Researchers showed that by changing only one-pixel it was possible to fool deep learning algorithms. Others 3-D printed a toy turtle with a texture engineered to make Google's object detection AI classify it as a rifle regardless of the angle from which the turtle was viewed. Creating the turtle required only low-cost commercially available 3-D printing technology.

A machine-tweaked image of a dog was shown to look like a cat to both computers and humans. A 2019 study reported that humans can guess how machines will classify adversarial images. Researchers discovered methods for perturbing the appearance of a stop sign such that an autonomous vehicle classified it as a merge or speed limit sign.

McAfee attacked Tesla's former Mobileye system, fooling it into driving 50 mph over the speed limit, simply by adding a two-inch strip of black tape to a speed limit sign.

Adversarial patterns on glasses or clothing designed to deceive facial-recognition systems or license-plate readers, have led to a niche industry of "stealth streetwear".

An adversarial attack on a neural network can allow an attacker to inject algorithms into the target system. Researchers can also create adversarial audio inputs to disguise commands to intelligent assistants in benign-seeming audio; a parallel literature explores human perception of such stimuli.

Clustering algorithms are used in security applications. Malware and computer virus analysis aims to identify malware families, and to generate specific detection signatures.

Attack modalities

Taxonomy

Attacks against (supervised) machine learning algorithms have been categorized along three primary axes: influence on the classifier, the security violation and their specificity.

• Classifier influence: An attack can influence the classifier by disrupting the classification phase. This may be preceded by an exploration phase to identify vulnerabilities. The attacker's capabilities might be restricted by the presence of data manipulation constraints.
• Security violation: An attack can supply malicious data that gets classified as legitimate. Malicious data supplied during training can cause legitimate data to be rejected after training.
• Specificity: A targeted attack attempts to allow a specific intrusion/disruption. Alternatively, an indiscriminate attack creates general mayhem.

This taxonomy has been extended into a more comprehensive threat model that allows explicit assumptions about the adversary's goal, knowledge of the attacked system, capability of manipulating the input data/system components, and on attack strategy. This taxonomy has further been extended to include dimensions for defense strategies against adversarial attacks.

Strategies

Below are some of the most commonly encountered attack scenarios:

Data poisoning

Poisoning consists of contaminating the training dataset. Given that learning algorithms are shaped by their training datasets, poisoning can effectively reprogram algorithms. Serious concerns have been raised especially for user-generated training data, e.g. for content recommendation or natural language models, especially given the ubiquity of fake accounts. To measure the scale of the risk, it suffices to note that Facebook reportedly removes around 7 billion fake accounts per year. In fact, data poisoning has been reported as the leading concern for industrial applications.

On social medias, disinformation campaigns are known to produce vast amounts of fabricated activities to bias recommendation and moderation algorithms, to push certain content over others.

A particular case of data poisoning is called backdoor attack, which aims to teaches a specific behavior for inputs with a given trigger, e.g. a small defect on images, sounds, videos or texts.

For instance, intrusion detection systems (IDSs) are often re-trained using collected data. An attacker may poison this data by injecting malicious samples during operation that subsequently disrupt retraining.

Byzantine attacks

As machine learning is scaled, it often relies on multiple computing machines. In federated learning, for instance, edge devices collaborate with a central server, typically by sending gradients or model parameters. However, some of these devices may deviate from their expected behavior, e.g. to harm the central server's model or to bias algorithms towards certain behaviors (e.g., amplifying the recommendation of disinformation content). On the other hand, if the training is performed on a single machine, then the model is very vulnerable to a failure of the machine, or an attack on the machine; the machine is a single point of failure. In fact, the machine owner may themselves insert provably undetectable backdoors.

The current leading solutions to make (distributed) learning algorithms provably resilient to a minority of malicious (a.k.a. Byzantine) participants are based on robust gradient aggregation rules. Nevertheless, in the context of heterogeneous honest participants, such as users with different consumption habits for recommendation algorithms or writing styles for language models, there are provable impossibility theorems on what any robust learning algorithm can guarantee.

Evasion

Evasion attacks consist of exploiting the imperfection of a trained model. For instance, spammers and hackers often attempt to evade detection by obfuscating the content of spam emails and malware. Samples are modified to evade detection; that is, to be classified as legitimate. This does not involve influence over the training data. A clear example of evasion is image-based spam in which the spam content is embedded within an attached image to evade textual analysis by anti-spam filters. Another example of evasion is given by spoofing attacks against biometric verification systems.

Evasion attacks can be generally split into two different categories: black box attacks and white box attacks.

Model extraction

Model extraction involves an adversary probing a black box machine learning system in order to extract the data it was trained on. This can cause issues when either the training data or the model itself is sensitive and confidential. For example, model extraction could be used to extract a proprietary stock trading model which the adversary could then use for their own financial benefit.

In the extreme case, model extraction can lead to model stealing, which corresponds to extracting a sufficient amount of data from the model to enable the complete reconstruction of the model.

On the other hand, membership inference is a targeted model extraction attack, which infers the owner of a data point, often by leveraging the overfitting resulting from poor machine learning practices. Concerningly, this is sometimes achievable even without knowledge or access to a target model's parameters, raising security concerns for models trained on sensitive data, including but not limited to medical records and/or personally identifiable information. With the emergence of transfer learning and public accessibility of many state of the art machine learning models, tech companies are increasingly drawn to create models based on public ones, giving attackers freely accessible information to the structure and type of model being used.

Specific attack types

There are a large variety of different adversarial attacks that can be used against machine learning systems. Many of these work on both deep learning systems as well as traditional machine learning models such as SVMs and linear regression. A high level sample of these attack types include:

• Adversarial Examples
• Trojan Attacks / Backdoor Attacks
• Model Inversion
• Membership Inference

Adversarial examples

An adversarial example refers to specially crafted input which is designed to look "normal" to humans but causes misclassification to a machine learning model. Often, a form of specially designed "noise" is used to elicit the misclassifications. Below are some current techniques for generating adversarial examples in the literature (by no means an exhaustive list).

• Gradient-based evasion attack
• Fast Gradient Sign Method (FGSM)
• Projected Gradient Descent (PGD)
• Carlini and Wagner (C&W) attack
• Adversarial patch attack

Black Box Attacks

Black box attacks in adversarial machine learning assumes that the adversary can only get outputs for provided inputs and has no knowledge of the model structure or parameters. In this case, the adversarial example is generated either using a model created from scratch, or without any model at all (excluding the ability to query the original model). In either case, the objective of these attacks are to create adversarial examples that are able to transfer to the black box model in question.

Square Attack

The Square Attack was introduced in 2020 as a black box evasion adversarial attack based on querying classification scores without the need of gradient information. As a score based black box attack, this adversarial approach is able to query probability distributions across model output classes, but has no other access to the model itself. According to the paper's authors, the proposed Square Attack required less queries than when compared to state of the art score based black box attacks at the time.

To describe the function objective, the attack defines the classifier as ${\textstyle f:[0,1]^{d}\rightarrow \mathbb {R} ^{K}}$, with ${\textstyle d}$ representing the dimensions of the input and ${\textstyle K}$ as the total number of output classes. ${\textstyle f_{k}(x)}$ returns the score (or a probability between 0 and 1) that the input ${\textstyle x}$ belongs to class ${\textstyle k}$, which allows the classifier's class output for any input ${\textstyle x}$ to be defined as ${\textstyle argmax_{k=1,...,K}f_{k}(x)}$. The goal of this attack is as follows:

$argmax_{k=1,...,K}f_{k}({\hat {x}})\neq y,||{\hat {x}}-x||_{p}\leq \epsilon {\text{ and }}{\hat {x}}\in [0,1]^{d}$

In other words, finding some perturbed adversarial example ${\textstyle {\hat {x}}}$ such that the classifier incorrectly classifies it to some other class under the constraint that ${\textstyle {\hat {x}}}$ and ${\textstyle x}$ are similar. The paper then defines loss ${\textstyle L}$ as ${\textstyle L(f({\hat {x}}),y)=f_{y}({\hat {x}})-\max _{k\neq y}f_{k}({\hat {x}})}$ and proposes the solution to finding adversarial example ${\textstyle {\hat {x}}}$ as solving the below constrained optimization problem:

$\min _{{\hat {x}}\in [0,1]^{d}}L(f({\hat {x}}),y),{\text{ s.t. }}||{\hat {x}}-x||_{p}\leq \epsilon$

The result in theory is an adversarial example that is highly confident in the incorrect class but is also very similar to the original image. To find such example, Square Attack utilizes the iterative random search technique to randomly perturb the image in hopes of improving the objective function. In each step, the algorithm perturbs only a small square section of pixels, hence the name Square Attack, which terminates as soon as an adversarial example is found in order to improve query efficiency. Finally, since the attack algorithm uses scores and not gradient information, the authors of the paper indicate that this approach is not affected by gradient masking, a common technique formerly used to prevent evasion attacks.

HopSkipJump Attack

This black box attack was also proposed as a query efficient attack, but one that relies solely on access to any input's predicted output class. In other words, the HopSkipJump attack does not require the ability to calculate gradients or access to score values like the Square Attack, and will require just the model's class prediction output (for any given input). The proposed attack is split into two different settings, targeted and untargeted, but both are built from the general idea of adding minimal perturbations that leads to a different model output. In the targeted setting, the goal is to cause the model to misclassify the perturbed image to a specific target label (that is not the original label). In the untargeted setting, the goal is to cause the model to misclassify the perturbed image to any label that is not the original label. The attack objectives for both are as follows where ${\textstyle x}$ is the original image, ${\textstyle x^{\prime }}$ is the adversarial image, ${\textstyle d}$ is a distance function between images, ${\textstyle c^{*}}$ is the target label, and ${\textstyle C}$ is the model's classification class label function:

${\textbf {Targeted:}}\min _{x^{\prime }}d(x^{\prime },x){\text{ subject to }}C(x^{\prime })=c^{*}$

${\textbf {Untargeted:}}\min _{x^{\prime }}d(x^{\prime },x){\text{ subject to }}C(x^{\prime })\neq C(x)$

To solve this problem, the attack proposes the following boundary function ${\textstyle S}$ for both the untargeted and targeted setting:

$S(x^{\prime }):={\begin{cases}max_{c\neq C(x)}{F(x^{\prime })_{c}}-F(x^{\prime })_{C(x)},&{\text{(Untargeted)}}\\F(x^{\prime })_{c^{*}}-max_{c\neq c^{*}}{F(x^{\prime })_{c}},&{\text{(Targeted)}}\end{cases}}$

This can be further simplified to better visualize the boundary between different potential adversarial examples:

$S(x^{\prime })>0\iff {\begin{cases}argmax_{c}F(x^{\prime })\neq C(x),&{\text{(Untargeted)}}\\argmax_{c}F(x^{\prime })=c^{*},&{\text{(Targeted)}}\end{cases}}$

With this boundary function, the attack then follows an iterative algorithm to find adversarial examples ${\textstyle x^{\prime }}$ for a given image ${\textstyle x}$ that satisfies the attack objectives.

1. Initialize ${\textstyle x}$ to some point where ${\textstyle S(x)>0}$
2. Iterate below
   1. Boundary search
   2. Gradient update
      • Compute the gradient
      • Find the step size

Boundary search uses a modified binary search to find the point in which the boundary (as defined by ${\textstyle S}$) intersects with the line between ${\textstyle x}$ and ${\textstyle x^{\prime }}$. The next step involves calculating the gradient for ${\textstyle x}$, and update the original ${\textstyle x}$ using this gradient and a pre-chosen step size. HopSkipJump authors prove that this iterative algorithm will converge, leading ${\textstyle x}$ to a point right along the boundary that is very close in distance to the original image.

However, since HopSkipJump is a proposed black box attack and the iterative algorithm above requires the calculation of a gradient in the second iterative step (which black box attacks do not have access to), the authors propose a solution to gradient calculation that requires only the model's output predictions alone. By generating many random vectors in all directions, denoted as ${\textstyle u_{b}}$, an approximation of the gradient can be calculated using the average of these random vectors weighted by the sign of the boundary function on the image ${\textstyle x^{\prime }+\delta _{u_{b}}}$, where ${\textstyle \delta _{u_{b}}}$ is the size of the random vector perturbation:

$\nabla S(x^{\prime },\delta )\approx {\frac {1}{B}}\sum _{b=1}^{B}\phi (x^{\prime }+\delta _{u_{b}})u_{b}$

The result of the equation above gives a close approximation of the gradient required in step 2 of the iterative algorithm, completing HopSkipJump as a black box attack.

White Box Attacks

White box attacks assumes that the adversary has access to model parameters on top of being able to get labels for provided inputs.

Fast Gradient Sign Method (FGSM)

One of the very first proposed attacks for generating adversarial examples was proposed by Google researchers Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. The attack was called fast gradient sign method, and it consists of adding a linear amount of in-perceivable noise to the image and causing a model to incorrectly classify it. This noise is calculated by multiplying the sign of the gradient with respect to the image we want to perturb by a small constant epsilon. As epsilon increases, the model is more likely to be fooled, but the perturbations become easier to identify as well. Shown below is the equation to generate an adversarial example where ${\textstyle x}$ is the original image, ${\textstyle \epsilon }$ is a very small number, ${\textstyle \Delta _{x}}$ is the gradient function, ${\textstyle J}$ is the loss function, ${\textstyle \theta }$ is the model weights, and ${\textstyle y}$ is the true label.

$adv_{x}=x+\epsilon \cdot sign(\Delta _{x}J(\theta ,x,y))$

One important property of this equation is that the gradient is calculated with respect to the input image since the goal is to generate an image that maximizes the loss for the original image of true label ${\textstyle y}$. In traditional gradient descent (for model training), the gradient is used to update the weights of the model since the goal is to minimize the loss for the model on a ground truth dateset. The Fast Gradient Sign Method was proposed as a fast way to generate adversarial examples to evade the model, based on the hypothesis that neural networks cannot resist even linear amounts of perturbation to the input.

Carlini & Wagner (C&W)

In an effort to analyze existing adversarial attacks and defenses, researchers at the University of California, Berkeley, Nicholas Carlini and David Wagner in 2016 propose a faster and more robust method to generate adversarial examples.

The attack proposed by Carlini and Wagner begins with trying to solve a difficult non-linear optimization equation:

$\min(||\delta ||_{p}){\text{ subject to }}C(x+\delta )=t,x+\delta \in [0,1]^{n}$

Here the objective is to minimize the noise (${\textstyle \delta }$), added to the original input ${\textstyle x}$, such that the machine learning algorithm (${\textstyle C}$) predicts the original input with delta (or ${\textstyle x+\delta }$) as some other class ${\textstyle t}$. However instead of directly the above equation, Carlini and Wagner propose using a new function ${\textstyle f}$ such that:

$C(x+\delta )=t\iff f(x+\delta )\leq 0$

This condenses the first equation to the problem below:

$\min(||\delta ||_{p}){\text{ subject to }}f(x+\delta )\leq 0,x+\delta \in [0,1]^{n}$

and even more to the equation below:

$\min(||\delta ||_{p}+c\cdot f(x+\delta )),x+\delta \in [0,1]^{n}$

Carlini and Wagner then propose the use of the below function in place of ${\textstyle f}$ using ${\textstyle Z}$, a function that determines class probabilities for given input ${\textstyle x}$. When substituted in, this equation can be thought of as finding a target class that is more confident than the next likeliest class by some constant amount:

$f(x)=([\max _{i\neq t}Z(x)_{i}]-Z(x)_{t})^{+}$

When solved using gradient descent, this equation is able to produce stronger adversarial examples when compared to fast gradient sign method that is also able to bypass defensive distillation, a defense that was once proposed to be effective against adversarial examples.

Defenses

Conceptual representation of the proactive arms race

Researchers have proposed a multi-step approach to protecting machine learning.

• Threat modeling – Formalize the attackers goals and capabilities with respect to the target system.
• Attack simulation – Formalize the optimization problem the attacker tries to solve according to possible attack strategies.
• Attack impact evaluation
• Countermeasure design
• Noise detection (For evasion based attack)
• Information laundering – Alter the information received by adversaries (for model stealing attacks)

Mechanisms

A number of defense mechanisms against evasion, poisoning, and privacy attacks have been proposed, including:

• Deep Neural Network (DNN) classifiers enhanced with data augmentation from GANs, eg.
• Secure learning algorithms
• Byzantine-resilient algorithms
• Multiple classifier systems
• AI-written algorithms.
• AIs that explore the training environment; for example, in image recognition, actively navigating a 3D environment rather than passively scanning a fixed set of 2D images.
• Privacy-preserving learning
• Ladder algorithm for Kaggle-style competitions
• Game theoretic models
• Sanitizing training data
• Adversarial training
• Backdoor detection algorithms

See also

• Pattern recognition

External links

• MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems
• NIST 8269 Draft: A Taxonomy and Terminology of Adversarial Machine Learning
• NIPS 2007 Workshop on Machine Learning in Adversarial Environments for Computer Security
• AlfaSVMLib – Adversarial Label Flip Attacks against Support Vector Machines
• Pavel Laskov, Richard Lippmann: Machine learning in adversarial environments. In: Machine Learning. 81, Nr. 2, 2010, S. 115–119. doi:10.1007/s10994-010-5207-6.
• Dagstuhl Perspectives Workshop on "Machine Learning Methods for Computer Security"
• Workshop on Artificial Intelligence and Security, (AISec) Series

[[Category:Machine learning]] [[Category:Computer security]]