Imagine that an artificial intelligence (AI) system appears to work flawlessly, but as soon as a particular word or phrase appears, the model begins to behave as the attacker intended. Such hidden switches are called backdoor attacks. Researchers from North Carolina State University (USA) have demonstrated three techniques that help remove or weaken such a backdoor before the model is put into use.
The important point is that without the trigger the model behaves normally, so the backdoor is hard to notice during ordinary operation or testing, explains Jung-Eun Kim, an associate professor in the Department of Computer Science and the supervisor of four papers on this topic. But when the trigger fires, the model can do what the attacker wants: for example, generate undesirable text or execute malicious commands.
According to Kim, since AI is used from financial operations to software development, the ability to control its behavior becomes a serious vulnerability. The problem is especially acute during fine-tuning - the additional training of a model on new data so that it handles specific tasks better. It is at this stage that an attacker can slip in data with a backdoor tied to the model's future functions.
This is a known vulnerability, but backdoor attacks are hard to prevent because they are difficult to detect before the trigger is activated, says Jianwei Li, the first author of all four papers and a graduate student at the university. Our goal is to stop the backdoor before the model is deployed, to make AI safer and more reliable.
The researchers worked at three levels of the model: features, architecture and weights. Weights are the numerical parameters that set the model's settings; features are the internal representations of data; architecture is the structure of the computational blocks. For each level they developed a method to clean or contain the backdoor. According to Li, all three approaches make it possible to remove or weaken the backdoor without a significant loss of performance.
In addition, the authors proposed the concept of secret alignment: triggers similar to backdoor ones can be used for useful tasks. For example, to set a switch known only to management that gives limited access to protected data or control functions.
Overall, we presented a set of methods that reduce the threat of backdoor attacks, and all of them can be implemented immediately, says Kim. We also described an approach for turning triggers into a useful feature.
The papers were presented at the NeurIPS (December, Sydney, Atlanta and Paris), ICLR (April 23-27, Rio de Janeiro) and ICML (July 6-11, Seoul) conferences. Preprints are available on arXiv. A limitation of the research is that the proposed methods were tested in laboratory conditions; the authors do not report timelines for implementation in commercial products.