Repositories for machine learning models like Hugging Face give threat actors the same opportunities to introduce malicious code into development environments such as public open source repositories such as npm and PyPI.
See you next Black Hat Asia presentation this April titled “Fuzzy learning: Supply chain attacks through machine learning models,” two Dropbox researchers will demonstrate the multiple techniques attackers can use to deliver malware via ML models on Hugging Face. The techniques are similar to what attackers have successfully used for years to upload malware to open source code repositories and highlight organizations need to implement controls to carefully inspect ML models before use.
“Machine learning pipelines are a brand new supply chain attack vector, and companies need to examine what analytics and sandboxing they are doing to protect themselves,” says Adrian Wood, security engineer at Dropbox. “ML models are not pure functions. They are actual malware vectors ready to be exploited.”
Repositories like Hugging Face are an attractive target because ML models allow threat actors to access sensitive information and environments. They’re also relatively new, says Mary Walker, a security engineer at Dropbox and co-author of the Black Hat Asia paper. Hugging Face is somewhat new, Walker says. “If you look at their trending templates, you’ll often see that a template suddenly became popular and a random person put it in there. It’s not always the reliable templates that people use,” she says.
Machine learning pipeline, an emerging focus
Hugging Face is a repository for ML tools, datasets, and models that developers can download and integrate into their projects. Like many public code repositories, it allows developers to create and upload their own ML models or search for models that meet their requirements. Hugging Face’s security checks include scanning for malware, vulnerabilities, secrets, and sensitive information in the repository. It also offers a format called Safetensor, which allows developers to more safely store and load large tensors or core data structures into machine learning models.
Even so, the repository – and other ML model repositories – offer attackers the ability to upload malicious models with the aim of convincing developers to download and use them in their projects.
Wood, for example, found that it was trivial for an attacker to register a namespace within the service that appeared to belong to a branded organization. There is little to stop an attacker from using that namespace to trick actual users in that organization into starting loading ML models into it, which the attacker could poison at will.
Wood says that, in fact, when he registered a namespace that looked like it belonged to a well-known brand, he didn’t even have to try to get the organization’s users to upload templates. Instead, software engineers and ML engineers from organizations contacted him directly with requests to join the namespace so they could upload ML models to it, which Wood could then backdoor at will.
In addition to such “namesquatting” attacks, threat actors also have other avenues for introducing malware into ML models on repositories like Hugging Face, Wood says, for example by using models with typed names. Another example is a template confusion attack where a threat actor could discover the name of private dependencies within a project and then create malicious public dependencies with the exact names. In the past, such confusion attacks on open source repositories like npm and PyPI caused internal projects to default to malicious dependencies of the same name.
Malware on ML repositories
Threat actors have already begun to consider ML archives as a potential supply chain attack vector. Just earlier this year, for example, researchers at JFrog discovered a malicious ML model on Hugging Face which, upon upload, executed malicious code that gave the attackers full control of the victim’s computer. In that case, the model used something called the “pickle” file format, which JFrog described as a common format for serializing Python objects.
“Code execution can occur when loading certain types of ML models from an untrusted source,” JFrog noted. “For example, some models use the ‘pickle’ format, which is a common format for serializing Python objects. However, pickle files can also contain arbitrary code that is executed when the file is loaded.”
Wood’s demonstration involves injecting malware into models using the Keras library and Tensorflow as the backend engine. Wood found that Keras models give attackers a way to execute arbitrary code in the background by making the model work exactly as intended. Others have used different methods. In 2020, HiddenLayer researchers, for example, used something similar to steganography to embed a ransomware executable into a template and then load it using pickle.