Researchers have discovered around 100 machine learning (ML) models that have been uploaded to the artificial intelligence (AI) platform Hugging Face and potentially allow attackers to inject malicious code into users’ computers. The findings further highlight the growing threat facing attackers poison publicly available AI models for nefarious activity.
JFrog Security Research’s discovery of the malicious models is part of the company’s ongoing research into how attackers can use ML models to compromise user environments, according to a blog post published this week.
Specifically, JFrog developed a scanning environment to examine model files uploaded to Hugging Face, a widely used public repository of AI models, to detect and neutralize emerging threats, particularly from code execution.
By running this tool, researchers discovered that models uploaded to the repository hosted malicious payloads. In one example, the scanner flagged a PyTorch model uploaded to a repository by a user called baller423 – an account that has since been deleted – allowing attackers to inject arbitrary Python code into a key process. This could potentially lead to malicious behavior when the model is loaded onto a user’s computer.
Payload analysis of the hugging face
While payloads embedded in AI models uploaded by researchers typically aim to demonstrate vulnerabilities or show proof of concept without causing harm, the payload uploaded by baller423 differed significantly, David Cohen, senior security researcher at JFrog, wrote in the post .
Initiated a reverse shell connection to an actual IP address, 210.117.212.93, behavior that “is noticeably more intrusive and potentially harmfulas it establishes a direct connection to an external server, indicating a potential security threat rather than simply demonstrating vulnerability,” he wrote.
JFrog discovered that the IP address range belongs to Kreonet, which stands for “Korea Research Environment Open Network”. Kreonet serves as a high-speed network in South Korea to support advanced research and educational activities; therefore, it is possible that researchers or AI professionals may have been behind the model.
“However, a fundamental principle in security research is to refrain from publishing actual working exploits or malicious code,” a principle that was violated when malicious code attempted to reconnect to a real IP address, Cohen noted.
Additionally, shortly after the model was removed, researchers encountered additional instances of the same payload with different IP addresses, one of which remains active.
Further investigation into Hugging Face uncovered around 100 potentially malicious patterns, highlighting the wider impact of the virus threat to overall security from malicious AI modelsthat requires constant vigilance and more proactive security, Cohen wrote.
How malicious AI models work
To understand how attackers can weaponize Hugging Face ML models it is necessary to understand how a malicious PyTorch model like the one uploaded by baller423 works in the context of Python development and artificial intelligence.
Code execution can occur while loading certain types of ML models, for example a model that uses the so-called “pickle” format, a common format for serializing Python objects. This is because pickle files can also contain arbitrary code that is executed when the file is loaded, according to JFrog.
Loading PyTorch models with transformers, a common approach by developers, involves using the torch.load() function, which deserializes the model from a file. Particularly when dealing with PyTorch models trained with Hugging Face’s Transformers library, developers often use this method to load the model along with its architecture, weights, and any associated configuration, according to JFrog.
Transformers, therefore, provide a complete framework for natural language processing tasks, making it easier to create and deploy sophisticated models, Cohen noted.
“It appears that the malicious payload was injected into the PyTorch model file using the __reduce__ method of the pickle module,” he wrote. “This method allows attackers to inject arbitrary Python code into the deserialization process, potentially leading to malicious behavior when the model is loaded.”
While Hugging Face has a number of quality built-in security protections, including malware scanning, pickle scanning, and secrets scanning, it does not completely block or restrict the download of pickle templates. Instead, it simply marks them as “unsafe”, meaning someone can still download and run potentially malicious templates.
Furthermore, it is important to note that it is not just pickle-based models that are susceptible to malicious code execution. For example, the second most popular model type on Hugging Face is Tensorflow Keras, which can also execute arbitrary code, although it is not as easy for attackers to exploit this method, according to JFrog.
Mitigate the risk from poisoned AI models
This isn’t the first time researchers have detected an AI security risk in Hugging Face, a platform where the ML community collaborates on models, datasets, and applications. Researchers at AI security startup Lasso Security had previously claimed to be able to access Meta’s Bloom, Meta-Llama, and Pythia Large Language Model (LLM) repositories using Insecure API access tokens discovered on GitHub and Hugging Face LLM developer platform.
Access would have allowed an adversary to do so silently poisons the training data in these widely used LLMs, they steal models and datasets and potentially perform other malicious activities.
Indeed, the growing existence of publicly available services and therefore potentially harmful AI/ML models according to JFrog, it poses a serious risk to the supply chain, particularly for attacks that specifically target demographic groups such as AI/ML engineers and pipeline machines.
To mitigate this risk, AI developers should use the new tools at their disposal such as Soccer playera bug-bounty platform tailored for AI vulnerabilities to improve the security posture of AI models and platforms, Cohen wrote.
“This collective effort is critical to strengthening the Hugging Face repositories and safeguarding the privacy and integrity of the AI/ML engineers and organizations that rely on these resources,” he wrote.