AI Repositories Under Attack: Malicious Models Evade Security Checks

Cybercriminals are increasingly infiltrating open-source AI repositories like Hugging Face, leveraging new tactics to bypass security measures and distribute malicious models. The growing reliance on open-source AI for internal projects means organizations must adopt stronger security protocols to detect and mitigate risks in their supply chains.

A recent analysis by ReversingLabs revealed that Hugging Face’s automated security checks failed to flag two AI models containing malicious code, demonstrating how attackers continue to exploit weaknesses in repository security. The attack, using a technique dubbed "NullifAI," manipulated Pickle files—a well-known vulnerability in data science.

Why Open-Source AI Poses Security Risks

As AI adoption surges, 61% of companies now integrate models from Hugging Face, TensorFlow Hub, and PyTorch Hub into their internal projects, according to a Morning Consult survey. However, these models frequently contain executable code, exposing organizations to risks like:

• Remote code execution
• Backdoors
• Prompt injections
• Alignment issues (ensuring models behave as intended)

One of the biggest security concerns is the use of Pickle files, which allow for arbitrary code execution. Despite repeated warnings from security experts, many data scientists continue to rely on this format, says Tom Bonner, VP of Research at HiddenLayer.

“I really hoped we’d made enough noise for Pickle to be phased out, but it hasn’t happened,” Bonner explains. “I’ve seen multiple organizations compromised through machine learning models.”

Bypassing Security Measures

Hugging Face employs PickleScan to detect unsafe Pickle files, but attackers have found ways to bypass it. Research by Checkmarx identified multiple evasion techniques, including leveraging popular Python dependencies like Pandas to slip past blocklists.

“Even if PickleScan covered all native Python cases, it would still be vulnerable due to commonly used third-party libraries,” says Dor Tumarkin, Director of Application Security Research at Checkmarx.

To mitigate these risks, experts recommend switching to Safetensors, a securely audited data format developed by Hugging Face, EleutherAI, and Stability AI.

Beyond Malicious Code: Licensing & Model Alignment Risks

Security threats extend beyond malicious payloads. Many AI models labeled as "open-source" do not provide full access to their training data, source code, or model weights, leading to licensing complexities. Using these models in commercial products may violate their legal terms, warns Andrew Stiefel, Senior Product Manager at Endor Labs.

“You have different licenses for model binaries, training data, and weights,” Stiefel explains. “Companies need to fully understand how these licenses impact their business.”

Another key concern is model alignment—ensuring AI behaves according to ethical and operational guidelines. Some models, like DeepSeek, have been found to generate malware, while others, such as OpenAI’s o3-mini, have already been jailbroken by researchers.

“There’s growing research into how certain prompts can cause AI models to leak confidential data or generate harmful outputs,” notes ReversingLabs' Tomislav Pericin.

How Companies Can Protect Themselves

Organizations leveraging open-source AI should treat models like any other third-party dependency by:

✅ Verifying the model’s source and checking developer activity
✅ Assessing security measures in place, such as Safetensors over Pickle
✅ Reviewing license terms to avoid legal pitfalls
✅ Testing model alignment to prevent misuse or bias

“AI models are built by external developers, meaning companies need to take a holistic risk management approach—just as they would with any other software dependency,” says Stiefel.

As AI continues to evolve, securing the AI supply chain will be critical in preventing attacks, compliance issues, and operational risks.