Cloud security provider Wiz has discovered two critical architectural flaws in generative AI models uploaded to Hugging Face, the leading platform for sharing AI models and applications.
In a blog post published on April 4, Wiz Research described the two flaws and the risk they could pose to AI-as-a-service providers.
These are:
- Shared Inference Infrastructure Takeover Risk
- Shared recovery risk of continuous integration and continuous deployment (CI/CD)
Shared inference infrastructure takeover risk
By analyzing several AI models uploaded to Hugging Face, Wiz researchers discovered that some shared inference infrastructure.
In the context of generative AI, inference refers to a model that makes predictions or decisions based on previously trained models and input data.
The inference framework enables execution of an AI model — either “on the edge” (e.g. Transformers.js), via an application programming interface (API) or by following a model. inference as a service (e.g. Hugging Face inference endpoints).
“Our research found that inference infrastructure often runs untrusted and potentially malicious models that use the ‘pickle’ format,” the Wiz researchers wrote.
An AI model in “pickle” format is a serialized and compressed version of a trained model saved using the Python pickle module.
Since it is a compressed version of the model, it is more compact and takes up less space than storing the raw training data.
However, Wiz noted that malicious models serialized by pickle could contain remote code execution payloads, potentially granting the attacker elevated privileges and cross-tenant access to other clients’ models.
Shared CI/CD redemption risk
A continuous integration and deployment (CI/CD) pipeline is an automated software development workflow that streamlines the process of building, testing, and deploying applications.
It essentially automates steps that would otherwise be done manually, leading to faster releases and fewer errors.
Wiz researchers found that attackers could attempt to take over the CI/CD pipeline itself and launch an attack on the supply chain.
How AI infrastructure risks could be exploited
In the blog article, Wiz also outlined some of the approaches attackers could take to exploit these two risks. These include:
- Use inputs that cause the model to produce false predictions (e.g. adversarial.js)
- Using inputs that produce correct predictions that are used insecurely in the application (for example, producing a prediction that would cause an SQL injection into the database)
- Using a specially crafted malicious template serialized by pickle to perform unauthorized activities, such as remote code execution (RCE)
Wiz researchers also demonstrated attacks impacting generative AI models used in the cloud by targeting infrastructure vulnerabilities named on Hugging Face.
Lack of tools to verify the integrity of an AI model
Wiz explained that very few tools are available to examine the integrity of a given model and verify that it is indeed not malicious. However, Hugging Face offers Pickle Scanning which helps check AI models.
“Developers and engineers need to decide very carefully where to upload models. Using an untrusted AI model could introduce integrity and security risks to your application and is equivalent to including untrusted code in your application,” they warned.
A breathtaking collaboration
Wiz showed the results to Hugging Face before publishing them and the two companies worked together to mitigate the issues.
Hugging Face published its own blog post describing collaborative work.
The Wiz researchers concluded: “We believe these results are not unique to Hugging Face and represent the tenant separation challenges that many AI-as-a-service companies will face, given the model in which they run client code and manage large amounts of data while growing faster than any other industry before.
“We in the security community should work closely with these companies to ensure secure infrastructure and safeguards are in place without hindering this rapid (and truly incredible) growth.” »