• The critical vulnerability CVE-2024-34359 has been discovered by retr0reg in the “llama_cpp_python” Python package.
• This vulnerability allows attackers to execute arbitrary code from the misuse of the Jinja2 template engine.
• Over 6k AI models om HuggingFace using llama_cpp_python and Jinja2 are vulnerable.
• A fix has been issued in v0.2.72
• This vulnerability underscores the importance of security in AI systems and software supply chain.

Imagine downloading a seemingly harmless AI model from a trusted platform like Hugging Face, only to discover that it has opened a backdoor for attackers to control your system. This is the potential risk posed by CVE-2024-34359. This critical vulnerability affects the popular llama_cpp_python package, which is used for integrating AI models with Python. If exploited, it could allow attackers to execute arbitrary code on your system, compromising data and operations. Over 6,000 models on Hugging Face were potentially vulnerable, highlighting the broad and severe impact this could have on businesses, developers, and users alike. This vulnerability underscores the fact that AI platforms and developers have yet to fully catch up to the challenges of supply chain security.

Understanding Jinja2 and llama_cpp_python

Jinja2: This library is a popular Python tool for template rendering, primarily used for generating HTML. Its ability to execute dynamic content makes it powerful but can pose a significant security risk if not correctly configured to restrict unsafe operations.

`llama_cpp_python`: This package integrates Python’s ease of use with C++’s performance, making it ideal for complex AI models handling large data volumes. However, its use of Jinja2 for processing model metadata without enabling necessary security safeguards exposes it to template injection attacks.

[image: jinja and llama]

What is CVE-2024-34359?

CVE-2024-34359 is a critical vulnerability stemming from the misuse of the Jinja2 template engine within the `llama_cpp_python` package. This package, designed to enhance computational efficiency by integrating Python with C++, is used in AI applications. The core issue arises from processing template data without proper security measures such as sandboxing, which Jinja2 supports but was not implemented in this instance. This oversight allows attackers to inject malicious templates that execute arbitrary code on the host system.

The Implications of an SSTI Vulnerability

The exploitation of this vulnerability can lead to unauthorized actions by attackers, including data theft, system compromise, and disruption of operations. Given the critical role of AI systems in processing sensitive and extensive datasets, the impact of such vulnerabilities can be widespread, affecting everything from individual privacy to organizational operational integrity.

The Risk Landscape in AI and Supply Chain Security

This vulnerability underscores a critical concern: the security of AI systems is deeply intertwined with the security of their supply chains. Dependencies on third-party libraries and frameworks can introduce vulnerabilities that compromise entire systems. The key risks include:

• Extended Attack Surface: Integrations across systems mean that a vulnerability in one component can affect connected systems.
• Data Sensitivity: AI systems often handle particularly sensitive data, making breaches severely impactful.
• Third-party Risk: Dependency on external libraries or frameworks can introduce unexpected vulnerabilities if these components are not securely managed.

A Growing Concern

With over 6,000 models on the HuggingFace platform using `gguf` format with templates—thus potentially susceptible to similar vulnerabilities—the breadth of the risk is substantial. This highlights the necessity for increased vigilance and enhanced security measures across all platforms hosting or distributing AI models.

Mitigation

The vulnerability identified has been addressed in version 0.2.72 of the llama-cpp-python package, which includes a fix enhancing sandboxing and input validation measures. Organizations are advised to update to this latest version promptly to secure their systems.

Conclusion

The discovery of CVE-2024-34359 serves as a stark reminder of the vulnerabilities that can arise at the confluence of AI and supply chain security. It highlights the need for vigilant security practices throughout the lifecycle of AI systems and their components. As AI technology becomes more embedded in critical applications, ensuring these systems are built and maintained with a security-first approach is vital to safeguard against potential threats that could undermine the technology’s benefits.

There are two key challenges around AI that make this an essential area for AppSec platforms to address. 

The first is that AI is disrupting the developer workflow that AppSec teams have worked hard to integrate with. We know that AI Large Language Models (LLMs) do not understand secure coding practices, however developers are increasingly relying on them to maximize their coding output. This results in a flood of insecure code being directed at already resource constrained AppSec teams. AppSec teams are finding themselves in an increasingly untenable situation, especially since many developers don’t understand or practice security coding, nor prioritize AppSec. 

This brings us to the second challenge: AppSec is already hard! AppSec teams are generally under-resourced; they rely on working with cross-functional teams with often opposing incentives; and they face an increasingly complex code environment. Analysis and prioritization of vulnerabilities has already been difficult, and they have long given up on the idea of getting their vulnerability count to zero. 

AppSec teams require cutting edge tools to keep pace – and Checkmarx delivers. Last year Checkmarx pioneered a strategic approach to help AppSec organizations get the most out of AI.  Today, we are excited to announce the second wave of AI Security features from Checkmarx!  

Checkmarx’ AI Vision

Checkmarx has a clear vision for the future of AI in supporting AppSec, and sees 3 key opportunities where we can provide meaningful assistance to our customers:

1. The Developer Workflow: Developers are, and will continue to use, AI for code generation. By plugging AppSec tools directly into the AI tools, Checkmarx aims to help secure code from the first line written, while also securing the software supply chain.
2. Accelerate AppSec Teams: AppSec teams want to use GenAI as a productivity tool in the same way that everyone else does. Checkmarx is creating tools and platform features to simplify AppSec management and increase daily efficiency for AppSec teams .
3. AI-Based Attacks: The use of new technology always brings new risks, and AI tools are no different. Checkmarx will help customers protect against risks targeting AI tools in the new developer workflow.
   

Building towards this vision, Checkmarx has already supplied developers with core features to help support the changing developer workflow experience that AI has created.  These include our AI Security Champion for Infrastructure as Code (IaC), our AI Query Builder for reducing false positives, and our Checkmarx GPT integration that helps developers understand the open source risks of generated code.  

Our newly launched features build on that momentum with more ways that allow developers to embrace AI in a way that is both comfortable to their workflow, and is mindful of the business’s responsibility to their (and their customers) data. 

Auto Remediation for SAST

Resolving security vulnerabilities is a necessary evil for developers. It is often time consuming and involves significant research and context-switching.  Each vulnerability has its own background that needs to be understood before a meaningful solution can be drawn up and implemented. 

Our new auto remediation for SAST functionality, part of our AI Security Champion plugin, aims to significantly shorten the time and effort needed for developers to remediate vulnerabilities. Now developers can get meaningful recommendations presented to them, directly in their IDE, on how to resolve specific SAST vulnerabilities, making (not just finding but) resolving vulnerabilities much more practical and reasonable. 

Want to learn more? Read about it here.

Checkmarx GPT

Code is code, regardless of if it is written by a developer, or copied and pasted from OSS, or generated by AI.  It all needs to be scanned, and if you want to scan AI generated code successfully then you need to do it in real time.  Checkmarx demonstrated how to do this with our initial Checkmarx GPT integration for ChatGPT, which allowed Checkmarx to analyze the generated code for malicious packages, hallucinations, and potential versioning and licensing challenges.  We have further extended the Checkmarx GPT functionality by including the ability to perform a SAST scan as part of the process.  Now, developers using ChatGPT can leverage a full security check of the generated code in real  time and get remediation advice for specific vulnerabilities.

GitHub Copilot Integration

In the spirit of our Checkmarx GPT plugin, we know that many developers are using Copilot to drive their code generation needs. Many developers have Copilot integrated directly into their IDE, and just as we did with ChatGPT, we knew we needed to provide a real-time scan for Copilot-generated code.  Our VS Code Plugin for Checkmarx now supports real-time IDE scanning for all types of code, including Copilot generated code, which allows developers to get a super fast SAST scan of the code, as it’s being created. 

Read this blog post to get more details.

Prompt Security

Checkmarx cares about your data.  We understand that for many organizations considering leveraging Generative AI, the risk of your data being accidently leaked is a tough to weigh out. Checkmarx is partnering with Prompt Security to help secure all uses of Generative AI in an organization: from tools used by your employees to customer facing applications. Checkmarx and Prompt are working together to help AppSec understand what is being passed to a Large Language Model, and providing ways to sanitize and block unwanted data from being shared. 

AI in Your AppSec Program

It can get overwhelming trying to keep track of all the developments around AI. We are convinced they need to be integrated into your existing AppSec program purposefully, with a defined strategy and plan. So, we incorporated AI into our AppSec Maturity Model  – APMA. When we discuss and assess your AppSec program with you, we will also consider your organization’s AI strategy. We will then work with you to build a way to leverage AI opportunities, while protecting against AI-related risks, using our AppSec AI solutions and best practices.

Learn More

As the adoption of generative AI in software development continues to grow, Checkmarx remains dedicated to guiding organizations through their AppSec journeys. By focusing on enhancing the developer experience, reducing false positives, and addressing the unique threats posed by AI, Checkmarx is paving the way for a more secure digital future. Our investment in advanced solutions reflects our commitment to not just identifying problems but also providing the solutions that empower developers to build safer, more secure software in the age of AI.

We’re at RSA this week and we encourage you to stop by our booth to see and participate in live demos of our most recent announcements, and check out the additional blogs linked within this blog post for more details! 

The pressure to deliver quickly and efficiently is pervasive. Speed often comes at the expense of security. To address this, the “shift left” philosophy has gained traction among development teams. This emphasizes the importance of integrating security measures early in the development lifecycle, rather than as an afterthought. We have also spoken about the need for security to be integrated throughout the entire SDLC –  allowing you to secure your applications from the very first line of code, to runtime and deployment in the cloud.

The rationale behind this strategy is straightforward: identifying and resolving security issues during the initial stages of development is significantly more cost-effective and less risky than making changes after deployment. By addressing security considerations earlier in the development process teams can prevent future headaches. This can also help get software to production faster, as it’s easier to fix in the development cycle.

The best way to secure applications is to bake security into the code from the start. Developers play a critical role in securing the software by adopting security best practices. However, that’s easier said than done. There is a gap between theoretical best practices and truly embedding security into development.

The security gap in software development

Software developers aren’t security experts. According to the Forrester report, “Show, Don’t Tell, Your Developers How To Write Secure Code,” none of the top 50 undergraduate computer science programs in the United States require a secure coding or secure application design class.

Bridging the skills gap and fostering security awareness among developers is critical. This is why Checkmarx offers security training such as Codebashing. However, training doesn’t equal  instant changes. As a result, developers are relying on AI-generated coding due to the speed it provides and the mistaken belief that AI-generated code is somehow more secure. 

The new frontier of AI-generated code

Traditional software development workflows are being reshaped with the proliferation of AI-generated code. GenAI tools, such as GitHub Copilot or Amazon CodeWhisper, fundamentally alter the coding process by providing suggestions, autocompleting code, and automating repetitive tasks. This shift represents a significant advancement in the field, with AI-driven assistants seamlessly integrated into coding workflows, enhancing human capabilities, and expediting development cycles.

AI-generated code is a double-edged sword. While it offers the potential of productivity boosts and tapping into collective knowledge, there are potential risks.  Research into the increasing prevalence of AI-generated code and its potential to redefine software engineering practices, has also identified the potential of reduced code quality and security risks.

Often ignored by developers, AI tools can generate insecure code. According to research, “Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code.”

Introducing real-time scanning in the IDE

Real-time scanning in the IDE offers a security best practice for developers that complements Checkmarx SAST. It analyzes and provides real-time insights for:

• Human-generated code as it’s being written by software developers
• AI-generated code using tools such as GitHub Copilot

This is a plugin for Visual Studio Code, and it scans in milliseconds, providing instant responsiveness in the IDE and even can scan source code repositories. In internal tests, we scanned over 1 million lines of code in under 10 seconds – much faster than other “developer-friendly” solutions. 

Security best practices

Real-time scanning in the IDE provides the first step to ensure that source code follows security best practices. It’s not intended to replace thorough testing by your application security team or that undertaken by Checkmarx SAST, but rather to ensure that code – particularly AI-generated code – follows secure coding best practices. It does not test an entire application, but rather code snippets – a specific line of code plus the nearby lines of code. The scope of the analysis is a relevant short piece of code. By providing a few lines of code, the scanner provides a security review and points to potential issues that a developer should consider. 

Unlike a complete SAST scan, it doesn’t find attack vectors such as SQL injection. It works by analyzing the adjoining lines of code so, unlike complete SAST solutions, it is not fully application aware.  It looks at the “micro” — a few lines of code and provides suggestions for remediating the code snippets. 

 This makes it easy for developers to fix their code as they are writing it. 

This is a win-win for security. By giving developers the opportunity to implement security best practices, it produces less and more accurate SAST results for the AppSec team.

How to get it

Real time insights are available in a freemium model. Users can get real time insights within a command line interface (CLI) executable available for free.

Additional features and real-time in-IDE scanning are available for customers with the “AI Security“ package. If you’re an existing customer, contact your account manager for more details. Not yet a customer?  Get a free demo.