Future Risk Predictions
In a recent presentation at Black Hat 2023, HackerOne Founder, Michiel Prins, and hacker, Joseph Thacker aka @rez0, discussed some of the most impactful risk predictions related to Generative AI and LLMs, including:
- Increased risk of preventable breaches
- Loss of revenue and brand reputation
- Increased cost of regulatory compliance
- Diminished competitiveness
- Reduced ROI on development investments
The Top Generative AI and LLM Risks According to hackers
According to hacker Gavin Klondike, “We’ve almost forgotten the last 30 years of cybersecurity lessons in developing some of this software.” The haste of GAI adoption has clouded many organizations’ judgment when it comes to the security of artificial intelligence. Security researcher Katie Paxton-Fear aka @InsiderPhD, believes, “this is a great opportunity to take a step back and bake some security in as this is developing and not bolting on security 10 years later.”
Prompt Injections
The OWASP Top 10 for LLM defines prompt injection as a vulnerability during which an attacker manipulates the operation of a trusted LLM through crafted inputs, either directly or indirectly. Thacker uses this example to help understand the power of prompt injection:
“If an attacker uses prompt injection to take control of the context for the LLM function call, they can exfiltrate data by calling the web browser feature and moving the data that are exfiltrated to the attacker’s side. Or, an attacker could email a prompt injection payload to an LLM tasked with reading and replying to emails.”
Ethical hacker, Roni Carta aka @arsene_lupin, points out that if developers are using ChatGPT to help install prompt packages on their computers, they can run into trouble when asking it to find libraries. Carta says, “ChatGPT hallucinates library names, which threat actors can then take advantage of by reverse-engineering the fake libraries.”
According to Thacker, “The jury is out on whether or not it’s solvable, but personally, I think it is.” He says the mitigation depends on the implementation and deployment of the prompt injection and, “of course, by testing.”
Agent Access Control
“LLMs are as good as their data,” says Thacker. “The most useful data is often private data.”
According to Thacker, this creates an extremely difficult problem in the form of agent access control. Access control issues are very common vulnerabilities found through the HackerOne platform every day. Where access control goes particularly wrong regarding AI agents is the mixing of data. Thacker says AI agents have a tendency to mix second-order data access with privileged actions, exposing the most sensitive information to potentially be exploited by bad actors.
The Evolution of the Hacker in the Age of Generative AI
Naturally, as new vulnerabilities emerge from the rapid adoption of Generative AI and LLMs, the role of the hacker is also evolving. During a panel featuring security experts from Zoom and Salesforce, hacker Tom Anthony predicted the change in how hackers approach processes with AI:
“At a recent Live Hacking Event with Zoom, there were easter eggs for hackers to find — and the hacker who solved them used LLMs to crack it. Hackers are able to use AI to speed up their processes by, for example, rapidly extending the word lists when trying to brute force systems.”
He also senses a distinct difference for hackers using automation, claiming AI will significantly uplevel the reading of source code. Anthony says, “Anywhere that companies are exposing source code, there will be systems reading, analyzing, and reporting in an automated fashion.”
There are even new tools for the education of hacking LLMs — and therefore for identifying the vulnerabilities created by them. Anthony uses “an online game for prompt injection where you work through levels, tricking the GPT model to give you secrets. It’s all developing so quickly.”
Use the Power of Hackers for Secure Generative AI
Even the most sophisticated security programs are unable to catch every vulnerability. HackerOne is committed to helping organizations secure their GAI and LLMs and to staying at the forefront of security trends and challenges. With HackerOne, organizations can:
Contact us today to learn more about how we can help take a secure approach to Generative AI.
