Google won’t fix this confirmed AI security issue—here’s why
Gmail users love the smart features that make using the world’s most popular email provider with 2.5 billion accounts such a breeze. The introduction of Gemini AI for Workspace, covering multiple Google products, only moved usability even further up the email agenda. But, as security researchers confirmed security vulnerabilities and demonstrated how attacks could occur across platforms like Gmail, Google Slides, and Google Drive, why did Google decide this was not a security issue and issue a “Won’t Fix (Intended Behavior)” ticket? I’ve been digging with the help of Google, and here’s what I’ve found and you need to know.
The Gmail AI Security Issue Explained
Across the course of 2024 there were multiple headlines that focused attention on AI-powered attacks against Gmail users, from the viral story about a security consultant who came oh so close to becoming yet another hacking statistic, to Google’s own security alerts being turned against users and, as the end o0f the year approached, a warning from Google itself about a second wave of attacks targeting Gmail users. But one technical security analysis caught my attention from earlier in the year that left me wondering just why one problem with potentially devastating security consequences was seemingly not being addressed: “Gemini is susceptible to indirect prompt injection attacks,” the report stated, and illustrating just how these attacks “can occur across platforms like Gmail, Google Slides, and Google Drive, enabling phishing attempts and behavioral manipulation of the chatbot.” Jason Martin and Kenneth Yeung, the security researchers involved in writing the detailed technical analysis, said that, as part of the responsible disclosure process, “this and other prompt injections in this blog were reported to Google, who decided not to track it as a security issue and marked the ticket as a Won’t Fix (Intended Behavior).”
With some people suggesting that Gmail users should disable smart features, and others asking how they can opt out of AI reading their private email messages, I thought it was worth talking to my contacts at Google as I dug deeper into what was going on here.
The Gmail Gemini Prompt Injection Problem In A Nutshell
I would, as always, recommend that you go and read the HiddenLayer Gemini AI security analysis in full, but here’s the security issue in as small a nutshell as I could get to fit. Like most large language models, Google’s Gemini AI is susceptible to what are known as indirect prompt injection attacks. “This means that under certain conditions,” the report said, “users can manipulate the assistant to produce misleading or unintended responses.” So far, so meh, unless you paid attention to the indirect bit of that. Indirect prompt injection vulnerabilities allow third-parties to take control of a language model by inserting the prompt into “less obvious channels” such as documents, emails or websites. So, when you then take into consideration that attackers could distribute malicious documents and emails to target accounts, compromising the integrity of the responses generated by the target Gemini instance, it starts getting, as Elon Musk might say, interesting. “Through detailed proof-of-concept examples,” the researchers explained” they were able to illustrate “how these attacks can occur across platforms like Gmail, Google Slides, and Google Drive.” Specifically, the report covered phishing via Gemini in Gmail, tampering with data in Google Slides and poisoning Google Drive locally and with shared documents. “These examples show that outputs from the Gemini for Workspace suite can be compromised,” the researchers said, “raising serious concerns about the integrity of this suite of products.”
Google Responds To Gmail Prompt Injection Attack Concerns
I approached my contacts within Gmail and a Google spokesperson told me:
“Defending against this class of attack has been an ongoing priority for us, and we’ve deployed numerous strong defenses to keep users safe, including safeguards to prevent prompt injection attacks and harmful or misleading responses. We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks.”
A more detailed conversation with my contacts revealed the following information that all Gmail users should take into consideration when thinking about security and Google’s AI resources.
- These vulnerabilities are not novel and are consistent in LLMs across the industry.
- When launching any new LLM-based experience, Google conducts internal and external security testing to meet user needs as well as its own standards regarding user safety.
- This includes security testing from the Google AI Red Team on prompt attacks, training data extraction, backdooring the model, adversarial examples, data poisoning and exfiltration.
- Google also includes AI in its Vulnerability Rewards Program, which includes a specific criteria for AI bug reports to assist the bug hunting community in effectively testing the safety and security of Google AI products.
- In addition, Gmail and Drive include strong spam filters and user input sanitization which help to mitigate against hostile injections of malicious code into Gemini.