AI tools like ChatGPT, Google Gemini and others being afflicted by malicious actors via injecting harmful instructions into leading GenAI tools. These were overlooked previously and attack methodology targets the browser extensions installed by various organizations.

The attack methodology named as ‘Man in Prompt’, exercise its attack with new class exploit targeting the AI tools as per LayerX’s researchers.

As per the research any browser extension, even without any special permissions, can access the prompts of both commercial and internal LLMs and inject them with prompts to steal data, exfiltrate it and cover their tracks. 

The exploit has been tested on all top commercial LLMs, with proof-of-concept demos provided for ChatGPT and Google Gemini. 

The question is how do they impact Users & organizations at large & how does the AI tools function within web browsers?

For organizations the implications can be high then expected as AI tools are most sought after and slowly organization across verticals are relying on AI tools.

The LLMs used and tested on many organizations are mostly trained ones. They carry huge data set of information which are mostly confidential and possibility of being vulnerable to such attack rises .

The attack methodology named as ‘Man in Prompt’, exercise its attack with new class exploit targeting the AI tools as per LayerX’s researchers. As per the research any browser extension, even without any special permissions, can access the prompts of both commercial and internal LLMs and inject them with prompts to steal data, exfiltrate it, and cover their tracks. 

The attack methodology named as ‘Man in Prompt’, exercise its attack with new class exploit targeting the AI tools as per LayerX’s researchers. As per the research any browser extension, even without any special permissions, can access the prompts of both commercial and internal LLMs and inject them with prompts to steal data, exfiltrate it, and cover their tracks. 

LayerX researcher termed this type of attack as ‘hacking copilots’ that are equipped to steal organizational information.

The prompts given are a part of the web page structure where input fields are known as the Document Object Model, or DOM. So virtually any browser extension with basic scripting access to the DOM can read or alter what users type into AI prompts, even without requiring special permissions.

Bad actors can use compromised extensions to carry out activities including manipulating a user’s input to the AI.

• Perform prompt injection attacks, altering the user’s input or inserting hidden instructions.
• Extract data directly from the prompt, response, or session.
• Compromise model integrity, tricking the LLM into revealing sensitive information or performing unintended actions

Understanding the attack scenario

Proof-of-concept attacks against major platforms

For ChatGPT, an extension with minimal declared permissions could inject a prompt, extract the AI’s response and remove chat history from the user’s view to reduce detection.

LayerX implemented an exploit that can steal internal data from corporate environments using Google Gemini via its integration into Google Workspace.

Over the last few months, Google has rolled out new integrations of its Gemini AI into Google Workspace. Currently, this feature is available to organizations using Workspace and paying users.

Gemini integration is implemented directly within the page as added code on top of the existing page. It modifies and directly writes to the web application’s Document Object Model (DOM), giving it control and access to all functionality within the application

These platforms are vulnerable to  any exploit which Layer X researchers showcased that without any special permissions shows how practically any user is vulnerable to such an attack. 

Threat mitigation

These kind of attacks creates a blind spot for traditional security tools like endpoint Data Loss Prevention (DLP) systems or Secure Web Gateways, as they lack visibility into these DOM-level interactions. Blocking AI tools by URL alone also won’t protect internal AI deployments.

LayerX advises organisations to adjust their security strategies towards inspecting in-browser behaviour.

Key recommendations include monitoring DOM interactions within AI tools to detect suspicious activity, blocking risky extensions based on their behavior rather than just their listed permissions, and actively preventing prompt tampering and data exfiltration in real-time at the browser layer.

(Source: https://layerxsecurity.com/blog/man-in-the-prompt-top-ai-tools-vulnerable-to-injection/)