Google Threat Intelligence Group (GTIG) has tracked and found how attackers have models pose as security researchers or firmware experts to perform analyses on embedded systems and protocols. The zeroday exploit set to target popular open-source web administration tool, generated using AI. Observations revealed hackers are deploying agentic tools to partially automate research and exploit validation.

This shifts AI from a passive assistant to a system that independently executes parts of offensive workflows.

Theis report provide insights derived from Mandiant incident response engagements, Gemini and GTIG’s proactive research. The highlights aim at the threat environment where AI serves dual purpose. On one hand to disrupt advance cyber threats from hackers and other AI tools acting as high value agents for cyber attacks. 

Here are key highlights of the threat research:

Vulnerability Discovery and Exploit Generation: For the first time, GTIG has identified a threat actor using a zero-day exploit that we believe was developed with AI. The criminal threat actor planned to use it in a mass exploitation event but our proactive counter discovery may have prevented its use. 

AI-Augmented Development for Defense Evasion: AI-driven coding has accelerated the development of infrastructure suites and polymorphic malware by adversaries. These AI-enabled development cycles facilitate defense evasion by enabling the creation of obfuscation networks and the integration of AI-generated decoy logic in malware that google have linked to suspected Russia-nexus threat actors.

Autonomous Malware Operations: AI-enabled malware, such as PROMPTSPY, signal a shift toward autonomous attack orchestration, where models interpret system states to dynamically generate commands and manipulate victim environments. Analysis of this malware revealed previously unreported capabilities and use cases for its integration with AI. 

AI-Augmented Research and IO: Adversaries continue to leverage AI as a high speed research assistant for attack lifecycle support, while shifting toward agentic workflows to operationalize autonomous attack frameworks. 

Obfuscated LLM Access: Threat actors now pursue anonymized, premium tier access to models through professionalized middleware and automated registration pipelines to illicitly bypass usage limits. This infrastructure enables large scale misuse of services while subsidizing operations through trial abuse and programmatic account cycling.

Supply Chain Attacks: Adversaries like “TeamPCP” (aka UNC6780) have begun targeting AI environments and software dependencies as an initial access vector. These supply chain attacks result in multiple types of machine learning (ML)-focused risks outlined in the Secure AI Framework (SAIF) taxonomy, namely Insecure Integrated Component (IIC) and Rogue Actions (RA). 

Hackers leveraging AI for vulnerability development and Zeroday exploitation

Cybercriminal groups are increasingly leveraging AI to support vulnerability discovery and exploit development.

Google Researchers observed threat actors planning large-scale exploitation campaigns using AI-assisted techniques.

A zero-day vulnerability was identified in a Python script capable of bypassing Two-Factor Authentication (2FA) in a popular open-source web administration tool. The exploit required valid user credentials but bypassed 2FA due to a hardcoded trust assumption within the application logic. Analysis suggests the vulnerability discovery and exploit development were likely assisted by an AI model due to:

• Structured and highly “textbook” Python coding style
• Excessive educational docstrings
• Hallucinated CVSS scoring
• LLM-like formatting patterns and helper classes

Unlike traditional vulnerabilities such as memory corruption or input validation flaws, this issue was a high-level semantic logic flaw difficult for conventional scanners to detect. Frontier AI models are becoming increasingly capable of:

• Understanding developer intent
• Identifying hardcoded security assumptions
• Detecting hidden logic inconsistencies
• Surfacing vulnerabilities missed by static analysis and fuzzing tools

The incident highlights the growing risk of AI-assisted zero-day discovery and exploitation by threat actors and as AI use datasets containing historical vulnerabilities to help models better reason about security flaws.

“For the first time, GTIG has identified a threat actor using a zero-day exploit that we believe was developed with AI,” GTIG researchers say.

What can be the consequences specifically at a time when new AI models unlike Anthropic’s Mythos, which were announced last month and appear to be good at finding such holes that Anthropic shared.

Rob Joyce, the former cybersecurity director of the National Security Agency, said that it can be difficult to know whether a human or machine wrote computer code, adding that, “A.I.-authored code does not announce itself.”

The Zeroday Defect

The report’s main findings involves a zero-day exploit that GTIG assessed was likely developed with AI assistance.

The vulnerability affected a popular open-source, web-based system administration tool and allowed two-factor authentication to be bypassed, although valid user credentials were still required.

The zero-day flaw was detected by the Google Threat Intelligence Group within the past few months and was exploited by “prominent cybercrime threat actors” in a script of the Python programming language.

Allow hackers to bypass two-factor authentication on “a popular open-source, web-based system administration tool,” though the hackers also would have needed access to valid credentials like user names and passwords to be successful, the company said.

Malware Evasion Techniques via AI

Hackers are also leveraging malware evasion techniques and sandbox evasions and other tricks to stay out of sight. As defenders increasingly rely on AI to accelerate and improve threat detection, a subtle but alarming new contest has emerged between attackers and defenders.

GTIG identified several malware families or tools with LLM-enabled obfuscation features, including PROMPTFLUX, HONESTCUE, CANFAIL, and LONGSTREAM.

Here is an example:

In June 2025, a malware sample was anonymously uploaded to VirusTotal from the Netherlands. At first glance, it looked incomplete. Some parts of the code weren’t fully functional, and it printed system information that would usually be exfiltrated to an external server.

The sample contained several sandbox evasion techniques and included an embedded TOR client, but otherwise resembled a test run, a specialized component or an early-stage experiment. What stood out, however, was a string embedded in the code that appeared to be written for an AI, not a human. It was crafted with the intention of influencing automated, AI-driven analysis, not to deceive a human looking at the code.

The malware includes a hardcoded C++ string, visible in the code snippet below:

In-memory prompt injection.

Hackers can leverage these emerging AI Evasion techniques to bypass AI-powered security systems by manipulating how Large Language Models (LLMs) interpret, analyze, and classify malicious content or activity.

How Attackers May Use AI Evasion Techniques

• Prompt Injection Attacks
  Attackers craft malicious inputs that manipulate AI models into ignoring security rules, revealing sensitive information, or executing unintended actions.
• Bypassing AI-Based Detection
  Threat actors can design malware, phishing emails, or malicious scripts in ways that appear legitimate to AI-powered detection systems.
• Manipulating Context & Intent
  AI systems rely heavily on context and language interpretation. Attackers may exploit ambiguous wording, hidden instructions, or layered prompts to confuse AI defenses.
• Generating Adaptive Malware
  AI-generated malware can dynamically modify behavior, code structure, or communication patterns to evade traditional and AI-driven security tools.
• Automating Social Engineering
  AI can help create highly convincing phishing messages, fake identities, and impersonation attempts that are harder for AI-based defenses to detect.

Conclusion: AI is significantly strengthening cybersecurity defenses.

Security teams are leveraging AI for real-time threat detection, behavioral analytics, automated incident response, vulnerability management, and proactive risk assessment. While attackers currently benefit from AI-driven automation and exploitation capabilities, defenders are expected to gain a stronger long-term advantage as AI evolves into a core component of secure software development, proactive cyber defense, and intelligent security operations.

Sources: https://cloud.google.com/blog/topics/threat-intelligence/ai-vulnerability-exploitation-initial-access

Sources: https://blog.checkpoint.com/artificial-intelligence/ai-evasion-the-next-frontier-of-malware-techniques/