Cybercriminals have begun refining malicious large language models (LLMs) using underground forum posts and breach dumps to tailor AI models for specific fraud schemes, threat intel firm Flashpoint warns.
More specifically, fraudsters are fine-tuning illicit LLMs — including WormGPT and FraudGPT — using malicious datasets such as breached credentials, scam scripts, and infostealer logs. As adversaries use these models to generate outputs, they gather user feedback to fine-tune responses, creating a loop where offensive capability keeps improving over time.
[ See also: Top 5 ways attackers use generative AI to exploit your systems ]
“This trend is particularly concerning because it demonstrates adversaries ‘closing the loop on model tuning’ — their offensive capabilities constantly improving over time through real-time feedback and illicit data,” Ian Gray, Flashpoint VP of cyber threat intelligence, tells CSO.
Flashpoint has also observed private chat groups where users submitted failed prompt attempts back to LLM developers, leading to rapid iteration and improved performance within days. In one instance, a user reported formatting issues with a financial fraud prompt, and shortly after, the developer shared an updated version with refined templates, Flashpoint observed.
“This adaptive and self-improving nature of malicious AI, fueled by compromised data and criminal collaboration, makes it an especially potent and difficult threat to counter,” Gray says.
Cybercriminals are tailoring AI models for specific fraud schemes, including generating phishing emails tailored by sector or language, as well as writing fake job posts, invoices, or verification prompts.
“Some vendors even market these tools with tiered pricing, API access, and private key licensing, mirroring the [legitimate] SaaS economy,” Flashpoint researchers found.
“This specialization leads to potentially greater success rates and automated complex attack stages,” Flashpoint’s Gray tells CSO.
Deepfake as a service goes mainstream
Cybercrime vendors are also lowering the barrier for creating synthetic video and voice, with deepfake as a service (DaaS) offerings that include:
- Custom face generation for dating scams
- Audio spoofing for voice verification fraud
- On-demand video avatars that lip-sync based on customer-submitted scripts
These services are increasingly offered with add-ons such as pre-loaded backstories,
matching fake documents, and automated scheduling for calls.
Prompt engineering as a service
Underground communities have also emerged around the art of crafting jailbreak prompts.
These “bypass builders” specialize in defeating guardrails of mainstream LLMs (e.g., ChatGPT or Gemini) to unlock restricted outputs such as social engineering scripts, step-by-step hacking tutorials, and bank fraud playbooks, including “know your customer” (KYC) bypass guides.
“This ‘prompt engineering as a service’ (PEaaS) lowers the barrier for entry, allowing a wider range of actors to leverage sophisticated AI capabilities through pre-packaged malicious prompts,” Gray warns.
“Together, these trends create an adaptive threat: tailored models become more potent when refined with illicit data, PEaaS expands the reach of threat actors, and the continuous refinement ensures constant evolution against defenses,” he says.
Deep dive
Flashpoint analysts tracked these developments in real-time across more than 100,000 illicit sources, monitoring everything from dark web marketplaces and Telegram groups to
underground LLM communities.
Between Jan. 1 and May 30, 2025, the researchers logged more than 2.5 million AI-related posts covering various nefarious tactics, including jailbreak prompts, deepfake service ads, phishing toolkits, and bespoke language models built for fraud and other forms of cybercrime.
Underground LLM tactics and strategies
Related research from Cisco Talos warns that cybercriminals continue to adopt LLMs to streamline their processes, write tools and scripts that can be used to compromise users, and generate content that can more easily bypass defenses.
Talos observed cybercriminals resorting to using uncensored LLMs or even custom-built criminal LLMs for illicit purposes.
Advertised features of malicious LLMs suggest that cybercriminals are linking these systems to various external tools to scan sites for vulnerabilities, verify stolen credit card numbers, and other malicious actions.
At the same time, adversaries are often jailbreaking legitimate models faster than LLM developers can secure them, Talos warns.
Defense against the dark (AI) arts
Flashpoint’s “AI and Threat Intelligence: The Defenders’ Guide” explains that while AI is a double-edged sword in cybersecurity, defenders who thoughtfully integrate AI into their threat intelligence and response workflows can outpace adversaries.
Enterprises need to balance automation with expert analysis, separating hype from reality, and continuously adapt to the rapidly evolving threat landscape.
“Defenders should start by viewing AI as an augmentation of human expertise, not a replacement,” Flashpoint’s Gray says. “This philosophy ensures AI strengthens existing workflows, driving value by reducing noise and accelerating decision-making, rather than creating new blind spots.”
Gray adds: “The organizing principle should enhance their collection advantage by utilizing AI to derive insights from high-signal data, accelerating discovery, and structuring unstructured content. Ultimately, the aim is to improve efficiency by empowering analysts with tools that assist their judgment, maintain human control, and provide context.”