The days of obvious, poorly written phishing emails are fading fast. A recent campaign uncovered by Microsoft Threat Intelligence shows how cybercriminals are using artificial intelligence to build more convincing, harder-to-detect scams. On August 18, Microsoft detected and blocked a credential phishing attack that targeted U.S. organizations using AI-generated code hidden inside a disguised file.
How the scam worked
The attack began with a fraudulent file-sharing email sent from a compromised small business account — a tactic that added legitimacy. The attachment looked like a six-page PDF named 23mb – PDF – 6 pages.svg. But the extension told the real story: this wasn’t a PDF at all, but a Scalable Vector Graphic (SVG) file.
SVG files can embed interactive code, making them a handy vehicle for attackers. In this case, the file was designed to resemble a business analytics dashboard, complete with fake charts. Hidden inside that display was a payload that redirected victims to a counterfeit login page aimed at harvesting credentials. Instead of using classic obfuscation tricks, the malicious code was disguised with ordinary business terms — “revenue,” “operations,” “risk” — to make it look like legitimate corporate data.
AI was writing the code
Microsoft researchers used their own AI tool, Security Copilot, to analyze the file. The verdict: the code was unlikely to have been written by hand. Its systematic, overly verbose, and impractical structure pointed to a large language model (LLM) — the same kind of AI that powers popular chatbots. In short, attackers had used AI to generate code that slipped past traditional signature-based defenses.
AI vs. AI on defense
The campaign didn’t succeed thanks to Microsoft’s defensive AI. Microsoft Defender for Office 365 blocked the attack by spotting behavioral red flags that were harder for the attackers’ AI to hide. These included emails self-addressed with hidden BCC recipients, suspicious file name and type combinations, and redirects to known malicious domains.
The case underscores an emerging reality: AI isn’t just a tool for defenders, but for attackers too. Security systems must now be able to recognize the subtle behavioral anomalies that even AI-generated phishing can’t fully mask.
Expert perspectives
Security professionals say this shift changes the frontline of defense. Anders Askasen, VP of Product Marketing at Radiant Logic, noted that with AI-driven phishing, “the frontline isn’t the payload, it’s the person behind the login.” He argued that organizations should invest in identity observability — consolidating identity data to detect accounts behaving out of character.
Andrew Obadiaru, CISO at Cobalt, added that AI has become “camouflage that blends seamlessly into enterprise workflows.” His recommendation: security teams should double down on behavioral detection, red-teaming against AI-assisted tactics, and shortening response times to incidents.
The bigger picture
This incident highlights a new phase in phishing: one where attackers and defenders both rely on AI. While AI-written code can help attackers make scams more convincing, AI-driven defense can spot hidden behavioral tells they leave behind. The lesson is clear — security strategies must adapt quickly, focusing on identity, behavior, and speed of response to stay ahead of AI-scaled deception.
Would your team be able to spot an AI-generated phishing attempt before clicking? Share your thoughts below.
