AI is reshaping cybersecurity on both fronts — slashing phishing-creation time to minutes, powering multi-million-dollar deepfake fraud, and opening a new frontier where AI systems themselves are the target. A close look at global data and India's fast-escalating digital arrest crisis.
Over the last year or so, one thing has become pretty clear while reading through breach reports and fraud data: AI has stopped being a “future threat” everyone talks about and become the thing actively shaping cybersecurity right now, on both sides. The same generative and agentic AI tools that are helping security teams triage alerts and contain breaches faster are also being used by attackers to write near-perfect phishing emails, clone someone's voice from a few seconds of audio, and pull off multi-step fraud in hours instead of the weeks it used to take.
This research looks at the issue from several angles rather than just one — how attacks are changing, how defense is actually improving, a newer risk most people haven't thought about yet (AI systems themselves getting attacked), and how all of this looks specifically in India, where deepfake-driven “digital arrest” scams have become a genuine national problem. It closes with practical, actionable takeaways.
The numbers here are honestly a bit alarming once pulled together. Sift, a fraud-detection firm, recorded a 466% jump in phishing reports in just the first quarter of 2025, alongside a 456% rise in AI-enabled scams and a 186% surge in compromised personal data — all in one quarter. IBM's 2025 breach research put a number on why this is happening: generative AI has cut the time it takes to write a convincing phishing email from around 16 hours down to about 5 minutes. That's not a small efficiency gain — it's a completely different economics for attackers.
Deepfakes have moved well past the “novelty” stage too. The most quoted case is Arup, where a finance employee made 15 fraudulent transfers amounting to $25.6 million after a video call where every other participant was an AI-generated executive. These cases aren't sporadic either — the World Economic Forum's Global Cybersecurity Outlook 2026 found that 73% of businesses faced cyber-enabled fraud in 2025, while Chainalysis reports AI-powered cryptocurrency scams cost $17 billion in losses for 2025, with government-impersonation deepfakes rising by more than 1,400% year-on-year.
What's most striking is how far AI has moved into phishing at the infrastructural level, not just individual fraud cases. ENISA's 2025 Threat Landscape report found that over 80% of social engineering attacks worldwide were conducted using AI-powered phishing campaigns. IBM's 2025 report adds that roughly 1 in 6 security breaches globally now involves attacker AI use, split evenly between AI-written phishing messages and AI-generated deepfakes. The FBI's Internet Crime Complaint Center logged over 22,000 AI-related complaints in 2025, totaling almost $893 million in losses — and investigators believe this is only the floor, since most victims never realize AI was involved.
It isn't just social engineering either. Several 2025 incident writeups describe business email compromise campaigns where attackers used AI to study a target company's internal writing style, org structure, and recent transactions before sending one highly specific, highly convincing fraudulent request — instead of the old spray-and-pray approach. One campaign targeted around 800 accounting firms with AI-generated emails referencing specific state registration details and achieved a 27% click rate, far above normal phishing benchmarks. Most email filters were built to catch mass-produced phishing — bad grammar, mismatched domains, generic greetings — and AI-written messages increasingly have none of those tells. Detection has to shift from pattern-matching to genuinely understanding context and behavior.
AI has also lowered the skill threshold for cyber-crime more broadly. Multiple 2025 reports describe AI assisting in vulnerability research and exploit generation, shrinking the gap between a bug's disclosure and a working exploit in the wild. Ransomware groups have reportedly begun using AI to craft targeted, industry-specific threats and to automate parts of the reconnaissance phase. None of it needs to be spectacular on its own — it's the cumulative efficiency gained at every stage of an attack that adds up.
There is good news here too. According to IBM's 2025 Cost of a Data Breach Report, the average cost of a data breach fell to $4.44 million, a 9% decrease from $4.88 million the previous year — the first decline in five years. The main driver is faster detection and containment enabled by AI-powered security tooling.
The gap between adopters and non-adopters is significant. Firms using AI extensively in security operations spend an average of $3.62 million per breach, versus $5.52 million for those that don't — a saving of roughly $1.9 million per breach. AI-assisted organizations also contained breaches roughly 80 days faster on average, and up to 190 days faster in some cases. The global average breach lifecycle dropped to 241 days, the lowest in nine years.
But there's a governance gap sitting right beneath this good news. 63% of breached organizations either had no formal AI governance policy or were still drafting one, and even among those with a policy, only about a third regularly audited for unauthorized “shadow AI” use. One in five organizations reported a breach specifically tied to shadow AI — AI tools running in their environment that nobody was tracking. In effect, companies are adopting AI for defense faster than they're governing it, and that gap is quietly becoming its own risk category.
Looking at which specific controls actually move the needle, AI clearly isn't a magic fix on its own — it works best as part of a larger stack. IBM ranks a DevSecOps approach, AI/ML-specific security insights, mature SIEM platforms, active threat-intel sharing, strong encryption, tested incident response plans, and zero-trust architecture among the top cost-reducers. Organizations that contained a breach in under 200 days paid about $3.61 million on average versus $5.49 million for slower responders — so AI's biggest defensive value shows up as speed, not as a standalone replacement for human security teams.
It's also not spread evenly. Healthcare had the longest breach lifecycles of any tracked sector, averaging around 279 days, largely because patient data systems tend to be fragmented. Regionally, the US moved in the opposite direction — average breach costs there hit a record $10.22 million, up 9%, driven by regulatory fines. India was grouped with Canada as one of the few major markets where costs rose rather than fell in 2025 — suggesting AI-driven savings are concentrated where both technical maturity and a favorable regulatory environment exist, which India doesn't fully have yet.
This is arguably the least talked-about angle of the story. As AI systems move from answering questions to actually doing things — sending emails, querying databases, calling APIs, running code — they open up a genuinely new kind of security risk. OWASP's 2025 Top 10 for LLM Applications ranks prompt injection as the #1 risk for the second edition running. Prompt injection happens when an attacker crafts input that the model treats as a new instruction rather than as data, because most current AI architectures process instructions and untrusted content through the same channel, with no clean way to separate “things to do” from “things to read.”
There are two flavors. Direct injection is when someone types adversarial instructions straight into a chat. Indirect injection — considered the more dangerous of the two — is when a model reads content from a webpage, document, email, or code repo, and hidden instructions buried in that content get treated as legitimate commands. In systems that combine browsing, plugins, or autonomous tool use, this escalates quickly, since the model may act on the injected instruction with real consequences: leaking data, sending unauthorized messages, or triggering unapproved tool calls.
Agentic AI — systems that plan multi-step actions on their own, retain memory across sessions, and call external tools — widens this risk further. OWASP's newer Top 10 for Agentic Applications, launched at Black Hat Europe 2025, lists risks unique to this category: excessive agency (giving an agent more tools or permissions than its task requires), tool misuse, memory or context poisoning across sessions, and even multi-agent attacks like spoofed identities between cooperating agents. Notably, even the vendors building these systems — Microsoft, Cisco, Anthropic — converge on the same message: there's no single fix. It takes layered defenses — least-privilege tool access, strict filtering, mandatory human approval for high-risk actions, and continuous red-teaming. Even the most hardened agentic systems from leading labs show non-zero attack success rates under sustained testing, confirming this is an active engineering problem, not a solved one.
To make this concrete: imagine an AI agent with permission to read email and documents is asked to summarize a shared file. If that file contains hidden text instructing the model to also forward specific data to an outside address, a poorly isolated agent might simply comply — because architecturally it can't reliably distinguish “content I was asked to summarize” from “commands I was asked to obey.” Researchers describe the underlying condition as a combination of three factors: access to private data, exposure to untrusted content, and the ability to communicate externally. When an agent has all three at once, prompt injection stops being an edge case and becomes an architectural risk. That's why standard advice calls for isolating untrusted input, requiring human approval before sensitive actions, and logging every tool call. A growing category of “guardrail” tools — real-time injection detectors, input/output safety classifiers — is built specifically to catch this, though independent testing shows they reduce attack success rather than eliminate it entirely.
For anyone deploying AI copilots or customer-facing agents, this changes the security conversation: the AI system isn't just a tool a human might misuse anymore — it's its own component with its own credentials and its own blast radius, and needs to be threat-modeled as one. This isn't purely hypothetical either: several public red-teaming exercises and bug bounty writeups through 2025 documented working prompt-injection chains against real deployed AI assistants, including cases where a model was tricked through a webpage or document into leaking conversation context it shouldn't have.
India is probably the clearest large-scale example of AI-enabled fraud colliding with rapid smartphone adoption and uneven digital literacy. “Digital arrest” scams — where fraudsters use AI-generated video and voice to pose as police officers, judges, or investigators on a video call, then pressure victims into paying to avoid a fake arrest — have gone from a rare tactic to a nationwide crisis in about two years. Law-enforcement data cited in industry analysis puts the number of these incidents at more than 92,000, while government figures put total cyber-fraud losses for India at roughly ₹22,495 crore (about $2.6–2.7 billion) for 2025, with reported cases up about 24% year-on-year to around 2.8 million.
Some figures genuinely stand out. A 2025 industry analysis found that 47% of Indian adults have either personally experienced or know someone who's experienced an AI voice-cloning or deepfake scam — almost double the global average of 25%. And 83% of Indian victims of AI voice scams lost actual money, with nearly half losing more than ₹50,000. Deepfake content in India is projected to have grown roughly ninefold year-on-year, from around 500,000 files in 2023 to an estimated 8 million in 2025 — driven by how cheap it's become to generate fake video and voice, and how much source material (a single WhatsApp or LinkedIn photo, essentially) is already available.
Digital arrest scams are only one piece of it, though — investment fraud is actually the single largest category by money lost. Fake trading apps, Ponzi-style schemes, and crypto scams running through WhatsApp and Telegram “stock tip” groups reportedly account for more than 75% of all cybercrime losses in India, often using deepfaked images and voices of well-known business figures or cricketers to make the pitch look legitimate. A few individual cases give a sense of scale: a Mangaluru woman lost around ₹1.80 crore, a retired Delhi doctor couple lost about ₹15 crore, and a 92-year-old in Delhi was drained of roughly ₹2 crore. These cases span retirees to serving government and security personnel — this is not just a “less tech-savvy people” problem. UPI fraud adds another layer — reported UPI fraud crossed ₹805 crore in just the first eight months of FY26, and survey data suggests around one in five UPI users has faced an attempted fraud, with just over half never even filing a report.
Zooming out, India's National Human Rights Commission estimates cyber fraud has cost citizens over ₹52,976 crore across the last six years, and the national cybercrime helpline (1930) received nearly 3.24 crore calls in 2025 alone — close to one call every single second of the year. Some government analysts believe the real number, accounting for underreporting, could be as high as ₹1.2 lakh crore — almost five times the official figure. Even taking just the reported numbers, the trend is clearly accelerating: digital-arrest-style fraud nearly tripled between 2022 and 2024 and kept climbing into 2025–26.
On the response side, things have scaled up, though funding trends are contradictory. I4C and the Ministry of Home Affairs reported deactivating around 1.2 million SIM cards and freezing about 1.33 million mule accounts in 2025, recovering roughly ₹5,489 crore. In February 2026, new rules came into force requiring platforms to take down flagged deepfake content within three hours of a government or court order. At the same time, however, India's cybersecurity capital budget dropped sharply — from about ₹1,900 crore in 2025–26 to around ₹790 crore for 2026–27, a cut of more than 58% — even as fraud losses kept climbing. That gap between the scale of the problem and the pace of funding deserves more attention than it's currently getting.
Two things seem to be converging. On the attacker side, tactics are shifting from generating better content (sharper phishing emails, better deepfakes) toward actual orchestration — autonomous scam operations combining synthetic voices, AI-scripted conversations, and multi-channel delivery with barely any human involvement. It's effectively industrializing social engineering the way earlier automation industrialized network scanning. On the defender side, organizations now have to secure not just their networks and endpoints, but the AI systems they've deployed themselves — copilots and agents that carry real permissions and real consequences when something goes wrong.
Traditional perimeter-based security doesn't map onto this well, because an AI agent's “perimeter” is essentially whatever tools and data it can reach at any given moment, and that shifts depending on the task. Regulation is starting to catch up — the EU AI Act's transparency rules take effect from August 2026, and OWASP, MITRE (through its ATLAS framework for adversarial AI threats), and NIST (through its AI Risk Management Framework) have converged on roughly the same best practices: least-privilege tool permissions, mandatory human approval for high-impact actions, continuous red-teaming, and treating AI output as untrusted data requiring the same sanitization as anything else from outside the system.
The security workforce itself isn't fully caught up either — most existing training paths were built for network, endpoint, and application security, and there isn't much standardized coverage yet for concepts like multi-agent trust boundaries or memory poisoning. As more setups move from single chatbots to multi-agent systems delegating tasks to each other, the attack surface doesn't just add up — it compounds. A manipulated agent low in a delegation chain can end up influencing decisions made by agents above it, a scenario with no clean equivalent in traditional IT security. Organizations that build AI governance and security testing in from the start — rather than bolting it on after an incident — are the ones most likely to realize the cost savings outlined in Section 2, rather than becoming the next shadow-AI breach statistic.
Our security team can help you investigate, contain, and remediate.
Contact Our Security Experts
