When autonomy is the attack surface: securing the defense-AI convergence

The convergence of defense and artificial intelligence is producing systems of remarkable capability — autonomous platforms that can sense, decide and act with minimal human input, at speeds and scales no human-directed force could match. It is also, simultaneously and unavoidably, producing a new and deeply worrying attack surface, because everything that makes these systems powerful also makes them targets worth attacking.

Networked autonomy multiplies both reach and risk in the same motion. A fleet of connected, intelligent systems can coordinate and react at a scale and tempo that traditional forces cannot, but the very connectivity and software dependence that enable that coordination also create vulnerabilities an adversary can probe, exploit and turn against their owner. Capability and exposure grow together.

An adversary who corrupts the intelligence directing a system may never need to defeat the system itself.

New ways to fail

Autonomous systems can be attacked in ways that conventional hardware simply cannot. Their perception can be spoofed or deceived, their underlying models manipulated or poisoned, their communications jammed, hijacked or fed false data. An adversary who successfully corrupts the intelligence directing a system may never need to defeat the system itself in any conventional sense — they can simply turn it.

The stakes of such a failure are unusually and uncomfortably high. When the compromised system can act directly in the physical world, a security failure is no longer merely a data breach to be disclosed and patched but a genuine safety and strategic one, potentially with irreversible consequences. The cost of getting security wrong is measured in a fundamentally different and graver currency here.

Security as a design constraint

This grim reality is forcing a real shift in how such systems must be conceived and built. Security can no longer be treated as an afterthought, layered on at the end of development once the capabilities work; it must be a first-order constraint that shapes the architecture from the very beginning. Robustness against manipulation and deception becomes as important a design goal as raw capability or performance.

That is a genuinely demanding standard to meet in practice. Building autonomy that is both powerful and resistant to subversion requires deep expertise spanning artificial intelligence, security engineering and the unforgiving, high-consequence requirements of defense — a combination of skills that is scarce, expensive and not easily assembled. The hardest part is making the two goals coexist.

Building trust into autonomy

The central challenge, then, is making autonomous systems trustworthy enough to deploy responsibly in the first place: resilient to determined attack, predictable and bounded in their behaviour, and constrained in what they are permitted to do without human confirmation. Securing autonomy is becoming every bit as important and difficult as building it, and the two cannot sensibly be separated.

How well the field collectively meets that challenge will shape not only narrow military effectiveness but the much broader and more consequential question of how much we can safely delegate to machines that act on their own in the world. The answer will define the limits of autonomy long after the current generation of systems is obsolete.