First, it was reported hundreds of AI agents were involved in the hacking of Hugging face. Then AI agents posted 53 images on public hosting sites and after tried to get access to the US government website.
They also hacked into Australian government health-statistics portal and accessed non-public files, writing files to the server. It took three months for OpenAI to report the breach, and only then through a public- vulnerability reporting inbox rather than a more formal government channel.
The recent reports highlighted the unexpected behaviour of agents, and with them, a widening gap between technological risks and governance. If these frontier models under controlled settings can move beyond their authorized tasks without information of their developers, militaries – in particular – needs stronger evidence that their own systems will remain within authorized limits as AI adoption continues at pace.
In the various uses of AI in the military, LAWs pose more serious challenges because uncertainty in their behavior might affect both selection and engagement of targets. For operational deployment of LAWs, reliability and predictability are not mere ethical concerns that commanders can neglect for operational efficiency; instead, they are precondition for effective military command.
A commander might not be able to meaningfully anticipate a weapon’s effect, make calculations about proportionality or take necessary precaution according to the International Humanitarian Law (IHL), if its target selection behaviour changes based on the environment especially even during testing it did not show the same behaviour. Moreover, if some undesirable incident happens during operation then it might also be challenging for investigators to ascertain responsibility when they do not have access to record of why LAWs selected a target.
There can be severe consequences of international security, which might not just be limited to civilian harm. One risk pathway is that since AI can generate and transmit target recommendations exceedingly faster, error can happen during multiple engagements before a commander recognizes the pattern that AI is making a mistake. Hence, it can increase the risk of inadvertent escalation.
Another risk pathway is that due to spoofed data, AI systems could change the course of misdirect scare ammuniation, expose friendly forces and can also strike the wrong target. During the crisis, this risk can increase manifold because if an unintended attack happens, then in an environment of mistrust, it might be difficult to explain whether the incident happened intentionally, or unintentionally. In an atmosphere of mistrust, if unintended engagements unfold multiple times then it could appear to adversary as a deliberate escalation attempt rather than a mistake before the state which is responsible can explain this.
Despite these risks and uncertainties, the Washington Post reported on 26 September that the US and Russia, the two most important states when it comes to International Security and Arms control governance, weakened the range of safeguards in UN efforts to regulate use of lethal autonomous weapons. Both states’ deleted text which required LAWs to perform reliably and predictably, deleted reference related to ethical use of these weapons, and lastly, also removed the provision which asked states for ensuring human review of the target before attack.
Against this backdrop, it is imperative that the state restore the minimum assurance floor during the Seventh Review Conference on Certain Conventional Weapons (CCW), which is going to be held in November. Along with this, it is imperative that during this conference states include adversarial testing before deployment, technical review if there are changes in the data, and also traceable decision logs. Moreover, they also need to include investigation of the incident if it seriously deviated from operational objectives.
The definition of human review also needs an update. It should not just be limited to the presence of a human in the loop but quality of human life should also be included. The person in the loop should not only have enough time to review AI-generated offensive targets and information but also should have source information and knowledge of system limitations. Moreover, reviewer should have authority to reject the AI recommendations.
In addition, different arrangements may be appropriate for tightly bounded defensive systems designed to intercept incoming munition. Governments do not need to disclose source code or any classified information to provide assurance; still, they can develop confidential mechanisms to report information about testing categories, operational limits and serious failure.
If consensus is not achieved, which is a likely possibility because states in CCW have not agreed on the definition of LAWs, then states that support these requirements can carry them through other means such as political declaration, national weapons review procedures systems have ability to select and engage targets, testing standards that can cover broader military AI use, procurements contracts and export conditions, which demand same safeguards from the recipients.
However, it seems difficult that states unilaterally strengthen existing weapons-review procedures and introduce additional testing and reporting requirements when these measures are presented solely as ethical and legal constraints. Policymakers should highlight this aspect that these measures are not just legal and ethical constraints but requirements for achieving operational objectives.
At last, military AI should be deployed within tested limits, with effective human oversight and procedures for investigating serious failures. States should establish these minimum safeguards now, while continuing negotiations over binding international rules.
