Autonomous weapons and human control over the use of force

  • Our news
  • Our values

From 16 to 20 November, the Palais des Nations in Geneva is hosting the Seventh Review Conference of the Convention on Certain Conventional Weapons. States Parties are discussing the possibility of opening negotiations on an international instrument to regulate the use of such weapons.

As weapons systems become increasingly autonomous, one question is becoming particularly pressing. How can those who authorise the use of force be held accountable when certain functions of the weapon are carried out autonomously? For C&V Defence, it is this issue of accountability that must be at the heart of the discussions in Geneva.

 

Human judgement in the use of force

Called upon to use force on behalf of the State, military personnel must exercise sound judgement in situations where their decisions may have life-or-death consequences. They must be able to assess whether, in the circumstances of the moment, a strike is consistent with the purpose and values of their mission. It is this capacity for judgement that distinguishes them from machines and may lead them to forgo a military advantage to spare lives.

But why do the new autonomous systems (implying those relying on an algorithm that models human logic to decide) undermine the ability to control the effects on civilian populations and the environment?

Indeed, prior to the use of AI, certain missiles – notably ballistic and cruise missiles – would autonomously follow their trajectory after launch without the operator being able to intervene at every stage. If civilians came close to the target, it might then be too late to abort the strike. The use of AI could therefore be regarded as a step forward, since, under certain conditions of use, the deployment of autonomous functions utilising AI could make it possible to assess (although imperfectly) the risk of collateral damage and thus adjust the precision of the strike or cancel the mission if it had been decided in advance that the consequences would be unacceptable.

Autonomy can thus provide better control over the execution of a strike that has already been ordered. But it can also intervene prior to its execution to assess, where appropriate, what is acceptable and what is not. What role, then, does human judgement retain in the decision to use force?

 

Accountability for the consequences of using an autonomous weapon

The question of accountability must indeed be raised whenever a weapons system is used, regardless of the technology employed. It becomes particularly acute when autonomous functions are involved in target selection or in triggering the strike.

However, it has so far been accepted that those who authorise the use of a weapons system must be held accountable for their decision. And before any technical debate takes place, it is this principle that must guide the discussion on the regulation of autonomous weapons.

A rule which stipulates that, for any breach of international humanitarian law or the law of armed conflict, it must remain possible to establish State responsibility and, where applicable, individual responsibility, in accordance with the rules applicable to each.

Consequently, just as with other weapons, the use of autonomous weapon systems to conduct a military operation must be assessed in the light of the law governing it, regardless of the cause in the name of which force is employed.

For those who authorise the use of autonomous weapons, this question of liability therefore raises a practical issue. What unacceptable consequences must they be able to anticipate before deciding to delegate the decision to an autonomous system, and what level of trust should they place in that autonomous system?

 

Anticipating the effects of an autonomous system

It is therefore essential that those authorising the use of an autonomous weapon can anticipate the effects it is likely to have, particularly on civilian populations and their environment.

For C&V Defence, this requirement must be considered from the very earliest stages of the system’s design. It is essential to define the conditions under which its AI-supported autonomous functions may be used and to verify, through representative scenarios, that their effects – primarily those likely to result in collateral damage – are sufficiently predictable and that the planned response measures are operationally viable.

However, in Geneva, the debate should not initially focus on technical issues but rather on the responsibilities that states intend to attach to the use of autonomous capabilities by the armed forces, including decision-support. It must lead to the affirmation that the autonomy of an AI-driven weapons system does not absolve users of responsibility for the consequences of its use.

 

For C&V Defence, this responsibility – affirmed and confirmed at United Nations level – must serve to legitimise and drive the development of autonomous systems, enabling us to understand and assess the level of trust that can be placed in these AI-based autonomous systems. It is indeed on this condition that those who decide to deploy these new capabilities will be able to continue to exercise their judgement and be accountable for their decisions by setting the limits of this autonomy.