Research index Find a question, concept, or source.

Loading the research index…

Machine War · IX

A probability score cannot carry moral responsibility for a human life.

A high-level analysis of AI-assisted targeting, distinction, proportionality, automation bias, uncertainty, civilian harm and meaningful human judgment.

3 min read 4 sections 6 sources 01 Sep 2026
  • algorithmic targeting
  • human judgment
  • distinction
  • proportionality
  • civilian protection

Short answer

AI can help organize large volumes of information, but a system’s confidence score cannot replace the legal and moral work of distinction, proportionality, precaution and contextual judgment. High-consequence decisions require people who can understand uncertainty, challenge the data, refuse the recommendation and remain accountable afterward.

ReviewedSeptember 1, 2026
Evidence6 page-specific references
Claim modeSourced synthesis
Source mixInstitutional / legal 2 · Academic / scholarly 1 · Research institute 1 · Public record / standard 1 · Archive / museum 1
MethodMethodology · Source map

01

Quantification can hide normative choices

A model output may look numerical and objective even though it depends on choices about data, labels, thresholds and acceptable error. Those choices can move moral judgment upstream into technical design.

Transparency requires making those assumptions contestable rather than treating the final score as neutral fact.

02

Civilian protection is not a confidence threshold

The law of armed conflict demands distinction and proportionality, but real civilian environments contain uncertainty, dual-use systems and cascading effects that resist simple classification.

A model can assist analysis without becoming the authority that decides what uncertainty society is willing to impose on another person.

03

Accountability must survive delegation

Responsibility should not disappear because a recommendation came from software. Institutions need traceability, review and post-event investigation that can identify who authorized the model, the threshold and the action.

04

Accountability cannot be transferred to a confidence score

International discussions of autonomous weapons have repeatedly returned to the same institutional principle: human responsibility for decisions involving weapons remains with people and organizations rather than migrating into software. The 2019 CCW guiding principles state this explicitly, while the ICRC frames loss of human control and judgment as a legal, humanitarian and ethical concern.

This is also why nominal human approval can be misleading. A person who receives only a machine conclusion, lacks contradictory evidence, cannot understand the system’s uncertainty, or has too little time to intervene may remain formally present while substantive judgment has already shifted elsewhere.

Historical casebook

1 case
Persian Gulf · 3 July 1988

Iran Air Flight 655 showed that advanced automation does not remove scenario-driven human error.

Before

USS Vincennes operated the Aegis combat system during a tense naval engagement in the Persian Gulf. Its combat information center combined radar tracks, identification signals, computer-assigned track numbers and human reports under severe time pressure.

Turning point

Iran Air Flight 655, a scheduled civilian Airbus, was misidentified as a threatening military aircraft and shot down, killing all 290 people aboard. Later official histories describe miscorrelated identification information, contradictory cues and a mistaken report that the aircraft was descending even though system data showed it climbing.

Aftermath

The official investigation and later Navy historical review treated the event as a chain of human, informational and contextual failures occurring inside a highly automated command environment rather than as a simple machine malfunction.

Historical significance

Sophisticated automation can organize information without resolving ambiguity. When time is compressed, interfaces and expectations can shape which evidence people notice, trust or misread.

Primary & supporting links

Read beyond the summary.

International Review of the Red Cross — erosion of humane judgment Academic / scholarlycambridge.org NATO CCDCOE — Tallinn Manual Research instituteccdcoe.org United Nations CCW — 2019 Guiding Principles on emerging technologies in lethal autonomous weapons systems Institutional / legaldocuments.unoda.org ICRC — Position on autonomous weapon systems Institutional / legalicrc.org U.S. Department of Defense — 2023 update to Directive 3000.09, Autonomy in Weapon Systems Public record / standarddefense.gov U.S. Naval History and Heritage Command — USS Vincennes tragedy and Iran Air Flight 655 Archive / museumhistory.navy.mil

Direct answers

Frequently asked questions

Can AI ever assist lawful targeting?

It can assist information processing, but lawful and ethical use still requires contextual human judgment, distinction, proportionality, precaution and accountability.

Why is this related to cognitive liberty?

Because both reject the idea that an opaque system may define a person conclusively without explanation, contestability or human responsibility.