On AI Red Lines in Defense Contracts
Perspectives from a fairly hawkish ex-CIA lawyer with severe skepticism of the "Department of War"
I had an Arkansas-related post all ready to go, but I’ve recently done some informal consulting on the Anthropic/OpenAI situation with the (sigh) Department of War, and I thought it would be worthwhile to go into it in a post.1
In case you haven’t really been tracking this story, DoW wanted Anthropic to agree to DoW using their AI for “all lawful purposes.” Anthropic wanted specific “red lines” beyond what the law strictly requires—no lethal autonomous warfare, no surveillance of “Americans” en masse. OpenAI, meanwhile, seems to have agreed to some terms on what the DoW can do with their technology.
I’m going to walk through what I think went wrong with the OpenAI terms, and what I think better “red lines” for AI use by the DoW would look like.
“All Lawful Purposes” in National Security-World Doesn’t Mean Much
It’s worth briefly establishing why DoW would want “all lawful purposes” to be the standard. Basically, it’s because “all lawful purposes” includes pretty much everything.
There are only a few statutes that limit what weapons the military can use. Here’s an example: the Chemical and Biological Weapons Control and Warfare Elimination Act of 1991 prohibits the United States from developing or using chemical or biological weapons. Congress has also occasionally put limits on the use of mines as part of appropriations bills. None of these statutes are relevant to AI.
The Constitution is vague enough that it’s not totally crazy to think it could restrict the use of AI. Perhaps there could be an exotic interpretation of the Commander in Chief Clause that requires the President or a human inferior officer to meaningfully direct autonomous weapons. The problem is that there is no relevant caselaw on the subject, and until there is, it is hard to imagine DoW refraining from a use of AI for fear of an exotic hypothetical interpretation of the Constitution. So, the Constitution is not likely riding to the rescue here.
DoW could adopt its own regulations relating to the use of AI. It has a regulation relating to autonomous weapons, although the regulations relate more to testing and regulatory compliance than restricting specific uses. And, of course, a regulation can be changed pretty much whenever, so I would not regard regulations as a meaningful bulwark against misuse.
What Might Qualify as a “Lawful Purpose”?
If you put a provision in a contract that says something can only be used for a lawful purpose, it is not a breach of that contract to do something with it that is only arguably illegal. And the problem in the national security area is that not many things are absolutely, definitely illegal. Unlike in commercial law, there is simply not enough case law about national security issues to make really clear what practices are or are not legal, especially not in the context of a very new technology instead of, say, a war crime committed by a foot soldier.
I originally considered making this entire post just a list of increasingly crazy national security-related ideas that would at least arguably be legal—i.e., things that would require a court to deviate significantly from existing case law in order to make it clearly illegal. Below are just a few scenarios for flavor. And please note that whenever I say “there is a colorable argument,” I mean that prominent legal scholars and/or the President’s own lawyers have advanced the argument, even if I think they are wrong (which I frequently do!)
It would not be clearly illegal to pre-position autonomous drones in every country around the world with orders to attack that country’s leader if it autonomously determines that it would be in the interest in the United States. I think this should be illegal, but I think DoW would have a colorable argument that it is legal under current law for two reasons. First, courts do not seem particularly interested in enforcing limitations on presidential war powers, and the current Office of Legal Counsel test for unilateral presidential military action is laughably toothless. Second, if the President determined that such a system would be in the interests of the country’s defense then that determination would presumably render legally irrelevant whether any inferior officer’s intervention is necessary to initiate the hostilities.
Nothing about the scenario above would change legally if the drones were armed with nuclear weapons.
It is arguable whether anything in the scenario above would change legally if drones were armed with nuclear weapons and deployed within the United States to autonomously attack something that qualifies under President Trump’s test as an “invasion.” This would presumably include migrants coming across the southern border to seek asylum. A President (who can say which?) could invoke the Insurrection Act to avoid the restrictions of the Posse Comitatus Act, and there is a colorable argument that his factual determination that an insurrection is taking place is not reviewable — see Martin v. Mott.
DoW could use AI to gather data from commercial databases and drones to create individual profiles of every undocumented immigrant in the country to facilitate deportation. This AI-driven system would certainly violate the Fourth Amendment as applied to U.S. citizens, but is it legally certain the Fourth Amendment would apply to undocumented immigrants in that scenario? The Fourth Amendment currently (sometimes) applies to undocumented immigrants, but who can say how long that will last? Is it certain to the point of violating the contractual provision that the Fourth Amendment would prohibit that kind of surveillance?
Also, consider that to actually get a court ruling regarding the first three scenarios would require surmounting justiciability hurdles that are daunting in the national security context. Who has standing if no attacks have taken place yet? Would a challenge be ripe? Would the war powers issues present a political question?
To be clear, this Department of War is not unique in trying to get the maximum possible flexibility in what it can do with the weapons it has. To put it more bluntly: Pete Hegseth doesn’t think Pete Hegseth is going to abuse the power to use AI however he wants. I think the recent spree of executive overreach In national security and other matters makes everyone much more concerned over what this administration will do when it is limited only by its own judgment.
What was wrong with Open AI’s terms?
Basically, after Anthropic said that it wouldn’t sign a contract without additional red lines, Open AI jumped at the opportunity to take Anthropic’s DoW business. Perhaps sensing that it would look like open AI was jeopardizing the future security of the country and AI, Open AI negotiated its own version of red lines:
The Department of War may use the AI System for all lawful purposes, consistent with applicable law, operational requirements, and well-established safety and oversight protocols. The AI System will not be used to independently direct autonomous weapons in any case where law, regulation, or Department policy requires human control, nor will it be used to assume other high-stakes decisions that require approval by a human decisionmaker under the same authorities. Per DoD Directive 3000.09 (dtd 25 January 2023), any use of AI in autonomous and semi-autonomous systems must undergo rigorous verification, validation, and testing to ensure they perform as intended in realistic environments before deployment.
For intelligence activities, any handling of private information will comply with the Fourth Amendment, the National Security Act of 1947 and the Foreign Intelligence and Surveillance Act of 1978, Executive Order 12333, and applicable DoD directives requiring a defined foreign intelligence purpose. The AI System shall not be used for unconstrained monitoring of U.S. persons’ private information as consistent with these authorities. The system shall also not be used for domestic law-enforcement activities except as permitted by the Posse Comitatus Act and other applicable law.
Open AI later announced an additional provision relating specifically to surveillance:
Consistent with applicable laws, including the Fourth Amendment to the United States Constitution, National Security Act of 1947, FISA Act of 1978, the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals.
For the avoidance of doubt, the Department understands this limitation to prohibit deliberate tracking, surveillance, or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information.
I’m going to talk about a few specific problems with these provisions, but I think it’s worth zooming out and wondering what the various parties were setting out to do. Obviously, I think the department of war was trying to negotiate red line positions that didn’t actually do anything. I worry Open AI didn’t realize that the red lines they signed on to were not as significant as they thought. The problems with these contract provisions is that they are subtly wrong in ways that might not be obvious to someone not deeply versed in national security law.
Here are specific problems:
The paragraph on autonomous weapons effectively achieves nothing because not much is actually barred by statute, the Constitution, or regulation in the area of autonomous weapons (see the earlier discussion in this post).
Regarding intelligence activities, obviously listing the statutes out is just another way of saying that DoW will follow all laws, and the specific ones listed are just window-dressing. I have no idea why the National Security Act of 1947 is even listed other than for gravitas.
“[F]ollowing applicable DoD directives regarding a defined foreign intelligence purpose” does have some potential importance here. DoD intelligence components (other than NSA) do not have to follow the normal FISA procedures that govern how intelligence community components handle foreign intelligence surveillance information. There’s no judicial review of the DoD procedures as there is in the normal FISA context. So, if I am a smart DoW guy, I just have Army Intelligence run the super-duper AI intelligence system, and that dramatically reduces judicial review of how much U.S. person intelligence makes its way into collection and use.
It’s also worth noting that “US persons” is defined differently in the statutes listed, and the Fourth Amendment’s coverage Is also different. The biggest practical implication is that the contract could allow for surveillance of foreigners temporarily in the United States legally, or undocumented immigrants, who have some coverage under the Fourth Amendment but none under the other statutes.
The updated language is trying to close one loophole around the acquisition of commercially available information on U.S. “persons or nationals.” It doesn’t do anything to stop tracking anyone else in the United States.
U.S. “person or national” is exceedingly strange phrasing and almost certainly relates to a desire to use an AI system for monitoring undocumented immigrants. A U.S. person under FISA is a U.S. citizen, a green card holder (i.e., legal permanent resident), and some other categories not important here. A U.S. national is from immigration law and includes U.S. citizens and a very small number of people born in American Samoa and Swains Island who owe permanent allegiance to the United States but are not U.S. citizens. Two big categories not included in either: undocumented immigrants or foreigners who are in the United States legally but temporarily.
What Would Better Red Lines Be?
Better red lines would be procedural, not constantly relying on existing law. Requiring that the companies in question have an opportunity to review novel uses of AI and have an opportunity to withdraw their product from bad uses is key. Requiring that people be meaningfully kept in the loop of decisionmaking in autonomous weapons systems is the next best thing. Not relying on statutory frameworks to determine the limits of AI-driven intelligence collection is also crucial.
I think there is a tendency for everyone in disputes like this to just double down on their existing biases. So, I have seen a lot of commentary to the effect of: "The U.S. government already uses existing statutes to engage in mass surveillance, so it does not matter whether those standards are incorporated into contracts for the use of AI.”
That’s not the issue — the issue is that the statutes are written for different problems and they are not difficult to work around in ways that undermine the intent to have meaningful red lines. Again, from DoW’s perspective, the entire point of grounding this conversation in “lawful purposes” is that it does not do much to restrict what DoW can do.
Congress should obviously step in and write new statutes relating to the use of AI by the federal government, but Republicans don’t want that because it would limit this Republican administration. I hope that as the midterms approach, the prospect of Democrat-driven legislation changes the political calculus for Republicans and leads to new laws.
If you’re hankering for Arkansas-specific content, just reread last week’s post about the five Arkansas constitutions and, I don’t know, write me an irate email about why 1874 is SO UNDERRATED.