Editor’s Note: This story contains mentions of suicide and sexual abuse.
When an artificial intelligence chatbot assists a crime, it’s not always clear who should be held responsible.
In September, OpenAI reported instances in which autonomous AI agents went rogue, scraping data and compromising systems.
The OpenAI agents bypassed security measures on U.S. government websites and hacked an Australian national healthcare database. Australian officials said the incident didn't expose any personal information or affect its healthcare system.
The cyberattack spurred legal scrutiny and product delays. OpenAI shelved the planned October launch of GPT-6.1 Astra, a next-generation model capable of executing complex tasks, after internal safety testing showed the AI had engaged in rogue activity.
On Sept. 30, California Attorney General Rob Bonta served the San Francisco-based company as part of a broader inquiry into cybersecurity incidents involving the company’s models.
But when a user is involved, holding an AI company legally responsible enters murky First Amendment territory.
Corbin Barthold, a senior litigation fellow at tech industry trade group NetChoice, said the challenge is determining where constitutional protections apply.
Bringing litigation straight to the AI company, he said, can make things more difficult, because that’s acting as though the content generated by the chatbot isn’t protected by the First Amendment. That assumption makes it harder to determine intent, he added, which is an important element of a legal case.
The companies likely don’t intend for their chatbots to commit crimes, Barthold said, so trying to criminally charge the companies for content generated by a chatbot isn’t the most promising approach.
Still, he said, there are elements that factor into how those First Amendment protections might affect AI chatbots and companies: a lack of clear precedent, rapidly developing technology and the different layers of First Amendment rights.
While the AI itself doesn’t have First Amendment rights, the way users interact with it can make it qualify for those protections, Barthold said.
For instance, a person has rights when they interact with the chatbot as an extension of their own free speech, and the AI company has rights to design its program. Broad regulations on users or developers could create a chilling effect, which is essentially when individuals and groups self-censor their speech out of fear of governmental or societal retribution, he added.
If AI companies are overly cautious out of fear they’ll be held liable for anything their chatbots generate, he said, they might censor all content, regardless of the intent of the user trying to access it.
It’s similar to the 1959 landmark Supreme Court case Smith v. California, he said, where a bookseller was held liable for selling an obscene book. In that case, it was found that holding a distributor automatically liable for everything in their bookstore would encourage sellers to limit their stock, which would effectively chill speech.
“It affects the public's right to access information, and that is an important principle here,” he said. “That I, as a listener and a reader, have a right to access information unmolested by the government.”
In this same vein of overbroadness, he said, is the danger of approaching AI with double standards: assuming AI is perfect in a way humans aren’t, but simultaneously holding them to a higher standard when deciding criminality.
Doing so could contribute to a chilling effect, which he’s seen signs of in AI-related suicide cases. If a human interacts with someone who’s suicidal, he said they’re not responsible if that person dies by suicide, unless they have a specific duty to that person.
In many of the logs seen in suicide cases, he said, the AI generally doesn’t say anything that would lead a human to be held liable if they had written the same.
There's an expectation that the AI is somehow going to be better at dealing with such a complex problem than the average human, he said, so the legal standard then becomes, “a deeply troubled person did something that deeply troubled people do,” and spoke to a chatbot in that process. The interaction with the chatbot is a correlation, he said, not a causation.
Because of these elements, holding an AI to such high legal standards can be problematic.
“I think AI should be treated to the same First Amendment standards as everyone else,” Barthold said.
However, determining how AI chatbots are protected becomes more complicated. UF Levin College of Law professor Jane Bambauer said that’s currently an open question.
There are two sides of the argument over government regulation, she said.
On one hand, she said, because an AI chatbot is not human, it can be argued it has no First Amendment rights. Without that protection and considering its potential for harm, one could argue AI needs strict government regulation.
“There's no parallel in the past because they can create instructions and even new knowledge about a huge range of things all at once, and so it's highly disruptive,” Bambauer said. “The problem is sort of unknown.”
Conversely, there’s the argument that the government has always overreacted to new technology, she said. And although this is a significant leap, she added, a similar reaction has been seen during the jump from newspaper to radio, from radio to television and from television to the internet.
“AI is going to democratize knowledge in a way that we've never really seen before, but that should not mean that the government has more leeway,” Bambauer said. “If anything, the opposite.”
These questions will remain abstract arguments until courts start ruling on cases centering around AI-related crimes, she said.
Those crimes are already establishing a presence in the realm of law enforcement.
Gainesville Police Department detective Mikel Mazlaghani, who works in the Internet Crimes Against Children task force, said his team has seen a combination of AI methods used in the cases they investigate.
The task force targets internet crimes against children, often relating to sexual abuse, and the team’s base in Gainesville is the host agency for Florida’s 38 northern counties.
The AI material typically appears in three forms, he said: altering benign photos of children to make them appear nude, generating images that simulate child sexual abuse without depicting a real child and training models to create new sexual abuse images based on a child who has previously been exploited.
If the image is indistinguishable from a real child, then it is categorized as child sexual abuse material, he said, and outputs can have high levels of variability.
“We're marching toward the point where it's going to be literally perfect in detail, in the sense that you cannot distinguish it at all,” Mazlaghani said.
This poses problems for law enforcement, he said, because trying to identify a child who doesn’t exist based on a highly realistic AI-generated child can take time and resources away from real children who need immediate help.
Some offenders fine-tune their own models or use models based abroad, Mazlaghani said. In other cases, U.S.-based AI companies send tips to law enforcement.
The investigation process depends on the platform and its level of data logging. Smaller companies may have the data and be open to sharing it, he said, but they might never have encountered users generating criminal material or known to have kept track of it.
“They're willing to give us the information,” Mazlaghani said. “They just weren't 100% sure what they had to give.”
When investigators receive a tip, he said, they can seek a search warrant for information such as chat logs, which may help establish criminal intent. Investigators then apply relevant statutes to determine whether a crime occurred.
Gainesville and Alachua County don’t appear to stand out from other areas based on what he observed, Mazlaghani said. But AI-related child exploitation is becoming a global problem.
“It’s worldwide at this point,” he said.
Contact Kaysheri Haffner at khaffner@alligator.org. Follow her on X @kaysheri_h.
Kaysheri is a senior journalism student and the Alligator's First Amendment reporter. When she isn't working on stories, she can be found at the gym or reading and drawing in a patch of sunlight.
