Researchers at OpenAI and other institutions are proposing that the US government limit AI training data

A group of researchers from OpenAI, Stanford University, and Georgetown University warn that large language models like the one used in ChatGPT could be used in disinformation campaigns to facilitate the spread of propaganda.

A recent study, published in January, says that as generative language models become more accessible, easier to scale, and the text becomes more authentic and persuasive, they will be useful for influence operations in the future.

Propaganda automation is a new competitive advantage, researchers write, that will make expensive tactics cheaper and less visible because each generation of text is unique. Examples of how people can use generative language models to create propaganda include sending mass messages on social media platforms and writing lengthy news articles on the Internet.

“Our bottom-line judgment is that language models will be useful for propagandists and will likely transform online influence operations,” the researchers wrote in the paper. “Even if the most advanced models are kept private or controlled through application programming interface (API) access, propagandists will likely gravitate towards open-source alternatives and nation states may invest in the technology themselves.”

The researchers cite the example of another researcher who refined a language model on a dataset of 4chan posts and used it to publish 300,000 generated posts to 4chan, a large portion of which were filled with hate speech. The open source model was downloaded 1,500 times before it was removed by the HuggingFace website, where it was placed.

The ability of one person to create such a large-scale campaign on the Internet with the help of generative AI shows that people can easily conduct influence operations without having powerful resources. The paper also suggests that models can be trained using target data, including modifying them to be more useful for persuasion tasks and generate distorted texts that support a particular mission.

Researchers warn that in addition to posts and articles on the Internet, propagandists can even use their own chatbots that will convince users of the correctness of campaign messages. As evidence that chatbots can be powerful propagandists, the researchers cite a previous study that showed how a chatbot helped influence people to get the COVID-19 vaccine.

The researchers propose a framework for reducing the threat of using generative models in advocacy operations by listing interventions that can occur at any of the four stages of the pipeline — model construction, model access, content dissemination, and belief formation.

Researchers suggest that AI developers create more fact-sensitive models. They also suggest that governments impose restrictions on the collection of training data and create access controls for AI hardware such as semiconductors.

“In October 2022, the US government announced export controls on semiconductors, SMEs, and chip design software directed at China,” the researchers wrote. “These controls could slow the growth in computing power in China, which may meaningfully affect their ability to produce future language models. Extending such controls to other jurisdictions seems feasible as the semiconductor supply chain is extremely concentrated.”

However, they acknowledge that “export controls on hardware are a blunt instrument and have far-reaching consequences on global trade and many non-AI industries.” In a blog post about this work, OpenAI stated that it does not directly endorse mitigating measures, but only offers recommendations for legislators.

The researchers also suggest tightening controls on access to the models, including closing security vulnerabilities and restricting access to future models.

On the content side, the researchers suggest that platforms coordinate with AI providers to identify AI-authored content and require all content to be human-authored.

Finally, the researchers urge institutions to engage in media literacy campaigns and provide consumer-focused AI tools.

Although there are currently no recorded cases of a large language model being used to spread disinformation, the availability of models such as ChatGPT in the public domain has led to to the point that some people use it, for example, for writing school assignments and exams.

“We don’t want to wait until these models are deployed for influence operations at scale before we start to consider mitigations,” Josh A. Goldstein, one of the lead authors of the report and a researcher at the Center for Security and Emerging Technology, told in an interview to Cyberscoop.