The model, when prompted to reason out loud, reasons to itself: I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.

[...]

Fake facts

Outputting false information is another problem. Luccioni says that models like GPT-4, which exist to predict the next word in a sentence, can’t be cured of coming up with fake facts — known as hallucinating. “You can’t rely on these kinds of models because there’s so much hallucination,” she says. And this remains a concern in the latest version, she says, although OpenAI says that it has improved safety in GPT-4.

Without access to the data used for training, OpenAI’s assurances about safety fall short for Luccioni. “You don’t know what the data is. So you can’t improve it. I mean, it’s just completely impossible to do science with a model like this,” she says.

The mystery about how GPT-4 was trained is also a concern for van Dis’s colleague at Amsterdam, psychologist Claudi Bockting. “It’s very hard as a human being to be accountable for something that you cannot oversee,” she says. “One of the concerns is they could be far more biased than for instance, the bias that human beings have by themselves.” Without being able to access the code behind GPT-4 it is impossible to see where the bias might have originated, or to remedy it, Luccioni explains.

Ethics discussions

Bockting and van Dis are also concerned that increasingly these AI systems are owned by big tech companies. They want to make sure the technology is properly tested and verified by scientists. “This is also an opportunity because collaboration with big tech can of course, speed up processes,” she adds.

Van Dis, Bockting and colleagues argued earlier this year for an urgent need to develop a set of ‘living’ guidelines to govern how AI and tools such as GPT-4 are used and developed. They are concerned that any legislation around AI technologies will struggle to keep up with the pace of development. Bockting and van Dis have convened an invitational summit at the University of Amsterdam on 11 April to discuss these concerns, with representatives from organizations including UNESCO’s science-ethics committee, Organisation for Economic Co-operation and Development and the World Economic Forum.

Despite the concern, GPT-4 and its future iterations will shake up science, says White. “I think it's actually going to be a huge infrastructure change in science, almost like the internet was a big change,” he says. It won’t replace scientists, he adds, but could help with some tasks. “I think we're going to start realizing we can connect papers, data programmes, libraries that we use and computational work or even robotic experiments.”