British mathematician Alan Turing in 1950 proposed a test to assess whether a machine can demonstrate human-like intelligence. In the classic version, a person communicates simultaneously with a live interlocutor and with a machine, not knowing which is which, and tries to determine where the machine is.
In a new study, philosopher Diana Proudfoot of the University of Canterbury (New Zealand), together with computer science doctoral student Sharon Temtsin and professor Christoph Bartneck, argue that many experiments called "Turing tests" do not actually follow the original design. The work was published in the journal Philosophy of Science. According to the authors, they were the first to implement Turing's original conceptual design experimentally.
The researchers studied Turing's texts from 1948 to 1952, as well as early commentaries and historical sources, to understand how the "imitation game" was supposed to work.
"Such features that are often associated with the Turing test, such as the five-minute limit and the idea that a machine passes the test if it deceives 30% of judges, were not conceived by Turing as passing rules," Bartneck explains. According to him, Turing envisioned a game with three participants - an interrogator, a human and a machine - with free questions and no fixed time limit. "This substantially changes the rules of the game, and therefore the results of the test."
To test their interpretation in practice, the researchers conducted an experiment with GPT-4-Turbo, an artificial intelligence language model developed by OpenAI (the company that created ChatGPT). Participants conversed freely with both a human and the AI until they were ready to decide which of them was the machine.
Of the 37 participants, 36 correctly identified GPT-4-Turbo - that is 97%. In addition, 33 participants (89%) took longer than five minutes to decide. The average conversation lasted 14 minutes, the longest - 46 minutes. The study raises the question of whether short tests give people enough time to meaningfully check an AI system.
"Turing's open format gives the interrogator the opportunity to check whether the system truly understands the topic, rather than simply producing convincing answers," says Bartneck. "If this is taken into account, we would argue that no large language model has yet demonstrated passing a test that follows the design of Turing's original work."
The researchers emphasize: their results do not mean that all current or future AI models will fail the test. The experiment used GPT-4-Turbo, and newer models may show different results.