For years, artificial intelligence has proven that it can outperform humans at analytical tasks, but is less able at skills like intuition and reasoning. Stanford University scientists investigated whether neural networks like GPT-3.5 can master Theory of Mind (ToM) tests designed to analyze cognitive ability to predict the actions of others. The results show that GPT’s ToM capability was introduced spontaneously in recent years, and the latest iteration delivered results comparable to those of a 9-year-old human.
The AI revolution is upon us as super-advanced machines continue to master the subtle art of being human at breathtaking (worrying) speed. It’s old news that AI has beaten humans at their own games, particularly things like chess and go, but there’s more to our brains than checking a king. There are more subtle skills like inference and intuition — muddy, almost subconscious concepts that help us understand and predict the actions of others.
But with the advent of advanced AI platforms like Open AI’s Generative Pre-Training Transformer (GPT), those lines between human and machine are also beginning to fade.
A new study conducted by Michal Kosinski, a computational psychologist at Stanford University, used multiple iterations of OpenAI’s GPT neural network – from GPT-1 to the latest GPT-3.5 – to generate “Theory of Mind” (ToM ) tests, a series of experiments first developed in 1978 to measure the complexity of a chimpanzee’s mind in order to predict the behavior of others.
These tests involve solving normal, everyday scenarios from which people can easily derive the result. For example, one scenario is to incorrectly label a bag of popcorn as “chocolate,” and then the test asks the AI to infer what the human’s reaction will be when the bag is opened. Kosinski’s team used “sanity checks” to analyze how well GPT networks understood the scenario and the predicted human response. The results were published online on the preprint server arXiv.
While early versions of GPT-1, first released in 2018, performed poorly in testing, the neural network showed amazing improvements spanning different iterations, and by November 2022 spontaneously developed the “Theory of Mind” capability of a 9-year-old people ( the release of the latest GPT-3.5). According to Kosinski, this could be a “game changer” for AI, as the ability to understand and predict human behavior would make these engines much more useful.
Kosinsky writes:
“The ability to ascribe the mental state of others would vastly improve AI’s ability to interact and communicate with humans (and each other) and allow it to develop other skills that rely on theory of mind, such as empathy, moral judgment or self-awareness.”
The ability to program empathy and morality could be a huge boon for things like self-driving cars, such as putting a driver at risk to save the life of a child crossing the street.
One question that remains is whether these neural networks participate in ToM intuition or bypass it by “using some unknown language patterns”. This could explain why this ability shows up in language-based models that strive to understand the subtle nuances of human language.
But that also begs the question: Can humans also perform this language trick and just don’t know it? Kosinski argues that by studying the cognitive abilities of these AI systems, we are essentially studying ourselves, as how the human mind works is still shrouded in scientific mysteries.
“The study of AI could provide insights into human cognition,” writes Kosinski. “As AI learns how to solve a wide range of problems, it may evolve mechanisms similar to those used by the human brain to solve the same problems.”
In other words, if you want to figure out how something works, build it from scratch.
Darren lives in Portland, has a cat, and writes/edits about science fiction and how our world works. You can find his previous stuff at Gizmodo and Paste if you look hard enough.
