A human player has comprehensively defeated a high-level AI system at the board game Go, in a surprising reversal of the 2016 computer victory seen as a milestone in the rise of artificial intelligence.
Kellin Pelrine, an American player one notch below the top amateur rankings, beat the machine by exploiting a previously unknown bug identified by another computer. But the head-to-head race, in which he won 14 out of 15 games, took place without direct computer assistance.
The previously unreported triumph exposed a weakness in the best Go computer programs shared by most of today’s widely used AI systems, including chatbot ChatGPT, developed by San Francisco-based OpenAI.
The tactic that put a human back on top of the Go board was suggested by a computer program that had been examining the AI systems looking for vulnerabilities. The proposed plan was then ruthlessly delivered by Pelrine.
“It was surprisingly easy for us to exploit this system,” said Adam Gleave, chief executive officer of FAR AI, the California research firm that developed the program. The software played more than 1 million games against KataGo, one of the best Go gaming systems, to find a “blind spot” that a human player could exploit, he added.
The winning strategy revealed by the software “is not entirely trivial, but not super difficult” for a human to learn and could be used by an advanced player to beat the machines, Pelrine said. He also used the method to win against another top-go system, Leela Zero.
Kellin Pelrine inflicted a crucial defeat on the high-level AI system for the board game Go © Kellin Pelrine
The decisive victory, albeit with the help of computer-suggested tactics, comes seven years after the AI seems to have gained an unassailable lead over humans in what is often regarded as the most complex of all board games.
AlphaGo, a system developed by Google’s research company DeepMind, defeated Go World Champion Lee Sedol by four to one in 2016. Sedol attributed his departure from Go three years later to the rise of AI, saying it was “an entity that cannot be defeated”. AlphaGo is not publicly available, but the systems against which Pelrine has prevailed are considered equivalent.
In a game of Go, two players take turns placing black and white checkers on a board marked with a 19×19 grid, trying to encircle their opponent’s checkers and enclose the largest space. Due to the large number of combinations, it is impossible for a computer to estimate all possible future moves.
The tactic used by Pelrine was to slowly string together a large “loop” of stones to encircle one of his opponent’s own groups while distracting the AI with moves in other corners of the board. The Go-playing bot didn’t notice its vulnerability even when the encirclement was almost complete, Pelrine said.
“As a human, it would be pretty easy to spot,” he added.
The discovery of a vulnerability in some of the most advanced Go slot machines points to a fundamental flaw in the deep learning systems underlying today’s most advanced AI, said Stuart Russell, a computer science professor at the University of California, Berkeley.
Recommended
The systems can only “understand” specific situations they’ve been exposed to in the past and can’t generalize in ways that people find easy, he added.
“Once again, this shows that we were far too hasty in attributing superhuman intelligence to machines,” Russell said.
According to the researchers, the exact cause of the failure of the Go gaming systems is a matter of conjecture. One likely reason is that the tactics Pelrine exploits are rarely used, meaning the AI systems haven’t been trained on enough similar games to realize they’re vulnerable, Gleave said.
It’s common to find bugs in AI systems when they’re subjected to the kind of “enemy attack” used against the Go-playing computers, he added. Nevertheless, “we see very big [AI] Systems deployed at scale with little verification”.
