IBM's Project Debater loses live debate to human champion

Artificially-intelligent debate software made by IBM fell just short of beating a former European debate champion, but the software’s ability to understand both sides of a problem is likely to win it enterprise fans.

IBM panel discusses the live debate.

In a precursor to the opening of IBM’s Think 2019 conference, Project Debater - an IBM Research-built system - took on human debate champion Harish Natarajan in a live forum in San Francisco.

Project Debater has only been out in the open since June 2018, but Big Blue is expending significant effort in developing its capabilities.

The reason for that is now clear: IBM sees the technology moving beyond debating and firmly into enterprise environments where it could assist people to understand complex problems and find an appropriate answer.

So, let's have an argument over AI...

Though IBM would no doubt have liked to beat the human -  as it has done previously against human chess and Jeopardy! champions - IBM Research was clear from the outset that today’s debate was intended as a capability demonstration more than anything.

“The vision around Project Debater is how do we develop the technology,” Project Debater manager Ranit Aharonov said.

“It’s not a question of is AI going to be better at debating humans; that’s not really an interesting goal.

“The goal of this demonstration and of building this technology was to set [a goal] that’s far away and that enabled us to develop technologies to find relevant information within a mass of text, organise it and bring it to a position that’s digestible to humans, in order to drive better decision making more quickly.

“So what’s going to be a win for us here is if people come out of this room saying, ‘we can see that this interaction is enriching the way we can make decisions in the future’.”

IBM appeared to achieve that goal.

Project Debater's physical representation. A panel of four data scientists oversaw its operation.

The art of persuasion is goes beyond facts

Natarajan - who took on the machine - said he was struck by the amount of information it could grasp, contextualise and form into a cogent argument.

“I think if you take some of those skills and add it to a human being who can use it in slightly more subtle ways, I think it could be incredibly powerful,” he said.

Natarajan took the victory based on the percentage of people whose opinions were altered by his arguments. He saw a 17 percent uplift in measures taken pre- and post-debate.

However, when audience members were asked which of the two debaters “better enriched your knowledge”, the answer came back clearly as Project Debater.

Watson will question you now

Project Debater uses Watson speech recognition to understand what its opponent is saying. It then tries to capture “the gist” of the opponent’s arguments before plumbing the depths of its memory for counterpoints.

“The system is capable of pinpointing relevant information within a massive collection - about 10 billion sentences are in the memory of this system,” Project Debater principal investigator Noam Slonim said.

“It needs to pinpoint these little pieces of text that are relevant to the topic, that are argumentative in nature, that hopefully support our side of the debate, and then somehow glue them together into a meaningful argument, which is very difficult for a machine to do.

“Another part of the system uses a unique production of more principled arguments that are relevant to the topic.

“Finally there is the listening and comprehension part,” he noted, getting the gist of nuanced arguments in order to be able to refute them.

Aharonov noted this was a key aspect that set Project Debater apart from other IBM technologies that have previously tackled AI challenges.

“If you think about grand challenges in the past around AI, these were often cases where there was a factual question and a clear winner [and right and wrong answer],” she said.

“When you think of a debate, this is something where the winner is not clear.”

Beyond binary arguments

Aharonov believed that would make the technology underpinning Project Debater attractive to enterprises.

“Technology like Project Debater can debate both sides so it could help you understand both sides of a problem, pros and cons, so you have a wider view of the topic and can make a better decision,” she said.

IBM Research director Dario Gil concurred: “We believe there is great potential in having artificial intelligence that can understand us [and] be more transparent and explainable. The more we can trust it, the more we can rely on it to help us make better decisions.

“There is hope behind the technology. We envision a future of the technology well beyond the podium, helping people to reason, to build well-informed arguments and to make better decisions.”

It isn’t clear when IBM might be able to commercialise elements of Project Debater for use in enterprise environments.

However, it mirrors the path that IBM Watson took to production usage. Watson owes its existence to IBM Research, where it was developed before being spun into commercial offerings.

System takes Aussie debate notes

The full debate is available online to replay.

It’s worth noting that Project Debater called on two pieces of Australiana in its bid to present a compelling argument for the topic, which was that preschool should be subsidised.

“Former Prime Minister Gough Whitlam said in 1973, ‘Preschool is the greatest single aid in removing or modifying the inequalities of background, environment, family income or family nationality”,” Project Debater told the audience during its opening remarks, borrowing from a 1972 speech.

Debater also went on to say cite a 2013 study “by the Melbourne Institute of Applied Economic and Social Research”, which it said “shows that attendance at preschool has a significant positive impact on later NAPLAN outcomes, particularly in the domains of numeracy, reading and spelling.”

These may be slightly complicated supporting points, given both lacked some context in the way they were delivered.

Gil noted prior to the debate kicking off that the machine was never going to be perfect.
“Project Debater was never trained on this topic,” he said.

“You may hear Project Debater repeat itself unnecessarily or make mistakes. This is because it’s an AI system, and AI systems are far from perfect.”

Ry Crozier is attending IBM Think 2019 in San Francisco as a guest of IBM.