Monday, June 27, 2011


Around April, a student asked me if I knew of a game called Cleverbot. I didn't. He tells me that the goal of the game is to convince Cleverbot that it is a robot by:
  1. Making Cleverbot admit that it's a robot
  2. Trick Cleverbot into saying that it's a robot.
He wanted me to try it, so I took a look. Turns out it's not a game. It's a chatbot that tries to mimic a human in conversation. Here are some of the screenshots from my chat with Cleverbot. You decide how human it appears (my text in black and Cleverbot's is in blue):

You can see the chat above either as a win or fail. You could see it as a fail for Cleverbot because I tricked it into saying that it is a robot. Or you could see it as a win for Cleverbot because Cleverbot seems to have figured out my logic and answered me correctly. However, I think it's a qualified win. Real humans would've understood the context and clarified their meaning by saying that "I am a robot, but by robot I really mean that I am human" so as to clarify that they knew that it was a trick to get them to say that they are robots.

So I continue. After a few more screens of chat we get:
In this context, the "Be quiet" appears to be an acknowledgement by Cleverbot that it is a robot. Behaviorally, it's very human. Logically, though, it admitted to being a robot. When I ask for clarification with "So you agree?" the "Yes" from Cleverbot confirms it. It's possible Cleverbot does not check previous questions for context and may just agree in general. Either way, it's another sign that it is not human.

I try another way of getting it to explicitly state or imply that it is a robot.
Seems Cleverbot employs misdirection tactics to avoid answering in a way that gives it away. So we continue with our chat. Somehow it brings up Harry Potter and we have this exchange:

Not bad! It guesses that Jean-Luc is a female. In English, Jean is more common as a female name. It falls for the I_am_a_robot trick for my name. What it shows though is there's some retention or some attempt to use context and our chat history for information to be used in responses. Very human!

Cleverbot came with the following notice.
PLEASE NOTE: Cleverbot learns from real people - things it says may seem inappropriate - use with discretion, and at YOUR OWN RISK
Apparently it has won a Turing Test competition. According to the site:
A special version of the Cleverbot application has won the BCS Machine Intelligence Competition 2010, after taking part in a quick-fire Turing Test.

Cleverbot was running with notably more power behind it than is possible for the online version, with 24 separate instances conferring on their answer.

10 volunteers talked for 2 minutes each using a plain text interface, and the whole of the event audience voted on 'how human' each conversation appeared to be.

Cleverbot achieved an average rating of 42.1% human!
I was very impressed! My prior experiences with chatbots is that you can quickly figure out whether it is a chatbot with about 2 or 3 questions. Most aren't on topic or use misdirection or generic answers. Cleverbot's use of chats with humans to respond gets it a little closer. Behaviorally, some of the ways it answers also seem human.

You can judge for yourself how convincing it is. You can talk to Cleverbot here.

No comments:

Post a Comment