Loebner Prize: an annual Turing test chatbot competition

Format and rules

Typically, the contest pairs each judge with two unseen conversational partners: one human and one chatbot. Conversations are text-only and limited in length. Judges attempt to decide which partner is human. Scoring ranks chatbots by how often they are mistaken for humans. Over the years the precise timing, number of judges and other rules have varied, but the essential idea echoes the original Turing test thought experiment.

History and prizes

The event was established in 1990 by Hugh Loebner. The first recorded winner of the Loebner Prize was an entry called "Whimsical Conversation." The competition has offered an annual award (often with a bronze medal and cash prize) and has historically announced a larger, higher-level prize reserved for a program that could pass a less restricted form of the Turing test. That larger prize has remained unclaimed, and the annual contest has continued as a stimulus to chatbot development.

Reception and criticisms

The Loebner Prize has attracted attention for promoting conversational agents, but it has also drawn critique from parts of the AI community. Common criticisms include:

• the short, game-like dialogue format favors scripted or evasive tactics over deep understanding;
• judges’ expectations and the quality of confederate humans can skew results;
• success in the contest does not necessarily indicate broad or robust intelligence.

Proponents argue the contest raises public awareness and provides a useful benchmark for incremental progress in language technology.

Significance and examples

While the Loebner Prize is not a definitive test of machine intelligence, it remains one of the better-known public experiments inspired by Alan Turing’s ideas. The contest has encouraged the creation and refinement of many conversational systems and helped popularize chatbots beyond academic settings. For background on the underlying concept, see the classic Turing test discussion and general resources on chatbots.