Download Data Entry Protocol and File Format

SCHEDULE ONE
Final Application Information and Rules for the 2001 Loebner Prize presented by the Science
Museum
(These Rules will be distributed with suitable amendments each year)
The Loebner Prize Medal and cash award is presented annually to the designer of the computer
system that best succeeds in passing a variant of the Turing Test.
The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Hugh Loebner and
was first held at the Boston Computer Museum, Boston, Massachusetts, USA in 1991. From 2001,
the Competition is to be run by the Science Museum, London.
In accordance with the requirements of the Donor, as published in the June 1994 Communications of
the ACM, the winner of the US$100,000 Gold Prize must be prepared to deal with audio visual input,
and appropriate competitions will be held once Competitors have reached Turing's 50:50 likelihood
level of being mistaken for a human. An intermediate Silver Prize of US$25,000 will be offered for
reaching this level in a text-only test. There is also an Annual Bronze Prize which is awarded to the
designer of the "most human computer" as rated by a panel of Judges. In 2001, this prize will be
worth $2,000.
There is no entry fee.
Applications must be accompanied by e-mail protocols recording interactions between the computer
system to be entered and one or more human subjects. Protocols must not exceed 2500 words.
Applications must be submitted by e-mail to the Science Museum. The Science Museum may request
an opportunity to interact with Computer Entries. Transcripts and scores from the [1991, 1992, 1993
and 1994] [and 1995 to 1999?] Competitions may be obtained by contacting the Cambridge Center for
Behavioral Studies, tel+1 978) 369 222, e-mail [email protected].
Please review the
accompanying rules carefully and provide the following information:














Name of Applicant:
Age of Applicant:
Proof of parental consent if aged under 18:
Professional title of Applicant (if applicable):
Affiliation (if applicable):
Contact person (if applicable):
Postal address (with postcode and country):
Fax number (with international country and area code):
Telephone number(s):
E-mail address(es):
Computer software to be employed (optional):
Media to be employed for submitting software:
Enclose a protocol recording interactions between the computer system to be entered and
one or more human subjects (maximum 2500 words):
E-mail completed applications to: [email protected]
Sabiha Foster, Science Museum
1
OFFICIAL RULES
2001 Loebner Prize Competition in Artificial Intelligence
Presented by the Science Museum
1.
The objective of the Loebner Prize Competition in Artificial Intelligence is to identify the
computer system that can best succeed in passing a modern variant of the Turing Test. Judges will
attempt to distinguish computer systems ("Computer Entries") from one or more human beings
("Human Confederates") based on interactions with these Computer Entries.
2.
Contestants may be individuals, organisations, businesses, schools, corporations, institutions,
or other entities. Individuals need not have institutional affiliations. Contestants may be of any
nationality or age. Parental consent is required for Contestants who are aged under 18. Contestants
may submit only one entry during any year.
3.
If there are two or more Computer Entries there will be a Loebner Prize contest. The Medal
and cash award will go to the Contestant whose Computer Entry wins the highest score, based on the
decision of the Judges.
4.
If there is only one Computer Entry, the Loebner Prize Medal and the US$2000 cash award
will be awarded to that Entry.
5.
If there are no Computer Entries in the 2001 Loebner Prize Competition, the US$2000 cash
prize will be added to the cash award for 2002, making the award for 2002 US$4000.
6.
If a Computer Entry passes the text-only Turing Test the Contestant will be awarded the Silver
Prize of a Silver Medal and US$25,000.
7.
If a Computer Entry passes the Turing Test (dealing with audio visual input) the Contestant will
be awarded the Gold Prize of US$100,000 and a Gold Medal.
8.
The closing date for applications for the 2001 Competition is 25 June 2001. No more than
eight finalists will be chosen to compete in a (simultaneous and real-time) competition on 13 October
2001 at the Science Museum, London (date and location subject to change). Pre-selection will be
carried out by the Science Museum. Any inappropriate responses will be disqualified. The decision of
the Science Museum will be final.
9.
The selection process may entail interaction between Science Museum personnel and
Computer Entries. Contestants will be notified of the selection decision by 31 July
2001.
10.
Judges will be selected by the Science Museum and may include children, people with
disabilities, experts in Psychology, Linguistics or Artificial Intelligence, etc and are intended to be in
part representative of the community at large and part of the cognitive science community.
11.
It is the task of the Computer Entries to respond to the communications of the Judges in such
a manner as to imitate the responses of a human but any inappropriate responses will be disqualified.
The decision of the Science Museum will be final. The Museum is a public venue and the Competition
is widely publicized, the transcripts will be published, and all Contestants must assume responsibility
for their own Computer Entries. As in any normal conversation, both participants have a role in
determining the direction of the conversation and are free to decide what topics they do or do not wish
to discuss. Computer Entries must be prepared to communicate for an indefinite period of time.
12.
Judges will have at least one opportunity to interact with each of the Computer Entries (on the
computer terminals available concurrently) during the Competition. Judges will be allowed unrestricted
communications. They will be informed that at least one of the terminals is controlled by Human
Confederates and that at least two of the terminals are controlled by Computer Entries.
13.
One or more referees may be present to limit communications of the Human Confederates.
-2-
14.
Judges will not be allowed to interact with each other and will be instructed to provide
individual ratings of each of the appropriateness and responsiveness of the replies.
15.
Computer Entries must communicate in the English language.
16.
Computer Entries may contain standard or customised software and hardware. The system
may be of any type as long as it contains no genetic material and as long as its replies are in no
manner controlled by human beings or other organic systems in real time.
17.
Entries will be required to run on hardware located at the Science Museum or an alternative
competition site or an associated site. NO TELECOMMUNICATIONS WILL BE PERMITTED.
Finalists chosen to participate may submit their Computer Entries as programs recorded on standard
machine-readable media (magnetic or optical storage) together with operating documentation.
Windows NT 4.0 (capable of running DOS in a window) computers will be available on-site for the
Contest.
18.
Appropriate steps will be taken to prevent the unauthorised duplication or publication of
Contestants' entries. However, neither the Science Museum nor any of its agents can guarantee
absolute security. Contestants, by entering this Competition, confirm their understanding of this and
agree that the Science Museum and its agents harmless should there be any unauthorised duplication
or publication of Computer Entries. Contestants who require absolute security will be allowed to
operate their Computer Entries and/or provide their own hardware on-site. Computer Entries may be
used by the Science Museum for demonstration or publicity purposes after the contest in a temporary
exhibition lasting no more than one month.
19.
Computer Entries will be required to record the conversations as text files on magnetic media.
The recordings will remain the property of the Science Museum, which will also retain the copyright on
transcripts or other representations, magnetic or otherwise, of the recordings. By entering the Contest
the Contestants shall assign copyright in the conversations recorded by the Computer Entries and in
the contribution to those conversations made by the Computer Entries during the Competition and
during the period of temporary display at the Science Museum.
20.
The Contestants will retain copyright in their Entries and will grant to the Science Museum a
licence to use the Computer Entries for the judging and promotion of the Competition and/or the
temporary exhibition but for no other purpose.
21.
Unless the Contestants agree otherwise, the Science Museum will wipe all Competition
Entries from the Competition hardware after the Competition and the temporary exhibition.
22.
Applications must be accompanied by e-mail transcripts recording actual interactions between
the system to be entered and one or more human beings. The protocols may not exceed 2500 words.
23.
The names ‘Loebner Prize’ and ‘Loebner Prize Competition’ may be used by contestants in
advertising only by advance written permission of the Science Museum. Advertising is subject to
approval by representatives of the Science Museum. Improper or misleading advertising may result in
revocation of the Prize and/or other actions.
24.
To avoid giving Judges clues as to the nature of the Computer Entries, Computer Entries run
from a shell should use the default I/O (stdin/stdout). Computer Entries run in a windowed
environment should use black on white in a 25x80 window or full screen mode, using the standard
font, and should seek to mimic the Confederate communications program (download or screendump
possible): it displays a menu bar with Connection Font TTY View and Help. Providing a View menu
option to set the title (for the title bar of the window) is desirable but not obligatory. Since ideally all
Computer Entries will be connected to the communications program over serial lines or (preferably)
via a network, like the Human Confederates, Computer Entries are advised to provide a commandline
or Communication menu option to communicate via a serial line or TCP socket, but are again not
obliged to do so.
25.
If in the opinion of the Judges a Computer Entry is not compliant with the Rules, the Computer
Entry will be disqualified.
-3-
26.
The Judges' decision will be final. Neither the Science Museum nor its agents accept liability
in connection with the Prize.
Data Entry Protocol and File Format
Key Entry – Terminal Display
Each Computer Entry program (also "program") will operate in two modes: "set-up" and "contest".
Upon initial execution each program will operate in set-up mode.
Set-up mode
Set-up mode will permit contest officials to prepare the program for the contest. To that end, while in
set-up mode the program will:1.
request a file name to append the transcripts of the interaction to;
2.
assume that "judge00" is at the terminal and prepare an initial comment;
3.
prompt with a sign "+" requesting a response from "judge00" (contest official);
4.
interact with "judge00" in the normal way with the sole difference that the "+" sign is used as
the prompt;
5.
write a transcript to the specified file as usual.
Contest mode
Upon receipt of the string ‘@@T[CR][CR]’ the program will toggle to ‘contest’ mode and behave as
follows:1.
The key sequence ‘@,@,T,[CR][CR]’ when entered in response to a ‘>’ or ‘+’ prompt
terminates the conversation with the current Judge, and will cause the program to clear the
screen, prompt with a ‘?’ and await the next Judge.
2.
The key sequence ‘@,@,nn,[CR][CR](nn=01-99)’ when entered in response to a ‘>’ or ‘?’
prompt will indicate that a new Judge, ‘JUDGEnn’, is now entering data. Each Judge will thus
identify himself/herself when moving to a new terminal.
3.
The program will make an initial response to the Judges' input.
4.
The program's response may be multiline.
5.
Upon completion of the program's response, the program will prompt the Judge for input.
6.
The computer will prompt the Judge with a ">" character and then echo on the screen,
character by character, the Judge's entry.
7.
Judges' questions and comments can be multiline. Each question or comment will be entered
one line at a time. Each line will be terminated by a carriage return [CR]. Judges will key in
questions and comments in response to a ">" prompt from the program.
8.
Entry of two consecutive carriage returns will indicate that the Judge's question or comment is
complete and that the terminal (program or human) must respond.
9.
The terminal will display character data in a monotype font (all characters of equal width).
10.
The key sequence ‘@,@,X,[CR],[CR]’ when entered in response to a ‘>’ or ‘?’ prompt will
cause the program to exit.
EXAMPLE: Assume program is responding to current interaction with Judge 3.
[some program comment-completed]
>@@04[CR](new Judge, number 4)
>[CR](two lines required)
-4-
Welcome Judge 4 (Comment by program (or human))
>Do you think that the [CR](multiline question from Judge 4)
>Republicans can succeed [CR] (question continued line 2)
>in winning the White House? [CR](question ended (first [CR])
>[CR](second [CR])
Only if Newt succeeds in. (answer, line 1)
Developing a more tolerant image. (answer, line 2)
>(Cursor waits for input ‘>’ prompt – Judge to respond)
Data File Format
Intent: Each program entered in the Loebner Prize Contest will produce a text file transcript of the
interactions with Judges. The file should be readable by standard text-reader programs.
1.
Each Computer Entry in the 2001 Loebner Prize Contest will append to a text file on disk
containing the transcript of keyboard input and program output. The program should discard
the existing contents of the file, but should open the file in append mode (the file should be
created automatically if it does not exist yet).
2.
The file will be named according to the input during set-up operation.
3.
The file will be in ASCII text format suitable for input into a standard word processing program.
4.
The first three lines are headers containing the following:
(c)2001 Science Museum, London, all rights reserved
[Program Name] [Contestant Name]
Start at: [YYYY/MM/DD HH:MM:SS]
5.
Each succeeding line will EITHER mirror one line displayed on the screen, preceded by the
source (‘JUDGEnn’) or ‘PROGRAM’) and time in brackets OR indicate a change of Judges:
‘***JUDGEnn***’.
EXAMPLE: For the above interaction:
***JUDGE04***
PROGRAM[14:12:25] Welcome judge 4
JUDGE04[14:12:32] Do you think that the
JUDGE04[14:12:39] Republicans can succeed
JUDGE04[14:12:55] in winning the White House?
PROGRAM[14:13:15] Only if Newt succeeds in
PROGRAM[14:13:17] developing a more tolerant image
-5-