How the game is played
A single game consists of an exchange of request-response message pairs, initiated by a first START request sent by the guesser agent to the describer agent. The first response of the describer returns a first hint. Each subsequent request sent by the guesser will be a guess of a city. Each subsequent response to an incorrect guess will contain the answer no. and a new hint. The response to a correct guess is a simple yes. This is the last message exchanged in the game.
An example gameplay is provided below:
Guesser: START
Describer: sea
Guesser: Sydney
Describer: no. yearly festival
Guesser: Rio de Janeiro
Describer: no. bridges
Guesser: Amsterdam
Describer: no. renaissance art
Guesser: Venice
Describer: yes
The hints
Each hint is a simple textual phrase in English. More specifically, the hint is always a simple noun phrase consisting of one to three words that are common nouns, adjectives, or connectors (such as ‘and’). The words might be inflected (e.g., in plural form). Guesser agents are supposed to include some logic capable of understanding these hints (or at least matching them or looking them up in a resource).
In response to an incorrect guess, the hint is preceded by the string prefix "no. ".
The guesses
Guesses should consist of EITHER of the following:
- a single city name (composed of one or more words) in English, e.g., "Rio de Janeiro" or "Venice" but not "Venezia";
- latitude-longitude coordinates for the city guessed, as integer numbers, e.g., 48,5 but not 48.1224,5.7783.
If both kinds of responses are provided simultaneously by the guesser then the service ignores the city name and will only take into account the latitude-longitude response.
Evaluation
Evaluation will consist of running the guesser agents a predefined number of times, each time playing a different game. The set of games to be used in the evaluation is predefined, is the same for all participants, and is kept secret by the organisers until the end of the evaluation.
The score for a guesser is derived by the number of guesses it submits. For example, the example game above would score 4. If a guesser cannot find the answer until all the hints run out, it will obtain a score of number_of_hints+5. The total score for the guesser is the sum of the scores of the individual games it played. The winner is the guesser with the lowest total score.
Specifications for the agent to be submitted
Input and Output
Your agent implementation is not required to access the API directly. Instead we are providing you with a complete test suite (that is going to be used for the evaluation too, so please double check to be compliant with it!) which will take care of handling the API request/response packing and unpacking and that will give your agent the hints and receive its guesses.
In order to do so, your agent will have to read the hints from the STDIN and write the guesses to the STDOUT. The test infrastructure will take care of delays and to wait for the responses to arrive. We will be open to any issue you will encounter and we will be helping you to overcome and get through those problems.
Your agent must include a loop which will wait for the STDIN and which, after computation, will write the guess on the STDOUT. This procedure is needed for ensuring that your program will keep the "status" of the game (i.e. keep the list of previous guesses and any other stuff that is needed for guessing the city correctly).
An example guesser agent could look like the following (this is written in Python but the main login can be implemented in different languages too):
#! /usr/bin/env python
import sys
stdin = sys.stdin
stdout = sys.stdout
is_game_ended = False
while not is_game_ended:
# 1. wait for hint --> read from STDIN
pipeIn = stdin.readline()
pipeIn = pipeIn.replace("\n", "")
# 2. Check for ending messages from the Wrapper
if pipeIn == "NO_MORE_HINTS":
# You run out of hints!
is_game_ended = True
elif pipeIn == "CITY_FOUND":
# You guessed right!
is_game_ended = True
city_found = True
else:
# Your own Guesser Implementation resides here!
guess = findTheNameOfTheCityBasedOnAIAndThisHint(pipeIn)
# Write the guess to the STDOUT
# Remember to include the '\n' for terminating the line!
stdout.write("{0}\n".format(guess))
# Flush the STDOUT in order to make it readable by the wrapper
stdout.flush()
This Guesser Agent works only by using this Wrapper Agent (another script written in Python). Right now it works only on *nix systems, but soon the Windows version will be uploaded. In order to use the system you simply have to provide the path of your guesser agent, your token and the number of the game you intend to play.
The syntax is the following:
./challenge.sh {guesser_agent_path} {your_token_here} {game_number_1_to_20}
You can download the zip file containing all the necessary files to run the test/evaluation suite here.
If don't want/need to use our test suite, you have to inform us on how to run the evaluation with your agent. In this case is highly recommended to send us precise information through email. If you feel more comfortable, we can also make a Skype or equivalent call to better understand your needs.
If you need further explanation on the evaluation process just drop us an email and we will be happy to clarify all of your doubts!
Programming languages and environment
We do not want to hinder your creativity by imposing any programming language on you. What we expect, however, is that your implementation should be able to run within the virtualisation environment we will use to run and evaluate submissions. We are going to create two different VM with both Windows and Linux (information about the version will be provided soon) and we'll get you access them for testing purposes. You will soon find here a detailed description of the requirements and a step-by-step tutorial.
The online describer
We provide an online describer agent for the purpose of testing your solution. The games (hints) provided by this describer are similar to those in the evaluation. (Note that for the evaluation we will use a different instance of this service, with the same API specification.)
The API itself is very simple, if you want to try it by yourself without using the provided test suite. If you want to have a look at the API description, you can simply reach
http://challenge.essence-network.com/describer/v1/
and you will be prompted with a brief explanation of the Challenge API endpoints.
In order to authenticate to the online server you have to add a token header to your request with your authentication token. If you don't have one (you should have received it with our first email), simply drop us an email and we'll be happy to send it to you!
For starting a game, you have to make a GET request with a REST client (both CLI or GUI) to this address:
http://challenge.essence-network.com/describer/v1/start/{game_number}
{game_number} is a number between 1 and 20. Right now 20 games are provided for testing purposes.
With the start/{game_number} you are provided with a {gameId} and the first hint to guess the city.
The server response is the following:
Your game identifier: {gameId} --- Max number of tries: 4 --- First hint: {first_hint}
To send your guess to the server you need to make a request to this API:
http://challenge.essence-network.com/describer/v1/play?gameId={gameId}&guess={city_name}&lat={latitude}&lon={longitude}
- {gameId} is the id you received with the first API request
- {city_name} is the english name of the city you try to guess
- {latitude} and {longitude} are the integer coordinates of the city you are guessing
For a better explanation on how the guess is evaluated please refer to the right section.
When you provide a guess the online server can answer in different ways. Basically it will give you feedback on the guess (yes/no), the number of remaining hints before closing the game and, of course, the next hint.
Possible answers are the following:
| Server Response | Response Explanation |
|---|---|
This game is closed (max number of tries achieved). |
You finished the hints for this game and thus you can't no more try. You have to start a new game. |
NO --- Next hint: winter --- Tries left: 3 |
Your guess was wrong! You are reminded with the remaining hints and a new hint is provided for you to try to guess. |
YES! --- Plese start a new game. |
Your agent did well! You successfully guessed the city! The only thing left to do now is to start a new game and try to get even better at City-Guessing! |
Resources available to guessers
We do enable you to use both online and offline resources. The only limits are that you HAVE TO declare to us what resources you are going to use with your agent and that you can only use online resources that are present in the following "whitelist":
If you need to access extra sites or resources elsewhere on the Internet, please contact us as soon as you can and we will add that to the list as soon as we check for its feasibility (no rules infrangements or other possible issues related to the resource to be added).
Submitting the workshop paper
Submissions must include a short paper of 4-8 pages describing the guesser agent submitted. The goal of this paper is to allow the Programme Committee as well as fellow participants to understand and appreciate the innovative theoretical or practical ideas implemented in the agent. The paper must be formatted using the Springer LNCS templates: download them here for LaTeX and here for Microsoft Word.
The content of the paper should address the following points:
- mandatory content:
- the description of the theoretical solution used by the agent (e.g., which AI methods were used and how they were adapted to this specific challenge) and the rationale for having chosen that approach,
- the main lines of implementation (language, libraries, high-level structure of the code),
- the list of knowledge bases and other resources or services used by the agent, whether online or offline;
- recommended content:
- pseudocode of the key algorithm(s) of the agent,
- references to key publications related to the solution;
- optional content:
- evaluation results if you have done quantitative evaluations.
Note that submissions will be evaluated based primarily on the performance of the submitted agent in our official evaluations. The quality of the paper will be taken into account only to the extent of respecting the criteria stated above. Submissions with a missing or largely insufficient accompanying paper (e.g., less than four pages, not addressing one of the mandatory items above) run the risk of being rejected.
Submit your paper to gabor dot bella at unitn dot it.
Frequently Asked Questions
Here's a list of the more common question & answer regarding the Challenge. If you are not going to find what you need, please drop an email and we'll help you to get through!
Q: What cities are you going to use for the evaluation?
A: In principle we use all cities, however, we focus on major cities that "everyone has heard of". We are not publishing a closed set of cities to choose from.
Q: How many games does the evaluation be composed by?
A: Every agent is going to be evaluated with at least 50 games. We are going to tune this number based on performance between agents in order to stress the difference between participants and ease the evaluation and scoring process.
Q: Does the agent need to make any vocabulary lookup or other pre processing before being consuming the hints received?
A: No, the hints that are sent are ready to be consumed. If you find any misspelled word or any other error while using the app, we apologise for the issue and we ask you to please report it to us and we'll correct it ASAP.
Questions
You can ask any questions regarding the Challenge specification in the comment by dropping and email at [email protected] and we will try to answer ASAP!