The 6th ESSENCE Workshop will be held on 21st August at IJCAI 2017 in Melbourne, Australia. The workshop is based on the ESSENCE Taboo City Challenge, and we are delighted to share discuss the results of the competition and award the winners at the event.
Background
Despite the impressive breakthroughs AI has achieved in recent years, we are still a long way from building intelligent systems that exhibit human-like commonsense reasoning capabilities and to interact naturally with human users using these capabilities. Based on the idea that humans obtain much of this commonsense knowledge through interaction with each other, we have been exploring in the ESSENCE project over the past four years how humans established shared semantics when communicating, and how we can replicate these capabilities in computational systems. In an effort to develop an appropriate challenge scenario and stimulate research and debate around this theme, we developed the Taboo City Challenge scenario, a simple game that requires an artificial agent to guess the name of a city that is described (without using Taboo words that are often used to describe the city, as in the classic board game) through simple textual hints (which we have elicited from human play).
The Challenge is designed to mimic a conversational agent/chatbot situation similar to, e.g., an intelligent user interface for an online travel engine where the customer would be describing their requirements for a travel destination and the engine wants to give appropriate suggestions. However, we introduce an artificial "obstruction" by prohibiting the use of descriptions that are too obvious (Taboo words) so that simple lookup of textual resources (e.g. through a Web search engine) is unlikely to yield good results. This means that the guesser agent has to relate the descriptions semantically to any background knowledge it may hold about different cities in the world, and utilise a deeper commonsense understanding to accurately identify the target city. In order to avoid focus on natural language processing problems related to the problem and focus on the reasoning side of the problem, we keep the format of textual hints extremely simple, permitting only simple noun phrases.
The restriction of the target domain to cities serves as a simplification from the general "concept guessing" problem to enable contestants to make progress on a single, limited domain, so that submitted solution do not need to involve general commonsense knowledge about the world. The infrastructure and data gathering process we developed can be extended to other domains easily to make the Challenge harder in the future.
Challenge
The challenge is to develop a computer program capable of guessing the name of a city from simple, increasingly informative textual hints. The winner is the agent that can find the most cities with the least amount of incorrect guesses. In the example below, the Describer (in our case, an automated service) gives a hint in every step, and the Guesser (to be developed by the participants) provides the name of a city that, in principle, matches the set of descriptions provided so far.
Guesser: START
Describer: sea
Guesser: Sydney
Describer: no. yearly festival
Guesser: Rio de Janeiro
Describer: no. bridges
Guesser: Amsterdam
Describer: no. renaissance art
Guesser: Venice
Describer: yes
Competition
In the run-up to the workshop, we ran an open competition where contestants could submit their own solutions to the problem without any restrictions on the methods or technology used. Although the problem was designed to be simpler than other commonsense reasoning challenges that have been proposed by the community, it turned out to be much more challenging than expected. You can find out more about the details of the competition and the results obtained at this page.
The GUESSence app
We have developed an Android app that is available for human play, and will be used to integrate automated guesser (and describer) agents in the future. It is available for free download from the Google Play Store.
Workshop
The Workshop will be run as a half-day event at the IJCAI 2017 Conference on 21 August and is the closing event of the competition. The programme of the one-day workshop will include:
- talks by successful participants, demos, and discussion;
- announcement of the winners and presentation of the awards;
- a panel discussion on AI challenges .
A detailled schedule of the workshop can be found here.
Organisers
Michael Rovatsos, University of Edinburgh, UK
Gábor Bella, University of Trento, Italy
Dagmar Gromann, CSIC, Barcelona, Spain
Programme Committee
Manuel Atencia, INRIA Grenoble, France
Gábor Bella, University of Trento, Italy
Katrien Beuls, Vrije Universiteit Brussel, Belgium
Aysenur Bilgin, University of Amsterdam, Netherlands
Jérôme Euzenat, INRIA Grenoble, France
Raquel Fernández, University of Amsterdam, Netherlands
Kobi Gal, Ben-Gurion University of the Negev, Israel
Fausto Giunchiglia, University of Trento, Italy
Dagmar Gromann, IIIA-CSIC, Spain
Fiona McNeill, Heriot-Watt University Edinburgh, UK
Roberto Navigli, University of Rome “La Sapienza”, Italy
Nir Oren, University of Aberdeen, UK
Iyad Rahwan, MIT, USA
Michael Rovatsos, University of Edinburgh, UK
Marco Schorlemmer, IIIA-CSIC Barcelona, Spain
Michael Spranger, Sony Computer Science Labs Tokyo, Japan
Remi van Trijp, Sony Computer Science Labs Paris, France
Advisory Board
Marcelo G. Armentano, UNICEN Buenos Aires, Argentina (Social Influence Analysis Contest)
Catholijn Jonker, Delft University of Technology, Netherlands (Automated Negotiating Agents Competition)
Diego Perez-Liebana, University of Essex, UK (General Video Game AI Competition)
Wolf Ketter, Erasmus University Rotterdam, Netherlands (Power Trading Agent Competition)
Gerhard Lakemeyer, RWTH Aachen, Germany (Winograd Schema Challenge)
Jochen Renz, Australian National University, Australia (Angry Birds Competition)
Pavel Shvaiko, University of Trento, Italy (Ontology Alignment Evaluation Initiative)