Comments on: Who Wants To Play “Jeopardy”? http://thenoisychannel.com/2009/04/27/who-wants-to-play-jeopardy/ Mon, 21 May 2012 05:21:00 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Transparent Text Symposium | The Noisy Channel http://thenoisychannel.com/2009/04/27/who-wants-to-play-jeopardy/comment-page-1/#comment-4438 Transparent Text Symposium | The Noisy Channel Sat, 19 Sep 2009 17:46:32 +0000 http://thenoisychannel.com/?p=2023#comment-4438 [...] some of its own researchers to the program, including David Ferrucci, who has been leading the Jeopardy project. There’s even an “Ignite-style” session where all attendees will have the [...] [...] some of its own researchers to the program, including David Ferrucci, who has been leading the Jeopardy project. There’s even an “Ignite-style” session where all attendees will have the [...]

]]> By: Daniel Tunkelang http://thenoisychannel.com/2009/04/27/who-wants-to-play-jeopardy/comment-page-1/#comment-2997 Daniel Tunkelang Mon, 27 Apr 2009 15:34:12 +0000 http://thenoisychannel.com/?p=2023#comment-2997 Mark, that's a good point. Though I suspect the early chess-playing programs made stupid mistakes too, at least from what I read of early evaluation heuristics. Ellen, you're right that I may be raising the bar too high. But I'd like, at a minimum, for the system to be able to express the query it believes the offered results answer, so that a user can compare that back-fitted query to his or her information need. The back-fitted query may be different that the original one, but I'd hope the user would be able to make the connection. Or perhaps the serendipity of the connection would itself be useful. In any case, I'd start by setting what I believe is an achievable goal: requiring that search results offer at least one user-understandable query that can be reverse engineered from them. That would be a big step toward transparency. Mark, that’s a good point. Though I suspect the early chess-playing programs made stupid mistakes too, at least from what I read of early evaluation heuristics.

Ellen, you’re right that I may be raising the bar too high. But I’d like, at a minimum, for the system to be able to express the query it believes the offered results answer, so that a user can compare that back-fitted query to his or her information need. The back-fitted query may be different that the original one, but I’d hope the user would be able to make the connection. Or perhaps the serendipity of the connection would itself be useful. In any case, I’d start by setting what I believe is an achievable goal: requiring that search results offer at least one user-understandable query that can be reverse engineered from them. That would be a big step toward transparency.

]]> By: Ellen Voorhees http://thenoisychannel.com/2009/04/27/who-wants-to-play-jeopardy/comment-page-1/#comment-2996 Ellen Voorhees Mon, 27 Apr 2009 14:46:34 +0000 http://thenoisychannel.com/?p=2023#comment-2996 People aren't necessarily very good at guessing the original query from a sampling of the relevant documents. In the TREC query track, the goal was to get very many different expressions ("queries") of the same information need ("topic") for topics that had already been used in an ad hoc task. One of the ways of getting new queries was to show someone unfamiliar with the original topic statement a sampling of five or so relevant documents and have them write a query for which those documents would be relevant. Some of those back-fitted queries are quite dissimilar from the original topic statement. Ellen People aren’t necessarily very good at guessing the original query from a sampling of the relevant documents.

In the TREC query track, the goal was to get very many different expressions (“queries”) of the same information need (“topic”) for topics that had already been used in an ad hoc task. One of the ways of getting new queries was to show someone unfamiliar with the original topic statement a sampling of five or so relevant documents and have them write a query for which those documents would be relevant. Some of those back-fitted queries are quite dissimilar from the original topic statement.

Ellen

]]> By: Mark Johnson http://thenoisychannel.com/2009/04/27/who-wants-to-play-jeopardy/comment-page-1/#comment-2995 Mark Johnson Mon, 27 Apr 2009 04:28:46 +0000 http://thenoisychannel.com/?p=2023#comment-2995 Thanks for alerting us to this. Very interesting! Ambiguity, double entendre, puns, and anologies are fascinating, but regular question answering systems still suck pretty hardcore. I wonder if research on the former will help the latter? BTW, one of the things I find most interesting about these systems (unlike chess-playing programs) is that they make mistakes that humans would think are completely ridiculous. I think success in this space shouldn't just be defined by beating humans, but by not making inane mistakes (e.g., when a question clearly calls for a thing of class X, the system gives an answer in class Y). Thanks for alerting us to this. Very interesting! Ambiguity, double entendre, puns, and anologies are fascinating, but regular question answering systems still suck pretty hardcore. I wonder if research on the former will help the latter?

BTW, one of the things I find most interesting about these systems (unlike chess-playing programs) is that they make mistakes that humans would think are completely ridiculous. I think success in this space shouldn’t just be defined by beating humans, but by not making inane mistakes (e.g., when a question clearly calls for a thing of class X, the system gives an answer in class Y).

]]>