Challenges in Information Retrieval and Language Modeling

Report of a Workshop held at the Center for Intelligent Information Retrieval,

University of Massachusetts Amherst, September 2002

James Allan (editor), Jay Aslam, Nicholas Belkin, Chris Buckley, Jamie Callan, Bruce Croft (editor), Sue Dumais,
Norbert Fuhr, Donna Harman, David J. Harper, Djoerd Hiemstra, Thomas Hofmann, Eduard Hovy, Wessel Kraaij,
John Lafferty, Victor Lavrenko, David Lewis, Liz Liddy, R. Manmatha, Andrew McCallum, Jay Ponte, John Prager,
Dragomir Radev, Philip Resnik, Stephen Robertson, Roni Rosenfeld, Salim Roukos, Mark Sanderson, Rich Schwartz,
Amit Singhal, Alan Smeaton, Howard Turtle, Ellen Voorhees, Ralph Weischedel, Jinxi Xu, ChengXiang Zhai

Summary 1. Introduction

Information retrieval (IR) research has reached a What is information retrieval? Early definitions, dating
point where it is appropriate to assess progress and to from the 1960’s, emphasize the very general nature of
define a research agenda for the next five to ten years. the task. For example, in Salton’s classic textbook * :
This report summarizes a discussion of IR research
challenges that took place at a recent workshop. Information retrieval is a field concerned with the
structure, analysis, organization, storage, searching,
The attendees of the workshop considered information and retrieval of information.
retrieval research in a range of areas chosen to give
broad coverage of topic areas that engage information In that textbook, information retrieval is assumed to
retrieval researchers. Those areas are retrieval also include database systems and question answering
models, cross-lingual retrieval, Web search, user systems, and information is construed to mean
modeling, filtering, topic detection and tracking, documents, references, text passages, or facts.
classification, summarization, question answering,
metasearch, distributed retrieval, multimedia Over the 1970’s and 1980’s, much of the research in IR
retrieval, information extraction, as well as test bed was focused on document retrieval, and the emphasis
requirements for future work. The potential use of on this task in the Text Retrieval Conference (TREC)
language modeling techniques in these areas was also evaluations of the 1990’s has further reinforced the
discussed. view that IR is synonymous with document retrieval.
Web search engines are, of course, the most common
example of this type of IR system.
The workshop identified major challenges within each
of those areas. The following are recurring themes
that ran throughout:
The enormous increase in the amount of online text
· User and context sensitive retrieval available and the demand for access to different types
· Multi-lingual and multi-media issues of information have, however, led to a renewed interest
· Better target tasks in a broad range of IR-related areas that go beyond
· Improved objective evaluations simple document retrieval, such as question answering,
· Substantially more labeled data topic detection and tracking, summarization,
· Greater variety of data sources multimedia retrieval (e.g., image, video and music),
· Improved formal models software engineering, chemical and biological
informatics, text structuring, text mining, and
Contextual retrieval and global information access genomics. Salton’s general definition is even more
applicable now than it has been in the past.
were identified as particularly important long-term
challenges. G. Salton, Automatic Information Organization and Retrieval,
*
McGraw-Hill, New York, 1968.

1
One common theme that has been used to distinguish These factors, plus the resurgence of interest in
IR-related research from research in database systems formal, statistical methods for language-related tasks
is that the information that is retrieved is derived such as IR, make this an appropriate time to reassess
from “unstructured” data sources. In the past, this and more clearly define the IR research agenda. To
distinction has been very clear, but if marked-up text respond to this need, a group of IR and language
is regarded as “semi-structured” and in the domain of technology researchers met at the University of
database systems, then the boundary between the two Massachusetts Amherst to discuss IR research
areas becomes less obvious. Given the number of challenges. This report summarizes those
papers in recent database conferences on nearest- discussions. The workshop was the second in a series
neighbor and similarity search, distributed search, funded by ARDA* that is focused on the language
Web search, and information extraction, it seems modeling approach to IR. For this workshop, the first
apparent that IR and database systems now have priority was to identify the research challenges
many common interests. overall and the second was to discuss how language
modeling could help to address these challenges.
The huge success of Web search engines such as
Google might lead some to question the need for The discussion of IR challenges is broken down by
extensive IR research. There are a number of possible topic. In each topic area, there is a short overview of
answers to this question, but here are some major the area followed by a list of “near-term” challenges
points: (3-5 year timeframe). In some cases, there is also a
discussion of how language modeling could be used
- Web search and IR are not equivalent. As and what resources would be required for progress.
mentioned previously, IR encompasses many The topic areas are: retrieval models; cross-lingual
types of information access. Web search is only retrieval; Web search; user modeling; filtering, TDT,
part (although an important part) of this spectrum and classification; summarization; question
of information systems. answering; metasearch and distributed retrieval;
multimedia retrieval; information extraction; and
- Web queries do not represent all information testbeds.
needs. A broad range of information access
technologies are being created to address the Before discussing this detailed list of research issues,
diversity of information needs of people in we first consider some possible longer-term
different contexts. If we focus only on the current challenges for a ten-year timeframe and beyond. This
mix of queries in Web search engine logs, many of type of challenge, although necessarily quite
those information needs will not be addressed. abstract, is valuable for illustrating general research
directions and emphasizing the importance of
collaborations.
- Web search engines are efective for some types of queries in some contexts. Retrieval experiments in the
TREC environment, and commercial success, demonstrate that, for a very popular type of query (find the
right home page), retrieving the pages containing all the query words and then ranking them according to
other features based on links, anchor text, URLs, and HTML tags is very effective. For other types of
queries, and in other environments (e.g. corporate), this approach to ranking is less successful.
* Advanced Research Development Agency. [Link]

2
2. Long-Term Challenges A definition of the second challenge is:

In the discussions of longer-term challenges at the Contextual retrieval: Combine search technologies
workshop, two main themes emerged. These were and knowledge about query and user context into a
global information access and contextual retrieval. A single framework in order to provide the most
definition of the first challenge is: “appropriate” answer for a user’s information
needs.
Global information access: Satisfy human
information needs through natural, eficient inter- In general, interactions with Web search engines
action with an automated system that leverages could be characterized as “one size fits all”. This
world-wide structured and unstructured data in any means that all queries are treated as simple Web
language. queries where the aim is to locate useful home pages
and the burden is placed on the user to scan and
On the World Wide Web alone, digital information is navigate the retrieved material to find the answers.
being accumulated at a phenomenal rate and in a vast There is no representation of user preferences, search
number of languages. By some estimates, the number context, or the task context.
of web pages written in Mandarin Chinese will
overtake those written in English in the not too Despite some recent attention to this problem, little
distant future. A “grand” challenge for IR will be to progress has been made due to the difficulty of
develop massively distributed, multi-lingual retrieval capturing and representing knowledge about users,
systems. Such systems would take as input an infor- context, and tasks in a general Web search
mation need, encoded in any language, and return environment. Future search engines should be able to
relevant results, encoded in any language. The design use context and query features to infer characteristics
of such a system would probably borrow from of the information need such as query type, answer
techniques in distributed retrieval, data fusion, and type, and answer level, and use these characteristics
cross-lingual IR, but it is unlikely that simply in retrieval models to rank potential answers such as
combining known techniques from these fields would sentences, passages, documents, or combinations of
yield such a system. What would the architecture of documents.
such a system be? What data sets could be used to
develop and test such a system? An example of contextual retrieval would be a
context aware, transmedia IR system. With such a
Wireless devices are being rapidly developed which system, if a user enters a query such as “Taj Mahal”,
can spontaneously form ad hoc peer-to-peer net- and if the user is at a desktop computer and has spent
works. A world in which such devices are ubiquitous time earlier planning a conference trip to India
gives rise to a number of challenges for IR. For (reading emails on the trip; examining travel web
example, one might fancifully imagine flying to an pages; placing entries in a diary), then the system
unknown city for a business trip, renting a car, will be aware of this context and will be more
opening one’s wireless PDA, and connecting to an ad inclined to retrieve pictures and videos of the Indian
hoc peer-to-peer network in order to submit requests mausoleum, while music from the jazz band with the
for information such as: (1) background on local same name would be less likely to be retrieved. The
points of interest, (2) reviews of local restaurants, and ranked output would contain images indexed by the
(3) “dossiers” on people who are likely to be query text, and images similar in visual content to
encountered.. How would such information needs be the retrieved text-tagged images. The high bandwidth
described? (Free text, structured queries, etc.) To of the machine's connection would also be taken into
whom would such requests be sent? How would the account when choosing images or videos to be
results obtained be combined and presented to the delivered. Indeed, the bandwidth may even be a
user? factor in the retrieval algorithm. On the other hand, if
a user were starting out on a long car trip (the system
being aware of this context from diaries and location
sensors) and the user has often

3
requested audio in such a situation, then the music document linkage, queries, users;
from the jazz band will be more likely to be retrieved. their information needs and the context in which the
It is this context awareness, making use of knowledge retrieval task is embedded. The reliance on formal
about the user, the current task, the history and future retrieval models is one of the great strengths of IR
plans, location and device, and the fact that retrieval research.
can be of any kind of digital information, regardless of
query type or mode, which makes this a long-term Information retrieval encompasses a broad range of
challenge. complex information seeking tasks and current
retrieval models capture only a small part of that
Other types of challenges can involve contextual complexity. Current models of text, for example, deal
information. Examples are: with relatively simple aspects of language (words,
phrases, names) and do not capture aspects of
1. Developing intelligent classification algorithms linguistic structure that may be important to specific
that will be able to unobtrusively elicit user retrieval tasks. Similarly, current models of the user
feedback, combine it with contextual and are very weak or, often, missing from current retrieval
historical evidence, and produce effective models. Current retrieval models generally focus on a
structured annotation of new data. narrow range of retrieval tasks (e.g., ad hoc retrieval
and filtering) while ignoring other important tasks
2. Summarizing content from a number of textual (e.g., browsing, known-item retrieval, questions
and semi structured sources, including databases answering, summarization).
and web pages, in the right way (language, format,
size, time) for a specific user, given a task and the Research aimed at developing more comprehensive
user profile. retrieval models is critical. The goal of a single
comprehensive model of retrieval is beyond our
In addition to these two major themes, it is clear that current level of understanding, but models that better
many of the major research opportunities of the future describe individual tasks or a range of related tasks are
will involve collaborations between fields. For possible and necessary to progress in the field.
example, IR and database collaborations will be
crucial for the development of probabilistic models for Near-Term Challenges
integrating unstructured, semi-structured and
structured data, for the design of effective distributed, 1. Models that incorporate the evolving information
heterogeneous information systems, and possibly for needs of users performing realistic tasks and the
systems that detect new and interesting information in utility of information as well as topicality
sensor-based streams. Collaborations with natural (relevance is more than topicality).
language processing and machine learning researchers
will be needed for advanced question answering
systems, and, of course, there are many exciting 2. More advanced models that, for example, estimate
opportunities in collaborations with the sciences (e.g. translation models or compute sub-topic models
bio-, geo-, astro-, ...). through clustering, currently require significantly
more computation than is practical for deployment
in many applications. Significant advances in the
3. Topic Discussions underlying estimation and inference algorithms
will make probabilistic models much more
3.1 Retrieval attractive for use in large scale systems.
Models 3. Generalization of current techniques to informa-
tion sources that may be significantly different
Formal retrieval models have formed the basis of IR from today’s document collections
research since the early 1960’s. A number of different 4. Models and tools for incorporating multiple
models have been developed to describe aspects of the sources of evidence (text, queries, relevance
retrieval task: document content and structure, inter-

4
 judgments, user context, annotations etc.).
CLIR has very recently achieved one major
5. Intrinsic and extrinsic evaluation techniques that milestone: cross-lingual document retrieval performs
measure the performance of retrieval and related essentially as accurately as monolingual retrieval.
technologies in the context of larger tasks, and This has been demonstrated in formal evaluations in
based principled error analysis to go beyond the TREC in 2000, where English queries were run
‘what’ and address ‘why.’ against a Chinese corpus, and 2001, where English
queries were against an Arabic corpus and CLEF*
6. Full elucidation of the relationships between the 2001, with French queries against an English corpus.
various modeling approaches.
Near-Term Challenges
Resource Requirements
1. Efective user functionality. The technology has
1. More refined taxonomy of search tasks and task been evaluated formally for accuracy in returning
models. lists of documents ranked by estimated relevance
to the user’s need. The next steps for effective
2. Data, lots of data (documents, queries, relevance user functionality are to incorporate effective user
judgments, user behavior, task-related data). feedback about their information need and to
provide readable translations of (parts of) the
Language Modeling retrieved documents to support document
selection. Systems should also provide better
Retrieval models that incorporate language modeling support for query formulation and reformulation
techniques have produced promising results over the based on some set of intermediate search
last four years. Simple language models have been results.
shown to incorporate document and collection
statistics in a more systematic way than earlier [Link] 2. New, more complex applications. We can apply
based techniques. Language models work as well as the technology to the next level of cross-lingual
the classical models using [Link], but further challenges. Cross-lingual (factoid) question
improvements are likely to require a broad range of answering would allow posing factoid questions
techniques in addition to language modeling. The (e.g., asking who, what organization, when, or
essence of the language modeling approach, which is where) in English and receive answers in English,
shared with more classical probabilistic approaches to based on documents (in English or another
IR, is that probabilistic modeling is taken to be the language) containing the answer. Cross-lingual
primary scientific tool. At present, this appears to be gisting/summarization would provide short
the most promising framework for advancing English summaries of documents in another
information retrieval to meet future challenges language.
presented by more diverse data sources and advanced
retrieval tasks 3. Languages with sparse data. The technology has
been developed and proved in the context of
3.2 Cross-Lingual Information Retrieval (CLIR) languages with many speakers, e.g., English,
Spanish, French, German, Chinese, and Arabic.
Though initially the Web was dominated by English One challenge now is developing ways to very
speakers, now less than half of existing web pages are quickly (a few weeks) and inexpensively (a few
in English. Accessing information in a host of person-weeks) find/create data for languages
languages is clearly important for many uses. where resources are minimal today (for example,
Farsi, Pashto, Hindi, or any of a large number of
In monolingual retrieval, the queries are in the same
language as the collection being accessed. The
purpose of CLIR is to support queries in one *Cross-Language Evaluation Forum held in Europe
language against a collection in other languages. [Link]

5
 possibilities). For the new challenges above, the following seem
crucial:
4. Massive improvements in monolingual retrieval
based on learning semantic relationships from 1. An effective, inexpensive way to collect far more
parallel and comparable corpora. The lessons (100 times more) queries and results (answers or
learned from CLIR suggest new ways to approach relevant documents)
monolingual retrieval. Given the successes to date
in CLIR, any improvements in monolingual 2. Appropriate resources for evaluations of CLIR
retrieval should generate comparable with low density languages
improvements across languages and vice versa
3. Testbeds (see Testbeds section) for delivering
5. Merging retrieval result lists from databases in effective functionality to users
multiple languages. This problem is related to the
meta-search problem, but has the additional Language Modeling
challenge that the statistics underlying the ranking
functions are built on different vocabularies and In addition to the language techniques used for
thus not directly comparable. monolingual retrieval, CLIR has capitalized on a
probabilistic translation model—e.g., a mixture
6. More tightly integrated models for CLIR. Most model of words from general language usage, and the
current approaches to CLIR are based on a product of probabilities of a term coming from a
shallow integration between translation models document and of that term being translated to a query
and retrieval models, where independence is term. The techniques use a mix of manual bilingual
assumed for both components. A promising dictionaries and statistical bilingual dictionaries
direction seems to be to explore models with less automatically derived from corpora of translated
strict independence assumptions, enabling the use documents. It is interesting that machine translation
of context information for translation directly. systems for the document language are not a
prerequisite.
7. More training data. Language modeling depends
on training data. Today there are several hundreds 3.3 Web Search
of pairs of query and relevant documents. For
factoid questions, there are roughly 1000 queries Search engines for the Web are one of the most
with correct answers. It is generally agreed that publicly visible realizations of information retrieval
developing those materials over the last 10 years technology. For some search tasks (e.g., home page
has spurred much productive research and led to finding), systems such as Google provide highly
major improvements in IR. A new challenge is to accurate results. However, the Web contains more
develop ways to collect a hundred times more than just home pages and searchers are interested in
training data for language modeling—e.g., more than just finding single pages. Significant chal-
100,000 pairs of query and relevant lenges remain for improving general Web search.
documents/answers rather than a few hundred.
Near-Term Challenges
Resource Requirements
1. Web Structure. The challenges are to define the
Major contributors to progress in CLIR have been actual document collection that we call the Web
formal evaluations and data that have been made and to understand how the uniqueness of the Web
available through TREC, CLEF, and the NIINACSIS collection structure affects retrieval methodology.
Test Collection for IR Systems (NCCTR).
- What is a "document" that can be retrieved and
deemed as relevant to a user's information need?
It could be a single Web page, a group

6
 of related Web pages (a Web site), a path 3. S e a rc h i n g . The challenge is to develop methods
through the Web, or a portal to a hidden database for exploiting all evidence, including the web
(e.g., Amazon). A different view or different structure, meta-data, user context information, to
retrievable “documents” might imply different find high quality information for users.
retrieval algorithms. For example, if we take a
whole Web site as a document, then we would - How can we capture and then exploit user
need to measure the relevance of the whole Web information and query context to better
site, rather than a single page. understand a user's information need? What
aspects of a user's context and history can we
- What is the boundary of the Web collection? capture? Can we exploit language models to
Should the collection include dynamically represent information need more precisely?
generated web pages? How is the global, public How can we formally model the interactive
web different from corporate intranets and retrieval process? How do we deal with
extranets? The differences may imply different variations of queries? How can we identify
approaches to IR applications on each. different types of queries and utilize the fact that
specific methods work well for specific query
- How does the hypertext structure, which types?
distinguishes the Web from a traditional
document collection, affect the evaluation - How do we represent a web document? How do
methodology and retrieval models? For we annotate web pages (“generating new
example, the independent relevance assumption evidence”) automatically or semi-automatically?
is clearly violated due to the links; a page
pointing to a relevant document can be regarded - The Web is an ideal environment for developing
partially relevant, even if it is not relevant by formal retrieval models that support multiple
itself. forms of evidence—e.g., multiple types of
document representation, use of document
2. C r a w l i n g a n d I n d e x i n g . The challenge is to structure, etc. The community has talked about
develop an architecture for information access that this, but never really done it (except possibly for
can ensure freshness and coverage of information the inference network model). How can
in a rapidly growing web. statistical language models be combined with
other structural evidence? How can
- It is especially challenging to maintain freshness domain/linguistic/context knowledge be
and coverage in a centralized search engine. The integrated with statistical methods?
current approach is to have different re-visit
frequencies for different types of pages/websites. - There is a growing “semantic Web” effort based
There is something inherently wrong with on widespread use of controlled vocabulary
waiting for a crawler to come around and pick metadata to improve the accuracy and
up your new content before it can be “found” by consistency of Web information services. The IR
people and as the web grows the issues of community has a long history of studying
freshness will get worse. controlled vocabulary metadata, especially how
to assign it automatically, how to evaluate its
- Would other alternative search architecture help quality, how to cope with missing or
maintain freshness and coverage? For example, inconsistent metadata, and how to map among
would a distributed search architecture have an different vocabularies or ontologies. The IR
advantage here? Would topic-specific dynamic community needs to take semantic Web efforts
crawling be useful? more seriously and contribute its expertise to
controlling the hype and solving the problems
surrounding this

7
 issue. Additionally, some information retrieval tasks are ill-
defined without taking the user into account:
- How can we measure “trust” of information summarization is meaningful only in the context of a
sources and perhaps incorporate this trust factor particular task and/or class of users. Other tasks, such
into retrieval? How can we trust the search as cross-lingual document retrieval, appear by classic
engine vendors not to take advantage of their (not user oriented) measures to be quite effective, but
powerful position in providing editorial no one knows how such systems would be used by a
influence on the material we access from the person.
web?
We claim that greater focus on the user will enable
4. Data Collection and Evaluation. The challenge is major advances in information retrieval technologies,
to develop a shared platform for gathering perhaps dwarfing those made possible by better core
appropriate data and to design evaluation methods algorithms. The focus may involve better modeling
that reflect realistic characteristics of web of the user's background and type, personalization,
environment. the context in which information access occurs, the
interactive information seeking process, explanation
- What can, and should, we log from Web of what happened, results presentation, or
searching in order to capture user context, how combinations of all of those.
can we provide this log data as a public utility
for researchers and what are the architectural Near-Term Challenges
implications for logging such data? What does
this mean in terms of privacy issues? 1. Shared data-gathering environment. It is very
difficult to gather information about users
- The collection is changing constantly, we cannot completely enough that it can be used in
freeze a copy, and the linkage characteristics are experiments. The process is expensive, labor
changing over time, so how can we evaluate on intensive, and requires expertise that is not
this “moving target”? universal in the information retrieval community.
It requires constructing a user's working
3.4 User Modeling environment that tightly integrates information
retrieval.
Much active research in information retrieval is
carried out by abstracting the user away from the The community should support one or more
problem: judgments are captured and held constant, groups in creating such a data-gathering
non-binary relevance is ignored as too complex, laboratory, defining what types of information
evaluation is on a single round of query-results need to be acquired (interaction histories,
without opportunity to adjust the query, and so on. annotations, explicit preferences, etc), and sharing
This abstraction has been incredibly successful in all results. This process would be cooperative
enabling research to advance rapidly, creating more among a large number of sites. As a result, it will
effective and efficient systems for retrieving and not be necessary for every group to gather these
organizing information. types of data in order to do experiments. The cost
would be amortized over many experiments.
However, those improvements in retrieval accuracy
appear to have dwindled in the past half dozen years. It 2. Test bed of IR interactions. One or more standard
may be that one reason researchers are unable to experimental data sets need to be created that are
advance beyond the current plateau is that the heavily annotated with infor mation about users,
evaluation model forces systems toward user-generic context, etc. Such a research collection should
approaches that are “good enough” for everyone, and allow evaluations that focus on the user in more
therefore “never great” for anyone. detail, but in a way that does not require
expensive user studies.

8
 Such a dataset would allow for some degree of retrieval system by automatic disambiguation or
comparability and comparison between results user-focused query expansion.
from different groups.
3.5 Filtering, TDT, and classification
3. Evaluation successfully incorporating the user.
The community needs to develop an evaluation Semi-structured text and media objects, created and
methodology that explicitly leverages information managed by users, form an increasingly large
about the user and the context of the information proportion of online data. Information access
retrieval process. Such an evaluation would have techniques that require schemata at data-entry time
the goal of dramatically improving effectiveness are not appropriate for this data. Semantics must be
for a particular user rather than providing merely imposed after the fact, in a dynamic task context, and
satisfactory results for an unknown user. often under the guidance of a nonprofessional user.
The imposed semantics may be as simple as “show
4. Privacy. Support for rich modeling of the user that me/don't show me,” as in topic tracking, message
also preserves privacy. routing, and content filtering systems. Or it may be as
complex as sorting documents into a hierarchical
5. Extended user models. Many, if not all of the other taxonomy, or producing richly linked hypertext.
areas discussed in this report explicitly demand or Systems for topic detection, terminology discovery,
implicitly require much more sophisticated named entity extraction, and unsupervised learning
models of users than are presently constructed for find novel semantic regularities across media types
any IR purposes. There is therefore a significant and time-varying data, aiding users in imposing
need for research in how to include characteristics semantics.
of such elements as users' goals, tasks, and
contexts in user models, as well as how to make Some of these technologies have been adopted
use of this knowledge in affecting IR system widely, for automated metadata assignment in
design and response. knowledge management systems, routing of email
and phone requests for help in customer relationship
6. Long term user models. As it becomes possible to management, categorization of trouble tickets to
gather and store more historical information about support data mining for process improvement, and
a user’s interactions, we need to develop models many others. Despite this commercial penetration,
that span tasks and applications. Such a model only a tiny fraction of text data is ever classified, and
would be in contrast to the short-term, single-task most applications require setup and maintenance by
models that are more common to date. computing professionals.

Language Modeling Near-Term Challenges

Very little has been done that explicitly uses language 1. User Models. Supervised machine learning has
models to capture the user. Arguably, past work allowed the production of classifiers from
modeling judged documents for relevance feedback is manually labeled example documents, reducing
a form of language modeling, but the connection is the need for rule writing by specialists.
tenuous. Unfortunately, the emphasis in machine learning
research on a tabula rasa approach means that a
One hope for language models is to represent the user burdensome number of documents often must be
by a probability distribution of interests (words and labeled. Better ways to elicit and incorporate user
phrases), actions (information seeking and use knowledge in machine learning algorithms, to
behavior), and annotations (judgments). This leverage domain-specific databases, and to gather
information, if appropriately captured, might implicit feedback from user actions, are all
plausibly be used to improve the accuracy of a needed to reduce the demands for labeling.
Promising new approaches to leveraging
unlabeled data, including pseudo-feedback, co-

9
 training, active learning, and transduction, need to be 3.6 Summarization
made more predictable in their effect.
Text Summarization is an active field of research in
In addition to algorithmic progress, we need a both the IR and NLP communities. Summarization is
better understanding of inherent properties of important for IR since it is a means to provide access
classification. How do, and how can, users and to large repositories of data in an efficient way. It
communities of users classify documents to shares some basic techniques with indexing, since
support their tasks? What properties do sets of both indexing and summarization are concerned with
classes (whether informal shared interests or identifying the essence of a document. High quality
structured hierarchical taxonomies) have, and how summarization requires sophisticated NLP techniques
are they connected with latent or link structures in addition, which are normally not studied in IR. In
inherent in the text data itself. particular, for domains in which the
aspects of interest can be pre-specified, summarization
2. Se mi -st ruc t ure d dat a. The fastest growing area in looks very much like Information Extraction.
text classification is in classifying real world Summarization is therefore a good challenge problem
entities by associated text: customers by their to bring together techniques from different areas.
comments, broken devices by partially textual
repair reports, students by their essay exams, and In comparison with IR, the field of summarization
so on. Most such applications to date rely on a suffers from the difficulty of defining a well-specified
known connection between text and real world and manageable task. Since truly reusable resources
entity. Broadening the range of such applications like the TREC test collections did not exist for
requires systems that can detect those connections summarization, it was hard to measure progress.
as well (“hardening” the text database), and in Importantly, the high amount of variance across
particular detecting textual records from multiple human summaries complicates evaluations. Improved,
sources are associated with the same entity. more tightly specified tasks are currently being
Techniques such as link analysis and information developed within the DUC program.
extraction, long of interest in the intelligence
community, need to be improved and extended to Near-Term Challenges
a wide range of text mining applications.
1. Define clearly specified summarization task(s) in
3. N ov e l t y De t e c t i on. Detection of novel data is an IR setting. Examples: headline-length
quickly gaining popularity as a necessary summarization, topic-based summarization.
complement to real-world filtering systems. The
problem has received relatively little attention, but 2. Move to a new genre, since producing text
the exploding growth of available information summaries is almost trivial for newswire and
makes redundancy a very real obstacle to a newspaper documents.
satisfactory user experience.
3. Move beyond extractive summarization.
Existing novelty detection systems are based Extractive summaries are clearly sub-optimal with
primarily on pairwise comparison of potentially respect to obtainable compression rate and overall
novel items to some form of “history,” coherence.
representing classes of items already known to the
user. Error rates of such systems can be exactly 4. Integrate the user’s prior knowledge into models.
predicted from the effectiveness of the comparison Users do not want in summaries material they
function, and this error rate rapidly increases with know already.
the size and richness of history. This makes
novelty detection very challenging in large-scale
filtering environments and
necessitates development of conceptually new
approaches to the problem.

10
5. Specification of a clear task. Suggested tasks have 3.7 Question Answering
included query biased summaries, news clustering
summaries, browsing tools for finding relevant A Question Answering (QA) system takes as input a
passages, and summarizing updates to the news. natural language question and a source collection,
and produces a targeted, contextualized natural
6. Development of an evaluation measure. The language answer. To build the answer it gathers
BLEU measure developed for machine translation relevant data, summary statistics, and relations from
may provide inspiration. It has been successfully the sources (which might be structured or
applied to headline-style summaries of unstructured), fuses or summarizes them into a single
documents. answer as appropriate, and also gathers information
about the epistemic status of the answer (including its
Resource Requirements reliability, recency, factuality/hypotheticality, etc.).
Building answers might also involve a dialog with the
1. Improved models that capture a user’s back- questioner to obtain clarification or otherwise refine
ground knowledge the question and answer. Specifically, QA:

2. Large data sets with example summaries - includes questions with short, factual answers
(preferably outside news domain)–e.g., the (“factoids”)
creation of a SummBank, similar to the one
created at the JHU summer workshop in 2001, but - includes questions with longer natural language
significantly larger. Ideally the SummBank would answers (passages and multi-passage summaries)
contain summaries of the same texts created from
different perspectives, at different compressions - includes answers that involve some data mining or
(from 30% down to headline only), and in gathering of summary statistics, possibly fused
different styles (fluent, telegraphic, bullet points, from multiple sources; summary statistics are
etc.). operations such as maximum, mean, clustering,
conjunctions and joins (thus including some
Language Modeling aspects of what databases provide)

Language Modeling has successfully been applied for - excludes answers that require complex reasoning
content selection, compression of sentences and with models that require no data or information
documents, generation of headlines and reverse- from external sources (thus not including
engineering the cut-and-paste process applied by mathematical story problems or solving college
human summarizers. Since it is highly unlikely that physics exam questions)
summarization can do without modeling higher order
structure, integrating linguistic intuitions into - excludes taking actions (not “make a flight
probabilistic models poses a particular challenge. reservation for me” or “what is the translation of
Also the scalability of increasingly complex models this document”, but does include “what is the
is an important issue for working systems providing earliest flight from LAX to LGA?”)
on-line summarization.
Near-Term Challenges

1. Improve performance of “factoid” QA to the point

that the general public would find it reliable and
useful.

11
2. Create systems that provide richer answers - Provide epistemic status of answer (including is
(beyond factoids): it a fact/opinion/hypothesis; at which times was
- Answers that require summary statistics. For it a valid answer; what is the trustworthiness of
example, use the web pages of a million the answer and the reliability of the source;
companies to answer the question “Which state etc.) The system should provide its confidence
in the country had the largest increase in high- in its own analysis and answer.
tech job openings last month?” Necessary
capabilities might include extracting from - Support interactive QA (dialogue) to clarify and
formatted and tabular data, mining structured explore the topic (here there may not be only
data sources, creating structured data from one answer but a series of answers, each
unstructured sources, ability to select the leading to the next). Systems should be able to
appropriate summary statistical operations, and detect when no answer is possible or when the
generating natural language answers from answer is not to be found in the resources at the
structured data. system's disposal; thus the system should know
when to reply "no answer available," "no
- Longer answers that have structure to them. This answer possible," or "question is inconsistent".
includes answers ranging from those with fairly This relates to other dialogue systems.
standard structure (biographies, event stories,
etc.) through those with some structure (causal - Take into account user models and session
accounts “What are the causes of the Korean models, and keep the models updated. Don't tell
war?”), pro/con advice (“should I take hormone the questioners what they already know.
replacement therapy?”) to ones that have very Remember what you have told them and don't
little structure as in general-purpose descriptions repeat. Give them answers in their preferred
of events and resources (“What is this movie style. Handle sequences of contextually
about?”) and objects (“Describe a telephone”). interrelated questions and responsive answers
Solutions might involve using a language model of varying complexity that need to be
and discourse model that depends on the understood within a larger model of the user's
question. information needs/requirements.

3. Create systems that leverage richer data sources: 5. Develop better evaluation methodologies that are
- Give answers that require integrating multiple scientific and repeatable. Is there an "optimal"
passages, and multiple data sources. answer? What does "optimal" mean?

- Give answers that require integrating structured Language Modeling

and unstructured data sources, including
databases, semi-structured text and numerical Current statistical/language models are proven useful
content. This might include the ability to extract for at least the first part of QA - locating
the desired capital city from a table of countries documents/passages with candidate answers. For the
with capitals. This work could be used to build second part, the accuracy required in pinpointing
(semi)structured knowledge bases. exact answers (for both factoids and for longer
answers that weave together factoids) demands more
- When using dynamic sources, track how an than current language models can support. One
answer changes over time. This relates directly suggestion is to extend language models to include
to novelty detection. more structured patterns (like information extraction
template patterns). (Most current work use hand-
4. Create systems that provide richer interaction with tailored special-purpose pattern matching for dates,
the human questioner, and also provide increased amounts, abbreviations, etc.)
transparency.

12
We envision a unified statistical approach in which to 1. Can a standard resource descriptor be devised
accurately integrate evidence, find the right answer, such that if a resource published its descriptor
and emit it in the best natural language. 3.8 (e.g., on the Web), it could participate in a generic
Metasearch and Distributed Retrieval distributed retrieval system? What data must be
present in such a descriptor? A language model of
Metasearch is the process of retrieving and combining the underlying data set? A semantic description of
information from multiple sources, and it is typically the content? A model of the required query syntax
studied in one of two forms: (1) data fusion, the which permits interoperability? Could it be useful
combination of information from multiple sources both for resource selection a n d metasearch data
that index an effectively common data set and (2) fusion?
collection fusion or distributed retrieval, the
combination of information from multiple sources 2. The performance of language modeling
that index effectively disjoint data sets. As more and techniques is, at present, on par with that of ad
more retrieval mechanisms become available over hoc techniques. Can a theoretically grounded
common data sets (e.g., the Web) and specialized data model of distributed IR be developed which
sets (e.g., medical and law libraries), the process of consistently outperforms ad hoc techniques?
identifying likely sources of relevant information,
retrieving information from those sources, and 3. The performance of distributed IR techniques is
effectively combining the information thus gathered approaching that of a “single database,” at least
will only grow in importance. A future wherein within research environments and when evaluated
ubiquitous mobile wireless devices exist, capable of for high precision. Can this be achieved in
forming ad hoc peer-to-peer networks and submitting practice? Furthermore, through the judicious use
and fielding requests for information, gives rise to a of resource selection, distributed IR should, in
new host of challenges and potential rewards. theory, outperform a “single database.” Can this
be achieved?
The issues typically addressed by a distributed
retrieval system include resource description, 4. The performance of search engines varies from
resource ranking, resource selection, searching, and query to query. The goal of metasearch is often to
merging of results. Many of the techniques developed outperform the ( a p r i o r i unknown) best
to address these issues are a d h o c in nature, though underlying search engine on a per query basis,
language modeling techniques have been successfully and this can typically be achieved when
employed to address resource description, ranking combining systems of similar performance.
and selection. However, this goal is often unachieved when
combining search engines of widely varying
Classic techniques for metasearch data fusion levels of performance. Can a metasearch
typically address the following issues: (1) relevance technique be developed which consistently
score normalization to a comparable space and (2) outperforms the best underlying search engine?
normalized score combination to obtain a final score Or can a technique be developed which is capable
from which a final ranking may be obtained. Other of distinguishing the “good” underlying systems
data fusion techniques that have been developed from “bad” on a per query basis?
include (1) modeling the problem as a multi-
candidate election and employing rank-aggregation 5. Techniques for data fusion typically assume (and
algorithms from Social Choice Theory (e.g., the often implicitly require) that the underlying search
Borda Count and Condorcet methods) and (2) various engines index an effectively common data set;
supervised learning techniques (e.g., boosting, techniques for distributed IR typically assume that
cranking, naive Bayes, etc.). the underlying search engines index effectively
disjoint data sets. Can techniques be developed
Near-Term Challenges which effectively combine the results of
underlying search engines that index data sets of
varying and unknown overlap? (Search

13
 engines on the Web fall within this category.) Can media objects. Automatic annotation may be one
the metasearch problem be modeled in a unified way of associating text with such media objects.
way such that data fusion and collection fusion are This may involve learning from examples how
merely two extremes of a single underlying text and media features are associated and then
problem? using that information for annotation.

3.9 Multimedia Retrieval 2. The context of user activity when a media object
is created may provide a good approach to
Devices for creating, storing, and transmitting indexing: for example, emails, diary entries or
multimedia information are increasing in prevalence location-based information associated with the
and capacity, and are decreasing in price. With little time that a photo was taken could be examined for
prospect of such changes slowing in the foreseeable pertinent text. Context at retrieval time will also
future, it is not hard to predict with some confidence be important: the location of the searcher (a
that content-based access to multimedia information particular country; a particular part of an office) or
(indexing, retrieval, browsing, summarization, etc.), is the type of device a user is using will inform the
set to become a significant problem for web and type or content of objects to be retrieved.
enterprise search as well as for personal retrieval.
3. One of the major reasons for the successes of
The problem space of this topic is large because the (text) IR has been the application of formal
types of objects to be retrieved are varied ranging models for indexing and retrieval. Applying such
from collections of audio (e.g., speech, music), images models to the multimedia domain has been
(e.g., photographs, clip art, scanned documents), video challenging partly because the features which are
(e.g., TV, camcorder, or security camera output), as often most useful do not easily lend themselves to
well as less common objects (e.g., vector drawings, indexing. The extraction of appropriate features
recorded pen strokes, and VRML). The methods which can be used for indexing is a challenge. So
available for indexing and retrieving such objects vary is the application of formal IR models to existing
significantly depending on their type, which has a features. Multimedia (image, video and audio) IR
strong impact on the forms of retrieval possible; tasks need to be formulated which cannot be
however, it is also clear that the forms of retrieval solved using text alone but will require advances
users will wish to conduct will vary for each media in finding such features and advances in applying
type. formal IR models to such tasks.

Near-Term Challenges 4. We also need to think of different kinds of tasks

which involve data mining and retrieval of time
The current challenge in multimedia retrieval centers sequences of images, video and audio from other
on indexing: given a non-text media object, the domains. Example: A few years ago there was a
following options are available. Text may be paper on collecting time sequenced images of
associated with the object (e.g. captions, juxtaposed storms from radars and then retrieving similar
text); part of the object might be convertable to text storms from a database of time sequenced images
(e.g. through speech recognition or OCR); metadata of storms to predict the future track of the storm.
might be assigned manually or media specific features This is basically a natural application of video
might be extractable. retrieval applied to this task.

1. Extracting good indexing features from most forms 5. To deal effectively with multimedia retrieval, one
of multimedia is hard (except within restricted must be able to handle multiple query and
domains). As an alternative, fragments of text or document modalities. In video, for example,
pieces of metadata may be located and used for moving images, speech, music, audio, and text
indexing. For certain media types, text may be (closed captions) can all contribute to effective
easy to come by, for others, however, little or no
text will be directly associated with

14
 retrieval. Integrating the different modalities in Information extraction is at the heart of much of the
principled ways is a challenge. anticipated progress in many fields. Question
answering, novelty detection, cross-lingual retrieval,
Resource Requirements and summarization all hope to leverage IR to improve
their effectiveness. IE is also viewed as a database-
In order to assess research effectively in multimedia filling technique that serves as a first step toward data
retrieval, task-related standardized databases on which mining and other decision support systems.
different groups can apply their algorithms are needed.
In text retrieval, it has been relatively straightforward Near-Term Challenges:
to obtain large collections of old newspaper texts
because the copyright owners do not see the raw text 1. Suficiently high accuracy of traditional entity
being of much value, however image, video, and extraction that it can be straightforwardly
speech libraries do see great value in their collections incorporated into systems that consume it. Current
and consequently are much more cautious in releasing accuracy in the low 90s percent may possibly be
their content. While it is not a research challenge, sufficient for use in isolation, but is not sufficient
obtaining large multimedia collections for widespread when these results are combined into n-ary
evaluation exercises is a practical and important step relations, or incorporated into question answering,
that needs to be addressed. We suggest that task novelty detection, or data mining—all of which
related image and video databases with appropriate cause errors to compound.
relevance judgments be included and made available
to groups for research purposes as is done with TREC. 2. Ability to extract literal meaning from text. This
Useful video collections could include news video (in could also be called automated proposition
multiple languages), collections of personal videos extraction. The University of Pennsylvania, BBN
and possibly movie collections. Image collections and New York University are creating a
would include image databases (maybe on specific “PropBank”, analogous to the “TreeBank” for
topics) along with annotated text - the use of library parsing. We should have as a goal to enable
image collections should also be explored. automated “PropBank”ing of new text. This
would be a major step toward automated computer
Language modeling “understanding” of language.

The application of information retrieval and other 3. Large-scale reference matching. In many
statistical machine learning techniques, analogous to important applications there are many thousands
language modeling, may be useful in multimedia or even millions of entity strings to be de-
retrieval. Language modeling has been successful in duplicated and matched. Performing this task
text related areas like speech, optical character efficiently and accurately requires new approaches
recognition and information retrieval. There is some and techniques. Also, many references (such as
evidence that some of these models may be useful in “police”) are generic and ambiguous. New
automatic annotation, combining image and text representations are needed that will facilitate top-
retrieval and image segmentation. down and knowledge-based disambiguation, or
useful consumption of representations that
preserve the ambiguity.
3.10 Information Extraction
4. Cross-lingual information extraction. This in-
Information extraction (IE) fills slots in an ontology or cludes the ability to combine evidence from
database by selecting and normalizing sub-segments passages in multiple languages when filling a
of human-readable text. Examples include find names database slot. (Both a British and a Jordanian
of entities and relationships between them. newspaper mention an event; use them both to
more accurately build a complete database record
of the event.) Cross-lingual IE also

15
 includes the ability to normalize entities across document retrieval, video retrieval, question-
multiple languages. (The entity "President Bush" answering, etc.
is written differently in Chinese than in English,
and furthermore, has multiple Chinese In addition to the evaluation exercises, these
transliterations in common use.) conferences created a number of significant data sets
that fueled further research in IR such as the use of
5. Relation extraction. What makes a database or language models for IR. The TREC events have
knowledge base most interesting is not lists of created a set of document collections (a few million
isolated entities, but the relations among those documents) with queries (a few thousand) with
entities. To make further progress in accurately corresponding relevance judgments (a few million).
extracting relations, new models are needed. These data sets have played a key role to promote
Relation extraction also includes event extraction. progress in the field. However, given the significant
increase of online content over the past few years
6. Confidence. Information extraction systems (the current Web is estimated to be about ten billion
should provide accurate estimates of its pages) and of the increasing rate of using search (tens
confidence in its own analysis and answer. of millions of queries per day), the current TREC
data sets are too small (perhaps a thousand-fold too
7. Robust ability to be trained from limited labeled small) to be representative of the “real world”.
data. Make efficient use of unlabeled data and
human interaction. Hence a community-based effort is needed to create a
more realistic (in scale and function) common data set
Language Modeling to fuel further research and increase the relevance of
the research activities to the commercial and
Language modeling has been at the heart of many of government activities in IR. We outline below some
the most successful information extraction systems. of the elements needed for creating a set of common
Language modeling has been applied to extraction resources including data sets, possibly annotated, and
tasks, particularly to name extraction, including testbed platforms for IR research.
hidden Markov models, maximum entropy models,
and conditional random fields. Such trained systems The first element is a data collection effort to capture
achieve performance comparable to the best real users performing real search tasks encompassing a
handcrafted, rule-based systems. Additionally, sufficiently large set of queries and corresponding set
language modeling is being applied to relation of retrieved documents; the acquisition of these data
extraction, co-reference resolution, and extraction of would allow the exploration and development of
the literal meaning of text. algorithms for learning retrieval models. To facilitate
the data collection effort, access to a state-of-the-art
There is much evidence that language modeling also (SOA) search capability is required to convince users
lies at the heart of future needed progress. The to use this testbed for their search tasks. One
models will need to be more sophisticated, make approach is to use a proxy or a meta-search engine to
more targeted approximations, and have improved a SOA search service and provide users with its
parameter estimation procedures. search results. This approach would rely on
instrumenting the proxy to log users’ interactions
3.11 Testbeds over a session including all results and ancillary
documents to create a complete data set of user
Over the previous decade, the IR research community queries and corresponding results and underlying
has benefited from a set of annual US government documents; the details of what needs to be logged
sponsored TREC conferences that provided a level would be determined through the IR community’s
field for evaluating algorithms and systems for IR. input. The scale of this data collection could easily be
The conferences have included exploration of several on the order of tens of thousands of
new areas such as spoken

16
users each performing hundreds of queries over a The above-shared common resources can be
period of several months. developed using a parallel approach in developing
the various elements with an appropriate mechanism
The management of user privacy and document IP for integration at a set of planned timelines to benefit
rights would require an experienced organization from the shared work. To enhance the chance of
such as the Linguistic Data Consortium (and/or the creating the shared resources in a timely fashion, the
Internet Archive) to coordinate the data collection creation and funding of specific projects to satisfy the
process, in addition to the actual system building various elements would be needed. NSF via its ITR
activity to create the instrumented platform to program may be a source of funding to help create
perform the data collection experiment. In addition, the Common Testbed for IR; in addition, other
the LDC could play the role of a data distribution agencies such as ARDA or DARPA may contribute to
center similar to the role it currently plays in similar the creation of this testbed. It is anticipated that such
data collection efforts for current human language an approach is required in the IR field to enable the
technologies research. The scope of the data next generation of research to be relevant to the new
collection can easily reach millions of queries with challenges and opportunities created by the explosion
corresponding tens of millions of retrieved of the Web.
documents with associated logs of user activity.

A second element for a common resource would be

the creation of a snapshot of a significant fraction of 4. Conclusion
the web (larger than say 10%, i.e. from 200 million to
2 billion pages) to be made available for research in
During the last decade, IR has been changed from a
conjunction with data logs based on real users doing
field for information specialists to a field for
real tasks. The instrumentation of the interface will be
everyone. The transition to electronic publishing and
based on the community’s input.
dissemination (the Web) when combined with large-
scale IR has truly put the world at our fingertips.
A third element of the common resource, would be However, information specialists and ordinary
the creation of a complete testbed framework which citizens alike are beginning to drown in information.
would enable researchers to incorporate new modules The next generation of IR tools must provide
into the testbed using the specified API set of the dramatically better capabilities to learn, anticipate,
testbed and to conduct user-based research and data and assist with the daily information gathering,
collection by introducing new functionality beyond analysis, and management needs of individuals and
the SOA search technology. The creation and groups over time-spans measured in decades. These
maintenance of a testbed for conducting plug and requirements require that the IR field rethink its basic
play experimentation would require significant effort assumptions and evaluation methodologies, because
but could be leveraged by several research groups to the approaches and experimental resources that
advance the SOA in IR. brought the field to its current level of success will
not be sufficient to reach the next level.
A fourth element would be the annotation of certain
data sets for specific research projects such as the
labeling of “aspect” information in a set of documents
(i.e. identifying the various subtopics in a document
relative to the query) retrieved by a query. The
identification of an efficient approach to the
annotation task would be a challenge for the specific
project that undertakes the task. The development of
annotation standards would increase the sharing of
effort across many teams.

17