Query Processing and Operations
MODULE-03
COB3 : To demonstrate the query processing and operations.
CO3 : Try to solve and process text and multimedia retrieval queriesand
their operations.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Different keywords are associated with the document set during
the process of indexing.
• These keywords generally consist of words, phrases, and other
characterizations of documents such as date created, author names,
and type of document.
• They are used by an IR system to build an inverted index which is
then consulted during the search.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• The queries formulated by users are compared to the set of index
keywords.
• Most IR systems also allow the use of Boolean and other operators
to build a complex query.
• The query language with these operators enriches the
expressiveness of a user’s information need.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Keyword-based queries are the simplest and most commonly used forms of IR
queries: the user just enters keyword combinations to retrieve documents.
• The query keyword terms are implicitly connected by a logical AND operator.
• A query such as ‘database concepts’ retrieves documents that contain both the words
‘database’ and ‘concepts’ at the top of the retrieved results. In addition, most systems also
retrieve documents that contain only ‘database’ or only ‘concepts’ in their text.
• Some systems remove most commonly occurring words (such as a, the, of, and
so on, called stopwords) as a preprocessing step before sending the filtered
query keywords to the IR engine.
• Most IR systems do not pay attention to the ordering of these words in the
query.
• All retrieval models provide support for keyword queries.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Single Word Queries:
• The most elementary query is a single word query.
• A word is defined in rather simpler way.
• The alphabet is split into letters and separators.
• Words carry a lot of meaning in natural languages; hence many models (such as
vector model) are completely structured around words.
• Results produced is a set of documents containing at least one of the words of the
query, if the query is in disjunctive form.
• Other popular mechanism of retrieving is based on the queries presented in
conjunctive form.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Context Queries
• Sometimes queries can be provided with context to present interest in better form.
• Phrase Queries
• When documents are represented using an inverted keyword index for searching, the relative
order of items in document is lost.
• To perform exact phrase retrieval, these phrases are encoded in inverted index or implemented
differently.
• This query consists of a sequence of words that make up a phase.
• It is generally enclosed within double quotes.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Context Queries
• Proximity Queries
• Proximity refers to search that accounts for how close within a record multiple items
should be to each other.
• Most commonly used proximity search option is a phase search that requires terms to
be in exact order.
• Other proximity operators can specify how close terms should be to each other. Some
will specify the order of search terms.
• Search engines use various operators names such as NEAR, ADJ (adjacent), or AFTER.
• However, providing support for complex proximity operators becomes expensive as it
requires time-consuming pre-processing of documents and so it is suitable for smaller
document collections rather than for web.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Boolean Queries
• The oldest way to combine keyword queries is to use Boolean operators
• A Boolean query has a syntax composed of
• atoms: basic queries that retrieve documents
• Boolean operators: work on their operands (which are sets of documents) and deliver
sets of documents
• This scheme is in general compositional: operators can be composed over the
results of other operators
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Boolean Queries
• A query syntax tree is naturally defined
• Consider the example of a query syntax tree below
• It will retrieve all the documents which contain the word translation as well as
either the word syntax or the word syntactic
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Boolean Queries
• The operators most commonly used, given two basic queries or boolean sub-
expressions e1 and e2, are:
• e1 OR e2: the query selects all documents which satisfy e1 or e2
• e1 AND e2: selects all documents which satisfy both e1 and e2
• e1 BUT e2: selects all documents which satisfy e1 but not e2
• NOT e2: the query selects all documents which not contain e2
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Boolean Queries
• With classic boolean systems, no ranking of the retrieved documents is provided
• A document either satisfies the boolean query or it does not
• This is quite a limitation because it does not allow for partial matching between a
document and a user query
• To overcome this limitation, the condition for retrieval must be relaxed
• For instance, a document which partially satisfies an AND condition might be
retrieved
• The NOT operator is usually not used alone as the complement of a set of
documents is the rest of the document collection
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Boolean Queries
• A fuzzy-boolean set of operators has been proposed
• The idea is that the meaning of AND and OR can be relaxed, so that they retrieve
more documents
• The documents are ranked higher when they have a larger number of elements in
common with the query
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Types of Queries in IR Systems
• Keyword Based Querying:
• Natural Language Queries:
• There are a few natural language search engines that aim to understand the structure
and meaning of queries written in natural language text, generally as a question or
narrative.
• This is an active area of research that employs techniques like shallow semantic
parsing of text, or query reformulations based on natural language understanding.
• The system tries to formulate answers for such queries from retrieved results. Some
search systems are starting to provide natural language interfaces to provide answers
to specific types of questions, such as definition and factoid questions, which ask for
definitions of technical terms or common facts that can be retrieved from specialized
databases.
• Such questions are usually easier to answer because there are strong linguistic
patterns giving clues to specific types of sentences—for example, ‘defined as’ or
‘refers to’. Semantic models can provide support for this query type.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Pattern Matching
• A pattern is a set of syntactic features that must be found in a text
segment.
• Those segments satisfying the pattern specifications are said to
match the pattern.
• We can search for documents containing segments which match a
given search pattern.
• Each system allows specifying some types of patterns.
• The more powerful the set of patterns allowed, the more involved
queries can the user formulate, in general.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Pattern Matching
• The most used types of patterns are:
1. Words: a string which must be a word in the text
2. Prefixes: a string which must form the beginning of a text word
3. Suffixes: a string which must form the termination of a text word
4. Substrings: a string which can appear within a text word
5. Ranges: a pair of strings which matches any word which
lexicographically lies between them
6. Allowing errors: a word together with an error threshold
7. Regular expressions: a rather general pattern built up by simple strings
8. Extended patterns: a more user-friendly query language to represent
some common cases of regular expressions
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• The text collections tend to have some structure built into them.
• The standardization of languages to represent structured texts has pushed
forward in this direction.
• Mixing contents and structure in queries allows posing very
powerful queries.
• Queries can be expressed using containment, proximity or other
restrictions on the structural elements.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Three main types of structures:
a) Form-like Fixed Structure
b) Hypertext Structure
c) Hierarchical Structure
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Fixed Structure
• The structure allowed in texts was traditionally quite restrictive.
• The documents had a fixed set of fields, and each field had some text
inside.
• Some fields were not present in all documents.
• Some documents could have text not classified under any field.
• They were not allowed to nest or overlap.
• Retrieval activity allowed: specifying that a given basic pattern was to be
found only in a given field.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Fixed Structure
• When the structure is very rigid, the content of some fields can be
interpreted as numbers, dates, etc.
• This idea leads naturally to the relational model, each field corresponding
to a column in the database table.
• There are several proposals that extend SQL to allow full-text retrieval.
• Among them we can mention proposals by the leading relational database vendors
such as Oracle and Sybase, as well as SFQL.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Hypertext Structure
• Hypertexts probably represent the opposite trend with respect to
structuring power.
• Retrieval from hypertext began as a merely navigational activity.
• That is, the user had to manually traverse the hypertext nodes following
links to search what he/she wanted.
• Some query tools allow querying hypertext based on their content and
their structure.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Hierarchical Structure
• An intermediate model which lies between fixed structure and hypertext
is the hierarchical structure.
• An example of a hierarchical structure: the page of a book and its
schematic view.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Structural Queries
• Hierarchical Structure
• An example of a query to the hierarchical structure presented.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Sometimes, query languages are used by applications to query text
databases.
• Because they are not intended for human use, we refer to them as
protocols rather than languages.
• The most important are/were:
• Z39.50
• Wide Area Information Service (WAIS)
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Z39.50 is a client-server protocol for searching and retrieving
information from remote databases.
• It is formally known as ANSI/NISO standard Z39.50 and its
international equivalent, ISO 23950.
• Its primary purpose is to allow a client program (called an Origin)
to search and retrieve data from a server (called a Target) in a
standardized way, even if the databases on the target side use
different software and native formats.
• It acts as a universal translator for library and bibliographic
databases.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Historical Context and Modern Relevance
• Origins: Developed in the 1980s, it was a groundbreaking solution for
the problem of "information silos" in libraries, museums, and
government institutions.
• Legacy: It was the backbone of large-scale union catalogs and library
networks (e.g., WorldCat via OCLC used Z39.50 extensively).
• Modern Use: While its use has declined in favor of simpler web-based
protocols like SRU (Search and Retrieve via URL) and RESTful APIs,
Z39.50 is still operational in many legacy systems, particularly in
national libraries, university library catalogs, and government
databases.
• SRU is often considered the "HTTP-friendly" version of Z39.50.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Key Features and Characteristics
• Stateful Protocol
• The server maintains the state of the interaction (session) with the client. It
remembers previous operations, like result sets, throughout a session.
• Asymmetric Operation:
• The client initiates all requests, and the server responds.
• Complex Query Support:
• Allows for very powerful, structured, and precise queries using attributes (e.g.,
find books where Author=Shakespeare AND Title=Hamlet AND Date > 2000).
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Key Features and Characteristics
• Multiple Result Sets:
• A single search can return multiple records, which are stored on the server as a
"result set." The client can then browse, sort, or retrieve these records in
chunks.
• Explain Facility:
• A client can query the server to understand its capabilities—what databases
are available, what search attributes it supports, and what record formats it
can return.
• Protocol Independence:
• Originally ran over the OSI network model but was adapted to run seamlessly
over TCP/IP, which is the standard for the internet.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Architecture: Client-Server Model
• The protocol follows a classic client-server
architecture with specific terminology:
• Origin (Client): The software that initiates the request.
This could be a library's online public access catalog
(OPAC) system searching other libraries, or a
dedicated client program.
• Target (Server): The software that waits for and
services requests from the Origin. It interfaces with
one or more underlying databases.
• Database: The actual collection of records
(bibliographic, full-text, etc.) on the server side.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Record Syntaxes
• A key strength of Z39.50 is its support for multiple output formats,
allowing a client to request data in a format it can understand.
• USMARC / MARC21: The standard machine-readable format for bibliographic
data. This was the most common syntax.
• XML:Modern, structured format (e.g., a MARCXML variant).
• SULMS (Summary University Library Management System): A simple,
human-readable text format.
• Dublin Core: A simpler metadata format, often used for cross-domain
information retrieval.
• SUTRS (Simple Unstructured Text Record Syntax): Plain text.
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)
Query Protocols
• Z39.50
• Advantages & Disadvantages
Advantage Disadvantage
Interoperability: Allows disparate systems to Complexity: The protocol is heavy, complex, and
communicate. difficult to implement fully.
Overhead: The ASN.1/BER encoding and stateful
Powerful Queries: Supports very precise, Boolean,
nature create significant network and processing
and proximity searching.
overhead.
Abstraction: The client doesn't need to know the Not Web-Friendly: Designed pre-web, it doesn't
server's native database schema. align well with stateless HTTP.
Rich Metadata: Designed to handle complex, Declining Support: Being replaced by lighter-
structured metadata records like MARC. weight APIs (REST, SRU).
Module-03(Query Processing and Operations) - INFORMATION
RETRIEVAL SYSTEM (ITDO7024)