Journal of Computer Applications (JCA)

ISSN: 0974-1925, Volume IV, Issue 1, 2011

Search Engines:A Study

1
Mr.K. Tarakeswar , 2 Ms. D. Kavitha

 [Link] based search engines.

Abstract— The Internet is a huge collection of data. To get the [Link] powered directories.
appropriate information from it, using a search engine is the [Link] search engines.
most effective way. Many Search Engines were introduced [Link] search engines.
since 1990. In this paper we present a brief study on search
engines. First, we present the definition of search engine, types
A. Crawler based Search Engines
of search engines and the general working process of a search
engine. Then we give an example for the working process with
Crawler based search engines contain three parts. The first
a description of the Google search engine architecture. Later, part is the ‘Crawler’ (bot or robot or spider). It is used to
we present a short description of the next generation search wander the web and create listings of web pages. The second
engines. Then we present comparisons among some major part is the ‘Index’, which is a huge collection of copies of
search engines. web pages and the third part is the ‘Search Engine Software’
Index Terms— Internet; Search Engine; working; architecture. which ranks the results. Because the crawler in this engine
searches the web constantly, it provides updated information.
I. INTRODUCTION Google, Live Search, Ask and most other search engines are
Access to various types of information is necessary these crawler based.
days. The World Wide Web (WWW) contains a lot of web B. Human Powered Directories
pages. To search for the information necessary for us from Human powered directories are search engines which depend
that huge collection of web pages, using a Search Engine will on humans for their web page listings. These types of search
provide with efficient results. Many web pages in the WWW engines get their listings of web pages from the submissions
contain inappropriate information. This is due to the made by the respective web page masters. The submission
inappropriate naming and unnecessary highlighting of the contains the address, title and a brief description of the site.
content of web pages by their web masters. This raises the Later, the submission is reviewed by editors. A directory
need of a search engine. Using a good Search Engine will searches for results only from the page descriptions
filter out the necessary and relevant information needed by submitted to it. This is an advantage because, as the pages are
the user. submitted manually, the quality of the content will be better
This paper presents an overview on the search engines. In the and more appropriate compared to the results retrieved by a
second section of this paper, we present the definition of crawler based search engine. But, the disadvantage is, any
search engine and we describe the types of search engines in change made to an already submitted web page will not be
the third section. We describe the general working of a search updated until it is submitted again. Also, the ranking of pages
engine in the fourth section and present an example for it in can’t be changed once ranking is done. Yahoo, dmoz and
the fifth section by explaining the Google search engine Galaxy are some examples.
architecture. In the sixth section we present a brief C. Hybrid Search Engines
explanation on the next generation search engines and in the Hybrid search engines include the features of crawler based
seventh section we present comparisons among some major search engines and human powered directories. Currently,
search engines. some search engines are using both features to provide
effective results. MSN, Google and Yahoo are some
examples.
II. DEFINITION OF SEARCH ENGINE D. Meta Search Engines
Definition 1: Search Engine is a program which searches the Meta search engines fetch results from other search engines.
database, gathers and reports the information which contains The fetched results are combined and ranked again according
the specified or related terms. to their relevancy. These search engines were useful when
Definition 2: The term Search Engine [11] refers to the each search engine had a significantly unique index and
process of searching files using the key words specified. The search engines were less savvy. Because the search has
key words found are returned and collated into the user improved a lot, the need for these has reduced. MetaCrawler
information. and MSN Search are some examples.

[Link] OF SEARCH ENGINES

Search Engines are of four types[6]. They are IV. WORKING OF SEARCH ENGINE
The working [3], [4], [5] of Search Engine involves three
Manuscript received Apr 18, 2011. basic tasks. They are,
[Link] the WWW and collecting the pages.
Mr.K. Tarakeswar, Department of Computer Science and Engineering,
G. Pulla Reddy Engineering College, Kurnool-518002, Andhra Pradesh,
[Link] the index of the words they find and where they
India. (e-mail : eshwartarak158@[Link])
were found.
D. Kavitha, Department of Computer Science and Engineering,G. Pulla [Link] users to search for words or a combination of
29
Reddy Engineering College, Kurnool-518002, Andhra Pradesh, India. them from the index by using efficient software.
(e-mail : dwaramkavithareddy@[Link])
 Search Engines:A Study

These tasks are performed by the three parts of a search page will be indexed. They guide a search engine in
engine. They are, choosing appropriate meaning for a word from the several
1) Crawler possible.
2) Index But, over reliance on meta tags leads to pages with popular
3) Search Engine Software. topics, but which have very less or irrelevant content. To
The working of a search engine is shown in the Fig.1. compensate this, the crawlers correlate the meta tags with the
A. Searching theWWW and collecting the pages page content. They reject the meta tags which don’t match
Definition of Computer Robot, Spider or Crawler: with the words in the page.
Computer Robots [10] are programs, which automate After collecting the information, it must be stored in a way
repetitive tasks at speeds impossible to be done by humans. useful to the user. The stored data is encoded by the search
The term ‘bot’ on the internet implies anything which engines to save the storage space. Two important
interfaces with the user or collects data. components are present in making the collected data
To present the result pages for a query a search engine must accessible to the user. They are mentioned below.
search and collect it. To find the web page from the millions 1. The information stored with the data
of web pages present, search engines use the software robots A search engine can store only the word and URL (Universal
called ‘Crawlers or Spiders’. They build lists of words found Resource Locator). This is a simple way of storage. In this
in the web pages. This process of building lists is called case, the results cannot be ranked for their relevancy. To
‘Web Crawling’. A lot of pages must be traced to collect a provide relevant results, weights can be assigned to the words
useful list of words. based on their locations in the page.
A spider chooses a list of heavily used servers and popular 2. The method by which the information is indexed
web pages as its starting point. It then begins with a popular2.. Indexing of words is made to allow the information to be
web site, indexes the words present in it and also follows accessed as fast as possible. An effective way is to use a
every other link present in that page. In this way, it quickly Hash table for indexing. In the Hash table indexing we apply
starts to travel spreading across widely used parts of the a formula for attaching a numerical value to the words. The
Internet. formula used must evenly distribute all the entries.
The crawler carefully chooses at each step about which page In a dictionary, more pages will be present for the words
to index. Some policies were introduced to guide the starting with the letter‘s’ than for the letter ‘z’. So, the time to
crawler. They are search for a word starting with‘s’ is more, compared with the
1. Selection policy: Selection policy states which pages to time taken to search for a word starting with‘z’. Hashing
download. evens out such differences. It also reduces the average search
2. Revisit policy: Revisit policy states when to check for time for an entry. The hash table will contain the hashed
changes in web pages. values and pointers to the actual data. Hence, using efficient
3. Politeness policy: Politeness policy states how to avoid indexing and effective storage methods provide quick and
overloading of web sites. better results for complicated queries also.
4. Parallelization policy: Parallelization policy states howC C. Providing results by using efficient search engine software
to coordinate the different web crawlers distributed. The third task is performed using search engine software.
Web Spider This software sifts through the results and ranks them
according to their relevancy. Some basic principles are
followed by all search engines to determine the relevancy of
results. They are,
x Principle 1: The location of key words in a web page is a
factor for determining the relevancy. The pages
containing the search term in its HTML (Hyper Text
Flowers
Art Markup Language) tag, at the beginning of the page, in
Travel the links or subheadings and meta tags are more relevant.
Theaters
Weather
School
Soccer
Builds list of x Principle 2: Frequency of key words in the page is another
Ci k words and notes factor for determining the relevancy. The page with
where they are more occurrences of a search term is said to be more
Builds index
based on its relevant.
own system of
Each search engine has its own method for assigning
weights. Because of this, for the same query, different search
Encodes data to
engines provide differently ordered results.
save space
1. Off Page Factors
Stores data for
users to access Off Page Factors[10] are also used to rank web pages. They do
not depend on the content of the page. They are
Figure1. Working of a Search Engine

B. Keeping an index of the words they find and where they x Factor 1: Look of the web page.
were [Link] describing this task, we give a brief Search engines infer a lot about the content of a page with a
explanation on Meta Tags. look of the page. Sophisticated techniques exist to find
Definition of Meta Tags: Meta Tags[7] allow a web page’s artificial, fake and useless links and remove them.
30
owner to mention key words and concepts under which his
 Journal of Computer Applications (JCA)
ISSN: 0974-1925, Volume IV, Issue 1, 2011
x Factor 2: Click Through Measurement. inverted index. The Sorter produces a list of wordIDs and
also offsets into the inverted index.
This determines the behavior of the user in relation to what C. Providing results by using efficient search engine
results they choose while searching. software.
The Dump Lexicon program takes the list generated by
V. THE GOOGLE SEARCH ENGINE ARCHITECTURE
Sorter along with the lexicon generated by the Indexer. It
In the Google search engine[1],[2], the three tasks of a search then produces the lexicon which is used by the Searcher. The
engine are performed as follows. The Google Architecture is Searcher is run by a web server. It uses the lexicon built by
shown in Fig. 2. the Dump Lexicon program, inverted index and page ranks to
efficiently answer the queries.
[Link] the WWW and collecting the pages
VI. NEXT GENERATION SEARCH ENGINES
The first task is performed using several distributed crawlers.
The URL Server will send the lists of URLs to be fetched to The next generation search engines are referred to as
the crawlers. The fetched web pages are sent to the Store Peer-to-Peer Search engines. They employ major types of
Server. The Store Server compresses the web pages and discovery methods which are mentioned below.
stores them in a repository. Each web page is assigned a x Selective forwarding systems.
‘docID’. It is assigned each time a new url is parsed out of a
page. x Flooding broadcast of queries.
x Centralized indexes and repositories.
Crawle
Store
x Decentralized hash table networks.
URL Server
Server x Distributed indexes and repositories.
Ancho
x Relevance driven network crawlers.
The Peer-to-Peer search implementation has two models.
URL Indexer Repository They are
Resolver
A. Centralized server-client model.
B. Decentralized model.
Lexic
Link
on
[Link] server-client model.
s
Barrel The Centralized server-client model[9] contains a single,
centralized server. It contains a directory of the shared files
Doc which are stored on the computers of users in the network.
Index
Sorter When a user searches for some file, the central server creates
a list of files from its database of files which belong to users
currently connected to the network. The server displays that
list of files to the user. After the user chooses the file, a direct
connection is setup with individual computers which contain
Search that file at that moment. Opennap, kazaa and eDonkey are
Page Rank
examples of Centralized server-client models.
Figure.2 The High Level Google Architecture
Advantages
[Link] an index of the words they find and where they
were found. x The single, centralized index locates files quickly and
efficiently.
In Google, Indexing is done by the Indexer and x The search requests are sent to all clients who have
Sorter. The Indexer reads the repository to uncompress the logged in to the network. So, the search will be as
documents. It then parses the documents. Every document is through as possible.
converted into a set of word occurrences which are referred
to as ‘Hits’. The Hits contains the words, their position in the Disadvantages
document, their font size and capitalization. These hits are x The centralized server results in a single point of failure.
distributed by the Indexer into a set of Barrels, thus creating a x As the centralized index is updated only periodically, the
partially sorted forward index. The Indexer also parses out client may receive outdated information.
the links in all web pages and stores the key information
about them in ‘Anchors’ file. [Link] model.
The URL Resolver reads the Anchors file, converts Decentralization of the network is made so that each peer can
relative URLs into absolute URLs which are then converted communicate as an equal to all the other peers. The
into docIDs. It also puts the anchor text into the forward Decentralized model[8] will not be having a single, central
index according to their docIDs. It generates a database of server. This model can be explained as follows.
links which are used to compute page ranks of all documents. Let there be some peers a, b, c, d, e, f etc., Whenever a peer
The Sorter takes the Barrels which are sorted by docIDs. ‘a’ enters the decentralized network, it connects to another
These are resorted according to their wordIDs to create the peer ‘b’ to announce that it is alive. The peer ‘b’ announces
to all other peers to which it is connected about the peer ‘a’

31
 Search Engines:A Study

being
alive. The other peers c, d, e, f etc., repeat this pattern. After x The network is harder to shutdown.
‘a’ announces that it is alive, it can send search requests to
Disadvantages
‘b’. ‘b’ will pass this request to c, d, e, f etc., If ‘c’ has a copy
of the file requested by ‘a’, ‘c’ sends a reply to ‘b’. ‘b’ passes x Searching is slower in a decentralized network.
this reply back to ‘a’. ‘a’ then opens a direct connection to ‘c’
x Because of the TTL, the request for a file can’t reach
and downloads the file. This scenario allows for an infinite
the node which will be having the file needed.
network. In practice, a time to live (TTL) is used to limit the
number of nodes reached by a request. Gnutella, mnet,
freenet and gnunet are examples of Decentralized model. VII COMPARISON OF SEARCH ENGINES

In this section, we present comparisons among some major

Advantages search engines based on some factors [12], which make a
x The problem of a single point of failure is eliminated. search engine provide satisfactory results. The results of the
comparisons are presented in the below table, Table I.

TABLE I COMPARISONS OF MAJOR SEARCH ENGINES

AltaVista Yahoo Google Ask Teoma MSN Search Bing

Links to a No Yes Yes No No No No

URL
Languages All or 41 languages 44 languages 6 languages 10 Languages 38 Languages 41
provided English languag
es

Similar No No Yes No No No No
pages
Boolean Yes Yes Yes Yes Yes Yes Yes
and Phrase
search
News and Yes Yes Yes Yes No Yes Yes
Multimedia
search
Stemming No Yes Yes No No Yes No

Other Yes Yes Yes Yes No Yes Yes

databases
provided
Word in Yes Yes Yes Yes Yes Yes Yes
URL

Search by Yes Yes Yes Yes No Yes Yes

File Type
Truncation Yes Yes No Yes No Yes Yes

Grouping No Yes Yes No Only Yes Yes

and Sorting Grouping of
Results results
Domain Yes Yes Yes Yes Yes Yes Yes
Search

Thumbnails No No Yes Yes No No Yes

of results

Personalize No Yes Yes Yes No No Yes

Date Limit Yes Yes Yes Yes Yes No No

Search

32
 Journal of Computer Applications (JCA)
ISSN: 0974-1925, Volume IV, Issue 1, 2011
[8] Gabor Vincze, Zoltan Pap & Robert Horvath,
VIII CONCLUSIONS “Peer-to-Peer based distributed file systems”,
International Journal of Internet Protocol Technology,
Concluding this paper, using a Search Engine is obviously Vol. 2, Number 2/2007, pp. 117-123, 2007.
good to gather the necessary information. Many search [9] L. Plissonneau, J. L. Costeux & P. Brown, “Detailed
engines have been developed to provide the best results for Analysis of eDonkey transfers on ADSL”, Proc. Second
users. The present day search engines provide a variety of Conference on Next Generation Internet Design and
results like geographic search, domain search, Engineering, 2006.
personalization etc., but, they are unable to present [10] S. Brin & L. Page, “The Anatomy of a Large-scale
Hypertextual Web Search Engine”, Proc. the Seventh
satisfactory results for scientists, analysts, research students World Wide Web Conference. Brisbane, Australia.
etc. To compensate this, the Peer-to-Peer search engines are [11] Wen-Jen Yu, Shrane Koung Chou, “A Bibliometric
being developed, which are referred to as the next generation Study of Search Engine Literature in the SSCI
search engines. The Peer-to-Peer search engines use the Database”, Journal of Software, Vol5, No 12 (2010),
major searching techniques like flooding broadcast of 1317-1322, Dec
queries, selective forwarding of queries, relevance driven [12] Ran Hock, (2010), “Major Search Engines – Features
network crawlers etc., They also use scalable and Guide”, [online] Available at:
[Link]
self-organizing algorithms and data structures and the results
provided by them will be more quick and efficient compared
to the present day search engines.
BIOGRAPHY

REFERENCES
K. Tarakeswar obtained his [Link]
[1] Sanjay Ghemawat, Howard Gobioff & Shun-Tak Leung, degree from Sri Krishna Devaraya
“The Google File System”, Proc. The Nineteenth ACM
Symposium on Operating Systems Principles, pp. 29-43, University, Anantapur in the year 2009.
2003. He is pursuing his [Link] in Computer
[2] William Yip & Dr. Liz Quiroga, “Google Page Rank Science and Engineering from Sri Krishna
Algorithm”, LIS 678 Personalized Information Delivery, Devaraya University, Anantapur, India. He presented a
Oct 11, 2008.
survey paper at a national level conference.
[3] Mark Levene, “An Introduction to Search Engines and
Web Navigation”, John Wiley & Sons, Inc., 2010.
[4] Fidel Cacheda, Diego Fernandez & Rafael Lopez, D. Kavitha obtained her [Link] degree
“Experiences on a Practical Course of Web Information from Sri Krishna Devaraya University,
Retrieval:Developing a Search Engine”, Proc. Second
International Workshop on Teaching and Learning of Anantapur and [Link] degree from
Information Retrieval, 2008. Jawaharlal Nehru Technological
[5] Curt Franklin, “How Internet Search Engines Work”, University, Anantapur, in the years 2001
[online] Available at: and 2005 respectively. She is pursuing
[Link]
basics/[Link] her Ph.D. from Sri Krishna Devaraya University, Anantapur,
[6] J. M. Kassim & M. Rahmany, “Introduction to Semantic India. She is working as an Associate Professor in the
Search Engine”, Proc. International Conference on Department of Computer Science and Engineering at G.
Electrical Engineering and Informatics, 2009.
Pulla Reddy Engineering College, Kurnool, Andhra Pradesh,
[7] Pegah Pishva & Mousa Majidi, “Study of HTML
Meta-Tags Utilization in Web-based Open-Access India. She presented six research papers in international
Journals”, Journal of Information Sciences and journals and five in national and international conferences so
Technology, Vol. 22 Number 3(4-2007), 2007. far. Her research areas include Computer Networks and
Network Security.

33