WEB AND SOCIAL MEDIA
MINING
MODULE - 1
WEB MINING
DEFINITION:
• Web mining refers to the process of extracting and discovering useful
information from the World Wide Web.
• It involves collecting data from various online sources, analyzing it, and
extracting knowledge or patterns to gain insights or make informed decisions.
• Web mining techniques utilize automated tools and algorithms to crawl,
retrieve, and process web content, structure, and usage data.
There are three main types of web mining:
1. Web Content Mining:
• Extracting information from web pages, documents, and other textual or multimedia content.
• Techniques:
• Natural language processing, text mining, information extraction, and sentiment analysis.
• Tasks:
• Web page classification, opinion mining, information retrieval, and content recommendation.
2. Web Structure Mining:
• Analyzing the structure and relationships of web pages, including hyperlinks and the link
structure of the web.
• Techniques:
• link analysis, graph theory, and clustering are used to uncover patterns, identify important pages
or hubs, and understand the organization and connectivity of the web.
• Tasks:
• web page ranking, web page categorization, and identifying communities or clusters of web
pages.
3. Web Usage Mining:
• Analyzing user behavior and interactions with websites.
• It involves capturing and analyzing web server logs, clickstream data, and other user-related
information to understand user preferences, browsing patterns, and trends.
• Techniques:
• session identification, user profiling, sequential pattern mining, and recommendation systems.
• Tasks:
• Personalization, user behavior analysis, and improving website design and usability.
APPLICATIONS:
• Web mining has numerous applications across various domains.
• For example,
• E-commerce companies can use web mining to personalize product recommendations based on
user browsing and purchase history.
• Search engines leverage web mining techniques to improve search results and ranking
algorithms.
• Market researchers can analyze web content and social media data to gain insights into
consumer opinions and trends.
• Additionally, web mining plays a crucial role in fraud detection, information retrieval, social
network analysis, and many other fields related to the World Wide Web.
WEB CRAWLING
• Also known as web scraping or web harvesting.
• Is the automated process of systematically browsing and retrieving information from websites.
• It involves accessing web pages, extracting their content, and storing it for further analysis or use.
• Web crawlers, also called spiders or bots, are programs that perform web crawling tasks.
PURPOSES:
• Building search engine indexes: Crawlers gather web pages' content to create searchable databases
for search engines like Google or Bing.
• Data aggregation: Crawlers collect data from multiple websites for purposes like market research,
price comparison, or content monitoring.
• Content extraction: Specific information, such as news articles, product details, or user reviews, can be
extracted from websites for analysis or repurposing.
• Website testing and validation: Crawlers can be used to check the functionality and performance of
websites by analyzing their structure and content.
• Web crawling should be performed ethically and in compliance with legal and ethical considerations,
such as respecting website terms of service, privacy policies, and copyright laws.
A Basic Crawler Algorithm
• Seed URLs: The web crawler starts with a list of seed URLs, which are the initial web pages it will
visit to begin the crawling process.
• Fetching: The crawler accesses the seed URLs and retrieves the corresponding web pages using
HTTP or HTTPS protocols. It sends requests to web servers and receives responses containing
HTML content.
• Parsing: The HTML content of the web pages is parsed to extract useful information. This
process involves analyzing the structure of the HTML document, identifying elements such as
links, text, images, and other data.
• URL Frontier: The crawler maintains a queue, known as the URL frontier, to store the discovered
URLs. It extracts URLs from the parsed web pages and adds them to the URL frontier for future
crawling.
• Crawling and Indexing: The crawler continues to follow the URLs in the frontier, visiting each web
page and repeating the fetching and parsing steps. This process is typically performed in a
breadth-first or depth-first manner, depending on the crawling strategy.
• Politeness and Respect: Web crawlers often implement politeness policies to avoid overloading
websites with excessive requests. They may respect [Link] files, which provide guidelines
for web crawlers regarding which pages to crawl or avoid.
• Data Storage: The extracted data from web pages can be stored in a structured format, such as
a database, or as unstructured text. It may include text content, metadata, links, images, or any
other relevant information.
A Basic Crawler Algorithm
• Breadth-First Crawlers
• Preferential Crawlers
BREATH FIRST CRAWLERS:
• Follows FIFO technique.
• does not imply that pages are visited in“random” order.
• only unvisited URLs are to be added to the frontier, this requires some data structure to be
maintained with visited URLs.
• The crawl history is a time-stamped list of URLs fetched by the crawler tracking its path through
the Web.
• A URL is entered into the history only after the corresponding page is fetched.
• This history may be used for post-crawl analysis and evaluation.
• Another important detail is the need to prevent duplicate URLs from being added to the frontier,
a separate hash table can be maintained to store the frontier URLs for fast look-up to check
whether a URL is already in it.
PREFERENTIAL CRAWLERS:
• Follows priority queue technique.
• priority based on an estimate of the value of the linked page.
• The time complexity of inserting a URL into the priority queue is O(logF), where F is the frontier
size (looking up the hash requires constant time).
• To dequeue a URL, it must first be removed from the priority queue (O(logF)) and then from the
hash table (again O(1)). Thus the parallel use of the two data structures yields a logarithmic total
cost per URL.
• Once the frontier’s maximum size is reached, only the best URLs are kept; the frontier must be
pruned after each new set of links is added.
IMPLEMENTATION ISSUES:
1. Fetching
2. Parsing
3. Stopword Removal and Stemming
4. Link Extraction and Canonicalization
5. Spider Traps
6. Page Repository
7. Concurrency
[Link]:
• To fetch pages, a crawler acts as a Web client; it sends an HTTP request to the server hosting
the page and reads the response.
• The client needs to timeout connections to prevent spending unnecessary time waiting for
responses from slow servers or reading huge pages.
• In fact, it is typical to restrict downloads to only the first 10-100 KB of data for each page.
• The client parses the response headers for status codes(three digit code ex:403)and
redirections (more URL address to the page).
• Error-checking and exception handling is important during the page fetching process since the
same code must deal with potentially millions of remote servers.
• Programming languages such as Java, Python and Perl provide simple programmatic
interfaces for fetching pages from the Web.
• However, one must be careful in using high-level interfaces where it may be harder to detect
lower-level problems.
• For example, a robust crawler in Perl should use the Socket module to send HTTP requests
rather than the higher-level LWP library (the World-Wide Web library for Perl). The latter does
not allow fine control of connection timeouts.
[Link]:
• Once (or while) a page is downloaded, the crawler parses its content, i.e., the HTTP
payload, and extracts information.
• The Document Object Model (DOM) establishes the structure of an HTML page as a
tag tree.
• HTML parsers build the tree in a depth-first manner, as the HTML source code of a
page is scanned linearly.(refer the below diagram)
• Even when HTML standards call for strict interpretation, de facto
standards(standards by industry) imposed by browser implementations are
very forgiving.
• This, together with the huge population of non-expert authors generating Web
pages, imposes significant complexity on a crawler's HTML parser.
• Many pages are published with missing required tags, tags improperly nested,
missing close tags, misspelled or missing attribute names and values, missing
quotes around attribute values, unescaped special characters, and so on.
• As an example, the double quotes character in HTML is reserved for tag syntax
and thus is forbidden in text. The special HTML entity & is to be used in its
place. However, only a small number of authors are aware of this, and a large
fraction of Web pages contains this illegal character.
• Crawlers must be forgiving in these cases.
• Preprocessing step taken by robust crawlers is to apply a tool such
as tidy to clean up the HTML content prior to parsing.
• However, if the crawler only needs to extract links within a page
and/or the text in the page, simpler parsers may suffice.
• The HTML parsers available in high-level languages such as Java and
Perl are becoming increasingly sophisticated and robust.
• A growing portion of Web pages are written in formats other than
HTML.
• Crawlers supporting large-scale search engines routinely parse and index
documents in many open and proprietary formats such as plain text, PDF,
Microsoft Word and Microsoft PowerPoint.
• Some formats present particular difficulties as they are written exclusively for
human interaction and thus are especially unfriendly to crawlers.
• For instance, some commercial sites use graphic animations in Flash; these are
difficult for a crawler to parse in order to extract links and their textual
content.
• Other examples include image maps and pages making heavy use of Javascript
for interaction.
• New challenges are going to come as new standards such as Scalable Vector
Graphics (SVG), Asynchronous Javascript and XML (AJAX), and other XML-based
languages gain popularity.
[Link] Removal and Stemming
• When parsing a Web page to extract the content or to score new URLs suggested by the page,
it is often helpful to remove so-called stopwords, i.e., terms such as articles and conjunctions,
which are so common that they hinder the discrimination of pages on the basis of content.
• Stemming(technique), by which morphological variants of terms are conflated into common
roots (stems).
• In a topical crawler where a link is scored based on the similarity between its source page and
the query, which helps improve the matches between the two sets and the accuracy of the
scoring function.
• Both stop-word removal and stemming are standard techniques in information retrieval.
[Link] Extraction and Canonicalization
• HTML parsers provide the functionality to identify tags and associated attribute-value pairs in
a given Web page.
• In order to extract hyperlink URLs from a page, we can use a parser to find anchor (<a>) tags
and grab the values of the associated href attributes.
• However, the URLs thus obtained need to be further processed.
• First, filtering may be necessary to exclude certain file types that are not to be crawled.
• This can be achieved with white lists (e.g., only follow links to text/html content pages) or
black lists (e.g., discard links to PDF files).
• The identification of a file type may rely on file extensions.
• However, they are often unreliable and sometimes missing altogether.
• We cannot afford to download a document and then decide whether we want it or not.
• A compromise is to send an HTTP HEAD request and inspect the content-type response
header, which is usually a more reliable label.
• Another type of filtering has to do with the static or dynamic nature of pages.
• A dynamic page (e.g., generated by a CGI script) may indicate a query interface for a database
or some other application in which a crawler may not be interested.
• Such pages were few and easily recognizable, e.g., by matching URLs against the /cgi-bin/
directory name.
• The use of dynamic content has become much more common; but it is very difficult
to recognize via URL inspection.
• So for these reasons Crawlers normally would not create query URLs autonomously (unless it
is designed to probe the so-called deep or hidden Web, which contain databases with query
interfaces), it will happily crawl URLs hard-coded in HTML source of parsed pages.
• There is one important exception to this strategy, the spider trap.
• Before links can be added to the frontier, relative URLs must be converted to absolute URLs.
• Ex: the relative URL news/[Link] in the page ([Link] is to
be transformed into the absolute form ([Link]
• Canonicalization (sometimes standardization or normalization) is a process for converting data
that has more than one possible representation into a "standard", "normal", or canonical
form.
• Next , convert the absolute URL into canonical form(different crawlers will have different
rules).
• For example, one crawler may always specify the port number within the URL (e.g.,
[Link] while another may specify the port number only when it is
not default 80 (the default HTTP port).
• A crawler may also need to use heuristics to detect when two URLs point to the same page in
order to minimize the likelihood that the same page is fetched multiple times.
[Link] Traps
• These are Web sites where the URLs of dynamically created links are modified based on the
sequence of actions taken by the browsing user (or crawler).
• Some e-commerce sites such as [Link] may use URLs to encode which sequence of
products each user views.
• Consider a dynamic page for product x, whose URL path is /x and that contains a link to product
y.
• The URL path for this link would be /x/y to indicate that the user is going from page x to page y.
• Now suppose the page for y has a link back to product x.
• The dynamically created URL path for this link would be /x/y/x, so that the crawler would think
this is a new page when in fact it is an already visited page with a new URL.
• As a side effect of a spider trap, the server may create an entry in a database every time the
user (or crawler) clicks on certain dynamic links.
• Thus a crawler could go on crawling inside the spider trap forever without actually fetching
any new content.
• Spider traps are not only harmful to the crawler, which wastes bandwidth and disk space to
download and store duplicate or useless data.
• They may be equally harmful to the server sites.
• Then the site may be disabled as a result.
• This is a type of denial of service attack carried out unwittingly by the crawler.
• Some cases a spider trap needs the client to send a cookie set by the server for the dynamic
URLs to be generated.
• So the problem is prevented if the crawler avoids accepting or sending any cookies.
• Proactive approach is necessary to defend a crawler against spider traps.
[Link] Repository
• A page repository may store the crawled pages as separate files.
• In this case each page must map to a unique file name.
• One way to do this is to map each page's URL to a compact string using some hashing function
with low probability of collisions, e.g., MD5.
• The resulting hash value is used as a (hopefully) unique file name.
• So, here occurs time and space wastage.
• Efficient solution is to combine many pages into one file.
• Special markup to separate and identify the pages within the file.
• This requires a separate look-up table to map URLs to file names and IDs within each file.
• A better method is to use a database to store the pages, indexed by (canonical) URLs.
• Since traditional RDBMSs impose high overhead, embedded databases such as the open-
source Berkeley DB are typically preferred for fast access.
• Many high-level languages such as Java and Perl provide simple APIs to manage Berkeley DB
files, for example as tied associative arrays.
• This way the storage management operations become nearly transparent to the crawler
code, which can treat the page repository as an in-memory data structure.
[Link]
• A crawler consumes three main resources: network, CPU, and disk.
• Each is a bottleneck with limits imposed by bandwidth, CPU speed, and disk seek/transfer times.
• The simple sequential crawler described in Sect. 8.1 makes a very inefficient use of these
resources because at any given time two of them are idle while the crawler attends to the third.
• The most straightforward way to speed-up a crawler is through concurrent processes or
threads.
• Multiprocessing may be somewhat easier to implement than multithreading depending on the
programming language and platform, but it may also incur a higher overhead due to the
involvement of the operating system in the management (creation and destruction) of child
processes.
• Whether threads or processes are used, a concurrent crawler may follow a standard parallel
computing model as illustrated in Fig. 8.3.
• Basically each thread or process works as an independent crawler, except for the fact that
access to the shared data structures (mainly the frontier, and possibly the page repository)
must be synchronized.
• In particular a frontier manager is responsible for locking and unlocking the frontier data
structures so that only one process or thread can write to them at one time.
• Both enqueueing and dequeuing are write operations.
• Additionally, the frontier manager would maintain and synchronize access to other shared
data structures such as the crawl history for fast look-up of visited URLs.
• It is a bit more complicated for a concurrent crawler to deal with an empty frontier than for a
sequential crawler.
• An empty frontier no longer implies that the crawler has reached a dead-end, because other
processes may be fetching pages and adding new URLs in the near future.
• The process or thread manager may deal with such a situation by sending a temporary sleep
signal to processes that report an empty frontier.
• The process manager needs to keep track of the number of sleeping processes; when all the
processes are asleep, the crawler must halt.
• The concurrent design can easily speed-up a crawler by a factor of 5 or 10.
• The concurrent architecture however does not scale up to the performance needs of a
commercial search engine.
Universal Crawlers
• Universal crawlers, also known as general-purpose or breadth-first crawlers, aim to traverse and
index a large portion of the web.
• Large-scale universal crawlers differ from the concurrent breadth-first crawlers described above
along two major dimensions:
1. Performance: They need to scale up to fetching and processing hundreds of thousands of
pages per second. This calls for several architectural improvements.
2. Policy: They strive to cover as much as possible of the most important pages on the Web,
while maintaining their index as fresh as possible. These goals are, of course, conflicting so that
the crawlers must be designed to achieve good tradeoffs between their objectives.
The main issues in meeting these requirements:
• Scalability
• Coverage vs Freshness vs Importance
• Scalability
• Change from the concurrent model is the use of asynchronous sockets in place of threads
or processes with synchronous sockets.
• Asynchronous sockets are non-blocking, so that a single process or thread can keep
hundreds of network connections open simultaneously and make efficient use of network
bandwidth.
• Instead, the sockets are polled to monitor their states.
• When an entire page has been fetched into memory, it is processed for link extraction and
indexing.
• This “pull” model eliminates contention for resources and the need for locks.
• The frontier manager can improve the efficiency of the crawler by maintaining several
parallel queues, where the URLs in each queue refer to a single server.
• In addition to spreading the load across many servers within any short time interval, this
approach allows to keep connections with servers alive over many page requests, thus
minimizing the overhead of TCP opening and closing handshakes.
• The crawler needs to resolve host names in URLs to IP addresses.
• The connections to the Domain Name System (DNS) servers for this purpose are one of the
major bottlenecks of a naïve crawler, which opens a new TCP connection to the DNS server for
each URL.
• To address this bottleneck, the crawler can take several steps.
• First, it can use UDP instead of TCP as the transport protocol for DNS requests.
• While UDP does not guarantee delivery of packets and a request can occasionally be dropped,
this is rare.
• On the other hand, UDP incurs no connection overhead with a significant speed-up over TCP.
• Second, the DNS server should employ a large, persistent, and fast (in-memory) cache.
• Finally, the pre-fetching of DNS requests can be carried out when links are extracted from a
page.
I
• Coverage vs Freshness vs Importance
• Not possible to crawl all contents of web.
• Only important pages can be crawled , where importance can be
based on several factors.
• a simple breadth-first crawling algorithm will tend to fetch the
pages with the highest PageRank
• [Link]
Universal crawlers
• Support universal search engines
• Large-scale
• Huge cost (network bandwidth) of crawl is
amortized over many queries from users
• Incremental updates to existing index and other
data repositories
Large-scale universal crawlers
▪ Two major issues:
Performance
Need to scale up to billions of pages
Policy
Need to trade-off coverage, freshness, and bias
(e.g. toward “important” pages)
Large-scale crawlers: scalability
• Need to minimize overhead of DNS lookups
• Need to optimize utilization of network bandwidth and
disk throughput (I/O is bottleneck)
• Use asynchronous sockets
– Multi-processing or multi-threading do not scale up to billions of
pages
– Non-blocking: hundreds of network connections open
simultaneously
– Polling socket to monitor completion of network transfers
Several parallel
queues to spread load DNS server using UDP
across servers (keep (less overhead than
connections alive) TCP), large persistent
in-memory cache, and
prefetching
High-level
architecture of a
scalable universal
crawler
Optimize use of
network bandwidth
Huge farm of crawl machines Optimize disk I/O throughput
Universal crawlers: Policy
• Coverage
– New pages get added all the time
– Can the crawler find every page?
• Freshness
– Pages change over time, get removed, etc.
– How frequently can a crawler revisit ?
• Trade-off!
– Focus on most “important” pages (crawler bias)?
– “Importance” is subjective
Maintaining a “fresh” collection
• Universal crawlers are never “done”
• High variance in rate and amount of page changes
• HTTP headers are notoriously unreliable
– Last-modified
– Expires
• Solution
– Estimate the probability that a previously visited page has
changed in the meanwhile
– Prioritize by this probability estimate
Estimating page change rates
• Algorithms for maintaining a crawl in which most pages
are fresher than a specified epoch
– Brewington & Cybenko; Cho, Garcia-Molina & Page
• Assumption: recent past predicts the future
(Ntoulas, Cho & Olston 2004)
– Frequency of change not a good predictor
– Degree of change is a better predictor
Do we need to crawl the entire Web?
• If we cover too much, it will get stale
There is an abundance of pages in the
Web
• For PageRank, pages with very low prestige are
largely useless
• What is the goal?
- General search engines: pages with high
- prestige News portals: pages that change often
- Vertical portals: pages on some topic
• What are appropriate priority measures in these
cases? Approximations?
Preferential crawlers
• Assume we can estimate for each page an importance
measure, I(p)
• Want to visit pages in order of decreasing I(p)
• Maintain the frontier as a priority queue sorted by I(p)
• Possible figures of merit:
– Precision ~
| p: crawled(p) & I(p) > threshold | / | p: crawled(p) |
– Recall ~
| p: crawled(p) & I(p) > threshold | / | p: I(p) > threshold |
Preferential crawlers
• Selective bias toward some pages, eg. most
“relevant”/topical, closest to seeds, most popular/largest
PageRank, unknown servers, highest rate/amount of
• change, etc…
Focused crawlers
• – Supervised learning: classifier based on labeled examples
Topical crawlers
- Best-first search based on similarity(topic, parent)
- Adaptive crawlers
• Reinforcement learning
• Evolutionary algorithms/artificial life
Preferential crawling algorithms:
Examples
• Breadth-First
– Exhaustively visit all links in order encountered
•
Best-N-First
- Priority queue sorted by similarity, explore top N at a time
• - Variants: DOM context, hub scores
PageRank
• – Priority queue sorted by keywords, PageRank
• SharkSearch
– Priority queue sorted by combination of similarity, anchor text,
similarity of parent, etc. (powerful cousin of FishSearch)
InfoSpiders
– Adaptive distributed algorithm using an evolving population of
learning agents
Preferential crawlers: Examples
• = PageRank
For I(p)
(estimated based on pages
crawled so far), we can find
high-PR pages faster than a
breadth-first crawler (Cho,
Garcia- Molina & Page Recall
1998)
Crawl size
Focused crawlers: Basic idea
• Naïve-Bayes classifier based on
example pages in desired topic,
c*
• Score(p) = Pr(c*|p)
– Soft focus: frontier is priority queue
using page score
– Hard focus:
• Find best leaf ĉ for p
• If an ancestor c’ of ĉ is in c* then
add links from p to frontier, else
discard
– Soft and hard focus work equally
well empirically
Example: Open Directory
Focused crawlers
• Can have multiple topics with as many classifiers, with
scores appropriately combined (Chakrabarti et al. 1999)
• Can use a distiller to find topical hubs periodically, and add
these to the frontier
• Can accelerate with the use of a critic (Chakrabarti et al.
2002)
• Can use alternative classifier algorithms to naïve-Bayes, e.g.
SVM and neural nets have reportedly performed better (Pant
& Srinivasan 2005)
Topical crawlers
• All we have is a topic (query, description,
keywords) and a set of seed pages (not necessarily
relevant)
• No labeled examples
• Must predict relevance of unvisited links to
prioritize
• Original idea: Menczer 1997, Menczer & Belew
1998
Topical locality
• Topical locality is a necessary condition for a topical crawler to
work, and for surfing to be a worthwhile activity for humans
• Links must encode semantic information, i.e. say something about
neighbor pages, not be random
• It is also a sufficient condition if we start from “good” seed pages
• Indeed we know that Web topical locality is strong :
– Indirectly (crawlers work and people surf the Web)
– From direct measurements (Davison 2000; Menczer 2004, 2005)
Naïve Best-First
BestFirst(topic, seed_urls) {
foreach link (seed_urls) {
Simplest topical crawler: enqueue(frontier, link);
Frontier is priority queue }
based on text similarity while (#frontier > 0 and visited < MAX_PAGES) {
link := dequeue_link_with_max_score(frontier);
between topic and parent doc := fetch_new_document(link);
page score := sim(topic, doc);
foreach outlink (extract_links(doc)) {
if (#frontier >= MAX_BUFFER) {
dequeue_link_with_min_score(frontier);
}
enqueue(frontier, outlink, score);
}
}
}
Best-first variations
• Many in literature, mostly stemming from different
ways to score unvisited URLs. E.g.:
– Giving more importance to certain HTML markup in parent
page
– Extending text representation of parent page with anchor text
from “grandparent” pages (SharkSearch)
– Limiting link context to less than entire page
– Exploiting topical locality (co-citation)
– Exploration vs exploitation: relax priorities
• Any of these can be (and many have been) combined
Evaluation:
• Evaluating a crawler involves metrics like:
• Coverage: The percentage of the target web that the crawler has indexed.
• Efficiency: How quickly the crawler can traverse and index web pages.
• Relevance: How well the collected data matches the intended focus or
topic.
• These metrics help in improving crawler performance and ensuring
effective indexing
Evaluation of topical crawlers
• Goal: build “better” crawlers to support applications
(Srinivasan & al. 2005)
• Build an unbiased evaluation framework
– Define common tasks of measurable difficulty
– Identify topics, relevant targets
– Identify appropriate performance measures
• Effectiveness: quality of crawler pages, order, etc.
• Efficiency: separate CPU & memory of crawler algorithms from
bandwidth & common utilities
Performance matrix
St Ç T Sct Ç Td
c d
target
pages Td Sct
target
å å c ( p,Dd target
c ( p,Dd
descriptions )
p Î S ct depth
)
pÎ S ct
Sct d=2
d=1
d=0
“recall” “precision”
Crawler ethics and conflicts
• Crawlers can cause trouble, even unwillingly, if not properly
designed to be “polite” and “ethical”
• For example, sending too many requests in rapid succession
to a single server can amount to a Denial of Service (DoS)
attack!
– Server administrator and users will be upset
– Crawler developer/admin IP address may be blacklisted
Crawler etiquette (important!)
• Identify yourself
– Use ‘User-Agent’ HTTP header to identify crawler, website with description of
crawler and contact information for crawler developer
– Use ‘From’ HTTP header to specify crawler developer email
– Do not disguise crawler as a browser by using their ‘User-Agent’ string
• Always check that HTTP requests are successful, and in case of error, use
HTTP error code to determine and immediately address problem
• Pay attention to anything that may lead to too many requests to any one server,
even unwillingly, e.g.:
– redirection loops
– spider traps
Crawler etiquette (important!)
• Spread the load, do not overwhelm a server
– Make sure that no more than some max. number of requests to any single
server per unit time, say < 1/second
• Honor the Robot Exclusion Protocol
– A server can specify which parts of its document tree any crawler is or is
not allowed to crawl by a file named ‘[Link]’ placed in the HTTP root
directory, e.g. [Link]
– Crawler should always check, parse, and obey this file before sending any
requests to a server
– More info at:
• [Link]
• [Link]
More crawler ethics issues
• Is compliance with robot exclusion a matter of
law?
– No! Compliance is voluntary, but if you do not comply,
you may be blocked
– Someone (unsuccessfully) sued Internet Archive over a
[Link] related issue
• Some crawlers disguise themselves
– Using false User-Agent
– Randomizing access frequency to look like a
human/browser
– Example: click fraud for ads
More crawler ethics issues
• Servers can disguise themselves, too
– Cloaking: present different content based on User-
Agent
– E.g. stuff keywords on version of page shown to search
engine crawler
– Search engines do not look kindly on this type of
“spamdexing” and remove from their index sites that
perform such abuse
• Case of [Link] made the news
Gray areas for crawler ethics
• If you write a crawler that unwillingly follows
links to ads, are you just being careless, or are you
violating terms of service, or are you violating the
law by defrauding advertisers?
– Is non-compliance with Google’s [Link] in this case
equivalent to click fraud?
• If you write a browser extension that performs
some useful service, should you comply with
robot exclusion?