MODULE - 5
SYLLABUS
• Regular expressions and Web Mining:
• Regular Expressions: Introduction/Motivation, Special Symbols
and Characters for REs, REs and Python.
• Web Mining: Introduction, Categories of Web Mining, Web
Mining Process, Web Scraping using ‘urllib’, ‘Beautifulsoup’
library, ‘find’ and ‘findall’ methods. Web Crawling, Types,
workflow for crawling.
REGULAR EXPRESSION
• Regular expressions (called REs, or regexes, or regex patterns) are essentially a tiny, highly
specialized programming language embedded inside Python and made available through
the re module.
• A regular expression is a special sequence of characters that helps youmatchor find other
string s or sets of string s, using a specialized syntax held ina pattern. Regular expressions
are widely used inUNIX world.
• Regular expressions are used to operate on strings.
• Meta Character: . ^ $ * + ? { } [ ] \ | ( )
• Backslash ‘\’ used to escape all the metacharacters so you can still match them in patterns.
• The module provides an interface to the regular expression engine, allowing you to compile
re
REs into objects and then perform matches with them
COMPILED OBJECT ANS MODULAR
FUNCTION
• prog = [Link](pattern)
• result = [Link](string)
PYTHON REGEX - PERFORING
MATCHING
• Object representing a compiled regular expression, Pattern objects have several
methods and attributes
• [Link](): Attempts to match a pattern at the beginning of a string.
• [Link](): Scans through the entire string and returns the first occurrence of
the pattern.
• [Link](): Returns all occurrences of the pattern in the string as a list.
• [Link](): Replaces parts of the string that match the pattern with another string.
• re. purge()-Clear the cache
.
SPECIAL MODIFIER
• + = match 1 or more
• ? = match 0 or 1 repetitions.
• * = match 0 or MORE repetitions
• $ = matches at the end of string
• ^ = matches start of a string
• | = matches either/or. Example x|y = will match either x or y
• [] = A set of characters in which we define range, or "variance"
• {x} = expect to see this amount of the preceding code.
• {x,y} = expect to see this x-y amounts of the precedng code
IDENTIFIER
• \d = any number
• \D = anything but a number
• \s = space
• \S = anything but a space
• \w = any letter
• \W = anything but a letter
• . = any character, except for a new line
• \b = space around whole words
• \. = period. must use a backslash, because ‘ . ‘ normally means any character.
RE
• Regular expressions are compiled into pattern objects, which
have methods for various operations such as searching for
pattern matches or performing string substitutions.
[[Link](‘ab*’)
• Module-Level Functions :group(),match() search() findall() sub()
¶ , , , .
WEB MINING
• Web Mining is the process of Data Mining techniques to
automatically discover and extract information from Web
documents and services.
• Categories:
• Web Content Mining
• Web Structured Mining
• Web Usage Mining
• n=[7484,9864234532,8618719912,8618819]
• \d,9|8|6|7+\d{9}
• Import re
• b=[Link](‘’)
• [Link](n)
APPLICATIONS
• 1. Web mining is used for social network analysis
• [Link] mining can learn user behavior, compute the effectiveness of a
specific Web site.
• [Link] mining is used for keyphrase extraction.
• 4. Fetch digital images.
• [Link] and Advertising -
• [Link] Detection
WEB MINING PROCESS
Data Data
Data collection Preprocessing integration
Pattern
Evaluation Visualization
discovery
• Python web scraping allows you to collect and parse data from websites
programmatically.
• With powerful libraries like urllib, Beautiful Soup, and MechanicalSoup, you can
fetch and manipulate HTML content effortlessly.
• You can scrape websites with Python by fetching HTML
content using urllib and extracting data using string methods or parsers like
Beautiful Soup.
• Beautiful Soup is a great choice for parsing HTML documents with Python
effectively.
• Data scraping may be illegal if it violates a website’s terms of use, so always
review the website’s acceptable use policy.
WEB CRAWLING
• Web crawling is the process of automatically navigating the
internet to gather information from websites.
• Python is a great choice for web crawling because it’s simple
to learn and has many helpful libraries. Tools like Scrapy,
BeautifulSoup, and Selenium make it easy to crawl websites
and collect data
HOW A WEB CRAWLER WORKS
• [Link] with Seed URLs
• 2. Requesting the Web Page
• 3. Parsing the HTML Content
• 4. Extracting Links
• 5. Following Links
• 6. Parsing the Blog Post
• 7. Extracting More Links
• 8. Storing Data
WEB CRAWLING VS. WEB SCRAPING
• The key difference between crawling and scraping is that web
crawling is used for data indexing, whereas web scraping is
used for data extraction.
• Web scraping, also referred to as web harvesting, is generally
more targeted than crawling.
TYPE OF WEB CRAWLING
• Focused web crawler: A focused web crawler's goal is to crawl content focused on a parameter,
such as content related to a single topic, or from a single type of domain. In order to do so, a
focused web crawler will discern which hyperlinks to follow based on probability.
• Incremental web crawler: An incremental web crawler is a type of crawler bot that revisits pages
in order to update indexes. It replaces old links with new URLs where applicable. This process
serves to reduce inconsistent document downloads.
• Distributed crawler: Distributed crawlers work on different websites simultaneously to fulfill
crawling duties.
• Parallel crawler: A parallel crawler is a type of crawl bot that runs multiple processes
simultaneously — or in parallel — to increase download efficiency.
BENEFITS
LIMITATION
• 1. No depth or page limit
• [Link] link handling
• 3. Single-threaded (serial) scraping
• 4. Lack of error handling or timeout management
• 5. No [Link] handling
• 6. No request delays
• 7. No support for dynamic content