Module -2
Processing Raw Text
[Link] D N, CSE(AI&ML),VVCE
• The chapter explains how to work with your own text sources
instead of just built-in corpora. It covers:
• Accessing text from local files and the Web.
• Processing text using tokenization, stemming, regular
expressions, and removing HTML markup.
• Producing and saving output in a formatted way.
• Along the way, it reinforces Python concepts like strings, files, and
text handling.
[Link] D N, CSE(AI&ML),VVCE
1-Accessing Text from the Web
Disk Electronic Books
• Project Gutenberg- we can browse the catalog of 25,000 free
online books at [Link] log/ and obtain a
URL to an ASCII text file
• Text number 2554 is an English translation of Crime and
Punishment, and we can access it as follows
[Link] D N, CSE(AI&ML),VVCE
• For our language processing, we want to break up the string into
words and punctuation-tokenization, and it produces our familiar
structure, a list of words and punctuation
• NLTK was needed for tokenization, but not for any other tasks of
opening a URL and reading it into a string.
[Link] D N, CSE(AI&ML),VVCE
We may not need all these contents
, so lets see how we can find the relevant content
[Link] D N, CSE(AI&ML),VVCE
Dealing with HTML
[Link] wont work use
beautiful soup
[Link] D N, CSE(AI&ML),VVCE
• Task: Fetch a BBC News story (“Blondes to die out in 200 years”) from a web
page.
• Process:
• Download HTML with urlopen().
• Clean the HTML using nltk.clean_html() ( this function is now deprecated —
BeautifulSoup is used today).
• Tokenize the cleaned text with nltk.word_tokenize().
• Trim off navigation/boilerplate by manually selecting the token range
[96:399].
• Wrap tokens in an [Link] object.
• Use .concordance("gene") to see the contexts where “gene” appears.
(beautifulsoup is a library used in parsing html /xml)
[Link] D N, CSE(AI&ML),VVCE
Processing Search Engine Results
• Web as a corpus: The internet can be seen as a vast collection of
unannotated text.
• They cover huge amounts of text, so even rare patterns can yield
many results.
• Easy to use for quickly checking linguistic theories or examples.
• Only simple queries (words/phrases), unlike complex searches
possible in a local corpus.
[Link] D N, CSE(AI&ML),VVCE
Processing RSS Feeds
• The blogosphere as a valuable source of raw text, now lets see
how Python can pull content from blogs using the Universal Feed
Parser library.
• RSS(Really Simple Syndication) is a web feed format (an XML
file) that websites use to publish updates — like news headlines,
blog posts, podcasts, or articles.
[Link] D N, CSE(AI&ML),VVCE
Reading Local Files
• open(filename, mode) → opens a file.
• 'r' = read mode
• 'rU' = universal newline mode
• .read() → reads the entire file content into a single string.
• [Link](path) → lists all files in the given directory (default "." = current
directory).
• for line in f: → iterates over each line in the file object.
• .strip() → removes whitespace (including \n) from start and end of a
string.
• [Link](resource_name) → finds the path of a corpus file in
NLTK’s data directory.
[Link] D N, CSE(AI&ML),VVCE
•Extracting Text from PDF, MSWord, and Other Binary Formats
•pypdf (formerly PyPDF2) → for extracting text from PDF files.
•pywin32 → for automating MS Word/Office applications on Windows
Capturing User Input:
s = input("Enter some text: ")
print("You typed", len(nltk.word_tokenize(s)), "words.")
[Link] D N, CSE(AI&ML),VVCE
2-Strings: Text Processing at the Lowest Level
Basic Operations with Strings
Defining Strings in Python:
monty = 'Monty Python'
circus = "Monty Python's Flying Circus"
Escaping quotes with backslash → needed if single quotes are
inside single-quoted string
circus = 'Monty Python\'s Flying Circus’ -correct
circus = 'Monty Python's Flying Circus' # Error
[Link] D N, CSE(AI&ML),VVCE
• Triple quoted string :
• Concatenation :
• Subtraction/division /wont work
[Link] D N, CSE(AI&ML),VVCE
• Printing Strings
# Single string
text = "Hello, world!"
print(text)
# Directly print a string literal
print("Python is fun!")
# Concatenate strings
name = "Alice"
print("Hello, " + name + "!")
# Repeat strings
print("Ha" * 3) # → HaHaHa
[Link] D N, CSE(AI&ML),VVCE
• Accessing Individual Characters
Check for:
Monty[0]
Monty[5]
Monty[7]
monty[-1]
monty[-3]
[Link] D N, CSE(AI&ML),VVCE
Accessing Substrings
a="monty python“
a[6:10]
a[-17:-10]
a[:5]
a[6:]
phrase = 'And now for something completely different’
if ‘thing' in phrase:
print("found")
[Link]("python")
[Link] D N, CSE(AI&ML),VVCE
More Operations on Strings
[Link] D N, CSE(AI&ML),VVCE
• Write a Python program to demonstrate different string methods.
Your program should:
• Take a sample string s = " Hello World, hello Python! ".
• Use the following string functions on it and display the results with
proper labels:
• find() and rfind()
• index() and rindex()
• join() with a list of words
• split() and splitlines()
• lower(), upper(), and title()
• strip()
• replace()
[Link] D N, CSE(AI&ML),VVCE
Differences between string & Lists
1. Both are sequences
• You can access elements with indexing ([]) and slicing ([:]).
• Example:
• query = "Who knows?"
• beatles = ["John", "Paul", "George", "Ringo"]
• print(query[2]) # 'o' (character at index 2)
• print(beatles[2]) # 'George' (element at index 2)
• print(query[:2]) # 'Wh'
• print(beatles[:2]) # ['John', 'Paul']
[Link] D N, CSE(AI&ML),VVCE
• 2. Concatenation (+)
• You can add strings together, and you can add lists together.
• But you cannot mix them directly:
• print(query + " I don't")
# "Who knows? I don't"
• print(beatles + ["Brian"])
# ['John', 'Paul', 'George', 'Ringo', 'Brian']
• print(beatles + "Brian")
• # TypeError: can only concatenate list (not "str") to list
[Link] D N, CSE(AI&ML),VVCE
3. Iteration granularity
• String iteration → gives you characters, one by one.
• List iteration → elements can be anything (words, sentences, even other lists).
• That’s why in NLP (Natural Language Processing), we often tokenize strings into
lists of words.
4. Mutability
Strings are immutable → once created, you cannot change them
whereas Lists are mutable → you can modify, add, or remove elements.
query = "Who knows?"
query[0] = "Fun" # Error: strings cannot be changed
beatles = ["John", "Paul", "George", "Ringo"]
beatles[0] = "John Lennon"
del beatles[-1]
print(beatles)
# ['John Lennon', 'Paul', 'George']
[Link] D N, CSE(AI&ML),VVCE
3-Text Processing with Unicode
• Why Unicode?
• “Plain text” doesn’t really exist — different languages need different
characters.
• ASCII → only supports 128 characters (English letters, digits, symbols).
• Extended Latin sets → add characters like ø, ñ, ő, ň.
• Unicode → universal system that supports over 1 million characters, each
with a code point (like \u0144 = ń).
• Unicode in Python:
• Code points are written like \uXXXX. In Python 3, all strings are Unicode by
default.
• a = u'\u0061' # Unicode for 'a'
• print(a) #a
[Link] D N, CSE(AI&ML),VVCE
• Encoding vs Decoding
• Decoding → converting bytes → Unicode (reading from a file).
• Encoding → converting Unicode → bytes (writing to a file).
• Common encodings:
• ascii → only English.
• latin2 → supports Central/Eastern European characters.
• utf-8 → supports all Unicode characters.
• Example (using codecs):
import codecs
f = [Link](path, encoding='latin2') # decode Latin-2 bytes into
Unicode
Unicode Escape
• unicode_escape shows non-ASCII characters as escape codes (\uXXXX
or \xXX).
[Link] D N, CSE(AI&ML),VVCE
• The unicodedata module shows the Unicode code point and
name of characters
• Hexadecimal string of codepoint
• Regular expressions (re) fully support Unicode.
[Link] D N, CSE(AI&ML),VVCE
• NLTK tokenizers accept Unicode and return Unicode tokens.
import nltk
text = u"świątowej na Dolny Śląsk"
print(nltk.word_tokenize(text))
# ['świątowej', 'na', 'Dolny', 'Śląsk']
• Local Encoding in Python Files -To make sure your Python file itself
handles special characters, add this at the top:
• # -*- coding: utf-8 -*-
• This tells Python to treat your source code as UTF-8, so you can write
characters like ś, ó, ł directly in the file.
[Link] D N, CSE(AI&ML),VVCE
4-Regular Expressions for Detecting Word
Patterns
• Regular expressions are patterns that let us search for specific sequences of
characters in text.
Examples:
• "ed$" → finds words ending with ed.
• "^un" → finds words starting with un.
• "a.b" → finds words with "a", then any character, then "b" (like "acb", "a9b").
• To use regular expressions in Python, we need to import the re library using:
import re.
• import re
• # [Link](pattern, string) checks if pattern exists inside string
• if [Link]("ed$", "abandoned"):
• print("Match!") # will print
[Link] D N, CSE(AI&ML),VVCE
Words ending with ed
[Link] D N, CSE(AI&ML),VVCE
Crossword helper:
• Suppose we want an 8-letter word where:
• 3rd letter = j
• 6th letter = t
• Regex pattern: ^..j..t..$
Special symbols:
^-start of the word, $-end of the word
.-any one character,?-previous character
[Link] D N, CSE(AI&ML),VVCE
• [Link]("^e-?mail$", "email")
• [Link]("^e-?mail$", "e-mail")
Matches:email,e-mail,
Doesnot match:emails, myemails,e---mail
[Link] D N, CSE(AI&ML),VVCE
Ranges and closures in regular expressions
• Ranges and Closures
• The T9 system is used for entering text on mobile phones
• 1. Square Brackets [ ]
• They define a set of allowed characters.
• Example:
• [ghi] → match either g, h, or i.
• [a-z] → match any lowercase letter.
• [0-9] → match any digit.
In the T9 example, ^[ghi][mno][jlk][def]$
• 2. Ranges
• Inside brackets, a-f means all characters from a to f.
• Example: [g-o] → any letter between g and o.
• Example: ^[g-o]+$ → whole word made only of letters g–o.
[Link] D N, CSE(AI&ML),VVCE
3. The Plus Sign + (Kleene Closure)
• Means “one or more of the preceding item.”
• Example: m+ → at least one m (mmm, mmmmm).
• Example: [ha]+ → any combination of h’s and a’s like ha, hahahaha, aaaahhh.
4. The Asterisk * (Kleene Closure)
• Means “zero or more of the preceding item.”
• Example: m* → could be empty, or m, or mmmmmm.
• Example: ^m*i*n*e*$ → matches mine, min, me, miiiiiiiinnnnneeee, etc.
5. The Question Mark ?
• Means “zero or one instance” (optional).
• Example: colou?r → matches color or colour.
6. Curly Braces { }
• Used for specifying repetitions:
• {n} → exactly n times
• {n,} → at least n times
• {,n} → up to n times
• {m,n} → between m and n times
• Example: [0-9]{4} → exactly 4 digits (like 2025).
[Link] D N, CSE(AI&ML),VVCE
• 7. Parentheses ( )
• Used for grouping.
• Example: (ed|ing)$ → word ending with ed or ing.
• Example: w(i|e|ai|oo)t → matches wit, wet, wait, woot.
• 8. The Pipe |
• Means “OR”.
• Example: cat|dog → matches either cat or dog.
• 9. The Caret ^
• At the start: means beginning of string.
• Inside brackets: means NOT.
• Example: ^[^aeiou]+$ → words made entirely of non-vowels (like grrr, zzz).
• 10. The Dollar Sign $
• Matches end of string.
• Example: ing$ → word ending in "ing"
• 11. Raw Strings in Python r''
• Prevent Python from treating \ as special (like \b = backspace).
• Example:
• Normal string: '\band\b' → would confuse Python.
• Raw string: r'\band\b' → safely passed to regex engine (means word-boundary "and").
[Link] D N, CSE(AI&ML),VVCE
5- Useful Applications of Regular Expressions
• 1. Extracting Word Pieces with [Link]()
[Link] D N, CSE(AI&ML),VVCE
2. Doing More with Word Pieces
[Link] D N, CSE(AI&ML),VVCE
[Link] D N, CSE(AI&ML),VVCE
3 Finding Word Stems
[Link] D N, CSE(AI&ML),VVCE
[Link] D N, CSE(AI&ML),VVCE
• Searching Tokenized Text
[Link] D N, CSE(AI&ML),VVCE
3.6 Normalizing Text
• We have done some of the normalization such as converting to
lower case, stemming , stripping affixes .
• A further step is to make sure that the resulting form is a known
word in a dictionary, a task known as lemmatization
[Link] D N, CSE(AI&ML),VVCE
Stemming:
• This program uses NLTK’s PorterStemmer to find all occurrences
of a word and its variations in a text, by comparing stems rather
than exact word forms
[Link] D N, CSE(AI&ML),VVCE
• Lemmatization = reducing a word to its dictionary form (lemma),
instead of just chopping off endings like a stemmer.
[Link] D N, CSE(AI&ML),VVCE
7- Regular Expressions for Tokenizing Text.
• Simple Approaches to Tokenization
[Link] D N, CSE(AI&ML),VVCE
• [Link](r'\W+', raw) splits text whenever it sees a chunk of non-
word characters.(to tokenize into words (only) this is best)
• [Link](r'\w+|\S\w*', raw)-captures punctuation attached to
words or standalone symbols.
[Link] D N, CSE(AI&ML),VVCE
• \w+(?:[-']\w+)* - goes through the string and matches any of
these patterns, giving a list of tokens that include:
• Words (including hyphenated words and contractions)
• Standalone punctuation
• Symbols
[Link] D N, CSE(AI&ML),VVCE
• Regular Expression Symbols
[Link] D N, CSE(AI&ML),VVCE
The regex uses several parts to tokenize text cleanly. (?:[A-Z]\.)+ matches abbreviations like “U.S.A.”, and the
?: makes it non-capturing so the full match is returned rather than just the last group. \w+(?:[-']\w+)* matches
regular words, hyphenated words, or contractions such as “poster-print” or “can't.” \$?\d+(?:\.\d+)?%? handles
numbers, currency like $12.40, and percentages like 82%, with non-capturing groups for decimal parts. \.\.\.
captures ellipses as single tokens, while [][.,;"'?():-_]matches individual punctuation marks.
[Link] the regex to be split over multiple lines with comments, [Link]()returns a
clean list of tokens instead of tuples, producing['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...']`.
[Link] D N, CSE(AI&ML),VVCE
8- Segmentation
• Segmentation is the process of dividing text into meaningful units,
such as sentences or words. Sentence segmentation splits text
into sentences, which is necessary before tokenizing words.
• Before tokenizing the text into words, we need to segment it into
sentences. NLTK facilitates this by including the Punkt sentence
segmenter
[Link] D N, CSE(AI&ML),VVCE
[Link] D N, CSE(AI&ML),VVCE
how to compute the “cost” of a given segmentation of text using the evaluate()
first, it uses segment() to split the text into words according to a segmentation string of 0s and 1s; second, it
calculates the cost as the sum of two parts:
text_size – the total number of words in the segmentation.
lexicon_size – the total number of characters in the set of unique words (the “lexicon”) needed to
reconstruct the text.
The objective is to minimize this cost: a lower value means the segmentation is more efficient, balancing
both the number of words and the size of the lexicon.
[Link] D N, CSE(AI&ML),VVCE
Non-deterministic search using simulated annealing: Begin searching with phrase
segmentations only; randomly perturb the zeros and ones proportional to the
“temperature”; with each iteration the temperature is lowered and the perturbation
of boundaries is reduced ,it is possible to automatically segment text into words with
a rea sonable degree of accuracy
[Link] D N, CSE(AI&ML),VVCE
9- Formatting: From Lists to Strings(SLT)
• 1. Converting Lists to Strings
• Often, text processing produces lists of words, e.g., ['We', 'called',
'him', 'Tortoise'].
• To display or save these lists, we convert them into strings using
join().
[Link] D N, CSE(AI&ML),VVCE
string formatting expressions
Placeholder
[Link] D N, CSE(AI&ML),VVCE
• The %s and %d symbols are called conversion specifiers. They
start with the % character and end with a conversion character
such as s (for string) or d (for decimal integer) The string
containing conversion specifiers is called a format string.
[Link] D N, CSE(AI&ML),VVCE
Lining Things Up
[Link] D N, CSE(AI&ML),VVCE