Working With Text Data Options & Customization
Topics Covered
● Introduction to Pandas and Text Data
● Text Data Handling in Pandas
● Common Operations for Text Data in Pandas
● Data Cleaning and Preprocessing Text Data
● Text Data Manipulation and Customization
● Advanced Text Data Handling in Pandas
Introduction to Pandas and Text Data
● Pandas is a powerful Python library used for data manipulation and
analysis.
● It provides data structures like DataFrames that allow easy handling
of structured data, including text data.
Text Data Handling in Pandas
● Pandas offers the Series and DataFrame data structures.
● The Series is a one-dimensional array-like object that can hold
different data types, including text (strings).
● The DataFrame is a two-dimensional tabular data structure consisting
of rows and columns, allowing for efficient handling of text data.
Example:
import pandas as pd
# Creating a DataFrame with text data
data = {'Text': ['Hello, how are you?', 'Pandas is great!', 'Text data
handling in Pandas is useful']}
df = [Link](data)
print(df)
Common Operations for Text Data in Pandas
● Pandas provide various string methods through the ‘str’ accessor for
text data manipulation.
● You can access these methods using the ‘.str’ attribute.
Example:
# Accessing text data and applying string methods
# Converting text to lowercase
df['Lowercase'] = df['Text'].[Link]()
# Finding the length of each text entry
df['Length'] = df['Text'].[Link]()
# Splitting text into words
df['Words'] = df['Text'].[Link]()
print(df)
Output:
Text \
0 Hello, how are you?
1 Pandas is great!
2 Text data handling in Pandas is useful
Lowercase Length \
0 hello, how are you? 19
1 pandas is great! 16
2 text data handling in pandas is useful 38
Words
0 [Hello,, how, are, you?]
1 [Pandas, is, great!]
2 [Text, data, handling, in, Pandas, is, useful]
Analogy:
#Analogy:- Counting the Number of Characters in a Text Column
import pandas as pd
# Sample DataFrame
data = {'Text': ['Hello, how are you?', 'Pandas is great!', 'Text data
handling in Pandas is useful']}
df = [Link](data)
# Counting characters using len() and the str accessor
df['Char_Count'] = df['Text'].[Link]()
print(df)
Output:
Text Char_Count
0 Hello, how are you? 19
1 Pandas is great! 16
2 Text data handling in Pandas is useful 38
#Analogy-2:- Converting Text to Uppercase in a DataFrame Column
# Converting text to uppercase
df['Uppercase_Text'] = df['Text'].[Link]()
print(df)
Text Char_Count \
0 Hello, how are you? 19
1 Pandas is great! 16
2 Text data handling in Pandas is useful 38
Uppercase_Text
0 HELLO, HOW ARE YOU?
1 PANDAS IS GREAT!
2 TEXT DATA HANDLING IN PANDAS IS USEFUL
#Analogy-3:- Extracting Words Starting with a Specific Letter
# Extracting words starting with 'P'
df['P_Words'] = df['Text'].[Link]().apply(lambda x: [word for word in x
if [Link]('P')])
print(df)
Output:
Text Char_Count \
0 Hello, how are you? 19
1 Pandas is great! 16
2 Text data handling in Pandas is useful 38
Uppercase_Text P_Words
0 HELLO, HOW ARE YOU? []
1 PANDAS IS GREAT! [Pandas]
2 TEXT DATA HANDLING IN PANDAS IS USEFUL [Pandas]
Data Cleaning and Preprocessing Text Data
● Data cleaning and preprocessing are crucial steps in text data
analysis. This involves handling missing values, removing duplicates,
and normalizing text.
Handling Missing Values:
● Pandas provides methods like ‘isnull()’ and ‘dropna()’ to handle
missing values in text data:
Example:
# Handling missing values
df['Text'].fillna('No text available', inplace=True) # Filling missing
values with a default text
Removing Duplicates:
● To remove duplicate rows based on text data
# Removing duplicate rows based on 'Text' column
df.drop_duplicates(subset='Text', keep='first', inplace=True)
Text Normalization:
● Text normalization involves converting text to a standard form, such
as converting all text to lowercase or stemming (reducing words to
their root form).
# Text normalization: converting text to lowercase
df['Normalized_Text'] = df['Text'].[Link]()
df['Normalized_Text']
0 hello, how are you?
1 pandas is great!
2 text data handling in pandas is useful
Name: Normalized_Text, dtype: object
Text Data Manipulation and Customization
● Text data manipulation involves various operations such as extracting
specific information, transforming text, and customizing data
according to requirements.
Code
#Analogy-1 : Text Data as a Bookshelf
import pandas as pd
# Sample DataFrame with book titles (text data)
data = {'book_title': ['The Great Gatsby', 'To Kill a Mockingbird',
'1984', 'Pride and Prejudice']}
df = [Link](data)
# Sorting book titles alphabetically
df_sorted = df.sort_values('book_title')
print(df_sorted)
output::
book_title
2 1984
3 Pride and Prejudice
0 The Great Gatsby
1 To Kill a Mockingbird
#Analogy-2 : Text Data as a Letter Editing
import pandas as pd
# Sample DataFrame with text data
data = {'letter_content': ['hello', 'world', 'python', 'pandas']}
df = [Link](data)
# Capitalizing first letter of each word
df['letter_content_capitalized'] = df['letter_content'].[Link]()
print(df)
Output::
letter_content letter_content_capitalized
0 hello Hello
1 world World
2 python Python
3 pandas Pandas
#Analogy-3 : Text Data as Newspaper Articles
import pandas as pd
# Sample DataFrame with news headlines (text data)
data = {'headline': ['Breaking: New discovery in science', 'Politics:
Election results announced', 'Technology: AI revolutionizing industries']}
df = [Link](data)
# Extracting category from headlines
df['category'] = df['headline'].[Link](':').str[0]
print(df)
Output:::
headline category
0 Breaking: New discovery in science Breaking
1 Politics: Election results announced Politics
2 Technology: AI revolutionizing industries Technology
Advanced Text Data Handling in Pandas
● Advanced text data handling involves tokenization, vectorization, and
dealing with multi-indexing to manage and process textual
information more efficiently.
Code:
#Analogy-1 : Newspaper Article Word Count
import pandas as pd
# Sample text data (analogous to a newspaper article)
article = "Pandas is a powerful library for data manipulation. Pandas
provides easy-to-use data structures."
# Tokenizing words and counting their frequency
words = [Link]()
word_freq = [Link](words).value_counts()
print(word_freq)
Output::
Pandas 2
data 2
is 1
a 1
powerful 1
library 1
for 1
manipulation. 1
provides 1
easy-to-use 1
structures. 1
dtype: int64
#Analogy-2 : Product Reviews Sentiment Analysis
from textblob import TextBlob
import pandas as pd
# Sample product review data (analogous to customer reviews)
reviews = [
"The product is fantastic and worth the price.",
"Okay product, but could be better.",
"Terrible product, not recommended."
]
# Performing sentiment analysis on reviews
sentiments = [TextBlob(review).[Link] for review in reviews]
df_sentiments = [Link]({'Review': reviews, 'Sentiment': sentiments})
print(df_sentiments)
Output::
Review Sentiment
0 The product is fantastic and worth the price. 0.35
1 Okay product, but could be better. 0.50
2 Terrible product, not recommended. -1.00
#Analogy-3 : Extracting Important Entities from Articles
import spacy
import pandas as pd
# Sample text data (analogous to a news article)
text = "Elon Musk, the CEO of SpaceX, announced plans to colonize Mars."
# Loading spaCy's English model
nlp = [Link]("en_core_web_sm")
# Performing Named Entity Recognition (NER)
doc = nlp(text)
entities = [([Link], ent.label_) for ent in [Link]]
df_entities = [Link](entities, columns=['Entity', 'Label'])
print(df_entities)
Output::
Entity Label
0 Elon Musk PERSON
1 Mars LOC