Module 2
Working with data
• Getting Data
• stdin and stdout
• Reading Files
• Scraping the Web
• Using APIs
• Exploring the Data
• Using Named Tuples
• Data classes
• Cleaning and Munging
• Manipulating Data
• Rescaling
• An Aside: tqdm
• Dimensionality Reduction
• The Curse of Dimensionality
• PCA
Getting Data: stdin and stdout
• Python’s [Link] (standard input) and [Link] (standard output)
are useful for processing data from the command line.
• They allow you to build data pipelines, where the output of one
program is used as input for another.
• [Link] → Reads input data (e.g., from a file, keyboard, or another
program).
• [Link] → Prints output (e.g., to the screen or another program).
• [Link] → Used for error messages (does not interfere with
standard output).
import sys
for line in [Link]: # Read from standard input
[Link]([Link]()) # Write to standard output
Reading Files
• Reading files is an essential skill for data science, allowing you to work
with text, CSVs, logs, and more.
• This section explains how to open, read, and process files efficiently
in Python.
Opening Files in Python
• The open() function is used to open a file in different modes:
Using with open()
• To avoid forgetting to close files, use the with statement
Counting Lines That Start with #
Extracting Domains from Email Addresses
• If you have a file where each line contains an email, you can extract
the domains:
Delimited Files
• Delimited files are text files where values are separated by a specific
delimiter (such as a comma ,, tab \t, or semicolon ;).
• The most common type is CSV (Comma-Separated Values).
Reading a CSV File in Python
• Python provides the built-in csv module to handle delimited files.
Writing to a CSV File
• To write data to a CSV file, use [Link]()
Reading as a Dictionary
Scraping the Web
• Web scraping is the process of automatically extracting data from
websites using programming.
• Instead of manually copying information from a webpage, you can
use Python to fetch the page content, extract useful data, and process
it.
• To collect data from news articles, e-commerce sites, social media, or
public APIs.
• To analyze trends, such as tracking product prices or monitoring
reviews.
• To gather large datasets for machine learning and data science
projects.
• Web scraping involves four key steps:
• Fetching the webpage: Using requests or urllib to download the HTML
content.
• Parsing the HTML: Using BeautifulSoup or lxml to extract specific elements.
• Navigating the structure: Finding data within tags like <div>, <span>, and <a>.
• Extracting and saving the data: Cleaning and storing it in a structured format
like CSV or a database.
HTML and Parsing Thereof
• When scraping the web, we first need to understand HTML
(HyperText Markup Language) because it's the structure of almost
every webpage.
• Parsing HTML means processing the raw HTML and extracting
meaningful data.
• Python provides libraries like BeautifulSoup to extract useful
information from raw HTML.
Basics of HTML Structure
• HTML is a structured format with tags that define elements on a
webpage
Fetching HTML Content
• First, we fetch the webpage content using the requests module.
• The BeautifulSoup library helps navigate and extract specific
elements.
• find specific HTML tags like <h1>, <p>, and <a>.
• Fetch HTML with [Link](url).text
• Parse it with BeautifulSoup(html, "[Link]")
• Use find() and find_all() to extract elements
• Extract attributes like href from links
• Use CSS selectors (select() and select_one()) for more precision
• Strip extra spaces using .strip()
Using APIs
• APIs (Application Programming Interfaces) allow you to
programmatically request data from web services in a structured
format.
• Instead of scraping websites for data, APIs provide a more efficient
and reliable way to access information.
JSON (and XML)
• APIs often return data in JSON (JavaScript Object Notation), which is
easy to parse in Python.
• JSON is similar to Python dictionaries and can be loaded using the
json module.
• Some APIs may return XML, which can be parsed using BeautifulSoup.
Using an Unauthenticated API
• Many APIs require authentication, but some allow limited access
without authentication.
• GitHub's API is one such example, allowing users to fetch public
repository data without logging in.
Fetching GitHub Repositories Without
Authentication
• The following Python script fetches public repositories of a GitHub
user
Extracting Repository Creation Dates
• The GitHub API provides a "created_at" timestamp for each
repository, but it's a string (e.g., "2013-07-05T02:02:28Z").
• Python does not have a built-in date parser for such formats, so we
need the python-dateutil package:
• Install python-dateutil
• pip install python-dateutil
• Parse Creation Dates
• We can use [Link]() to convert these date strings into Python
datetime objects
• Counter([Link] for date in dates): Counts how many repositories
were created in each month.
• Counter([Link]() for date in dates): Counts how many
repositories were created on each weekday (0 = Monday, 6 = Sunday).
Finding APIs
• Check the developer section of websites.
• Search for python <API_name> wrapper to find existing Python
libraries.
• Use API directories like:
• ProgrammableWeb
• Python API Directory
• If no API exists, web scraping might be the only option.
Exploring Your Data
• To explore the data to understand its structure, characteristics, and
potential issues.
Exploring One-Dimensional Data