Python for Data Science & Engineering: A Comprehensive Cheat Sheet
Introduction
Welcome to your essential quick-reference guide for data science and engineering in
Python. In the fast-paced world of data, having key commands and functions at your
fingertips is crucial for efficiency and accuracy. This cheat sheet consolidates the most
important libraries, functions, and commands for a wide range of common data tasks, from
foundational data cleaning and manipulation to advanced statistical simulation and large-
scale data management. The goal of this document is to provide a practical, at-a-glance
resource to streamline your everyday work and problem-solving.
--------------------------------------------------------------------------------
1. Data Cleaning & Manipulation (Pandas)
Data cleaning is a foundational, non-negotiable step in any data workflow. It involves
identifying and correcting errors, handling missing values, and transforming data into a
usable format for analysis and modeling. The pandas library is the primary tool for these
tasks in Python, offering a powerful and flexible DataFrame object for structured data. The
following sections detail the most critical functions for inspecting, cleaning, and
transforming dataframes to ensure your data is accurate and reliable.
1.1. Common Data Cleaning Tasks & Functions
This table provides a high-level overview of common data preparation tasks and the
primary pandas functions used to accomplish them.
Task Function(s) Used Description
Remove
drop_duplicates() Remove duplicate rows from a dataframe.
duplicates
Removes rows with any missing values. Pro-Tip: Use this
Drop missing cautiously on datasets where missing values are sparse,
dropna()
values as it can lead to significant data loss. The subset
parameter is often a safer choice for targeted cleaning.
Impute
Fill in missing values in a dataframe with a specified value
missing fillna()
or method.
values
Casts a column to a specified data type (e.g.,
df['col'].astype('int')). This is crucial for memory
Convert data
astype() optimization (e.g., converting floats to integers where
types
appropriate) and to enable correct mathematical or
categorical operations.
groupby() creates grouped objects based on a column's
Group and values, while agg() performs multiple aggregate functions
aggregate groupby(), agg() (e.g., sum, mean, count) simultaneously. Used together,
data they are the cornerstone of descriptive statistical analysis
in Pandas.
Filter data loc[], iloc[] Filter data in a dataframe using various methods.
Merge data merge(), concat() Merge data from multiple dataframes.
1.2. Data Inspection
Every robust analysis begins with a thorough inspection of the data. These commands are
your reconnaissance tools—use them to quickly assess data dimensions, identify data
type mismatches, and quantify missing values before you write a single line of
transformation code.
Function/Method Description
Preview the first 5 rows of the DataFrame to get a quick look at
[Link]()
the data.
[Link]() Read the last five rows of the DataFrame.
Check the dimensions of the dataset, returning a tuple of
[Link]
(rows, columns).
[Link] Return an array of all column names in the DataFrame.
Provide a concise summary of the DataFrame's structure,
[Link]()
including column data types and non-null value counts.
[Link] Look at the data types for each individual column.
Count the total number of missing (NaN) values for each
[Link]().sum()
column.
Return a single boolean value indicating if any missing values
df[cols].isnull().[Link]()
exist in the dataset.
1.3. Handling Missing or Bad Data
Empty cells, incorrect formats, or erroneous values can compromise your analysis. Use
these methods to address such issues systematically.
Function/Method Description
Removes rows with any missing values. Using inplace=True
[Link]() modifies the DataFrame directly. Pro-Tip: The subset
parameter is often a safer choice for targeted cleaning.
Remove only the rows where a value is missing in a specific
[Link](subset=['col'])
column.
Replace all empty cells in the DataFrame with a specified
[Link](value)
value.
Replace empty cells in a specific column with the mean
df['col'].fillna(df['col'].mean())
value of that column.
Convert a column to the proper datetime format, correcting
pd.to_datetime(df['date'])
potential formatting inconsistencies.
Replace a specific, incorrect value at a given index and col
[Link][index, 'col'] = value
label with a new value.
1.4. Handling Duplicates
Duplicate records can skew analytical results and lead to incorrect conclusions. These
functions are essential for de-duplication.
Function/Method Description
Returns a boolean Series indicating duplicate rows. By
default, the first occurrence of a set of duplicated rows is
[Link]()
marked as False; all subsequent occurrences are
marked True.
Remove all duplicate rows from the DataFrame, keeping
df.drop_duplicates(inplace=True)
only the first occurrence.
1.5. Handling Outliers & Categorical Data
Preparing data for machine learning often requires addressing outliers and converting non-
numeric data. Outliers can disproportionately affect model performance, while categorical
variables must be encoded into a numerical format using libraries like
[Link] or pandas.
Class/Method Library Purpose
Scales data by removing the median and scaling
RobustScaler [Link] according to the interquartile range, making it
robust to outliers.
Converts categorical variables into numerical
pd.get_dummies() pandas dummy/indicator variables, a necessary step for
many modeling algorithms.
With a clean and well-structured dataset, you can move on to more advanced statistical
analysis.
--------------------------------------------------------------------------------
2. Statistical Modeling & Simulation
Beyond cleaning and preparation, Python’s ecosystem offers powerful libraries for
advanced statistical analysis and simulation. These techniques allow you to uncover latent
structures in your data and model the impact of uncertainty. This section provides a
reference for two powerful methods: Factor Analysis for dimensionality reduction and
Monte Carlo simulation for modeling probabilistic outcomes.
2.1. Factor Analysis (factor_analyzer)
Factor Analysis is an exploratory data analysis method used to find underlying, unobserved
latent variables (or "factors") from a set of observed variables. It is highly effective for
reducing the number of variables in a dataset while retaining essential information.
Setup & Installation
To get started, install the factor_analyzer package using pip:
pip install factor_analyzer
Core Classes and Functions
The factor_analyzer library provides a straightforward interface for performing factor
analysis and evaluating data suitability.
Component Module Purpose
The main class used to
FactorAnalyzer() factor_analyzer create and configure the
factor analysis object.
Performs Bartlett's test of
sphericity to check if
observed variables
calculate_bartlett_sphericity(df) factor_analyzer.factor_analyzer intercorrelate. A
significant p-value (e.g.,
0.0) is required to
proceed.
Performs the Kaiser-
Meyer-Olkin (KMO) test to
measure the suitability of
calculate_kmo(df) factor_analyzer.factor_analyzer
the data for factor
analysis. A value > 0.6 is
considered adequate.
Performs the factor
analysis on the dataframe
[Link](df, n_factors,
FactorAnalyzer method with a specified number
rotation)
of factors and rotation
method (e.g., 'varimax').
Retrieves the eigenvalues
for each potential factor.
fa.get_eigenvalues() FactorAnalyzer method Eigenvalues > 1 are
typically retained (Kaiser
Criterion).
An attribute that returns
fa.loadings_ FactorAnalyzer attribute the factor loadings matrix,
which shows the
correlation between each
observed variable and the
underlying factors.
Returns the variance
explained by each factor,
fa.get_factor_variance() FactorAnalyzer method including SS Loadings,
Proportion Variance, and
Cumulative Variance.
2.2. Monte Carlo Simulation
A Monte Carlo simulation is a computational technique used to model the probability of
different outcomes in processes that are influenced by random variables. By repeatedly
sampling from probability distributions, it helps quantify the impact of risk and uncertainty.
Key Libraries and Functions
Building a Monte Carlo simulation in Python typically involves generating random numbers
and visualizing the results.
Library Function Purpose in Simulation
Generates random samples from a continuous
[Link](-1,
numpy uniform distribution within a specified range
1)
(e.g., between -1 and 1).
Simulates a discrete random event with integer
random [Link](0,1)
outcomes, such as a coin flip (0 or 1).
Visualizes the simulation results, such as
[Link](), [Link](),
[Link] plotting the convergence of a probability
[Link]()
estimate over many iterations.
These analytical techniques rely on having the infrastructure to process and store data,
especially at a large scale.
--------------------------------------------------------------------------------
3. Data Lake & Big Data Commands
Effective data analysis at scale depends on robust infrastructure for storing and processing
large volumes of information. A modern data stack often follows a multi-layered
architecture—typically Bronze (raw data), Silver (cleaned data), and Gold (aggregated,
business-ready data)—to ensure data quality and auditability. This section provides
essential commands for working with Docker for containerization and the Hadoop
Distributed File System (HDFS) for storage in this environment.
3.1. Container Management (Docker)
Use Docker to eliminate "it works on my machine" problems. It containerizes your entire
environment, ensuring absolute reproducibility for your data pipelines and making it simple
to deploy a complex stack including services like HDFS.
Command Description
Pulls the latest images for all services defined in the docker-
docker-compose pull
[Link] file.
Starts all services defined in the [Link] file in
docker-compose up -d
the background (detached mode).
Lists all currently running Docker containers, showing their
docker ps
IDs, names, and status.
docker-compose exec Opens an interactive bash shell inside a running container
<service_name> bash specified by its service_name (e.g., namenode).
3.2. Hadoop Distributed File System (HDFS)
HDFS is a distributed file system designed to store very large datasets across clusters of
commodity hardware. It is a core component of the Hadoop ecosystem, and its hdfs dfs
command-line interface is used to interact with data, including files stored in connected
object storage like S3 or MinIO.
Command Description
Lists the files and directories at a given path in HDFS or a connected
hdfs dfs -ls <path>
system (e.g., s3a://bronze/).
hdfs dfs -mkdir -p Creates a new directory at the specified path. The -p flag ensures that
<path> any necessary parent directories are also created.
Copies a file from a source to a destination. Typically used to move data
hdfs dfs -cp
between data lake layers, such as copying a processed file from a
<source>
s3a://gold/ path into HDFS (e.g., hdfs dfs -cp
<destination>
s3a://gold/kpi_summary.parquet /datalake/gold/).
hadoop distcp A specialized tool for large-scale, inter/intra-cluster data copying,
<source> optimized for performance and reliability. Use this for moving entire
<destination> directories between layers.
hdfs dfs -get
Copies a file from an HDFS path down to the local filesystem of the
<hdfs_path>
container or machine you are on.
<local_path>
This cheat sheet serves as a practical, daily reference for the modern data professional,
covering the key tools from data preparation to large-scale infrastructure management.