Data
Pre-processing
Deepak Bhatlavande
– Data Scientist
DA vs BI vs BA
• Data Analyst vs. BI Analyst:
Both roles focus on data, but Data Analysts are more concerned with analyzing raw data
to find insights, while BI Analysts focus on creating and managing BI solutions to visualize and
report on data.
• BI Analyst vs. Business Analyst:
BI Analysts work closely with data and reporting tools, while Business Analysts focus
more on understanding business needs and translating them into technical or process solutions.
• Data Analyst vs. Business Analyst:
Data Analysts dive into data to find insights, while Business Analysts focus on
understanding and improving business processes.
Different Types of Analytics
• Descriptive analytics
Analyzes past data to answer questions like "What has
happened?" or "What is happening?
• Diagnostic analytics
Analyzes past data to identify the root causes of an outcome,
answering questions like "Why did this happen?"
• Predictive analytics
Uses historical data, machine learning, and other techniques
to predict future events or outcomes, answering questions like "What
may happen next?"
• Prescriptive analytics
Uses data, mathematical algorithms, and business rules to
suggest the best course of action for a specific scenario, answering
questions like "What should I do?" .
Stages in Data Analytics
BUSINESS ISSUE DATA COLLECTION DATA PRE-PROCESSING COLLECT INSIGHT VIZ 4 DIS
The Data Analysis Process
• The first step in the data analysis process is to define the objectives and formulate clear, specific questions that your
analysis aims to answer. This step is crucial as it sets the direction for the entire process. It involves understanding the
problem or situation at hand, identifying the data needed to address it, and defining the metrics or indicators to measure
the outcomes.
• Once the objectives and questions are defined, the next step is to collect the relevant data. This can be done through
various methods such as surveys, interviews, observations, or extracting from existing databases. The data collected can
be quantitative (numerical) or qualitative (non-numerical), depending on the nature of the problem and the questions
being asked.
• Data cleaning, also known as data cleansing, is a critical step in the data analysis process. It involves checking the data for
errors and inconsistencies, and correcting or removing them. This step ensures the quality and reliability of the data,
which is crucial for obtaining accurate and meaningful results from the analysis.
• Once the data is cleaned, it's time for the actual analysis. This involves applying statistical or mathematical techniques to
the data to discover patterns, relationships, or trends. There are various tools and software available for this purpose, such
as Python, R, Excel, etc.
• After the data is analyzed, the next step is to interpret the results and visualize them in a way that is easy to understand.
This could involve creating charts, graphs, or other visual representations of the data. Data visualization helps to make
complex data more understandable and provides a clear picture of the findings.
• The final step in the data analysis process is data storytelling. This involves presenting the findings of the analysis in a
narrative form that is engaging and easy to understand. Data storytelling is crucial for communicating the results to non-
technical audiences and for making data-driven decisions.
DATA TYPE
Data types refer to the classification of data based on the kind of values they represent. Mainly types of data are
broadly categorized into quantitative and qualitative, along with some subcategories.
Quantitative Data BIG DATA
• Discrete Data – Whole Number
• Continuous Data – Date / Time
Qualitative Data
• Nominal Data
• Ordinal Data
• Binary Data
• Interval Data
• Ratio Data
• Time-Series Data / Cross sectional
• Panel or Longitudinal Data
Structured vs. Unstructured Data vs. Semi-Structured Data
Data Pre-processing
Data Wrangling (Data Munging)
The process of cleaning and transforming raw data into a useful format
for analysis.
Data Cleaning
A subset of data preprocessing and data wrangling focused specifically on
identifying and correcting (or removing) errors and inconsistencies in data.
Data Mining
The process of discovering patterns, correlations, and insights from large sets of
data using statistical, mathematical, and computational techniques.
KEY Difference:-
Data Preprocessing: Broad term covering all steps needed to prepare data for machine learning models.
Data Wrangling: Focused on transforming and integrating data from various sources into a usable format.
Data Cleaning: A specific aspect of preprocessing and wrangling that deals with correcting data quality issues.
Data Mining: Focused on extracting useful patterns and insights from processed data.
Data Collection
o Primary Data collection oSecondary Data collection
Primary data collection is the process of Secondary data collection is the process of
collecting information directly from users. This collecting information from other sources, such as
type of data collection is usually done through public records or databases.
surveys or interviews
Simple Random Sampling (SRS)
Descriptive Statistics (EDA)
Descriptive statistics summarizes or describes the characteristics of a data set.
o Descriptive statistics consists of three basic categories of measures: measures of central tendency, measures of
variability (or spread), and frequency distribution.
o Measures of central tendency include the mean, median, and mode, while measures of variability
include standard deviation, variance, minimum and maximum variables, kurtosis, and skewness.
Central Tendency – Measures of Variability – Distribution –
Mean | Median | Mode Variance | Standard Deviation Skewness | kurtosis
| Range
Variance = Σ(Xi−X)2 / N
𝑆𝐷 = Σ(Xi−X)2 /N
Range = Max – Min
Skewness (Power 3)
• Skewness measures the asymmetry of the distribution around its mean. The power of 3 is used in the formula to
emphasize the direction and degree of asymmetry.
• When you raise deviations from the mean (xi−xˉ) to the power of 3, the resulting value will be positive for
deviations to the right (positive side) of the mean and negative for deviations to the left (negative side) of the
mean.
• Cubing the deviations gives more weight to larger deviations and highlights the skewness by amplifying the
effect of the direction of the asymmetry.
Kurtosis (Power 4)
• Kurtosis measures the “Peekedness" of the distribution, or how heavy or light the tails of the distribution are
compared to a normal distribution. The power of 4 is used to highlight the presence of outliers.
• Raising deviations to the power of 4 makes all deviations positive, whether they are above or below the mean,
thus focusing on the magnitude of the deviations.
• The use of the fourth power gives even more weight to extreme values (outliers) than the third power does,
emphasizing the tails of the distribution.
• The use of the fourth power gives even more weight to extreme values (outliers) than the third power does,
emphasizing the tails of the distribution.
Histogram
A histogram is a graph that shows the frequency of numerical data using rectangles. The height of a rectangle
(the vertical axis) represents the distribution frequency of a variable (the amount, or how often that variable appears).
Skewness
Mode < Median < Mean
Mode = Median = Mean
Mode > Median > Mean
Kurtosis
The last descriptive statistic is kurtosis, which provides information for the degree of peakedness of a data
distribution. Peakedness in a data distribution is the degree to which data values are concentrated around the mean. Datasets
with high kurtosis tend to have a distinct peak near the mean and tend to decline rapidly, and have heavy tails. Datasets with
low kurtosis tend to have a flat top near the mean rather than a sharp peak.
Types of Kurtosis
[Link]:
1. A normal distribution has a kurtosis of 3,
which is often adjusted to 0 (excess kurtosis)
for comparison purposes.
2. It has moderate tail thickness.
[Link]:
1. Higher kurtosis (>3 or positive excess
kurtosis).
2. Heavy tails and a sharp peak.
[Link]:
1. Lower kurtosis (<3 or negative excess
kurtosis).
2. Light tails and a flatter peak.