602 -Data Analytics using Python

Unit-1: Fundamentals of Data Analytics:

1.1 Exploratory Data Analysis (EDA)

1.1.1 Types of Exploratory Data Analysis:

1.1.2 Univariate Analysis

1.1.3 Bivariate Analysis

1.1.4 Multivariate Analysis

1.1.5 Handling Missing Data and Outliers

1.2 Understanding the Data:

1.2.1 Quantitative Data: Discrete and Continuous

1.2.2 Qualitative Data: Non-numerical (Normal and Ordinal)

1.3 Spread of Data

1.3.1 Normal Distribution

1.3.2 Skewed Distribution

1.3.3 Skewness and Kurtiosis

1.1 Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is an important step in data science and data
analytics as it visualizes data to understand its main features, find patterns and
discover how different parts of the data are connected.

Why Exploratory Data Analysis Important

1. Helps to understand the dataset by showing how many features it has, what
type of data each feature contains and how the data is distributed.
2. Helps to identify hidden patterns and relationships between different data
points which help us in and model building.

1|Page SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

3. Allows to identify errors or unusual data points (outliers) that could affect our
results.
4. The insights gained from EDA help us to identify most important features for
building models and guide us on how to prepare them for better performance.
5. By understanding the data it helps us in choosing best modeling techniques
and adjusting them for better results.

1.1.1 Types of Exploratory Data Analysis:

There are various types of EDA based on nature of records. Depending on the
number of columns we are analyzing we can divide EDA into three types:

1. Univariate Analysis
Univariate analysis focuses on studying one variable to understand its
characteristics. It helps to describe data and find patterns within a single feature.
Various common methods like histograms are used to show data distribution, box
plots to detect outliers and understand data spread and bar charts for categorical
data. Summary statistics like mean, median, mode, variance and standard
deviation helps in describing the central tendency and spread of the data.

2. Bivariate Analysis
Bivariate Analysis focuses on identifying relationship between two variables to
find connections, correlations and dependencies. It helps to understand how two
variables interact with each other. Some key techniques include:
 Scatter plots which visualize the relationship between two continuous
variables.
 Correlation coefficient measures how strongly two variables are related which
commonly use Pearson's correlation for linear relationships.
 Cross-tabulation or contingency tables shows the frequency distribution of two
categorical variables and help to understand their relationship.
 Line graphs are useful for comparing two variables over time in time series
data to identify trends or patterns.
 Covariance measures how two variables change together but it is paired with
the correlation coefficient for a clearer and more standardized understanding
of the relationship.

3. Multivariate Analysis
Multivariate Analysis identify relationships between two or more variables in the
dataset and aims to understand how variables interact with one another which is
important for statistical modeling techniques. It include techniques like:
2|Page SHREE UTTAR GUJARAT BCA COLLEGE
 602 -Data Analytics using Python

 Pair plots which shows the relationships between multiple variables at once
and helps in understanding how they interact.
 Another technique is Principal Component Analysis (PCA) which reduces
the complexity of large datasets by simplifying them while keeping the most
important information.
 Spatial Analysis is used for geographical data by using maps and spatial
plotting to understand the geographical distribution of variables.
 Time Series Analysis is used for datasets that involve time-based data and it
involves understanding and modeling patterns and trends over time. Common
techniques include line plots, autocorrelation analysis, moving averages
and ARIMA models.

Steps for Performing Exploratory Data Analysis

It involves a series of steps to help us understand the data, uncover patterns,
identify anomalies, test hypotheses and ensure the data is clean and ready for
further analysis. It can be done using different tools like:
 In Python, Pandas is used to clean, filter and manipulate data. Matplotlib helps
to create basic visualizations while Seaborn makes more attractive plots. For
interactive visualizations Plotly is a good choice.
 In R, ggplot2 is used for creating complex plots, dplyr helps with data
manipulation and tidyr makes sure our data is organized and easy to work
with.

Its step includes:

Step 1: Understanding the Problem and the Data
The first step in any data analysis project is to fully understand the problem we're
solving and the data we have. This includes asking key questions like:
1. What is the business goal or research question?
2. What are the variables in the data and what do they represent?
3. What types of data (numerical, categorical, text, etc.) do you have?
4. Are there any known data quality issues or limitations?
5. Are there any domain-specific concerns or restrictions?
By understanding the problem and the data, we can plan our analysis more
effectively, avoid incorrect assumptions and ensure accurate conclusions.

Step 2: Importing and Inspecting the Data

After understanding the problem and the data, next step is to import the data into
our analysis environment such as Python, R or a spreadsheet tool. It’s important to
find data to gain an basic understanding of its structure, variable types and any
potential issues. Here’s what we can do:
3|Page SHREE UTTAR GUJARAT BCA COLLEGE
 602 -Data Analytics using Python

1. Load the data into our environment carefully to avoid errors or truncations.
2. Check the size of the data like number of rows and columns to understand its
complexity.
3. Check for missing values and see how they are distributed across variables
since missing data can impact the quality of your analysis.
4. Identify data types for each variable like numerical, categorical, etc which will
help in the next steps of data manipulation and analysis.
5. Look for errors or inconsistencies such as invalid values, mismatched units or
outliers which could show major issues with the data.
By completing these tasks we'll be prepared to clean and analyze the data more
effectively.

Step 3: Handling Missing Data

Missing data is common in many datasets and can affect the quality of our
analysis. During EDA it's important to identify and handle missing data properly
to avoid biased or misleading results. Here’s how to handle it:
1. Understand the patterns and possible causes of missing data. Is it missing
completely at random (MCAR), missing at random (MAR) or missing not at
random (MNAR). Identifying this helps us to find best way to handle the
missing data.
2. Decide whether to remove missing data or impute (fill in) the missing values.
Removing data can lead to biased outcomes if the missing data isn’t MCAR.
Filling values helps to preserve data but should be done carefully.
3. Use appropriate imputation methods like mean or median
imputation, regression imputation or machine learning techniques
like KNN or decision trees based on the data’s characteristics.
4. Consider the impact of missing data. Even after imputing, missing data can
cause uncertainty and bias so understands the result with caution.
Properly handling of missing data improves the accuracy of our analysis and
prevents misleading conclusions.

Step 4: Exploring Data Characteristics

After addressing missing data we find the characteristics of our data by checking
the distribution, central tendency and variability of our variables and identifying
outliers or anomalies. This helps in selecting appropriate analysis methods and
finding major data issues. We should calculate summary statistics like mean,
median, mode, standard deviation, skewness and kurtosis for numerical variables.
These provide an overview of the data’s distribution and helps us to identify any
irregular patterns or issues.

4|Page SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

Step 5: Performing Data Transformation

Data transformation is an important step in EDA as it prepares our data for accurate
analysis and modeling. Depending on our data's characteristics and analysis needs,
we may need to transform it to ensure it's in the right format. Common
transformation techniques include:
1. Scaling or normalizing numerical variables like min-max
scaling or standardization.
2. Encoding categorical variables for machine learning like one-hot
encoding or label encoding.
3. Applying mathematical transformations like logarithmic square root to correct
skewness or non-linearity.
4. Creating new variables from existing ones like calculating ratios or combining
variables.
5. Aggregating or grouping data based on specific variables or conditions.

Step 6: Visualizing Relationship of Data

Visualization helps to find relationships between variables and identify patterns or
trends that may not be seen from summary statistics alone.
1. For categorical variables, create frequency tables, bar plots and pie charts to
understand the distribution of categories and identify imbalances or unusual
patterns.
2. For numerical variables generate histograms, box plots, violin plots and
density plots to visualize distribution, shape, spread and potential outliers.
3. To find relationships between variables use scatter plots, correlation matrices
or statistical tests like Pearson’s correlation coefficient or Spearman’s rank
correlation.

Step 7: Handling Outliers

Outliers are data points that differs from the rest of the data may caused by errors
in measurement or data entry. Detecting and handling outliers is important because
they can skew our analysis and affect model performance. We can identify outliers
using methods like interquartile range (IQR), Z-scores or domain-specific rules.
Once identified it can be removed or adjusted depending on the context. Properly
managing outliers shows our analysis is accurate and reliable.

Step 8: Communicate Findings and Insights

The final step in EDA is to communicate our findings clearly. This involves
summarizing the analysis, pointing out key discoveries and presenting our results
in a clear way.
1. Clearly state the goals and scope of your analysis.
5|Page SHREE UTTAR GUJARAT BCA COLLEGE
 602 -Data Analytics using Python

2. Provide context and background to help others understand your approach.

3. Use visualizations to support our findings and make them easier to understand.
4. Highlight key insights, patterns or anomalies discovered.
5. Mention any limitations or challenges faced during the analysis.
6. Suggest next steps or areas that need further investigation.
Effective communication is important to ensure that our EDA efforts make an
impact and that stakeholders understand and act on our insights. By following these
steps and using the right tools, EDA helps in increasing the quality of our data,
leading to more informed decisions and successful outcomes in any data-driven
project.

4. Handling Missing Data and Outliers

Handling Missing Data
Missing data can occur due to various reasons, such as data entry errors or system
malfunctions, and can bias estimates and reduce statistical power if not handled
properly.
Common methods include:
 Deletion Methods:
o Listwise Deletion (Complete-Case Analysis): Removes entire rows with any
missing values. This is simple but can lead to significant data loss and introduce
bias if the data is not Missing Completely at Random (MCAR).
o Pairwise Deletion: Uses all available data for each specific calculation. This
retains more data than listwise deletion but can lead to an inconsistent set of cases
across different analyses.
 Imputation Techniques: Replacing missing values with estimated ones is often
preferred to preserve sample size.
o Simple Imputation: Uses a central tendency measure (mean, median, or mode).
The median is often a better choice than the mean for skewed data because it is less
sensitive to outliers.
o Advanced Imputation:
o Regression Imputation: Predicts missing values using a regression model based
on other variables in the dataset.
6|Page SHREE UTTAR GUJARAT BCA COLLEGE
 602 -Data Analytics using Python

o K-Nearest Neighbors (KNN) Imputation: Uses values from "similar" data points
(neighbors) to estimate the missing value.
o Multiple Imputation (e.g., MICE): Creates several complete datasets with
plausible values, analyzes them separately, and combines the results. This method
accounts for the uncertainty in the imputed values and has stronger statistical
properties.

Handling Outliers
Outliers are data points that significantly differ from other observations and can
skew results and lead to misleading conclusions.
Common methods for detection and handling include:
 Detection Methods:
o Graphical Methods: Visualizations like box plots and scatter plots help in
identifying unusual data points.
o Statistical Methods:
o Z-score: Identifies outliers based on a threshold (e.g., more than 3 standard
deviations from the mean).
o Interquartile Range (IQR): Flags values outside 1.5 times the IQR below the first
quartile or above the third quartile.
 Handling Techniques:
o Removal: Exclude the outlier from the dataset, which is suitable if the outlier is a
clear error (e.g., a data entry mistake).
o Transformation: Apply mathematical transformations (e.g., log or square root) to
the data to reduce the influence of extreme values and make the distribution more
symmetrical.
o Capping/Winsorization: Modify the outlier's value to an acceptable maximum or
minimum bound (e.g., setting it to the value of the 95th percentile).
o Robust Statistical Methods: Use models and analyses (like robust regression) that
are inherently less sensitive to the presence of outliers.

7|Page SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

Ultimately, the best approach for handling both missing data and outliers requires a
careful understanding of the data, the reason for the issue, and the specific context
of the analysis to ensure the integrity and reliability of the results.

1.2 Understanding the Data:

1.2.1 Quantitative Data
Quantitative data represents numerical values that indicate "how much" or "how
many." It is divided into two sub-types:
 Discrete Data: This consists of distinct, separate values that can be counted. It
cannot be divided into smaller parts (e.g., you cannot have half a person).
o Examples: The number of students in a class, the number of cars in a parking lot,
or the number of goals scored in a match.
 Continuous Data: This represents measurements that can take any value within a
range. It can be infinitely divided into smaller increments (decimals and fractions).
o Examples: A person's height (175.5 cm), the temperature (22.4°C), or the time
taken to complete a task.

1.2.2 Qualitative Data

Qualitative data (also known as Categorical Data) describes attributes or
characteristics that are non-numerical. It is divided into two sub-types:
 Nominal Data: This data is used for labeling variables without any quantitative
value or inherent order. One category is not "higher" or "better" than another.
o Examples: Eye color (Blue, Brown, Green), Gender, or Marital Status.
 Ordinal Data: This data has a clear, natural order or ranking. While the order
matters, the mathematical difference between the ranks is not consistent or
measurable.
o Examples: Customer satisfaction ratings (Poor, Fair, Good, Excellent),
Socioeconomic status (Low, Middle, High), or Education level (High School,
Bachelors, Masters).

8|Page SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

Summary Table

Data Type Sub-type Key Characteristic Example

Quantitative Discrete Countable whole numbers 5 Chairs

Continuous Measurable decimals/ranges 5.75 Kilometers

Qualitative Nominal Named categories (No order) Type of Fruit

Ordinal Ordered categories (Ranked) "Star" Ratings (1-5)

1.3 Spread of Data

The spread of data, also known as dispersion, measures how far apart or varied the
values in a dataset are from each other or from the center (mean/median). Key
measures include Range (max - min), Interquartile Range (IQR) (spread of
middle 50%), Variance (average squared deviation from the mean), and Standard
Deviation (square root of variance, in original units). A small spread means data is
clustered (e.g., low SD); a large spread means data is dispersed (e.g., high SD).
Common Measures of Spread

 Range: Simplest measure; difference between the highest and lowest value
in the set.
 Interquartile Range (IQR): The range of the middle 50% of data (Q3 -
Q1), showing typical variation.
 Variance (σ): Average of the squared differences from the mean; useful for
understanding overall variability but in squared units.
 Standard Deviation (σ): The most common measure; the square root of
variance, expressed in the same units as the data, indicating typical distance
from the mean.

Why It's Important

 Context for Central Tendency: It shows how well the mean or median
represents the data. A large spread means the center value is less typical.

9|Page SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

 Identifies Variation: Reveals if data points are consistent (small spread) or

widely different (large spread).
 Risk & Consistency: In finance or quality control, low spread indicates
more stability and predictability.

Example
If two supermarkets have an average wait time of 5 minutes (mean):

 Supermarket A has wait times of 4, 5, 6 mins (Small Spread/SD) – very

consistent.
 Supermarket B has wait times of 1, 5, 9 mins (Large Spread/SD) – less
consistent, more varied.

1.3.1 Normal Distribution

The normal distribution, also known as the Gaussian distribution or bell curve, is
a continuous probability distribution symmetric about its mean.

 Symmetry: The left side of the distribution is a mirror image of the right
side.
 Central Tendency: The mean, median, and mode are all equal and located
at the center peak.
 Empirical Rule (68-95-99.7): Approximately 68% of data falls within one
standard deviation (σ) of the mean, 95% within two, and 99.7% within
three.
 Total Area: The total area under the curve is exactly 1.0.

1.3.2 Skewed Distribution

Skewness measures the degree of asymmetry in a distribution, typically caused by
extreme values or outliers.

 Positive (Right) Skew: The distribution has a long tail extending toward the
right (higher values). Most data points cluster on the left side.
o Relationship: Mean > Median > Mode.
 Negative (Left) Skew: The distribution has a long tail extending toward the
left (lower values). Most data points cluster on the right side.
o Relationship: Mean < Median < Mode.

1.3.3 Skewness and Kurtosis

These measures provide a complete picture of a distribution's shape.

10 | P a g e SHREE UTTAR GUJARAT BCA COLLEGE

 602 -Data Analytics using Python

 Skewness: Indicates the direction and degree of lack of symmetry.

o Normal Value: 0
 Kurtosis: Measures "tailedness" or the presence of outliers rather than just
peakedness.
o Mesokurtic: Normal distribution, with a kurtosis of 3 (or excess
kurtosis of 0).
o Leptokurtic (Positive Kurtosis): Sharper peak and "fat" tails,
indicating more frequent extreme outliers.
o Platykurtic (Negative Kurtosis): Flatter peak and "thin" tails,
indicating fewer extreme outliers and a more stable distribution.
 Difference Between Skewness and Kurtosis

Sr.
No. Skewness Kurtosis

It indicates the shape and size of It indicates the frequencies of

variation on either side of the central distribution at the central
1. value. value.

The measure differences of skewness

It indicates the concentration
tell us about the magnitude and
of items at the central part of a
direction of the asymmetry of a
distribution.
2. distribution.

It studies the divergence of the

It indicates how far the distribution
given distribution from the
differs from the normal distribution.
3. normal distribution.

The measure of skewness studies the

It indicates the concentration
extent to which deviation clusters is are
of items.
4. above or below the average.

In an asymmetrical distribution, the

deviation below or above an average is No such distribution takes pl
5. not equal.

11 | P a g e SHREE UTTAR GUJARAT BCA COLLEGE