Data Exploration &

Visualization Dr. SELVA KUMAR S

Unit - 4 ASSISTANT PROFESSOR
DEPT. OF CSE
Big Data and Analytics by Seema Acharya and Subhashini Chellappan
Copyright 2015, WILEY INDIA PVT. LTD.
Agenda

Univariate analysis

Bivariate analysis

Multivariate analysis

Time Series Analysis (TSA)

Fundamentals of TSA - characteristics of TSA

Time based indexing

Visualizing time series

Grouping and resampling time series data.

Types of Analysis
• Univariate Analysis
-Mainly for Description
• Bivariate Analysis
-Determining the empirical relationship between the two variables.
• Multivariate Analysis
-Determining the empirical relationship among multiple variables.
Choosing the Statistical Technique
Univariate analysis
• Analysis on a single type of dataset is called univariate analysis.
• Simplest form of analyzing data: Data has only one type of variable.
• The main purpose of univariate analysis is to take data, summarize that data, and find
patterns among the values.
• Several techniques that describe the patterns found in univariate data
• Central tendency: the mean, mode, and median.
• Dispersion: the range, variance, maximum and minimum quartiles
• Interquartile range and standard deviation
Example-1
#import libraries
import [Link] as plt
import seaborn as sns
import pandas as pd
# loading dataset as Pandas dataframe
df = pd.read_csv("[Link]")
[Link]()
Example-1
• #distribution plot
• [Link](df,size=5).map([Link],"height").add_legend()
• We can observe that the maximum height of maximum cars ranges from 53 to 57.

• #distribution plot
• [Link](df,size=5).map([Link],"price").add_legend()
• We can say that the price ranges from 5,000 to 45,000

• #boxplot for price of cars

• [Link](x="price",data=df)
• [Link]()
Example-2
• Let's consider a univariate analysis using a dataset that contains multiple variables. For instance,
suppose we have a dataset that includes information about students' math scores, study hours,
and their ages.
• We'll perform univariate analysis on one variable—'Study Hours.’
Example-2
Example 3
• import seaborn as sns
• import [Link] as plt

• # Load a sample dataset from Seaborn

• tips_data = sns.load_dataset("tips")

• # Univariate analysis using a histogram

• [Link](tips_data["total_bill"], kde=True, color='skyblue')
• [Link]('Distribution of Total Bill Amount')
• [Link]('Total Bill Amount')
• [Link]('Frequency')
• [Link]()
Example 4
• import seaborn as sns
• import [Link] as plt

• # Load a sample dataset from Seaborn

• iris_data = sns.load_dataset("iris")

• # Univariate analysis using a box plot

• [Link](x=iris_data["species"], y=iris_data["sepal_length"], palette="Set3")
• [Link]('Distribution of Sepal Length by Species')
• [Link]('Species')
• [Link]('Sepal Length')
• [Link]()
Exercise-1
• Scenario: You work as a data analyst for an e-commerce company.
The marketing team is interested in understanding the distribution of
purchase amounts made by customers to tailor promotional
strategies effectively.
• Question: Using univariate analysis, explore and describe the
distribution of purchase amounts made by customers in the dataset.
Exercise-2
• You've been hired as a data analyst for a healthcare organization. The
management wants insights into the distribution of patient ages to
better understand the demographics they serve.
Bivariate analysis
• Bivariate Analysis involves the analysis of exactly two variables.
• It is used to find out whether there is a relationship between two different
variables.
• Generally, bivariate analysis helps us to predict a value for one variable
(that is, a dependent variable) if we are aware of the value of the
independent variable.
Importance
• Bivariate analysis helps identify trends and patterns.
• Bivariate analysis helps identify cause and effect
relationships.
• It helps researchers make predictions.
• It helps inform decision-making: Business, public
policy, and healthcare decision-making can benefit from
bivariate analysis.
Example -5
• # plot the relationship between “horsepower” and ”price”
• [Link](df["price"], df["horsepower"])
• [Link]("Scatter Plot for horsepower vs price")
• [Link]("horsepower")
• [Link]("price")

• #boxplot to visualize the distribution of "price" with types of

• "drive-wheels"
• [Link](x="drive-wheels", y="price",data=df)
Example - 6
Example-7
• import pandas as pd
• import numpy as np
• from [Link] import pearsonr

• # Generating hypothetical data

• [Link](42)
• age = [Link](20, 80, 100) # Simulating ages between 20 and 80 years
• blood_pressure = age * 0.5 + [Link](0, 10, 100) # Simulating blood pressure

• # Creating a DataFrame
• health_data = [Link]({'Age': age, 'Blood Pressure': blood_pressure})

• # Calculating Pearson's correlation coefficient

• correlation, p_value = pearsonr(health_data['Age'], health_data['Blood Pressure'])
• print(f"Pearson's correlation coefficient: {correlation:.2f}")
• print(f"P-value: {p_value:.4f}")
Pearson's correlation coefficient: 0.71
P-value: 0.0000
Exercise-3
• Scenario: You're working as a data analyst for a retail company that
wants to understand the relationship between the amount spent by
customers and the time they spend on the company's website. This
information will aid in optimizing the website experience and
marketing strategies.
• Question: Using a bivariate analysis technique, explore the
relationship between the time spent on the website (in minutes) and
the amount spent by customers (in dollars) to derive insights for the
retail company.
Exercise-4
• Suppose you're analyzing the relationship between advertising
expenditure and product sales for a range of products in a retail store.
Describe how a bivariate analysis can help uncover insights into the
effectiveness of advertising campaigns and their impact on product
sales. Provide an approach and potential findings that could emerge
from such an analysis.
Exercise-5
• Suppose you're analyzing the relationship between data on total
rainfall and total number of plants in different regions given below.
Describe how a bivariate analysis can help uncover insights
Multivariate Analysis
• Multivariate analysis is the analysis of three or more variables.
• This allows us to look at correlations and attempt to make
predictions for future behavior more accurately.
• It's a set of techniques that helps us make sense of complex
information by examining multiple variables at once.
Example-8
• One common way of plotting multivariate data is to make a matrix scatter plot, known as
a pair plot.
• A matrix plot or pair plot shows each pair of variables plotted against each other.
• # pair plot with plot type regression
• [Link](df,vars = ['normalized-losses', 'price','horsepower'],
• kind="reg")
• [Link]()
Example-9
• #pair plot (matrix scatterplot) of few columns
• [Link](style="ticks", color_codes=True)
• [Link](df,vars = ['symboling', 'normalized-losses','wheelbase'],
• hue="drive-wheels")
• [Link]()
Example-10
• [Link](correlation,xticklabels=[Link], yticklabels=[Link])
Multivariate data analysis technique
• There are many different techniques for multivariate analysis, and
they can be divided into two categories:
• Dependence techniques : Dependence methods are used when one
or some of the variables are dependent on others.
• Interdependence techniques :Interdependence methods are used to
understand the structural makeup and underlying patterns within a
dataset.
Multivariate Analysis Techniques
• Multiple linear regression
• Multiple logistic regression
• Multivariate analysis of variance (MANOVA)
• Factor analysis
• Cluster analysis
Multiple Linear Regression
• Multiple linear regression is a dependence method.
• It looks at the relationship between one dependent variable and two
or more independent variables.
• A multiple regression model will tell you the extent to which each
independent variable has a linear relationship with the dependent
variable.
• Example: crop growth is dependent variable and independent
variables could be rainfall, temperature, amount of sunlight, and
amount of fertilizer added to the soil.
Multiple Logistic Regression
• Logistic regression analysis is used to calculate (and predict) the
probability of a binary event occurring.
• Based on a set of independent variables, logistic regression can
predict how likely it is that a certain scenario will arise.
• It is also used for classification.
• Example:
1. filters used to classify email as “spam” or “not spam.”
2. Insurance : Claim or Not claim bases on independent variable
age, health condition, occupation etc.
Multivariate Analysis of Variance (MANOVA)
• Multivariate analysis of variance (MANOVA) is used to measure the effect
of multiple independent variables on two or more dependent variables.
• The independent variables are categorical.
• Example:
• Categorical independent variables could be:
• Engine type, categorized as E1, E2, or E3
• Material used for the rocket exterior, categorized as M1, M2, or M3
• Type of fuel used to power the rocket, categorized as F1, F2, or F3
• Your metric dependent variables are speed in kilometers per hour. This
should help you to find the optimal design solution for your rocket.
Factor Analysis
• Factor analysis is an interdependence technique.
• It seeks to reduce the number of variables in a dataset.
• If you have too many variables, it can be difficult to find patterns and lead
to overfitting.
• Factor analysis works by detecting sets of variables which correlate highly
with each other.
• These variables may then be condensed into a single variable.
• Example: You might also have data on how happy they were with customer
service, how much they like a certain product, and how likely they are to
recommend the product to a friend. Each of these variables could be
grouped into the single factor “customer satisfaction”
Cluster Analysis
• Another interdependence technique, cluster analysis is used to group
similar items within a dataset into clusters.
• This is measured in terms of intracluster and intercluster distance.
• Cluster analysis helps you to understand how data in your sample is
distributed, and to find patterns.
• Example :As a healthcare analyst, you might use cluster analysis to
explore whether certain lifestyle factors or geographical locations are
associated with higher or lower cases of certain illnesses.
Outlining Simpson's paradox
• The decisions differ when we segregate the data into groups and
apply statistical measures, or when we aggregate it together and then
apply statistical measures.
• This kind of anomalous behavior in the results of the same dataset is
generally called Simpson's paradox
Simpson's paradox

• Confounding variables are something we don't see in the data table but they can
be determined by background analysis of the data.
• Data is never purely objective and neither is the final plot. Therefore, we must
consider whether we are getting the whole story when dealing with a set of data.
Correlation does not imply causation
• Correlation reveals how strongly a pair of variables are related to
each other and change together.
• Causation explains that any change in the value of one variable will
cause a difference in the amount of another variable.
• In this case, one variable makes the other variable happen. This
phenomenon is known as cause and effect.
• We should not form our conclusions too quickly based on correlation.
It is essential to invest some time in finding the underlying factors of
the data in order to understand any critical, hidden factors.
Time Series Analysis
• An ordered sequence of timestamp values at equally spaced intervals is referred to as a time series.
• Time series data is in the form of a sequence of quantitative observations about a system or process
and is made at successive points in time.
• Analysis of a time series is used in many applications such as sales forecasting, utility studies,
budget analysis, economic forecasting, inventory studies, and so on.
• Two important key phrases here—
• a collection of observations and
• sequentially in time
Understanding Time Series Dataset
• A time series is a collection of observations made sequentially in time.
• Two important key phrases : a collection of observations and
Sequentially in time.
Example-1
• [Link](figsize=(16, 8))
• g = [Link](data=zero_mean_series)
• g.set_title('Zero mean model')
• g.set_xlabel('Time index')
• [Link]()
Example-2
Univariate Time series
• When we capture a sequence of observations for the same variable over a particular duration of
time.
• In general, in a univariate time series, the observations are taken over regular time periods.
• Example: change in temperature over time throughout a day.
Characteristics of Time series data
• When looking at time series data, it is essential to see if there is any trend.
• Time series data may contain a notable amount of outliers.
• Some data in time series tends to repeat over a certain interval in some patterns : seasonality.
• Sometimes, there is an uneven change in time series data : abrupt changes.
• Some series tend to follow constant variance over time.
Example-3
Exercise-1
• Consider a dataset containing monthly sales data for a retail store
over a period of two years. The dataset has two columns: "Date" and
"Sales." Your task is to perform time series analysis on this data.
[Link] the dataset and convert the "Date" column to the datetime
format.
[Link] the "Date" column as the index of the DataFrame.
[Link] the time series to visualize the sales pattern over the two-year
period.
[Link] and plot the 7-day rolling mean of sales to identify trends.
[Link] whether there are any seasonal patterns in the data.
Visualizing Time series
• import [Link] as plt
• import seaborn as sns
• [Link](rc={'[Link]':(11, 4)})
• [Link]['[Link]'] = (8,5)
• [Link]['[Link]'] = 150
• df_power['Consumption'].plot(linewidth=0.5)
Visualizing Time series
• cols_to_plot = ['Consumption', 'Solar', 'Wind']
• axes = df_power[cols_to_plot].plot(marker='.', alpha=0.5,
• linestyle='None',figsize=(14, 6), subplots=True)
• for ax in axes:
• ax.set_ylabel('Daily Totals (GWh)')
Visualizing Time series
• ax = df_power.loc['2016', 'Consumption'].plot()
• ax.set_ylabel('Daily Consumption (GWh)');
Visualizing Time series
• ax = df_power.loc['2016-12', 'Consumption'].plot(marker='o', linestyle='-')
• ax.set_ylabel('Daily Consumption (GWh)');
Visualizing Time series
• ax = df_power.loc['2016-12-23':'2016-12-30',
• 'Consumption'].plot(marker='o', linestyle='-')
• ax.set_ylabel('Daily Consumption (GWh)');
Example-3
• You are provided with a time series dataset containing monthly electricity
consumption data for a residential area over several years. Your task is to
perform an analysis to identify and interpret the trend, seasonal pattern,
and potential cyclical behavior in the data.

• Load the dataset and convert the "Date" column to the datetime format.
• Set the "Date" column as the index of the DataFrame.
• Plot the time series to visualize the overall pattern.
• Use the seasonal_decompose function from the statsmodels library to
decompose the time series into trend, seasonal, and residual components.
• Interpret and explain the identified trend, seasonal pattern, and any
potential cyclical behavior based on the decomposition results.
Solution
Exercise-2
• You have been given a dataset representing
quarterly sales data for a retail business over
the past decade. Your objective is to analyze
the time series data and identify the trend,
seasonal pattern, and any potential cyclical
behavior.

• Load the dataset and convert the "Date"

column to the datetime format.
• Set the "Date" column as the index of the
DataFrame.
• Plot the time series to visualize the overall
pattern in the sales data.
• Apply a suitable method (e.g., moving
averages or seasonal decomposition) to
identify and separate the trend, seasonal, and
residual components.
• Interpret the identified trend, seasonal
pattern, and any observed cyclical behavior.
Grouping Time series data
• Group the data by months
• fig, axes = [Link](3, 1, figsize=(8, 7), sharex=True)
• for name, ax in zip(['Consumption', 'Solar', 'Wind'], axes):
• [Link](data=df_power, x='Month', y=name, ax=ax)
• ax.set_ylabel('GWh')
• ax.set_title(name)
• if ax != axes[-1]:
• ax.set_xlabel('')
Grouping Time series data
• Group the consumption of electricity by the day of the week
• [Link](data=df_power, x='Weekday Name', y='Consumption');
Example - 4
Exercise-5
• You are given a time series dataset containing hourly electricity consumption data
for a residential area over a month. Your goal is to perform time series analysis
and create a grouped summary to better understand consumption patterns.

• Load the dataset and convert the "Timestamp" column to the datetime format.
• Set the "Timestamp" column as the index of the DataFrame.
• Plot the original time series to visualize the hourly electricity consumption
pattern.
• Group the data by day and calculate the daily total electricity consumption.
• Plot the original time series along with the daily total electricity consumption to
observe any patterns or trends.