STATISTICAL ANALYSIS OF DATA
Statistics deals with large dataset and Central tendency is used for the understanding and
analysis purpose of data.
“Central tendency” is stated as the summary of a dataset in a single value that represents the
entire distribution of data domain (or dataset).
Some important functions which we will use in future programs in this module are
i. mean ( ) →returns the mean of the data
ii. median ( ) →returns the median of the data
iii. mode ( ) →returns the mode of the data
iv. variance ( ) →returns the variance of the data
v. stdev ( ) →returns the standard deviation of the data
1. Mean: It is a quotient obtained by dividing the total of the values of a variable by the total
number of their observations or items.
M = ∑ fx / n
where M = Mean , ∑ = Sum total of the scores, f = Frequency of the distribution, x = Scores, n =
Total number of cases.
Example: The set S = {5,10,15,20,30} Mean of set S = 5+10+15+20+30/5 = 80/5 = 16
Program: There are 25 students in a class. Their heights are given below. Write a Python
Program to find the mean.
Heights → 145, 151, 152, 149, 147, 152, 151,149, 152, 151, 147, 148, 155, 147,152,151,149,145,
147, 152,146, 148, 150, 152, 151.
2. Median:
The median is another measure of central tendency.
It is positional value of the variables which divides the group into two equal parts, one
part comprising all values greater than median and other part smaller than median.
Example: Following series shows marks in mathematics of students learning AI
17 32 35 15 21 41 32 11 10 20 27 28 30
We arrange this data in an ascending or descending order.
10, 11, 15, 17, 20, 21, 27, 28, 30, 32, 32, 35, 40
As 27 is in the middle of this data position wise, therefore Median = 27
PROGRAM:
3. Mode:
It is the value which occurs most frequently in the data series.
It represents the highest bar in a bar chart or histogram.
Example-4: Age of 15 students of a class
Age (years) 22, 24, 17, 18, 17, 19, 18, 21, 20, 21, 20, 23, 22, 22, 22,22,21,24
• We arrange this series in ascending order as 17,17,18,18,19,20,20,21,21,22,22,22,
• An inspection of the series shows that 22 occurs most frequently, hence Mode=22
PROGRAM:
4. Variance and Standard Deviation
Variance and standard deviation are the measures of dispersion (quartiles, percentiles,
ranges), they provide information on the spread of the data around the centre.
Some important facts about variance and standard deviation
• A small variance indicates that the data points tend to be very close to the mean, and to each
other.
• A high variance indicates that the data points are very spread out from the mean, and from
one another.
• A low standard deviation indicates that the data points tend to be very close to the mean.
• A high standard deviation indicates that the data points are spread out over a large range of
values.
PROGRAM:
REPRESENTATION OF DATA
It is a technique for presenting large volumes of data in a manner that enables the user to
interpret the important data with minimum effort and time.
Data representation techniques are broadly classified in two ways:
(i) Non-Graphical technique: Tabular form and case form.
(ii) Graphical Technique:
The visual display of statistical data in the form of points, lines, dots and other geometrical
forms is known as Data visualization.
Most commonly used graphs and the way they are drawn.
i. Line graphs
ii. Bar diagrams
iii. Pie diagram
iv. Scatter Plots
v. Histogram, etc.
Data Visualization is possible in python using the library Matplotlib which is a
comprehensive library that can be used to create a wide variety of plots, including line
plots, bar charts, histograms, scatter plots, and more.
pyplot is a submodule of Matplotlib that provides a MATLAB-like interface to the library. It
also provides a number of convenience functions that make it easy to create simple plots.
In the program we have to import the library import [Link]
Some of the common functions and its description is given below:
1. LINE GRAPH
To represent continuous data.
It allows us to visualize trends and changes in data points over time.
Line graphs are suitable for data that can take on any value within a specific range.
Line chart is plotted in python using the function plot ( ).
Attributes of plot function which are used inside plot ( ) function are:
PROGRAM:
PROGRAM:
2. Bar Graph:
A bar chart or bar graph is a graph that presents categorical data with rectangular bars with
heights or lengths proportional to the values that they represent.
Bar chart is plotted in python using the function bar ( ).
Attributes of bar function which are used inside bar ( ) functions are:
Example:
Program : Write a program to draw a bar chart to visualize the comparative rainfall data for 12
months in Tamil Nadu using the CSV file "[Link]".
3. Histogram:
Histograms are graphical representations of data distribution, with vertical rectangles
depicting the frequencies of different value ranges.
These intervals are also called logical ranges or bins.
Histogram is plotted in python using the function hist ( ).
Limitation: They can only represent one data distribution per axis.
Example : Given a dataset containing the heights of girls in class XII, construct a histogram to visualize
the distribution of heights. 141,145,142,147,144,148,141,142,149,144,143,149,146,141, 147, 142,
143.
4. Scatter Graph
Scatter plots visually represent relationships between two variables by plotting data points
along both the x and y axes.
Scatterplot is plotted using the function scatter ( ).
Program: Write a program to draw a scatter chart to visualize the comparative rainfall data for 12
months in Tamil Nadu using the CSV file "[Link]".
5. Pie Chart:
A pie chart is a circular graph divided into segments or sections, each representing a relative
proportion or percentage of the total.
It is used to visualize data from a small table.
However, zero values cannot be depicted in pie charts.
Pie Chart is plotted using the function pie ( ).
INTRODUCTION TO MATRICES
In mathematics, matrix (plural matrices) is a rectangular arrangement of numbers.
The numbers are arranged in tabular form as rows and columns.
We denote matrices by capital letters, for example:
A matrix has m rows and n columns. It is called a matrix of order m × n.
Each individual element is represented as aij where i represents row and j represents column.
In general aij, is an element lying in the ith row and jth column.
Operations on Matrices:
i. Addition
ii. Difference
iii. Transpose
Applications of matrices in AI
Matrices are used throughout the field of machine learning for computing:
• Image Processing – Digital images can be represented using matrices. In a computer, every
image is kept as a matrix of integers called a Channel.
Recommender systems.
In Natural Language processing, vectors depict the distribution of a particular word in a
document. Vectors are one-dimensional matrices.
DATA PREPROCESSING: It involves several processes to clean, transform, reduce, integrate, and
normalize data:
1. Data Cleaning
(i) Missing Data: Missing data occurs when values are absent from the dataset, which can
happen due to various reasons.
(ii) Outliers: Outliers are data points that significantly differ from the rest of the data, often
due to errors or rare events.
(iii) Inconsistent Data
(iv) Duplicate Data
2. Data Transformation: Categorical variables are converted to Numerical variable. New
features are identified and existing features are modified if needed.
3. Data Reduction: If data set is too large to handle sampling techniques are applied.
4. Data Integration and Normalization: If data is stored in multiple sources or formats, they are
merged or aggregated together.
5. Feature Selection: It means the target variables are selected and irrelevant data are
removed.
DATA IN MODELLING & EVALUATION
After the data is pre-processed, it is split into two --Training dataset and Testing dataset. The
training set is used to train the machine learning models, while the testing set is used to
evaluate the performance of the trained models.
Techniques such as train-test split, cross-validation, and error analysis are employed to estimate
the model's generalization ability and identify areas for improvement.
Different types of evaluation techniques are applied on the model depending on the data.
For classification problems, metrics like accuracy, precision, recall, F1-score, and ROC
curve are commonly used.
For regression problems, metrics like mean squared error (MSE), root mean squared
error (RMSE), mean absolute error (MAE), and R-squared are often used.
---∞---