0% found this document useful (0 votes)
6 views15 pages

Data Preprocessing & Eda - Colab

The document describes the process of loading and preprocessing a heart disease dataset using pandas. It includes data exploration, encoding categorical variables, and handling outliers for various features such as Age, RestingBP, and Cholesterol. The final dataset consists of 918 entries and 20 columns after transformations and cleaning steps.
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
6 views15 pages

Data Preprocessing & Eda - Colab

The document describes the process of loading and preprocessing a heart disease dataset using pandas. It includes data exploration, encoding categorical variables, and handling outliers for various features such as Age, RestingBP, and Cholesterol. The final dataset consists of 918 entries and 20 columns after transformations and cleaning steps.
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

import pandas as pd

df = pd.read_csv('/content/[Link]')

df

Age Sex ChestPainType RestingBP Cholesterol FastingBS RestingECG MaxHR ExerciseAngina Oldpeak ST_Slope HeartD

0 40 M ATA 140 289 0 Normal 172 N 0.0 Up

1 49 F NAP 160 180 0 Normal 156 N 1.0 Flat

2 37 M ATA 130 283 0 ST 98 N 0.0 Up

3 48 F ASY 138 214 0 Normal 108 Y 1.5 Flat

4 54 M NAP 150 195 0 Normal 122 N 0.0 Up

... ... ... ... ... ... ... ... ... ... ... ...

913 45 M TA 110 264 0 Normal 132 N 1.2 Flat

914 68 M ASY 144 193 1 Normal 141 N 3.4 Flat

915 57 M ASY 130 131 0 Normal 115 Y 1.2 Flat

916 57 F ATA 130 236 0 LVH 174 N 0.0 Flat

917 38 M NAP 138 175 0 Normal 173 N 0.0 Up

918 rows × 12 columns

Next steps: Generate code with df New interactive sheet

[Link]()

<class '[Link]'>
RangeIndex: 918 entries, 0 to 917
Data columns (total 12 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Age 918 non-null int64
1 Sex 918 non-null object
2 ChestPainType 918 non-null object
3 RestingBP 918 non-null int64
4 Cholesterol 918 non-null int64
5 FastingBS 918 non-null int64
6 RestingECG 918 non-null object
7 MaxHR 918 non-null int64
8 ExerciseAngina 918 non-null object
9 Oldpeak 918 non-null float64
10 ST_Slope 918 non-null object
11 HeartDisease 918 non-null int64
dtypes: float64(1), int64(6), object(5)
memory usage: 86.2+ KB

[Link]()

Age RestingBP Cholesterol FastingBS MaxHR Oldpeak HeartDisease

count 918.000000 918.000000 918.000000 918.000000 918.000000 918.000000 918.000000

mean 53.510893 132.396514 198.799564 0.233115 136.809368 0.887364 0.553377

std 9.432617 18.514154 109.384145 0.423046 25.460334 1.066570 0.497414

min 28.000000 0.000000 0.000000 0.000000 60.000000 -2.600000 0.000000

25% 47.000000 120.000000 173.250000 0.000000 120.000000 0.000000 0.000000

50% 54.000000 130.000000 223.000000 0.000000 138.000000 0.600000 1.000000

75% 60.000000 140.000000 267.000000 0.000000 156.000000 1.500000 1.000000

max 77.000000 200.000000 603.000000 1.000000 202.000000 6.200000 1.000000

[Link](include='O')
Sex ChestPainType RestingECG ExerciseAngina ST_Slope

count 918 918 918 918 918

unique 2 4 3 2 3

top M ASY Normal N Flat

freq 725 496 552 547 460

Data preprocessing & EDA**

Encoding

data=pd.get_dummies([Link],prefix='Gender',drop_first=True)
df=[Link]([data,df],axis=1)
[Link](columns = 'Sex',inplace = True)

df

RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST ChestPainType_ATA ChestPainType_

0 True False False True False True Fa

1 True False False True False False T

2 False True False False True True Fa

3 True False True True False False Fa

4 True False False True False False T

... ... ... ... ... ... ...

913 True False False True False False Fa

914 True False False True False False Fa

915 True False True True False False Fa

916 False False False False False True Fa

917 True False False True False False T

918 rows × 19 columns

Next steps: Generate code with df New interactive sheet

data=pd.get_dummies([Link],prefix='ChestPainType',drop_first=True)
df=[Link]([data,df],axis=1)
[Link](columns = 'ChestPainType',inplace = True)

df

RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST ChestPainType_ATA ChestPainType_

0 True False False True False True Fa

1 True False False True False False T

2 False True False False True True Fa

3 True False True True False False Fa

4 True False False True False False T

... ... ... ... ... ... ...

913 True False False True False False Fa

914 True False False True False False Fa

915 True False True True False False Fa

916 False False False False False True Fa

917 True False False True False False T

918 rows × 19 columns

Next steps: Generate code with df New interactive sheet


if 'RestingECG' in [Link]:
data = pd.get_dummies([Link], prefix='RestingECG', drop_first=True)
df = [Link]([data, df], axis=1)
[Link](columns='RestingECG', inplace=True,)

df

ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

918 rows × 20 columns

Next steps: Generate code with df New interactive sheet

data=pd.get_dummies([Link],prefix='ExerciseAngina',drop_first=True)
df=[Link]([data,df],axis=1)
[Link]('ExerciseAngina',axis=1,inplace=True)
[Link]()

a_Y RestingECG_Normal RestingECG_ST ChestPainType_ATA ChestPainType_NAP ChestPainType_TA Gender_M Gend

alse True False True False False True

alse True False False True False False

alse False True True False False True

True True False False False False False

alse True False False True False True

Next steps: Generate code with df New interactive sheet

data=pd.get_dummies(df.ST_Slope,prefix='ST_Slope',drop_first=True)
df=[Link]([data,df],axis=1)
[Link]('ST_Slope',axis=1,inplace=True)

df
ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

918 rows × 20 columns

Next steps: Generate code with df New interactive sheet

Outlier Handling

from scipy import stats

[Link]([Link])

np.float64(13.0)

IQR_Age = [Link]([Link])

import numpy as np

Q1_Age = [Link]([Link],25)
Q3_Age = [Link]([Link],75)
print(Q1_Age)
print(Q3_Age)

47.0
60.0

Q1_Age - 1.5 * IQR_Age

np.float64(27.5)

Q3_Age + 1.5 * IQR_Age

np.float64(79.5)

LB_Age = Q1_Age - 1.5 * (IQR_Age)


UB_Age = Q3_Age + 1.5 * (IQR_Age)

df[([Link]>LB_Age)&([Link]<UB_Age)]
ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

918 rows × 20 columns

[Link][[Link]>UB_Age,'Age']

Age

dtype: int64

[Link][[Link]<LB_Age,'Age']

Age

dtype: int64

SO THERE ARE NO NULL VALUES IN AGE COLUMN

[Link]([Link])

np.float64(20.0)

[Link]([Link])

np.float64(93.75)

[Link]([Link])

np.float64(0.0)

[Link]([Link])

np.float64(36.0)

[Link]([Link])

np.float64(1.5)

[Link]([Link])

np.float64(1.0)

IQR_RestingBP = [Link]([Link])

Q1_RestingBP = [Link]([Link],25)
Q3_RestingBP = [Link]([Link],75)
print(Q1_RestingBP)
print(Q3_RestingBP)

120.0
140.0

LB_RestingBP = Q1_RestingBP - 1.5 * (IQR_RestingBP)


UB_RestingBP = Q3_RestingBP + 1.5 * (IQR_RestingBP)
print(LB_RestingBP)
print(UB_RestingBP)

90.0
170.0

LB_RestingBP

np.float64(90.0)

UB_RestingBP

np.float64(170.0)

df[([Link]>LB_RestingBP)&([Link]<UB_RestingBP)]

ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

876 rows × 20 columns

[Link][[Link]>UB_RestingBP,'RestingBP']
RestingBP

109 190

123 180

189 180

190 180

241 200

274 180

275 180

278 180

365 200

372 185

399 200

411 180

423 180

475 178

550 172

585 180

592 190

673 174

702 178

725 180

732 200

759 192

774 178

780 180

855 180

880 172

dtype: int64

[Link][[Link]<LB_RestingBP,'RestingBP']

RestingBP

314 80

449 0

dtype: int64

import seaborn as sns

[Link](x=[Link])
<Axes: xlabel='RestingBP'>

((918-876)/918)*100 #Percentage of outliers

4.57516339869281

RESTINGBP=4.5 PERCENTAGE

[Link]()

130.0

[Link][[Link]>UB_RestingBP,'RestingBP'] = 130

[Link][[Link]<LB_RestingBP,'RestingBP'] = 130

[Link][[Link]>UB_RestingBP,'RestingBP']

RestingBP

dtype: int64

[Link]([Link])

np.float64(93.75)

IQR_Cholesterol = [Link]([Link])

Q1_Cholesterol = [Link]([Link],25)
Q3_Cholesterol = [Link]([Link],75)
print(Q1_Cholesterol)
print(Q3_Cholesterol)

173.25
267.0

LB_Cholesterol = Q1_Cholesterol - 1.5 * IQR_Cholesterol


UB_Cholesterol = Q3_Cholesterol + 1.5 * IQR_Cholesterol
print(LB_Cholesterol)
print(UB_Cholesterol)

32.625
407.625

LB_Cholesterol

np.float64(32.625)

UB_Cholesterol

np.float64(407.625)
df[([Link]>LB_Cholesterol)&([Link]<UB_Cholesterol)]

gECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST ChestP

False False True False

False False True False

True False False True

False True True False

False False True False

... ... ... ...

False False True False

False False True False

False True True False

False False False False

False False True False

[Link][[Link]>UB_Cholesterol,'Cholesterol']

Cholesterol

28 468

30 518

69 412

76 529

103 466

149 603

250 491

496 458

616 564

667 417

796 409

dtype: int64

[Link][[Link]<LB_Cholesterol,'Cholesterol']

Cholesterol

293 0

294 0

295 0

296 0

297 0

... ...

514 0

515 0

518 0

535 0

536 0

172 rows × 1 columns

dtype: int64

[Link][[Link]>UB_Cholesterol,'Cholesterol']
Cholesterol

28 468

30 518

69 412

76 529

103 466

149 603

250 491

496 458

616 564

667 417

796 409

dtype: int64

import seaborn as sns

[Link](x=[Link])

<Axes: xlabel='Cholesterol'>

((918-735)/918)*100 #Percentage of outliers

19.934640522875817

CHOLESTEROL PERCENTAGE=19.93

[Link]([Link])

np.float64(0.0)

Since iqr of FastingBS column is 0, so on outliers are there in this column

IQR_MaxHR = [Link]([Link])

Q1_MaxHR = [Link]([Link],25)
Q3_MaxHR = [Link]([Link],75)

LB_MaxHR = Q1_MaxHR - 1.5 * IQR_MaxHR


UB_MaxHR= Q3_MaxHR + 1.5 * IQR_MaxHR
print(LB_MaxHR)
print(UB_MaxHR)

66.0
210.0
LB_MaxHR

np.float64(66.0)

UB_MaxHR

np.float64(210.0)

df[([Link]>LB_MaxHR)&([Link]<UB_MaxHR)]

ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

916 rows × 20 columns

[Link][[Link]<LB_MaxHR,'MaxHR']

MaxHR

370 63

390 60

dtype: int64

[Link][[Link]>UB_MaxHR,'MaxHR']

MaxHR

dtype: int64

import seaborn as sns

[Link](x=[Link])

<Axes: xlabel='MaxHR'>
((918-916)/918)*100 #Percentage of outliers

0.2178649237472767

[Link]()

138.0

[Link][[Link]>UB_MaxHR,'MaxHR'] = 138

[Link](x=[Link])

<Axes: xlabel='MaxHR'>

IQR_MaxHR = [Link]([Link])

Q1_Oldpeak = [Link]([Link],25)
Q3_Oldpeak = [Link]([Link],75)
print(LB_Oldpeak)
print(UB_Oldpeak)

-2.25
3.75

LB_Oldpeak

np.float64(-2.25)

UB_Oldpeak

np.float64(3.75)

df[([Link]>LB_Oldpeak)&([Link]<UB_Oldpeak)]
ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

902 rows × 20 columns

[Link][[Link]<LB_Oldpeak,'Oldpeak']

Oldpeak

324 -2.6

dtype: float64

[Link][[Link]>UB_Oldpeak,'Oldpeak']

Oldpeak

68 4.0

166 5.0

500 4.0

521 4.0

537 4.0

559 4.0

624 4.0

702 4.2

732 4.0

771 5.6

775 3.8

791 4.2

850 6.2

900 4.4

908 4.0

dtype: float64

import seaborn as sns

[Link](x=[Link])
<Axes: xlabel='Oldpeak'>

((918-902)/918)*100 #Percentage of outliers

1.7429193899782136

[Link]()

0.6

SCALING

from [Link] import MinMaxScaler


scale=MinMaxScaler()
df[['Age','RestingBP','Cholesterol','MaxHR']] = scale.fit_transform(df[['Age','RestingBP','Cholesterol','MaxHR']])

df

ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False

917 False True True False False True False

918 rows × 20 columns

Next steps: Generate code with df New interactive sheet

#Target and predictor seperation


X=[Link][:,:-1]
y=[Link]

X
ST_Slope_Flat ST_Slope_Up RestingECG_Normal RestingECG_ST ExerciseAngina_Y RestingECG_Normal RestingECG_ST Chest

0 False True True False False True False

1 True False True False False True False

2 False True False True False False True

3 True False True False True True False

4 False True True False False True False

... ... ... ... ... ... ... ...

913 True False True False False True False

914 True False True False False True False

915 True False True False True True False

916 True False False False False False False


Next steps: Generate code with X New interactive sheet
917 False True True False False True False

918 rows × 19 columns


y

HeartDisease

0 0

1 1

2 0

3 1

4 0

... ...

913 1

914 1

You might also like