Scribd
Upload
233 views5 pages

Essential Pandas & Sklearn Commands

This document contains notes on machine learning concepts and processes. It outlines steps for exploratory data analysis, including handling missing values, outliers, and feature engineering. It also discusses preprocessing such as scaling, encoding categorical data, and splitting data into training and test sets. Model building is covered with examples of linear regression, including fitting a model to training data and making predictions on test data.

Uploaded by

naveen katta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
233 views5 pages

Essential Pandas & Sklearn Commands

This document contains notes on machine learning concepts and processes. It outlines steps for exploratory data analysis, including handling missing values, outliers, and feature engineering. It also discusses preprocessing such as scaling, encoding categorical data, and splitting data into training and test sets. Model building is covered with examples of linear regression, including fitting a model to training data and making predictions on test data.

Uploaded by

naveen katta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
  • Introduction to Machine Learning Notes
  • Data Handling Techniques
  • Advanced Data Transformations
  • Data Preprocessing in Sklearn
  • Implementation of Machine Learning Models

Machine Learning Notes

1. All the Import Modules Commands :

import numpy as np
import pandas as pd
import seaborn as sns
import [Link] as plt

2. All the commands for Eda :

[Link]() / [Link]().sum()
[Link]()
[Link]()
[Link]( axis = 0,1 ) #0 for row and 1 for column
[Link]()

 To calculate mean :-
df['column_name'].mean()

 To fill missing values by mean :-


x = df['column_name'].mean()
df['column_name'].fillna(x, inplace=True)

 To read a csv file :-


df = pd.read_csv('[Link]')
df["column_name"].unique()
df["column_name"].value_counts()

 To replace a string by nan value :-


df['column_name'].replace("string",[Link],inplace =True)
df['column_name'] = df['column_name'].astype("float")

 To create a new df with specific data type :-


# df_cat / df_num = df with categorical / numerical data
df_cat = df.select_dtypes(object)
df_num = df.select_dtypes(['int64','float64'])  
 Steps to handle missing values :
#step1 - use replace 
df['column_name'].replace("string",[Link],inplace =True)

#step2 - change the datatype to float
df['column_name'] = df['column_name'].astype("float")

#step3 - calculate the mean for the cols
x = df['column_name'].mean()

#step4 - use fillna
df['column_name'].fillna(x, inplace=True)

 Label Encoder :
from [Link] import LabelEncoder

for col in df_cat:
   le=LabelEncoder()
   df_cat[col] = le.fit_transform(df_cat[col])

 To drop columns and rows :

[Link]('column_name', axis = 1)  #for a single column
[Link](['column_name','column_name'],axis=1) #multiple 
[Link](index_number) #to drop a Row

 To handle outliers :
#Step1-: Make boxplot with two variable
Eg :- [Link](data=df,x='price',y='make')

#Step2-: Filter out the outliers
Eg :- df[(df['make']=='dodge') & (df['price']>10000)]

#Step3-: Drop the outliers
Eg :- [Link](29,inplace=True)

 Feature engineering : It is used to reduce the columns / features in the


data frame. Eg : if a data set has height and width
column ,we can create a new column = area ; a=l*b
and then remove height and width columns .
 Skewness and handling Skewness :
from [Link] import skew

To find skewness of a column :


skew(df_num['column_name'])

Using for loop & plotting graph :


for col in df_num:
   print(col)
   print(skew(df_num[col]))

   [Link]()
   [Link](df_num[col])
   [Link]()

#to find correlation
df_num.corr()
[Link](df_num.corr(), annot=True)

WE SHOULD NOT REMOVE THE SKEWNESS FOR THE COLUMN WHICH HAS
VERY HIGH CO-RELATION WITH TARGET, BECAUSE IF WE DO THAT THEN
THEIR CO-RELATION WITH THE TARGET WILL ALSO BE CHANGE.
ALSO NEVER REMOVE SKEWNESS OF A NEGATIVE COLUMNS , IT WILL GIVE
YOU A NAN VALUE.

 To Handle Skewness either find the Square root or log of that


column :
df_num['column_name']= [Link](df_num['column_name'])

 Scaling :-
1. MinMax Scaler
from [Link] import MinMaxScaler
for col in df_new:
   ms = MinMaxScaler()
   df_new[col]=ms.fit_transform(df_new[[col]])

2. Standard Scaler
from [Link] import StandardScaler
for col in df_new:
   sc = StandardScaler()
   df_new[col]=sc.fit_transform(df_new[[col]])

 Requirements for working with data in Sklearn :-

 Feature and response should be seperated objects


 Feature and response should be Numeric
 Feature and response should be numpy array
 Feature and response should have specific shape (2D)

x = [Link][:,:-1].values #Features -> independent Variable
y = [Link][:,-1].values  # Response-> dependent variable

 Taking care of missing data :-

from [Link] import SimpleImputer

#step1: define the missing value & strategy
si = SimpleImputer(missing_values=[Link], strategy='mean'
)

#step2: select the col that has missing values
[Link](x[:,1:3])

#step3: fill the value using transform method to selected 
cols and save it back
x[:,1:3] = [Link](x[:,1:3])

 Encoding categorical data ( One Hot Encoder ) : -


from [Link] import ColumnTransformer
from [Link] import OneHotEncoder

ct = ColumnTransformer(transformers= [('encoder',
OneHotEncoder(), [0])], remainder=' passthrough ')

#selecting and apply change at the same time
x = [Link](ct.fit_transform(x))

 Splitting the dataset into the training set and test set :-
from sklearn.model_selection import train_test_split

xtrain, xtest, ytrain, ytest = train_test_split(x,y, 
test_size=0.2, random_state = 1)

 Feature Scaling :-
from [Link] import StandardScaler

sc = StandardScaler()
xtrain[:,3:] = sc.fit_transform(xtrain[:,3:])
xtest[:,3:]  = sc.fit_transform(xtest[:,3:])

 Linear regression model :-


#step 1-: Select a model from sklearn
from sklearn.linear_model import LinearRegression

#step 2 -: Create an object of your model
linreg = LinearRegression()

#step 3 -: Train your model
[Link](xtrain, ytrain)

#step 4: Predict the value
ypred = [Link](xtest)

Common questions

Powered by AI

Missing data can be handled using SimpleImputer by first defining the missing value and strategy, such as the mean, median, or most frequent value. The steps include: defining an instance of SimpleImputer with the chosen strategy, selecting the columns that contain missing values by using the fit method, and finally, filling the missing values by transforming the data using the transform method .

Separating features and response variables is crucial for predictive modeling as it clearly distinguishes between inputs (independent variables) and outputs (dependent variable). In Python, this is achieved using libraries like pandas, where features and response are separated into two objects, often using iloc to slice the dataset. For example, features are stored as x = df.iloc[:,:-1].values and response as y = df.iloc[:,-1].values .

Feature scaling adjusts the range of features to a standard scale without distorting differences in the ranges of values. This is necessary as many machine learning algorithms perform better or converge faster with features on a relatively similar scale. Techniques include MinMaxScaler and StandardScaler from Scikit-learn. MinMaxScaler scales features to a predefined range, typically 0 to 1, while StandardScaler standardizes features by removing the mean and scaling to unit variance .

One Hot Encoding can be applied to categorical data using Scikit-learn's ColumnTransformer and OneHotEncoder by first specifying the columns to transform and then fitting the transformer to the data. This process is necessary because machine learning models require numerical input, and One Hot Encoding converts categorical variables into a format suitable for the model (binary vectors).

Skewness can impact model accuracy by distorting statistical assumptions like normality, which many algorithms rely on. Skewed distributions often lead to poorly estimated model parameters. In Python, skewness can be handled by applying transformations such as the square root or log to make the data distribution more symmetrical before fitting models, while ensuring no negative values are transformed to avoid NaNs .

Removing skewness from columns highly correlated with the target variable can inadvertently weaken or alter these correlations, which are crucial for predictive power. High correlation with the target often signifies that skewness is not arbitrary but potentially significant. Therefore, maintaining these distributions can enhance the model's ability to discover the true relationships in the data .

Handling data skewness is essential because skewed data can lead to misrepresentations in model training and prediction. Skewness should not be removed from columns with high correlation with the target variable, as it can alter these correlations. To address skewness, one can use transformations like the square root or log transformation on the skewed column .

EDA steps with pandas include checking for missing values using df.isna(), understanding data types and data summary with df.info() and df.describe(), handling missing data with df.dropna() and df.fillna(), and exploring value counts and unique values using df["column_name"].value_counts() and df["column_name"].unique(). EDA is vital because it helps in understanding the structure, quality, and insights of the data, which informs the direction of subsequent modeling .

Feature engineering involves creating new features from existing data to improve a model's performance. It reduces the number of features and can capture more relevant patterns. For example, if a dataset has height and width columns, a new column named area can be created by calculating the product of height and width, and then the original columns can be removed .

To create a predictive model using Scikit-learn's Linear Regression, the steps are: first, select and import Linear Regression from Scikit-learn. Next, prepare the data by splitting into x (features) and y (target), followed by train-test splitting using train_test_split. Scale the features if necessary. Subsequently, create an instance of the LinearRegression model and fit it to the training data using linreg.fit(xtrain, ytrain). Finally, predict on the test data using ypred = linreg.predict(xtest).

Machine Learning Notes 1. All the Import Modules Commands : import numpy as np import pandas as pd import seaborn as sns impo
 Steps to handle missing values : #step1 - use replace  df['column_name'].replace("string",np.nan,inplace =True) #step2 -
Skewness and handling Skewness : from scipy.stats import skew To find skewness of a column : skew(df_num['column_name']) Usi
from sklearn.preprocessing import StandardScaler for col in df_new:    sc = StandardScaler()    df_new[col]=sc.fit_transform(
from sklearn.model_selection import train_test_split xtrain, xtest, ytrain, ytest = train_test_split(x,y,  test_size=0.2, ran

You might also like