Data Science Process: Definition, Steps,
and Code Examples
What is Data Science Process?
The Data Science Process is a systematic and iterative approach used to extract meaningful
insights and build predictive models from data. It combines knowledge from statistics,
computer science, and domain expertise to solve real-world problems. The process typically
includes problem definition, data collection, data cleaning, exploratory data analysis, feature
engineering, model training, evaluation, deployment, and monitoring.
1. Problem Definition
Start with a clear understanding of the problem you are trying to solve. Define your
objectives, success criteria, and understand the impact of the solution.
Example: Predict survival on the Titanic based on features such as age, sex, and passenger
class.
2. Data Collection
Gather data from relevant sources. These may include public datasets, APIs, company
databases, or sensors.
Code Example:
import seaborn as sns
titanic = sns.load_dataset('titanic')
3. Data Cleaning and Preprocessing
Handle missing data, remove duplicates, correct inconsistencies, and prepare the data for
modeling. Encoding categorical variables and scaling may be necessary.
Code Example:
[Link](['deck', 'embark_town', 'alive'], axis=1, inplace=True)
titanic['age'].fillna(titanic['age'].median(), inplace=True)
titanic['embarked'].fillna(titanic['embarked'].mode()[0], inplace=True)
titanic = pd.get_dummies(titanic, drop_first=True)
4. Exploratory Data Analysis (EDA)
Visualize and summarize data to uncover patterns and relationships between features.
Code Example:
[Link](x='class', y='survived', data=sns.load_dataset('titanic'))
[Link]('Survival Rate by Class')
[Link]()
5. Feature Engineering
Create new features or transform existing ones to improve model performance. This step
can include one-hot encoding, normalization, binning, and more.
Code Example:
titanic = pd.get_dummies(titanic, drop_first=True)
6. Model Selection and Training
Choose an appropriate algorithm and train the model on the dataset.
Code Example:
from [Link] import RandomForestClassifier
model = RandomForestClassifier()
[Link](X_train, y_train)
7. Model Evaluation
Evaluate the model using metrics such as accuracy, precision, recall, and F1-score. These
metrics help assess the model’s predictive performance.
Code Example:
from [Link] import classification_report
print(classification_report(y_test, y_pred))
8. Model Deployment
Deploy the model to a production environment where it can make predictions on new data.
Code Example:
import joblib
[Link](model, '[Link]')
model = [Link]('[Link]')
9. Monitoring and Maintenance
Continuously monitor model performance to ensure accuracy over time. Retrain or update
the model as needed.
Tools: MLflow, Prometheus, Grafana, Apache Airflow
PROGRAM FOR DATA SCIENCE PROCESS
# Import required libraries
import pandas as pd
import seaborn as sns
import [Link] as plt
from sklearn.model_selection import train_test_split
from [Link] import RandomForestClassifier
from [Link] import accuracy_score, classification_report
import joblib
# Problem Definition: Predict survival of Titanic passengers (binary classification)
# Data Collection
titanic = sns.load_dataset('titanic')
# Data Cleaning
[Link](['deck', 'embark_town', 'alive'], axis=1, inplace=True)
titanic['age'] = titanic['age'].fillna(titanic['age'].median())
titanic['embarked'] = titanic['embarked'].fillna(titanic['embarked'].mode()[0])
[Link](inplace=True) # Remove any remaining rows with missing data
# EDA (Exploratory Data Analysis)
[Link](x='class', y='survived', data=titanic)
[Link]('Survival Rate by Class')
[Link]()
[Link](figsize=(10, 6))
[Link]([Link](numeric_only=True), annot=True, cmap='coolwarm')
[Link]('Feature Correlation Heatmap')
[Link]()
# Feature Engineering
titanic = pd.get_dummies(titanic, drop_first=True)
# Model Training
X = [Link]('survived', axis=1)
y = titanic['survived']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
[Link](X_train, y_train)
# Model Evaluation
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Classification Report:\n", classification_report(y_test, y_pred))
# Deployment
[Link](model, 'titanic_model.pkl') # Save the model
# Load and use the model
loaded_model = [Link]('titanic_model.pkl')
new_predictions = loaded_model.predict(X_test)
print("Predictions on test set (first 10):", new_predictions[:10])
OUTPUT
Accuracy: 0.8268156424581006
Classification Report:
precision recall f1-score support
0 0.84 0.88 0.86 105
1 0.81 0.76 0.78 74
accuracy 0.83 179
macro avg 0.82 0.82 0.82 179
weighted avg 0.83 0.83 0.83 179