Data Science Process

Unit II

DR. RISHI KUMAR

Introduction
The data science process is a systematic approach to analysing and
interpreting data to extract meaningful insights and solve real-world
problems.

This data science process helps businesses and organisations make data-
driven decisions, improve operations, and innovate by turning raw data
into actionable knowledge.

It involves understanding the problem at hand, collecting relevant data,

cleaning and preparing that data, exploring it to find patterns and
relationships, building predictive models, and finally, using these models to
make informed decisions.
Understanding the Data Science Process
1. Understanding the Problem
Defining the objectives: What is the end goal? Are you trying
to predict future sales, classify customer reviews, or detect
fraud?
Identifying constraints: What are the limitations? These
could be data availability, time constraints, or computational
resources.
Gathering domain knowledge: Understanding the context of
the problem helps in making informed decisions throughout
the process.
In the data science process, when defining the objectives, it’s important to be as specific as
possible.
2. Data Collection
Web Scraping: This method is useful for gathering data from public websites. It requires
knowledge of web technologies and legal considerations to avoid violating terms of service.
Python libraries like Beautiful Soup and Scrapy are commonly used for web scraping tasks.

APIs: Many online platforms, such as social media sites and financial data providers, offer APIs
to access their data. Using APIs ensures that you get structured data, often in real-time, which
is crucial for time-sensitive analyses.

Databases: Internal databases are gold mines of historical data. SQL is the go-to language for
querying relational databases, while NoSQL databases like MongoDB are used for unstructured
data.

Surveys and Sensors: Surveys are effective for collecting user opinions and feedback, while
sensors are invaluable in IoT applications for gathering real-time data from physical devices.

Data collection is not just about gathering large volumes of data, it’s about gathering quality
data. The source of data significantly impacts the subsequent analysis.
 3. Data Preparation
Raw data in the data science process is often messy and needs to be cleaned and prepared before
analysis. Data preparation involves:.
Data cleaning: Handling data such as missing values, removing
duplicates, and correcting errors.

Data transformation: Converting data into a suitable

format, such as normalising or standardising values

Feature engineering: Creating new features from

existing data that can improve the performance of
machine learning models.
4. Exploratory Data Analysis (EDA)
EDA is a crucial step in the data science process where you explore the data to uncover
patterns, anomalies, and relationships. This involves:

Descriptive statistics: Calculating mean, median, standard deviation, etc. Descriptive

statistics provide a summary of the central tendency, dispersion, and shape of the
dataset’s distribution.
Visualisation: Creating plots such as histograms, scatter plots, and box plots to visualise
data distributions and relationships.

Correlation analysis: Identifying relationships between different variables.

Exploratory Data Analysis (EDA) is not just a preliminary step but a vital part of the data
science process. It helps in forming hypotheses about the data and guides the selection
of appropriate models.
5. Data
Modelling
After understanding the data, the next step in the data science process is to build
predictive models. This involves selecting and applying appropriate algorithms.
Common techniques include:

Supervised learning: Algorithms like linear regression, decision trees, and

neural networks are used when the outcome is known.

Unsupervised learning: Techniques like clustering and dimensionality reduction

are used when the outcome is unknown.

Model evaluation: Using metrics such as accuracy, precision, recall, and F1

score to evaluate model performance.
Choosing the right model depends on the problem at hand and the nature of the data. In
supervised learning, the goal is to learn a mapping from input features to output labels. For
example:
Linear regression is suitable for predicting continuous outcomes, such as house prices.
Decision trees and random forests are versatile and can handle both classification and
regression tasks.
Neural networks are powerful for complex tasks like image and speech recognition.
In unsupervised learning, the goal is to find hidden patterns in the data. For instance:
Clustering algorithms like K-means group similar data points together, which can be useful for
customer segmentation.
Dimensionality reduction techniques like PCA (Principal Component Analysis) reduce the
number of features while preserving the essential patterns in the data.
Model evaluation is critical to ensure that the model performs well, not just on the training data
but also on unseen data. Techniques like cross-validation help in assessing the model’s
generalizability
6. Model Deployment
Once you have a satisfactory model, the next step in the data science process is to
deploy it in a real-world environment.

Model integration: Integrating the model into an application or system.

API development: Creating APIs to allow other systems to access the model.
Continuous monitoring: Regularly checking the model’s performance to ensure it
continues to perform well over time.
Model deployment is where the theoretical models meet the practical application.
Integrating the model into existing systems requires collaboration with software
developers to ensure seamless integration.
Creating APIs allows other systems to interact with the model, making it accessible
to various applications.

Continuous monitoring is crucial because the performance of models can degrade

over time due to changes in data patterns. Setting up automated monitoring and
alerting systems helps in identifying and addressing any issues promptly.
7. Communicating Results

Creating clear and intuitive visualisations to

convey findings

Writing comprehensive reports that

explain the methodology, findings, and
recommendations.

Delivering presentations to stakeholders to

discuss the insights and potential actions.
The data science process involves several critical steps, from understanding the problem
to deploying the model and communicating the results. To ensure success, follow these
best practices at each stage.

• Understand the Business Context

• Ensure Data Quality
• Document the Process
• Perform Thorough Exploratory Data Analysis (EDA)
• Choose the Right Models and Techniques
• Focus on Feature Engineering
• Implement Robust Model Evaluation
• Ensure Ethical and Responsible Data Science
• Facilitate Collaboration
• Communicate Insights Effectively
 ❖ Programming Languages: ❖ Data Collection Tools:
❖ Python ❖ BeautifulSoup
Tools and ❖ Data Science with R ❖ Scrapy
Programming
Frameworks for the ❖ Integrated Development
Data Science ❖ Data Manipulation Libraries: Environments (IDEs):
Process ❖ Pandas ❖ Jupyter Notebooks
❖ NumPy ❖ Spyder
Data science involves
a range of tools and ❖ Data Visualization Tools: ❖ Collaboration and Version
❖ Matplotlib Control:
frameworks that help ❖ Seaborn ❖ Git
you collect, process, ❖ Tableau ❖ GitHub
analyse, and visualise
❖ Machine Learning Frameworks: ❖ APIs and Data Access:
data. Here are some ❖ scikit-learn ❖ RESTful APIs
of the key tools and ❖ TensorFlow ❖ SQLAlchemy
❖ Keras
frameworks used in ❖ Big Data Tools: ❖ Data Storage and Retrieval:
the data science ❖ Hadoop ❖ SQL
process: ❖ Spark ❖ NoSQL Databases (e.g.,
MongoDB)