Data Science

with
Visualization
By
Prof. Madhusmita Behera

Department of
Computer Science & Engineering
Department of Computer Science & Engineering [Link] [Link]
Department of Computer Science & Engineering [Link]
 Module 1
• Introduction to Data Science:
Data science in a big data world, Benefits and uses of data science
and big data, Facets of data, The data science process, The big data
ecosystem and data science.
• The Data Science Process:
Overview of the data science process, Defining research goals and
creating a project charter, Retrieving data, Cleaning, integrating and
transforming data, Exploratory data analysis, Build the models, Presenting
findings and building applications on top of them.
Chapter 1: [Link]
science/chapter-1
Chapter 2: [Link]
science/chapter-2
Department of Computer Science & Engineering [Link]
 Data science in a big data world
• Big data refers to very large and complex datasets that traditional tools like
RDBMS can't handle well.
• The characteristics of big data are often referred to as the three Vs:
• Volume —How much data is there?
• Variety —How diverse are different types of data?
• Velocity —At what speed is new data generated?
• To work with such data, specialized methods are needed—this is where
data science comes in.
• Data science combines statistics, computer science, and tools like Python,
Hadoop, and Spark to analyze big data and extract useful insights.
• Data scientists are different from traditional statisticians because they can
handle big data and apply machine learning and programming techniques.
As data keeps growing, working with big data is becoming a key part of
every data scientist’s job.
• [Link]

Department of Computer Science & Engineering [Link]

 1.1. Benefits and uses of data science and big data
• Data science and big data are widely used across many fields—
commercial, government, education, and NGOs.
• Businesses use them for customer insights, personalized ads (e.g.,
Google AdSense), hiring, and financial predictions.
• Governments use data science for fraud detection, public projects,
and surveillance (e.g., NSA data collection).
• NGOs apply it to improve fundraising and outreach (e.g., WWF,
DataKind).
• Universities use big data in research and to enhance learning
through online platforms like Coursera, edX, and Udacity.
Overall, data science helps make smarter decisions and unlock
valuable insights from large, complex data.

Department of Computer Science & Engineering [Link]

 1.2. Facets of data
• In data science and big data, Various types of data are encountered,
each requiring specific tools and techniques for effective handling
and analysis.
• The main categories of data are these:
• Structured
• Unstructured
• Natural language
• Machine-generated
• Graph-based
• Audio, video, and images
• Streaming

Department of Computer Science & Engineering [Link]

 1.2.1. STRUCTURED DATA
• Structured data follows a fixed
format and is easy to store in
tables or databases, often
managed using SQL. However,
some structured data, like
hierarchical data (e.g., family
trees), may not fit well in
traditional databases.
• In reality, most data is
unstructured.

Department of Computer Science & Engineering [Link]

 1.2.2. UNSTRUCTURED DATA
• Unstructured data
doesn’t follow a fixed
format, making it hard
to organize or analyze.
• For example, emails
have some structure,
but their content
varies widely, making
tasks like identifying
complaints difficult.
• Language differences
add to the complexity.
Department of Computer Science & Engineering [Link]
 1.2.3. NATURAL LANGUAGE

• Natural language is a complex form of unstructured data that’s hard

to process due to its ambiguity.

• While techniques like sentiment analysis and summarization exist,

models often struggle to understand meaning across different
contexts—just like humans do, as the same words can mean
different things depending on tone or emotion.

Department of Computer Science & Engineering [Link]

 1.2.4. MACHINE-GENERATED DATA
• Machine-generated data is created
automatically by systems or devices
without human input and is rapidly
growing due to the rise of the Internet of
Things (IoT).
• Analyzing this data requires powerful,
scalable tools because of its large volume
and speed, though it's less effective for
highly connected data where
relationships matter.
• The machine data shown in the figure
would fit nicely in a classic table-
structured database. This isn’t the best
approach for highly interconnected or
“networked” data, where the
relationships between entities have a
valuable role to play.

Department of Computer Science & Engineering [Link]

 1.2.5. GRAPH-BASED OR NETWORK DATA
• Graph data refers to data modeled using
graph theory, where relationships
between objects are represented using
nodes, edges and properties.
• It's ideal for analyzing networks, like
social media connections (e.g., LinkedIn,
Twitter). Multiple overlapping graphs can
reveal deeper insights.
• Graph databases store this kind of data
and use special query languages like
SPARQL.
• Though powerful, graph data can be
complex to analyze, especially compared
to more straightforward data types.

Department of Computer Science & Engineering [Link]

 1.2.6. AUDIO, IMAGE, AND VIDEO
• Audio, image, and video data are hard for computers to analyze,
even though humans find them easy to understand.
• For example, MLBAM (Major League Baseball Advanced Media)
announced in 2014 that they’ll increase video capture to
approximately 7 TB per game for the purpose of live, in-game
analytics. High-speed cameras at stadiums will capture ball and
athlete movements to calculate in real time, for example, the path
taken by a defender relative to two baselines.
• DeepMind developed an AI that learns to play video games by
analyzing the video feed using deep learning, showing how such
data can be used for powerful AI applications.

Department of Computer Science & Engineering [Link]

 1.2.7. STREAMING DATA
• Streaming data is any type of data that arrives continuously in real
time, triggered by events rather than in batches.

• It requires special handling and fast processing.

• Examples include Twitter trends, live sports updates, and stock

market feeds.

Department of Computer Science & Engineering [Link]

 1.3. The data science process
• The data
science process
typically
consists of six
steps.

Department of Computer Science & Engineering [Link]

 1.3.1 Setting the research goal

• Setting the research goal: Define the problem and establish the
objective of the analysis.

• Data science projects in organizations start with a project charter

that outlines the goal, business benefits, required data and
resources, timeline, and expected results.

Department of Computer Science & Engineering [Link]

 1.3.2 Retrieving data

• Collect relevant and accurate data from appropriate sources.

• The second step is collecting data as planned in the project charter.

You check if the data exists, is accessible, and of good quality. It can
come from various sources like databases, Excel files, or third-party
providers.

Department of Computer Science & Engineering [Link]

 1.3.3 Data preparation

• Data preparation improves data quality for analysis. It includes:

• Data cleansing (removing errors and inconsistencies),

• Data integration (combining data from different sources), and

• Data transformation (converting data into the right format for modeling).

Department of Computer Science & Engineering [Link]

 1.3.4 Data exploration

• Analyze and visualize the data to discover patterns and insights.

• Data exploration, also called Exploratory Data Analysis (EDA),

involves understanding the relationships between variables, data
distribution, and identifying outliers using descriptive statistics,
visual tools, and simple models.

Department of Computer Science & Engineering [Link]

 1.3.5 Data modeling

• Build and evaluate predictive or analytical models.

• Use models and domain knowledge to answer research questions

by selecting and applying techniques like statistics or machine

learning. This involves selecting variables, running the model, and

checking its performance.

Department of Computer Science & Engineering [Link]

 1.3.6 Presentation & Automation

• Share the results through presentations or reports, and sometimes

automate the process so businesses can reuse the insights or
integrate them into operations.

• Communicate results and automate workflows for deployment.

Department of Computer Science & Engineering [Link]

 Example
• Setting the research goal: Identify factors causing customers to leave and
predict future churn.
• Retrieving data: Gather customer usage records, billing history, and
service interaction logs from the company's database.
• Data preparation: Handle missing values in billing info, convert
categorical features like "Contract Type" to numerical form.
• Data exploration: Discover that customers with monthly contracts are
more likely to churn than those with yearly contracts.
• Data modeling: Use logistic regression and random forest models to
predict the likelihood of a customer churning.
• Presentation and automation: Present churn predictions via dashboards
and deploy the model to send alerts when high-risk customers are
detected.

Department of Computer Science & Engineering [Link]

 1.5 The big data ecosystem and data science
• The big data ecosystem includes many tools and technologies that
can be overwhelming due to rapid innovation. Grouping them by
similar functions helps make sense of them. Data scientists use only
a subset.
• A Mind Map

Department of Computer Science & Engineering [Link]

Department of Computer Science & Engineering [Link]
 1.4.1 Distributed File Systems
• A distributed file system operates across multiple servers, unlike a normal file
system.
• It supports common file operations like storing, reading, deleting, and securing
files.
• Advantages:
• Can store files larger than a single computer disk.
• Automatically replicates files across servers for redundancy and performance.
• Scales easily, removing limits of single-server memory or storage.
• Earlier, scaling was done by upgrading to more powerful servers (vertical scaling).
• Now, scaling is achieved by adding more smaller servers (horizontal scaling), making
it almost limitless.
• Hadoop File System (HDFS) is the most well-known and widely used distributed file
system. HDFS is an open-source version of the Google File System.
• Other examples of distributed file systems include:
• Red Hat Cluster File System
• Ceph File System
• Tachyon File System

Department of Computer Science & Engineering [Link]

 1.4.2 Distributed programming framework
• After storing data in a distributed file system, the next step is to process it
efficiently.
• In distributed systems, programs are moved to where the data resides, not
the other way around.
• Using general-purpose languages like C, Python, or Java for distributed
tasks can be complex due to:
• Handling job failures
• Managing subprocess results
• Coordinating distributed tasks
• Distributed programming frameworks simplify this by:
• Automating fault tolerance and task management
• Providing tools to work seamlessly with distributed data
• Many open-source frameworks are available to ease distributed data
processing.

Department of Computer Science & Engineering [Link]

 1.4.3 Data integration framework
• Data integration is used to move data from one source to another in
big data systems.
• It is required after setting up a distributed file system.
• Apache Sqoop and Apache Flume are commonly used tools for data
transfer.
• The process is similar to the ETL (Extract, Transform, Load) process
in traditional data warehouses.
• It helps in collecting and preparing data from various sources for
analysis or storage.

Department of Computer Science & Engineering [Link]

 1.4.4 Machine learning frameworks
• Machine learning, statistics, and applied mathematics are used to
extract insights from data.
• Algorithms can now be implemented by deriving formulas, coding
them, and applying them to data.
• Modern data volumes are too large for a single computer to handle
efficiently.
• Many old algorithms are inefficient and don’t scale well with today's
data sizes due to high time complexity.
• Specialized frameworks and libraries are now essential for large-scale
data analysis.
Department of Computer Science & Engineering [Link]
 1.4.4 Machine learning frameworks
• Scikit-learn is the most popular Python library for general machine learning
tasks.
• PyBrain is a library for neural networks, which mimic the learning processes
of the human brain.
• NLTK (Natural Language Toolkit) is used for working with natural language
data.
• Pylearn2 is another machine learning library, though less mature than Scikit-
learn.
• TensorFlow is a powerful deep learning library developed by Google.
• Apache Spark is a machine learning engine designed for real-time learning on
large datasets.
Department of Computer Science & Engineering [Link]
 1.4.5 NoSQL databases
• NoSQL databases are used to store and manage huge amounts of
data.
• Traditional relational databases like Oracle SQL, MySQL, and Sybase
IQ were commonly used before NoSQL.
• NoSQL stands for "Not Only SQL", indicating it extends beyond the
capabilities of traditional SQL.
• The shift to NoSQL is due to limitations in traditional databases, not
just SQL language constraints.
• Traditional databases struggle to scale, especially with big data
characteristics like volume, variety, and velocity.
• NoSQL databases support scalable storage, real-time processing,
and flexible data formats.

Department of Computer Science & Engineering [Link]

 1.4.5 NoSQL databases
• Many different types of databases have arisen, but they can be categorized
into the following types:
• Column databases store data in columns for faster queries, often using cell-wise
storage while keeping table-like structures.
• Document stores store data as flexible documents instead of tables, allowing
dynamic schemas.
• Streaming data is processed in real time instead of batches, using tools like Storm.
• Key-value stores assign a unique key to each value, offering high scalability but
requiring more developer control.
• SQL on Hadoop runs batch queries using a SQL-like language on top of MapReduce.
• NewSQL combines NoSQL scalability with relational database features, using SQL and
relational models.
• Graph databases store data using graph structures, making them ideal for
relationships like those in social networks.

Department of Computer Science & Engineering [Link]

 1.4.6 Scheduling tools
• Scheduling tools are used to automate repetitive tasks in data
processing.
• They can trigger jobs based on specific events, such as adding a new
file to a folder.
• These tools are similar to CRON on Linux but tailored for big data
needs.
• They can initiate tasks like MapReduce jobs when new data
becomes available.

Department of Computer Science & Engineering [Link]

 1.4.7 Benchmarking tools
• Benchmarking tools help optimize big data systems using
standardized profiling suites.
• These suites are based on typical big data job workloads.
• Benchmarking is usually done by IT infrastructure professionals, not
data scientists.
• Optimizing infrastructure through benchmarking can lead to
significant cost savings.
• For instance, a 10% improvement on a 100-server cluster saves the
cost of 10 servers.

Department of Computer Science & Engineering [Link]

 1.4.8 System deployment
• System deployment involves setting up big data infrastructure and
applications.
• Deployment tools help automate the installation and configuration
of big data components.
• These tools make it easier for engineers to manage complex system
setup.
• System deployment is typically handled by infrastructure engineers,
not data scientists.

Department of Computer Science & Engineering [Link]

 1.4.9 Service programming
• Service programming allows applications to share their outputs with
other systems.
• It is useful when others want to use your big data application's results
without knowing its internal architecture.
• Service tools expose big data applications as services accessible to
other applications.
• Data scientists may need to expose their models through services.
• REST (Representational State Transfer) is a common example used to
deliver data to websites.

Department of Computer Science & Engineering [Link]

 1.4.10 Security
• Security is important to control who can access your data.
• Fine-grained access control is needed without managing it separately
for each application.
• Big data security tools provide centralized and detailed access
control.
• Security in big data systems is a specialized area handled by security
experts.
• Data scientists usually interact with security only as data consumers,
not implementers.

Department of Computer Science & Engineering [Link]

 Chapter 2: The Data Science Process

• Overview of the data science process

• Defining research goals and creating a project charter
• Retrieving data
• Cleaning, integrating and transforming data
• Exploratory data analysis
• Build the models
• Presenting findings and building applications on top of them.

Department of Computer Science & Engineering [Link]

Department of Computer Science & Engineering [Link]
 2.1 Overview of the data science process
1. The first step is to define a clear research goal so that everyone understands
what the project is about, how it will be done, and why it's important.
2. The second step is to gather raw data from the right sources, making sure
you have access to it for analysis.
3. The third step is data preparation, where raw data is cleaned, combined, and
transformed into a usable format for analysis and modeling.
4. The fourth step is data exploration, where you analyze and visualize the data
to understand patterns, relationships, and anomalies before modeling.
5. The final step is model building, where you use the prepared data to
generate insights or make predictions, often favoring simple, effective
models over complex ones.
6. The final step is presenting your results clearly and, if needed, automating
the process to support better decisions and ongoing improvements.

Department of Computer Science & Engineering [Link]

 2.1.1 Don’t be a slave to the process
• Not all data science projects follow a strict process—it often
depends on the company, the project, and the team.
• While complex projects may need a structured approach, others
may benefit from agile, iterative methods.
• It’s common to revisit earlier steps as new insights or data needs
emerge during the process.
• Flexibility is key, especially in large organizations where getting
additional data can be slow and requires approvals.

Department of Computer Science & Engineering [Link]

 2.2 Step 1: Defining research goals and creating a
project charter
• The first step is to define a clear
research goal so that everyone
understands what the project is
about, how it will be done, and
why it's important.

• 2.2.1 Spend time understanding

the goals and context of your
research.
• 2.2.2 Create a project charter.

Department of Computer Science & Engineering [Link]

 2.2.1 Spend time understanding the goals and
context of your research.

• Take time to clearly understand the research goals and business

context.

• Knowing exactly what the business expects and how your work will
be used is critical to project success.

• Many data scientists fail by skipping this step and misaligning their
work with business needs.

Department of Computer Science & Engineering [Link]

 2.2.2 Create a project charter.
• Once you understand the business problem, define the expected
deliverables clearly and get formal agreement—usually documented
in a project charter, which is essential for major projects.
• A project charter requires teamwork, and your input covers at least
the following:
• A clear research goal
• The project mission and context
• How you’re going to perform your analysis
• What resources you expect to use
• Proof that it’s an achievable project, or proof of concepts
• Deliverables and a measure of success
• A timeline

Department of Computer Science & Engineering [Link]

 Step 2: Retrieving data
Data acquisition in data science
involves collecting needed data, often
from existing sources or third parties. It
can be in various formats and usually
requires cleaning before use.
• 2.3.1 Start with data stored within the
company.
• 2.3.2 Don’t be afraid to shop around.
• 2.3.3 Do data quality checks now to
prevent problems later.
Department of Computer Science & Engineering [Link]
 2.3.1 Start with data stored within the company.
• Begin by evaluating data already available within your company.
• Company-maintained data may already be cleaned and stored properly.
• Company data is typically stored in structured systems such as databases, data
warehouses, data marts, or data lakes.
• Databases are primarily used for storing data.
• Data warehouses are designed for querying and analysis.
• A data mart is a focused subset of a data warehouse that serves a specific business function.
• Data lakes store unprocessed, raw data in its original format.
• Useful data may still exist in spreadsheets on employees’ computers.
• Finding internal data can be hard due to scattered storage and poor
documentation.
• Employee turnover may lead to lost knowledge about data.
• Many companies enforce data access control through strict digital and physical
barriers known as "Chinese walls.“
• Chinese walls” protect sensitive information, especially customer data and
getting access may involve delays and internal approvals.
Department of Computer Science & Engineering [Link]
 2.3.2 Don’t be afraid to shop around.

• If internal data is lacking, explore external data sources.

• Many companies, like Nielsen and GFK, sell industry-specific data.
• Platforms like Twitter, LinkedIn, and Facebook offer data to support their ecosystem.
• Some governments and institutions provide free, high-quality public datasets.
• Public data can include statistics on accidents, drug use, or demographics.
• External data is useful for enriching company data or for personal learning projects.
Department of Computer Science & Engineering [Link]
 2.3.3 Do data quality checks now to prevent
problems later.
• Data correction and cleaning can take up to 80% of project time.
• Early data quality checks during retrieval help avoid bigger issues later.
• During retrieval, verify data types and match data with source
documents.
• In the preparation phase, fix typos and standardize values (e.g., "USQ" to
"USA").
• The exploratory phase focuses on analyzing distributions, correlations,
and outliers.
• You may need to revisit earlier phases if issues like outliers suggest data
errors.
• Data quality improves gradually across the retrieval, preparation, and
exploration stages.

Department of Computer Science & Engineering [Link]

 2.4 Step 3: Cleansing, integrating, and transforming
data
• After data retrieval, the data often
needs cleaning and transformation
before modeling.
• Fixing issues early improves model
performance.
• Data must be in the right format, and
errors may need to be handled
programmatically.
• Common steps include
• Cleansing
• Integration
• Transformation.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data
• Data cleansing is a step in data science focused on
removing errors from data.
• Its goal is to make data a true and consistent
representation of real-world processes.
• Interpretation errors occur when data values don’t
make logical sense (e.g., age over 300).
• Inconsistency errors happen when similar data is
represented differently across sources.
• Examples include using “Female” in one table and
“F” in another, or mixing currencies like Pounds
and Dollars.
• Many simple errors can be found with basic
checks, often called “low-hanging fruit”.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data
• The encircled point
influences the model heavily
and is worth investigating
because it can point to a
region where you don’t have
enough data or might
indicate an error in the data,
but it also can be a valid data
point.
• At the data cleansing stage,
these advanced methods are,
however, rarely applied and
often regarded by certain
data scientists as overkill.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data: Data Entry Errors
• Data entry and collection are prone to
errors, especially when done manually.
• Human errors include typos and loss of
focus during data entry.
• Machine-generated data can also
contain errors, such as transmission
issues or bugs in ETL processes.
• Small datasets can be checked manually
for errors.
• For variables with limited values, errors
can be detected using frequency tables.
• Example: Incorrect entries like “Godo”
instead of “Good” reveal data entry
mistakes.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data: Redundant Whitespace
• Redundant whitespaces are difficult to spot but can cause serious data errors.
• They often lead to issues like mismatched keys or missing data in output files.
• A common problem is when strings like “FR” and “FR ” are treated as different.
• These issues often stem from poor cleaning during the ETL phase.
• Debugging whitespace errors can be time-consuming and hard to explain to
stakeholders.
• Most programming languages offer functions to remove extra spaces.
• Example: In Python, the strip() function removes leading and trailing whitespaces
easily.
• Capital letter mismatches can cause data comparison issues. “Brazil” and “brazil”
are treated as different in most languages. Use of functions like .lower() in
Python to standardize text. "Brazil".lower() == "brazil".lower() returns True.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data: Impossible values and sanity
checks, Outlier
• Sanity checks help identify impossible or unrealistic data values.
• Examples include ages over 299 or height over 3 meters. These checks
are based on logical or physical limits. They can be implemented using
simple rules like: 0 <= age <= 120
• Outlier: An outlier is a data point that differs significantly from others.
• It may follow a different pattern or logic than the rest of the data.
• Outliers can be detected using plots or min-max value tables.
• In a normal distribution, most values cluster around the mean.
• Values far from the mean may be outliers, especially if the data is expected to be
normally distributed.
• Outliers can seriously affect modeling results and should be investigated early.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data: Dealing with missing values

• Missing values aren't always wrong but

must be handled carefully.
• Some models can't process missing data
directly.
• Causes may include data collection errors
or ETL issues.
• The handling method depends on the
context and data type.
• If data is limited, removing missing
entries may not be ideal.
• Imputation using distributions is possible
when the variable is stable.
• Sometimes, missing values imply a logical
value like zero (e.g., no promotion = 0).hh

Department of Computer Science & Engineering [Link]

 Figure 2.6 Distribution plots are helpful in detecting
outliers and helping you understand the variable.

Department of Computer Science & Engineering [Link]

 2.4.1 Cleansing data: Deviations from a code book
• A code book defines the structure, meaning, and valid values of
your data.
• It includes metadata like variable names, value meanings, and data
types.
• Errors can be found by comparing dataset values against the code
book using set operations.
• Values in the dataset that don’t match the code book should be
corrected.
• Using sets and proper data structures improves error detection and
performance.
• For multiple checks, store code book values in a table and use
difference operations to find mismatches.
Department of Computer Science & Engineering [Link]
 2.4.1 Cleansing data: Different units of
measurement, Different levels of aggregation
• Different units of measurement: Data integration requires checking for
consistent units of measurement.
• Different sources may use different units (e.g., gallons vs. liters).
• Mixing units without conversion can lead to incorrect analysis.
• Simple unit conversions can resolve these discrepancies.
• Different levels of aggregation: Different levels of aggregation mean data
is grouped at varying scales (e.g., weekly vs. work week).
• This mismatch is similar to having different measurement units.
• It can be fixed by summarizing (aggregating) or expanding data to match levels.
• Detecting such differences is usually straightforward.
• Cleaning these issues early makes data integration smoother.

Department of Computer Science & Engineering [Link]

 2.4.2 Correct errors as early as possible
• Correct data errors as early as possible in the collection process.
• Fixing errors at the source reduces repeated cleaning in future projects.
• Data errors can mislead decision-makers and result in costly mistakes.
• Errors may reveal broken business processes, like misuse of coupon systems.
• Faulty equipment or sensors can also introduce data errors.
• Software bugs and mismatched settings between systems can corrupt data.
• Data scientists often can’t control data collection, so may need to fix issues in
code.
• Always keep a backup of the original data to recover from cleaning mistakes.
• Real-time data ("flowing data") is harder to back up and may need live
adjustments.
• The biggest challenge is often combining multiple data sources meaningfully.

Department of Computer Science & Engineering [Link]

 2.4.3 Combining data from different data sources
• Data often comes from multiple sources with different formats and
structures.
• Integration focuses on combining data from sources like databases,
Excel files, and text documents.
• The different ways of combining data:
• The first is joining, where you add extra information to each row from
another table using a common key.
• The second is appending (or stacking), where you add new rows from one
table to another that has the same structure.
• When combining data, you can either create a new physical table or a
virtual table, called a view.

Department of Computer Science & Engineering [Link]

 2.4.3 Combining data from different data sources: Joining
tables, Appending tables

Department of Computer Science & Engineering [Link]

 2.4.3 Combining data from different data
sources:Using views to simulate data joins and
appends

Department of Computer Science & Engineering [Link]

 2.4.3 Combining data from different data
sources:Enriching aggregated measures

• Data enrichment involves adding calculated or aggregated values to a dataset.

• Examples include total sales or percentage of stock sold in a region.
• These extra measures provide better context and insight.
• Aggregated data can show each product’s contribution within its category.
• Relative measures (like % of sales) often improve model performance more than
raw numbers.

Department of Computer Science & Engineering [Link]

 2.4.4 Transforming data
• Some models need
data in a specific
format or structure.
• After cleaning and
integrating data, the
next step is data
transformation.
• Transformation
ensures the data is
ready and suitable
for modeling.

Department of Computer Science & Engineering [Link]

 2.4.4 Transforming data: Reducing the number of
variables
• Too many variables can make a model complex and less efficient.
• Some variables may be redundant and not add value to the model.
• Reducing variables simplifies the model and improves performance.
• Techniques using Euclidean distance work best with fewer than 10 variables.

Department of Computer Science & Engineering [Link]

 2.4.4 Transforming data: Reducing the number of
variables
• Data scientists use techniques
like Principal Component
Analysis (PCA) to reduce
variables while retaining key
information. For example, two
components (combinations of
original variables) can explain
over 50% of the variation in a
dataset. This simplifies analysis
and reveals hidden patterns or
groupings.

Department of Computer Science & Engineering [Link]

 2.4.4 Transforming data: Turning variables into
dummies
• Dummy variables convert
categorical data into binary
(0 or 1) format, making it
easier for models to process.
For example, a "Weekdays"
column can be split into
separate columns like
"Monday", "Tuesday", etc.,
with 1 indicating the
presence of that day. This
step is part of cleaning and
transforming data before
modeling.

Department of Computer Science & Engineering [Link]

 2.5 Step 4: Exploratory data analysis

• EDA helps understand data deeply using

visual techniques.
• Focus is on identifying patterns, trends,
and relationships.
• Use graphs and plots to make information
easier to grasp.
• Stay open-minded and curious during this
phase.
• Often reveals anomalies missed earlier,
prompting data correction.

Department of Computer Science & Engineering [Link]

 2.5 Step 4: Exploratory data analysis
• Use a variety of charts: line graphs, histograms, Sankey
diagrams, network graphs, etc.
• Composite graphs (combining multiple simple graphs)
can provide deeper insights.
• Interactive or animated visuals can enhance
understanding and engagement.
• Mike Bostock’s website offers great interactive
examples (mostly for data presentation).

Department of Computer Science & Engineering [Link]

 Example: A Pareto diagram

Department of Computer Science & Engineering [Link]

 Example: brushing and linking.
• Brushing and linking connects multiple graphs or tables interactively.
• Selecting data in one graph highlights related data in other graphs.
• Helps reveal correlations and patterns across multiple views.
• Enhances interactive exploration and insight discovery.
• Example: Selecting countries with high scores in one chart reflects in others.

Department of Computer Science & Engineering [Link]

 Example: Histogram
• Divides a variable into discrete
categories (bins).
• Displays the number of
occurrences in each category.
• Useful for understanding
frequency distribution.

Department of Computer Science & Engineering [Link]

 Example; Boxplot
• Shows distribution
characteristics (not
frequency).
• Visualizes minimum,
maximum, median, and
quartiles.
• Useful for spotting outliers
and distribution spread
within categories.

Department of Computer Science & Engineering [Link]

 2.6 Step 5: Build the models
• Once the data is clean and well-understood, the next step is model
building.
• The purpose is to predict outcomes, classify objects, and understand
system behavior.
• This phase is goal-driven, unlike exploratory analysis which is open-
ended.
• The different techniques that are used in this phase is
• Machine learning
• Data mining
• Statistics
• Most models consist of the following main steps:
• Selection of a modeling technique and variables to enter in the model.
• Execution of the model.
• Diagnosis and model comparison

Department of Computer Science & Engineering [Link]

 2.6.1 Model and variable selection

• Choose relevant variables based on insights from exploratory data

analysis.
• Select a suitable modeling technique (statistical, machine learning,
or data mining) based on the problem.
• Performance Consideration:
• Must the model be moved to a production environment and, if so, would it
be easy to implement?
• How difficult is the maintenance on the model: how long will it remain
relevant if left untouched?
• Does the model need to be easy to explain?

Department of Computer Science & Engineering [Link]

 2.6.2 Model execution: Example of StartsModel Library.

Department of Computer Science & Engineering [Link]

 2.6.2 Model execution
• Model Fit
• R-squared shows how much variation in the data is explained by the model.
• Adjusted R-squared adds a penalty for using too many variables, discouraging overly
complex models.
• A simple model is preferred over a complex one if both perform similarly.
• An R-squared of 0.893 is considered very good, especially in business models.
• In competitions, model fit often needs to be in the 90%+ range.
• In research, even models with low R-squared (<0.2) can be valuable if predictor variables
show strong influence.
• Predictor variables have a coefficient :
• In linear models, each predictor variable has a coefficient that shows how much it affects the
target variable.
• Example: If x1 increases by 1 unit, y increases by 0.7658 (based on the coefficient).
• A strong predictor can be extremely valuable, even if the overall model is weak.
• In scientific research, identifying an influential factor (like a gene linked to cancer) is more
important than having a perfect model.
• The focus in research is often on understanding relationships, not just prediction accuracy.
• To confirm if a predictor has a real impact, we measure its significance (typically using the p-
value).

Department of Computer Science & Engineering [Link]

 2.6.2 Model execution
• Predictor significance:
• Coefficients show the effect of a variable, but may not prove statistical
significance.
• The p-value indicates the likelihood that a variable’s influence is due to
chance.
• A p-value < 0.05 is commonly considered statistically significant.
• This means there’s a 5% chance the predictor has no real effect.
• The threshold is arbitrary, and depends on how much risk you're willing to
accept.
• Other common thresholds:
• p < 0.01 → Extremely significant
• p < 0.1 → Marginally significant

Department of Computer Science & Engineering [Link]

 2.6.2 Model execution
• Linear regression works if you want to predict a value, but what if you
want to classify something?
• Then you go to classification models, the best known among them being
k-nearest neighbors.

Department of Computer Science & Engineering [Link]

 2.6.3 Model diagnostics and model comparison
• Multiple models are built and compared to select the best one.
• A holdout sample is a reserved portion of the dataset not used during model
training.
• The holdout data is used to evaluate model performance on unseen data.
• This helps test the generalizability of the model.
• Only a portion of the data is used to build the model; the rest is held back for
testing.
• Error measures (e.g., Mean Square Error) are used to assess performance.
• The model with the lowest error on the holdout sample is typically preferred.
• Mean square error is a simple measure: check for every prediction how far it
was from the truth, square this error, and add up the error of every prediction.

Department of Computer Science & Engineering [Link]

 Example
• compares the performance of two models
to predict the order size from the price. The
first model is size = 3 * price and the second
model is size = 10.
• To estimate the models, we use 800
randomly chosen observations out of 1,000
(or 80%), without showing the other 20% of
data to the model.
• Once the model is trained, we predict the
values for the other 20% of the variables
based on those for which we already know
the true value, and calculate the model
error with an error measure.
• Then we choose the model with the lowest
error. In this example we chose model 1
because it has the lowest total error.

Department of Computer Science & Engineering [Link]

 2.7 Step 6: Presenting findings and building
applications on top of them
• After building a well-performing model, the final step is to present your
findings clearly to stakeholders, turning technical results into actionable
insights.
• If your work is valuable, people may request it repeatedly.
• Automate models to save time and ensure consistent results.
• Often, only model scoring or report updates are needed.
• Use tools to update reports, Excel files, or PowerPoint decks
automatically.
• Communication, presentation, and storytelling are critical at this stage.
• Invest time in learning soft skills, as they ensure your work is heard and
acted upon.

Department of Computer Science & Engineering [Link]

Department of Computer Science & Engineering [Link]