Data Science

with
Visualization
By
Prof. Madhusmita Behera

Department of
Computer Science & Engineering
Department of Computer Science & Engineering [Link] [Link]
Department of Computer Science & Engineering [Link]
 Module 1
• Introduction to Data Science:
Data science in a big data world, Benefits and uses of data science
and big data, Facets of data, The data science process, The big data
ecosystem and data science.
• The Data Science Process:
Overview of the data science process, Defining research goals and
creating a project charter, Retrieving data, Cleaning, integrating and
transforming data, Exploratory data analysis, Build the models, Presenting
findings and building applications on top of them.
Chapter 1: [Link]
science/chapter-1
Chapter 2: [Link]
science/chapter-2
Department of Computer Science & Engineering [Link]
 Data science in a big data world
• Big data refers to very large and complex datasets that traditional tools like
RDBMS can't handle well.
• The characteristics of big data are often referred to as the three Vs:
• Volume —How much data is there?
• Variety —How diverse are different types of data?
• Velocity —At what speed is new data generated?
• To work with such data, specialized methods are needed—this is where
data science comes in.
• Data science combines statistics, computer science, and tools like Python,
Hadoop, and Spark to analyze big data and extract useful insights.
• Data scientists are different from traditional statisticians because they can
handle big data and apply machine learning and programming techniques.
As data keeps growing, working with big data is becoming a key part of
every data scientist’s job.
• [Link]

Department of Computer Science & Engineering [Link]

 1.1. Benefits and uses of data science and big data
• Data science and big data are widely used across many fields—
commercial, government, education, and NGOs.
• Businesses use them for customer insights, personalized ads (e.g.,
Google AdSense), hiring, and financial predictions.
• Governments use data science for fraud detection, public projects,
and surveillance (e.g., NSA data collection).
• NGOs apply it to improve fundraising and outreach (e.g., WWF,
DataKind).
• Universities use big data in research and to enhance learning
through online platforms like Coursera, edX, and Udacity.
Overall, data science helps make smarter decisions and unlock
valuable insights from large, complex data.

Department of Computer Science & Engineering [Link]

 1.2. Facets of data
• In data science and big data, Various types of data are encountered,
each requiring specific tools and techniques for effective handling
and analysis.
• The main categories of data are these:
• Structured
• Unstructured
• Natural language
• Machine-generated
• Graph-based
• Audio, video, and images
• Streaming

Department of Computer Science & Engineering [Link]

 1.2.1. STRUCTURED DATA
• Structured data follows a fixed
format and is easy to store in
tables or databases, often
managed using SQL. However,
some structured data, like
hierarchical data (e.g., family
trees), may not fit well in
traditional databases.
• In reality, most data is
unstructured.

Department of Computer Science & Engineering [Link]

 1.2.2. UNSTRUCTURED DATA
• Unstructured data
doesn’t follow a fixed
format, making it hard
to organize or analyze.
• For example, emails
have some structure,
but their content
varies widely, making
tasks like identifying
complaints difficult.
• Language differences
add to the complexity.
Department of Computer Science & Engineering [Link]
 1.2.3. NATURAL LANGUAGE

• Natural language is a complex form of unstructured data that’s hard

to process due to its ambiguity.

• While techniques like sentiment analysis and summarization exist,

models often struggle to understand meaning across different
contexts—just like humans do, as the same words can mean
different things depending on tone or emotion.

Department of Computer Science & Engineering [Link]

 1.2.4. MACHINE-GENERATED DATA
• Machine-generated data is created
automatically by systems or devices
without human input and is rapidly
growing due to the rise of the Internet of
Things (IoT).
• Analyzing this data requires powerful,
scalable tools because of its large volume
and speed, though it's less effective for
highly connected data where
relationships matter.
• The machine data shown in the figure
would fit nicely in a classic table-
structured database. This isn’t the best
approach for highly interconnected or
“networked” data, where the
relationships between entities have a
valuable role to play.

Department of Computer Science & Engineering [Link]

 1.2.5. GRAPH-BASED OR NETWORK DATA
• Graph data refers to data modeled using
graph theory, where relationships
between objects are represented using
nodes, edges and properties.
• It's ideal for analyzing networks, like
social media connections (e.g., LinkedIn,
Twitter). Multiple overlapping graphs can
reveal deeper insights.
• Graph databases store this kind of data
and use special query languages like
SPARQL.
• Though powerful, graph data can be
complex to analyze, especially compared
to more straightforward data types.

Department of Computer Science & Engineering [Link]

 1.2.6. AUDIO, IMAGE, AND VIDEO
• Audio, image, and video data are hard for computers to analyze,
even though humans find them easy to understand.
• For example, MLBAM (Major League Baseball Advanced Media)
announced in 2014 that they’ll increase video capture to
approximately 7 TB per game for the purpose of live, in-game
analytics. High-speed cameras at stadiums will capture ball and
athlete movements to calculate in real time, for example, the path
taken by a defender relative to two baselines.
• DeepMind developed an AI that learns to play video games by
analyzing the video feed using deep learning, showing how such
data can be used for powerful AI applications.

Department of Computer Science & Engineering [Link]

 1.2.7. STREAMING DATA
• Streaming data is any type of data that arrives continuously in real
time, triggered by events rather than in batches.

• It requires special handling and fast processing.

• Examples include Twitter trends, live sports updates, and stock

market feeds.

Department of Computer Science & Engineering [Link]

 1.3. The data science process
• The data
science process
typically
consists of six
steps.

Department of Computer Science & Engineering [Link]

 1.3.1 Setting the research goal

• Setting the research goal: Define the problem and establish the
objective of the analysis.

• Data science projects in organizations start with a project charter

that outlines the goal, business benefits, required data and
resources, timeline, and expected results.

Department of Computer Science & Engineering [Link]

 1.3.2 Retrieving data

• Collect relevant and accurate data from appropriate sources.

• The second step is collecting data as planned in the project charter.

You check if the data exists, is accessible, and of good quality. It can
come from various sources like databases, Excel files, or third-party
providers.

Department of Computer Science & Engineering [Link]

 1.3.3 Data preparation

• Data preparation improves data quality for analysis. It includes:

• Data cleansing (removing errors and inconsistencies),

• Data integration (combining data from different sources), and

• Data transformation (converting data into the right format for modeling).

Department of Computer Science & Engineering [Link]

 1.3.4 Data exploration

• Analyze and visualize the data to discover patterns and insights.

• Data exploration, also called Exploratory Data Analysis (EDA),

involves understanding the relationships between variables, data
distribution, and identifying outliers using descriptive statistics,
visual tools, and simple models.

Department of Computer Science & Engineering [Link]

 1.3.5 Data modeling

• Build and evaluate predictive or analytical models.

• Use models and domain knowledge to answer research questions

by selecting and applying techniques like statistics or machine

learning. This involves selecting variables, running the model, and

checking its performance.

Department of Computer Science & Engineering [Link]

 1.3.6 Presentation & Automation

• Share the results through presentations or reports, and sometimes

automate the process so businesses can reuse the insights or
integrate them into operations.

• Communicate results and automate workflows for deployment.

Department of Computer Science & Engineering [Link]

 Example
• Setting the research goal: Identify factors causing customers to leave and
predict future churn.
• Retrieving data: Gather customer usage records, billing history, and
service interaction logs from the company's database.
• Data preparation: Handle missing values in billing info, convert
categorical features like "Contract Type" to numerical form.
• Data exploration: Discover that customers with monthly contracts are
more likely to churn than those with yearly contracts.
• Data modeling: Use logistic regression and random forest models to
predict the likelihood of a customer churning.
• Presentation and automation: Present churn predictions via dashboards
and deploy the model to send alerts when high-risk customers are
detected.

Department of Computer Science & Engineering [Link]

 1.5 The big data ecosystem and data science
• The big data ecosystem includes many tools and technologies that
can be overwhelming due to rapid innovation. Grouping them by
similar functions helps make sense of them. Data scientists use only
a subset.
• A Mind Map

Department of Computer Science & Engineering [Link]

Department of Computer Science & Engineering [Link]
 1.4.1 Distributed File Systems
• A distributed file system operates across multiple servers, unlike a normal file
system.
• It supports common file operations like storing, reading, deleting, and securing
files.
• Advantages:
• Can store files larger than a single computer disk.
• Automatically replicates files across servers for redundancy and performance.
• Scales easily, removing limits of single-server memory or storage.
• Earlier, scaling was done by upgrading to more powerful servers (vertical scaling).
• Now, scaling is achieved by adding more smaller servers (horizontal scaling), making
it almost limitless.
• Hadoop File System (HDFS) is the most well-known and widely used distributed file
system. HDFS is an open-source version of the Google File System.
• Other examples of distributed file systems include:
• Red Hat Cluster File System
• Ceph File System
• Tachyon File System

Department of Computer Science & Engineering [Link]

 1.4.2 Distributed programming framework
• After storing data in a distributed file system, the next step is to process it
efficiently.
• In distributed systems, programs are moved to where the data resides, not
the other way around.
• Using general-purpose languages like C, Python, or Java for distributed
tasks can be complex due to:
• Handling job failures
• Managing subprocess results
• Coordinating distributed tasks
• Distributed programming frameworks simplify this by:
• Automating fault tolerance and task management
• Providing tools to work seamlessly with distributed data
• Many open-source frameworks are available to ease distributed data
processing.

Department of Computer Science & Engineering [Link]

 1.4.3 Data integration framework
• Data integration is used to move data from one source to another in
big data systems.
• It is required after setting up a distributed file system.
• Apache Sqoop and Apache Flume are commonly used tools for data
transfer.
• The process is similar to the ETL (Extract, Transform, Load) process
in traditional data warehouses.
• It helps in collecting and preparing data from various sources for
analysis or storage.

Department of Computer Science & Engineering [Link]

 1.4.4 Machine learning frameworks
• Machine learning, statistics, and applied mathematics are used to
extract insights from data.
• Algorithms can now be implemented by deriving formulas, coding
them, and applying them to data.
• Modern data volumes are too large for a single computer to handle
efficiently.
• Many old algorithms are inefficient and don’t scale well with today's
data sizes due to high time complexity.
• Specialized frameworks and libraries are now essential for large-scale
data analysis.
Department of Computer Science & Engineering [Link]
 1.4.4 Machine learning frameworks
• Scikit-learn is the most popular Python library for general machine learning
tasks.
• PyBrain is a library for neural networks, which mimic the learning processes
of the human brain.
• NLTK (Natural Language Toolkit) is used for working with natural language
data.
• Pylearn2 is another machine learning library, though less mature than Scikit-
learn.
• TensorFlow is a powerful deep learning library developed by Google.
• Apache Spark is a machine learning engine designed for real-time learning on
large datasets.
Department of Computer Science & Engineering [Link]
 1.4.5 NoSQL databases
• NoSQL databases are used to store and manage huge amounts of
data.
• Traditional relational databases like Oracle SQL, MySQL, and Sybase
IQ were commonly used before NoSQL.
• NoSQL stands for "Not Only SQL", indicating it extends beyond the
capabilities of traditional SQL.
• The shift to NoSQL is due to limitations in traditional databases, not
just SQL language constraints.
• Traditional databases struggle to scale, especially with big data
characteristics like volume, variety, and velocity.
• NoSQL databases support scalable storage, real-time processing,
and flexible data formats.

Department of Computer Science & Engineering [Link]

 1.4.5 NoSQL databases
• Many different types of databases have arisen, but they can be categorized
into the following types:
• Column databases store data in columns for faster queries, often using cell-wise
storage while keeping table-like structures.
• Document stores store data as flexible documents instead of tables, allowing
dynamic schemas.
• Streaming data is processed in real time instead of batches, using tools like Storm.
• Key-value stores assign a unique key to each value, offering high scalability but
requiring more developer control.
• SQL on Hadoop runs batch queries using a SQL-like language on top of MapReduce.
• NewSQL combines NoSQL scalability with relational database features, using SQL and
relational models.
• Graph databases store data using graph structures, making them ideal for
relationships like those in social networks.

Department of Computer Science & Engineering [Link]

 1.4.6 Scheduling tools
• Scheduling tools are used to automate repetitive tasks in data
processing.
• They can trigger jobs based on specific events, such as adding a new
file to a folder.
• These tools are similar to CRON on Linux but tailored for big data
needs.
• They can initiate tasks like MapReduce jobs when new data
becomes available.

Department of Computer Science & Engineering [Link]

 1.4.7 Benchmarking tools
• Benchmarking tools help optimize big data systems using
standardized profiling suites.
• These suites are based on typical big data job workloads.
• Benchmarking is usually done by IT infrastructure professionals, not
data scientists.
• Optimizing infrastructure through benchmarking can lead to
significant cost savings.
• For instance, a 10% improvement on a 100-server cluster saves the
cost of 10 servers.

Department of Computer Science & Engineering [Link]

 1.4.8 System deployment
• System deployment involves setting up big data infrastructure and
applications.
• Deployment tools help automate the installation and configuration
of big data components.
• These tools make it easier for engineers to manage complex system
setup.
• System deployment is typically handled by infrastructure engineers,
not data scientists.

Department of Computer Science & Engineering [Link]

 1.4.9 Service programming
• Service programming allows applications to share their outputs with
other systems.
• It is useful when others want to use your big data application's results
without knowing its internal architecture.
• Service tools expose big data applications as services accessible to
other applications.
• Data scientists may need to expose their models through services.
• REST (Representational State Transfer) is a common example used to
deliver data to websites.

Department of Computer Science & Engineering [Link]

 1.4.10 Security
• Security is important to control who can access your data.
• Fine-grained access control is needed without managing it separately
for each application.
• Big data security tools provide centralized and detailed access
control.
• Security in big data systems is a specialized area handled by security
experts.
• Data scientists usually interact with security only as data consumers,
not implementers.

Department of Computer Science & Engineering [Link]

 Chapter 2: The Data Science Process

• Overview of the data science process

• Defining research goals and creating a project charter
• Retrieving data
• Cleaning, integrating and transforming data
• Exploratory data analysis
• Build the models
• Presenting findings and building applications on top of them.

Department of Computer Science & Engineering [Link]

Department of Computer Science & Engineering [Link]
 2.1 Overview of the data science process
1. The first step is to define a clear research goal so that everyone understands
what the project is about, how it will be done, and why it's important.
2. The second step is to gather raw data from the right sources, making sure
you have access to it for analysis.
3. The third step is data preparation, where raw data is cleaned, combined, and
transformed into a usable format for analysis and modeling.
4. The fourth step is data exploration, where you analyze and visualize the data
to understand patterns, relationships, and anomalies before modeling.
5. The final step is model building, where you use the prepared data to
generate insights or make predictions, often favoring simple, effective
models over complex ones.
6. The final step is presenting your results clearly and, if needed, automating
the process to support better decisions and ongoing improvements.

Department of Computer Science & Engineering [Link]

 2.1.1 Don’t be a slave to the process
• Not all data science projects follow a strict process—it often
depends on the company, the project, and the team.
• While complex projects may need a structured approach, others
may benefit from agile, iterative methods.
• It’s common to revisit earlier steps as new insights or data needs
emerge during the process.
• Flexibility is key, especially in large organizations where getting
additional data can be slow and requires approvals.

Department of Computer Science & Engineering [Link]

 2.2 Step 1: Defining research goals and creating a
project charter
• The first step is to define a clear
research goal so that everyone
understands what the project is
about, how it will be done, and
why it's important.

• 2.2.1 Spend time understanding

the goals and context of your
research.
• 2.2.2 Create a project charter.

Department of Computer Science & Engineering [Link]

 2.2.1 Spend time understanding the goals and
context of your research.

• Take time to clearly understand the research goals and business

context.

• Knowing exactly what the business expects and how your work will
be used is critical to project success.

• Many data scientists fail by skipping this step and misaligning their
work with business needs.

Department of Computer Science & Engineering [Link]

 2.2.2 Create a project charter.
• Once you understand the business problem, define the expected
deliverables clearly and get formal agreement—usually documented
in a project charter, which is essential for major projects.
• A project charter requires teamwork, and your input covers at least
the following:
• A clear research goal
• The project mission and context
• How you’re going to perform your analysis
• What resources you expect to use
• Proof that it’s an achievable project, or proof of concepts
• Deliverables and a measure of success
• A timeline

Department of Computer Science & Engineering [Link]

 Step 2: Retrieving data
•j

Department of Computer Science & Engineering [Link]