INTRODUCTION TO DATA SCIENCE

23CSD105
 Course Objectives

1. Knowledge and expertise to become a data scientist.

2. Essential concepts of statistics and machine learning that are vital for data
science;
3. Significance of exploratory data analysis (EDA) in data science.
4. Critically evaluate data visualizations presented on the dashboards
5. Suitability and limitations of tools and techniques related to data science
process
 Course Outcomes:
CO1: Understand the fundamental concepts and applications of data science.
CO2: Apply machine learning techniques to solve data science problems.
CO3: Compare and contrast relational and NoSQL databases.
CO4: Utilize graph databases for data analysis.
CO5: Create interactive data visualizations.
 UNIT I
Introduction to Data science:
➢ Benefits and uses
➢Facets of data
➢Data science process in brief
➢Big data ecosystem and data science
Data Science process:
➢Overview
➢Defining goals and creating project charter
➢Retrieving data
➢Cleansing
➢Integrating and transforming data
➢Exploratory analysis
➢Model building
➢Presenting findings and building applications on top of them
Part 1: Introduction to Data science
 What are Data, Information, and Knowledge
What is Data?
Data refers to raw, unprocessed facts and
figures that lack context or interpretation
on their own. This information, in its raw
form, can be difficult to understand or
apply without extra processing.
Text, multimedia (pictures, videos,
audio), and numerical numbers are all
acceptable formats for data collection.

Insight gained from

Graph showing
Temperature readings the graph that
Example temperature trends
recorded temperature increases
over time
in the summer
 What is Information?
• Information is data that has been processed, organized, or structured to convey meaning and significance.

• Unlike raw data, information is more comprehensible and provides context that aids in understanding the data.

• To illustrate, consider a dataset that includes the average temperature, humidity, and wind direction of a city.
When this data is processed and presented in an organized manner, it becomes information that describes the
city’s climate conditions.

• The transformation from data to information generally involves several key steps:

• 1. Data Collection: Gathering raw data from various sources, such as weather stations, satellites, or sensors.

• 2. Data Cleaning: Ensuring the data is accurate, consistent, and free from errors or outliers.

• 3. Data Analysis: Applying statistical methods and computational algorithms to identify patterns, correlations, and
trends within the data.

• 4. Data Interpretation: Making sense of the analyzed data by providing context and explaining what the data
signifies.

• 5. Data Presentation: Organizing the data in a coherent and visually appealing manner, such as charts, graphs, or
reports, to effectively communicate the information.
 What is Knowledge?
• Knowledge is information that has undergone further analysis, synthesis, and refinement, resulting in a
deeper understanding and more profound insights.

• Knowledge builds on information by adding experience, context, interpretation, and judgment, allowing it to
be applied to solve problems, develop new products, or create innovative solutions.

• It is the culmination of a continuous learning process, where raw data is transformed into information and
subsequently into knowledge, empowering you to make informed decisions and take effective actions.

• The process of transforming information into knowledge involves several key steps:

1. Critical Analysis: Evaluating and interpreting information to understand its implications and relevance.

2. Synthesis: Combining different pieces of information to form a comprehensive understanding or new

concepts.

3. Refinement: Continuously updating and improving knowledge based on new data, insights, and
experiences.

4. Application: Using knowledge to address real-world problems, innovate, and create value
 Data Science Introduction

• Data science is about making sense of the vast amounts of data generated around us.
• Data science helps business uncovering patterns, trends, and insights hidden within numbers,
text, images, and more.
• It combines the power of mathematics, programming, and domain expertise to answer
questions, solve problems, and even make prediction about the future trend or requirements.
• Data science is an interdisciplinary field that focuses on extracting knowledge and insights
from structured and unstructured data using various scientific methods, processes, algorithms,
and systems.
• Simply put, it's the process of turning raw data into valuable information. It involves using
statistics, computer science, and knowledge of the specific area you're working in. Think of it
as detective work where you use data to uncover patterns, make predictions, and inform
decision-making.
 Data Science Skills

• Programming Languages: Python, R, SQL.

• Mathematics: Linear Algebra, Statistics, Probability.
• Machine Learning: Supervised and unsupervised learning, deep learning basics.
• Data Manipulation: Pandas, NumPy, data wrangling techniques.
• Data Visualization: Matplotlib, Seaborn, Tableau, Power BI.
• Big Data Tools: Hadoop, Spark, Hive.
• Databases: SQL, NoSQL, data querying and management.
• Cloud Computing: AWS, Azure, Google Cloud.
• Version Control: Git, GitHub, GitLab.
• Domain Knowledge: Industry-specific expertise for problem-solving.
• Soft Skills: Communication, teamwork, and critical thinking.
Data Science Tools and Library: There are various tools required to
analyze data, build models, and derive insights.
• Jupyter Notebook: Interactive environment for coding and documentation.
• Google Colab: Cloud-based Jupyter Notebook for collaborative coding.
• TensorFlow: Deep learning framework for building neural networks.
• PyTorch: Popular library for machine learning and deep learning.
• Scikit-learn: Tools for predictive data analysis and machine learning.
• Docker: Containerization for reproducible environments.
• Kubernetes: Managing and scaling containerized applications.
• Apache Kafka: Real-time data streaming and processing.
• Tableau: A powerful tool for creating interactive and shareable data visualizations.
• Power BI: A business intelligence tool for visualizing data and generating insights.
• Keras: A user-friendly library for designing and training deep learning models.
 Career Opportunities in Data Science
• Data Scientist: Analyze and interpret complex data to drive business decisions.
• Data Analyst: Focus on analyzing and visualizing data to identify patterns and insights.
• Machine Learning Engineer: Develop and deploy machine learning models for automation
and predictions.
• Data Engineer: Build and maintain data pipelines, ensuring data is clean and accessible.
• Business Intelligence (BI) Analyst: Create dashboards and reports to support strategic
decisions.
• AI Research Scientist: Conduct research to develop advanced AI algorithms and solutions.
• Big Data Specialist: Handle and analyze massive datasets using tools like Hadoop and Spark.
• Product Analyst: Evaluate product performance and customer behavior using data.
• Quantitative Analyst: Analyze financial data to assess risks and forecast trends.
 How is Data Science different from traditional statistics?
• While traditional statistics focuses on
analyzing data to understand
relationships and make predictions, Data
Science expands on this by incorporating
advanced computational techniques,
such as machine learning and big data
processing.
• Data Science also emphasizes extracting
insights from large, complex datasets
that may include unstructured data like
text and images.
 Importance and Applications of Data Science
Data Science plays a crucial role in transforming raw
data into actionable insights.
• Healthcare: Improving patient care, predicting
disease outbreaks, and optimizing treatment plans.
• Finance: Fraud detection, risk management, and
algorithmic trading.

• Marketing: Personalized marketing strategies,

customer segmentation, and sentiment analysis.

• E-commerce: Recommendation systems, inventory

management, and sales forecasting.

• Transportation: Route optimization, predictive

maintenance, and autonomous driving.
Data Science Life Cycle
Facets of data
 [Link] data
• Structured data is arranged in rows and column
format. It helps for application to retrieve and
process data easily. Database management system
is used for storing structured data.

• The term structured data refers to data that is

identifiable because it is organized in a structure.
The most common form of structured data or
records is a database where specific information
is stored based on a methodology of columns and
rows.

• Structured data is also searchable by data type

within content. Structured data is understood by
computers and is also efficiently organized for
human readers.
 [Link] data
• Unstructured data is data that does not follow a specified format. Row and columns
are not used for unstructured data. Therefore it is difficult to retrieve required
information. Unstructured data has no identifiable structure.

• The unstructured data can be in the form of Text: (Documents, email messages,
customer feedbacks), audio, video, images. Email is an example of unstructured data.

• Even today in most of the organizations more than 80 % of the data are in
unstructured form. This carries lots of information. But extracting information from
these various sources is a very big challenge.

• Characteristics of unstructured data:

1. There is no structural restriction or binding for the data.

2. Data can be of any type.
3. Unstructured data does not follow any structural rules.
4. There are no predefined formats, restriction or sequence for unstructured data.
5. Since there is no structural binding for unstructured data, it is unpredictable in
nature.
 [Link] Language
• Natural language is a special type of unstructured data.

• Natural language processing enables machines to recognize characters, words and sentences, then apply meaning and
understanding to that information. This helps machines to understand language as humans do.

• Natural language processing is the driving force behind machine intelligence in many modern real-world applications. The
natural language processing community has had success in entity recognition, topic recognition, summarization, text completion
and sentiment analysis.

•For natural language processing to help machines understand human language, it must go through speech recognition, natural
language understanding and machine translation. It is an iterative process comprised of several layers of text analysis.
 [Link] Generated Data
• Machine-generated data is an information that is created without human
interaction as a result of a computer process or application activity. This means
that data entered manually by an end-user is not recognized to be machine-
generated.

• Machine data contains a definitive record of all activity and behavior of our
customers, users, transactions, applications, servers, networks, factory machinery
and so on.

• It's configuration data, data from APIs and message queues, change events, the
output of diagnostic commands and call detail records, sensor data from remote
equipment and more.
• Examples of machine data are web server logs, call detail records, network event
logs and telemetry.

• Both Machine-to-Machine (M2M) and Human-to-Machine (H2M) interactions

generate machine data. Machine data is generated continuously by every
processor-based system, as well as many consumer-oriented systems.

• It can be either structured or unstructured. In recent years, the increase of

machine data has surged. The expansion of mobile devices, virtual servers and
desktops, as well as cloud- based services and RFID technologies, is making IT
infrastructures more complex.
[Link]-based or Networked data
 [Link],image and Video
• Audio, image and video are data types that pose specific challenges to a data scientist. Tasks that are trivial for humans, such as
recognizing objects in pictures, turn out to be challenging for computers.

•The terms audio and video commonly refers to the time-based media storage format for sound/music and moving pictures
information. Audio and video digital recording, also referred as audio and video codecs, can be uncompressed, lossless
compressed or lossy compressed depending on the desired quality and use cases.

• It is important to remark that multimedia data is one of the most important sources of information and knowledge; the
integration, transformation and indexing of multimedia data bring significant challenges in data management and analysis.
Many challenges have to be addressed including big data, multidisciplinary nature of Data Science and heterogeneity.

• Data Science is playing an important role to address these challenges in multimedia data. Multimedia data usually contains
various forms of media, such as text, image, video, geographic coordinates and even pulse waveforms, which come from
multiple sources. Data Science can be a key instrument covering big data, machine learning and data mining solutions to store,
handle and analyze such heterogeneous data.
[Link] Data
 Data science process in brief
Data science process consists of six stages :

1. Discovery or Setting the research goal

2. Retrieving data

3. Data preparation

4. Data exploration

5. Data modeling

6. Presentation and automation

 Data science process in brief
Step 1: Discovery or Defining research goal

This step involves acquiring data from all the identified internal and external
sources, which helps to answer the business question.

Step 2: Retrieving data

It collection of data which required for project. This is the process of gaining a
business understanding of the data user have and deciphering what each piece
of data means. This could entail determining exactly what data is required and
the best methods for obtaining it. This also entails determining what each of
the data points means in terms of the company. If we have given a data set
from a client, for example, we shall need to know what each column and row
represents.

Step 3: Data preparation

Data can have many inconsistencies like missing values, blank columns, an
incorrect data format, which needs to be cleaned. We need to process, explore
and condition data before modeling. The cleandata, gives the better
predictions.
 Data science process in brief
Step 4: Data exploration

Data exploration is related to deeper understanding of data. Try to understand how

variables interact with each other, the distribution of the data and whether there
are outliers. To achieve this use descriptive statistics, visual techniques and simple
modeling. This steps is also called as Exploratory Data Analysis.

• Step 5: Data modeling

In this step, the actual model building process starts. Here, Data scientist
distributes datasets for training and testing. Techniques like association,
classification and clustering are applied to the training data set. The model, once
prepared, is tested against the "testing" dataset.

• Step 6: Presentation and automation

Deliver the final baselined model with reports, code and technical documents in
this stage. Model is deployed into a real-time production environment after
thorough testing. In this stage, the key findings are communicated to all
stakeholders. This helps to decide if the project results are a success or a failure
based on the inputs from the model.
Big data ecosystem and data science
 Big data ecosystem and data science
What is Big Data?
Big Data refers to the vast volumes of data generated at high velocity from a
variety of sources. This data is characterized by the three V’s: Volume,
Velocity, and Variety.
Volume: Big Data involves large datasets that are too complex for traditional
data processing tools to handle. These datasets can range from terabytes to
petabytes of information.
Velocity: Big Data is generated in real-time or near real-time, requiring fast
processing to extract meaningful insights.
Variety: The data comes in multiple forms, including structured data (like
databases), semi-structured data (like XML files), and unstructured data (like
text, images, and videos).
Data Scientists use their expertise to:
[Link]: They examine complex datasets to identify patterns, trends, and
correlations.

[Link]: Using statistical models and machine learning algorithms, they

create predictive models that can forecast future trends or behaviors.

[Link]: They translate data findings into actionable business strategies

and decisions.
 Big data ecosystem and data science
A big data ecosystem is a system of tools and components that help manage and analyze large amounts of data. Data
science is the field of study that uses these tools to extract insights from data. Data scientists use these insights to help
organizations make better decisions.
Big data ecosystem
• A system of tools and components that help manage and analyze large amounts of data
• Includes computing, storing, and analyzing big data
• Uses distributed storage systems, scalable processing frameworks, and real-time data streaming tools
• Helps organizations make data-driven decisions
Data science
• The field of study that uses tools to extract insights from data
• Uses statistics, scientific computing, and other methods to analyze data
• Helps organizations make better decisions, such as improving marketing, customer service, and supply chain
management
Big data ecosystem components
• Hadoop Distributed File System (HDFS): A file system that manages large data sets
• HBase: A column-based NoSQL database that runs on top of HDFS
• Apache Spark: A scalable processing framework
• Apache Kafka: A real-time data streaming tool
Big data ecosystem and data science
Part 2: Data science process
 Data science process

What is the Data Science Process?

The data science process is the

systematic journey that converts raw
data into actionable insights. Right
from identifying the problem, and
decoding the data to building models,
coming up with the results, and
finally deploying solutions, all the
steps play a very crucial role in
extracting value from the given data.
 Steps for Data Science Processes:
Step 1: Define the Problem and Create a Project Charter
Clearly defining the research goals is the first step in the Data Science Process. A project
charter outlines the objectives, resources, deliverables, and timeline, ensuring that all
stakeholders are aligned.

Step 2: Retrieve Data

Data can be stored in databases, data warehouses, or data lakes within an organization.
Accessing this data often involves navigating company policies and requesting permissions.

Step 3: Data Cleansing, Integration, and Transformation

Data cleaning ensures that errors, inconsistencies, and outliers are removed. Data integration
combines datasets from different sources, while data transformation prepares the data for
modeling by reshaping variables or creating new features.
 Steps for Data Science Processes:
Step 4: Exploratory Data Analysis (EDA)
During EDA, various graphical techniques like scatter plots, histograms, and box plots are
used to visualize data and identify trends. This phase helps in selecting the right modeling
techniques.

Step 5: Build Models

In this step, machine learning or deep learning models are built to make predictions or
classifications based on the data. The choice of algorithm depends on the complexity of the
problem and the type of data.

Step 6: Present Findings and Deploy Models

Once the analysis is complete, results are presented to stakeholders. Models are deployed into
production systems to automate decision-making or support ongoing analysis.
Tools for Data Science Process
 Usage of Data Science Process
The Data Science Process is a systematic approach to solving data-related problems and
consists of the following steps:
1. Problem Definition: Clearly defining the problem and identifying the goal of the analysis.
2. Data Collection: Gathering and acquiring data from various sources, including data
cleaning and preparation.
3. Data Exploration: Exploring the data to gain insights and identify trends, patterns, and
relationships.
4. Data Modeling: Building mathematical models and algorithms to solve problems and make
predictions.
5. Evaluation: Evaluating the model’s performance and accuracy using appropriate metrics.
6. Deployment: Deploying the model in a production environment to make predictions or
automate decision-making processes.
7. Monitoring and Maintenance: Monitoring the model’s performance over time and making
updates as needed to improve accuracy.
 Challenges in the Data Science Process
[Link] Quality and Availability: Data quality can affect the accuracy of the models developed and therefore, it is
important to ensure that the data is accurate, complete, and consistent. Data availability can also be an issue, as the
data required for analysis may not be readily available or accessible.

[Link] in Data and Algorithms: Bias can exist in data due to sampling techniques, measurement errors, or
imbalanced datasets, which can affect the accuracy of models. Algorithms can also perpetuate existing societal
biases, leading to unfair or discriminatory outcomes.

[Link] Overfitting and Underfitting: Overfitting occurs when a model is too complex and fits the training data
too well, but fails to generalize to new data. On the other hand, underfitting occurs when a model is too simple and
is not able to capture the underlying relationships in the data.

[Link] Interpretability: Complex models can be difficult to interpret and understand, making it challenging to
explain the model’s decisions and decisions. This can be an issue when it comes to making business decisions or
gaining stakeholder buy-in.

[Link] and Ethical Considerations: Data science often involves the collection and analysis of sensitive
personal information, leading to privacy and ethical concerns. It is important to consider privacy implications and
ensure that data is used in a responsible and ethical manner.
 Brainstorming session
Idea Pitch: Come up with a small data science problem that you want to solve.
• What is the objective of your project?
• What kind of data would you need?
• How would you measure success?
 Example
Problem Statement:
Many educational institutions struggle to identify students at risk of poor academic performance early on. A data-driven approach
can help predict student outcomes and allow for timely interventions.

What is the objective of your project?

The objective is to develop a predictive model that can identify students at risk of poor academic performance based on historical
data. This model will help educators take early intervention measures, such as personalized learning plans and academic support,
to improve student outcomes.

What kind of data would you need?

To build an effective model, the following data would be required:
•Demographic Data: Age, gender, socioeconomic background
•Academic Records: Past grades, exam scores, assignment performance
•Attendance Data: Number of classes attended per subject
•Study Habits: Average study hours per week, self-learning activities
•Extracurricular Activities: Participation in sports, clubs, or cultural events
•Behavioral Factors: Time spent on educational platforms, engagement in class discussions
•Parental Involvement: Frequency of parent-teacher meetings, support in studies

How would you measure success?

Success will be measured based on the following metrics:
•Model Accuracy: Percentage of correctly predicted student performance levels (e.g., High, Medium, Low).
•Teacher & Institutional Feedback: Evaluating whether the model’s insights help educators make better academic decisions.
End of unit 1