Foundations of Data Science Unit1 -1-

Q1. Need for Data Science

Today lots of data are generated via various electronic gadgets. Earlier lots of data were stored in
unstructured as well as structured format and lots of data flow happen. If you look today, data is created
faster than it is imagined. While travelling by road, we can recognize lots of data being created, for
example, vehicles speed, traffic light switching, Google map, etc. which get captured through satellites and
transmitted to handheld devices in real time. It guides by showing several alternative paths with traffic
intensity and a favorable path to take. We also get a suggestion on which route to take and which route to
avoid. So there is a lot of data being created as we do our day-to-day activity. The problem is that we are
not doing anything with the data. We are not able to analyze due to inefficient scientific insights.

The main purpose of data science is to compute better decision making. In some situations, there
are always tricky decisions to be made, on which prediction techniques are suitable for specific conditions.
For example, can we predict better traffic rout for reach destination? can we predict delays in the case of
an airline? Can we protect demand for a certain product? Another area is business in wearable sales.

Data science is crucial today because it enables businesses and organizations to extract valuable
insights from vast amounts of data, leading to better decision-making, increased efficiency, and improved
outcomes across various sectors.

Q2. What Is Data Science?

Data Science is also known as data-driven science, which makes use of scientific methods,
processes, and systems to extract knowledge or insights from data in various forms, i.e. either structured
or unstructured. Data Science uses the most advanced hardware, programming systems, and algorithms to
solve problems that have to do with data. It is where artificial intelligence is going.

Data science is the study of data that helps us derive useful insight for business decision making.
Data Science is all about using tools, techniques, and creativity to uncover insights hidden within data. It
combines math, computer science, and domain expertise to tackle real-world challenges in a variety of
fields.

Data Science processes the raw data and solve business problems and even make prediction about
the future trend or requirement. For example, from the huge raw data of a company, data science can help
answer following question:

 What do customer want?

 How can we improve our services?
 What will the upcoming trend in sales?
 How much stock they need for upcoming festival.

Data science involves these key steps:

Data Collection: Gathering raw data from various sources, such as databases, sensors, or user interactions.
Data Cleaning: Ensuring the data is accurate, complete, and ready for analysis.
Data Analysis: Applying statistical and computational methods to identify patterns, trends, or relationships.
Data Visualization: Creating charts, graphs, and dashboards to present findings clearly.
Decision-Making: Using insights to inform strategies, create solutions, or predict outcomes.
Foundations of Data Science Unit1 -2-
Q3. Data Science Process
The data science process is a systematic approach to extracting insights and knowledge from data,
involving several key stages: defining the problem, collecting and preparing data, exploring and analyzing
data, building models, evaluating those models, and finally, deploying and communicating the results. This
structured framework ensures a reliable and efficient path from raw data to actionable insights.

Here's a more detailed breakdown of the process:

Problem Definition:
 Clearly define the business problem or question that needs to be addressed.
 Identify the specific goals and objectives of the data science project.
 Determine the key performance indicators (KPIs) to measure success.
Data Collection:
 Gather data from various sources, both internal and external.
 This may include databases, spreadsheets, APIs, and web scraping.
 Consider the types of data required (structured, unstructured) and their formats.
Data Preparation:
 Clean and preprocess the data to handle inconsistencies, missing values, and errors.
 This stage may involve data transformation, integration, and dimensionality reduction.
 Ensure the data is in a suitable format for analysis and modeling.
Data Exploration and Analysis:
 Explore the data using descriptive statistics and visualizations to understand its characteristics.
 Identify patterns, trends, and relationships within the data.
 Use exploratory data analysis (EDA) techniques to gain insights and formulate hypotheses.
Modeling:
 Select and apply appropriate machine learning algorithms or statistical models.
 Train and validate the models using the prepared data.
 Optimize the model parameters to achieve desired performance.
Evaluation:
 Evaluate the performance of the models using appropriate metrics.
 Compare different models to identify the best-performing one.
 Iterate on the modeling and evaluation process to improve results.
Deployment and Communication:
 Deploy the chosen model into a production environment.
 Integrate the model into existing systems or applications.
 Communicate the findings and insights to stakeholders through visualizations and reports.
Q4. Business Intelligence and Data Science
Business intelligence (BI) and data science are both essential for making data-driven decisions. Data
science is the study of finding patterns and forecasts through sophisticated analytics, machine learning,
and algorithms. In contrast, the main function of business intelligence is to provide historical data so that
companies can make well-informed operational decisions.
Foundations of Data Science Unit1 -3-
What is Data Science?
Data science is basically a field in which information and knowledge are extracted from the data by
using various scientific methods, algorithms, and processes. It can thus be defined as a combination of
various mathematical tools, algorithms, statistics, and machine learning techniques which are thus used to
find the hidden patterns and insights from the data which help in the decision-making process. Data
science deals with both structured as well as unstructured data. It is related to both data mining and big
data. Data science involves studying the historic trends and thus using its conclusions to redefine present
trends and also predict future trends. Technologies

What is Business Intelligence?

Business intelligence(BI) is a set of technologies, applications, and processes that are used by
enterprises for business data analysis. It is used for the conversion of raw data into meaningful information
which is thus used for business decision-making and profitable actions. It deals with the analysis of
structured and sometimes unstructured data which paves the way for new and profitable business
opportunities. It supports decision-making based on facts rather than assumption-based decision-making.
Thus it has a direct impact on the business decisions of an enterprise. Business intelligence tools enhance
the chances of an enterprise to enter a new market as well as help in studying the impact of marketing
efforts.

Q5. Prerequisites for a Data Scientist

Discover the path to a successful career in data science by understanding its broad applications
across various industries and exploring the specifics of classes and careers in this expansive field. Learn
about the relevant skills such as knowledge of programming languages and artificial intelligence, required
to excel in roles like Data Analyst and Machine Learning Engineer.

Key Insights

 Data science is a field that requires proficiency in areas such as mathematics, computer programming,
and artificial intelligence.
 Various sectors such as banking, marketing, advertising, and healthcare, rely heavily on data science
expertise.
 Data science skills include mastering programming languages like Python and R, understanding
probability and statistics, and training in artificial intelligence and machine learning techniques.
 Prerequisites for learning data science may include a strong background in mathematics, familiarity
with object-oriented programming (OOP), and SQL.
 Data science professionals such as Data Scientists and Data Analysts play a crucial role in providing
actionable insights to stakeholders, emphasizing the importance and relevance of data science in
today's industries.

Q6. Tools and Skills Needed for Data scientists

Data scientists need a mix of technical and soft skills, including proficiency in programming
languages like Python and R, strong statistical and mathematical abilities, expertise in machine learning,
and effective communication and problem-solving skills. They also need to be familiar with data
visualization tools, database management systems, and cloud computing platforms.
Foundations of Data Science Unit1 -4-
Technical Skills:

Programming: Python (with libraries like Pandas, NumPy, and scikit-learn) and R are essential for data
manipulation, analysis, and model building.

Databases: SQL is crucial for querying and managing data stored in relational databases.

Machine Learning: Understanding and applying machine learning algorithms and techniques (e.g.,
supervised, unsupervised, reinforcement learning) is fundamental.

Data Visualization: Tools like Tableau, Power BI, and Matplotlib help in presenting data insights
effectively.

Big Data Technologies: Familiarity with tools like Hadoop and Spark is necessary for handling large
datasets.

Cloud Computing: Platforms like AWS, Azure, and Google Cloud are increasingly important for data storage,
processing, and deployment.

Statistics and Mathematics: A solid foundation in statistics, probability, linear algebra, and calculus is
crucial for data analysis and model building.

Soft Skills:

Communication: Data scientists need to effectively communicate their findings to both technical and non-
technical audiences.

Problem-Solving: A data scientist's core role is to solve business problems using data-driven insights.

Business Acumen: Understanding the business context and objectives is vital for formulating relevant
questions and interpreting results.

Tools

Programming Languages: Python, R, SQL.

Data Visualization Tools: Tableau, Power BI, Matplotlib.
Machine Learning Libraries: Scikit-learn, TensorFlow, PyTorch.
Data Manipulation Libraries: Pandas, NumPy.
Big Data Platforms: Hadoop, Spark.
Cloud Platforms: AWS, Azure, Google Cloud.
Version Control: Git.
Integrated Development Environments (IDEs): Jupyter Notebooks, PyCharm.
Q7. Applications of Data Science in various fields
Data Science is the deep study of a large quantity of data, which involves extracting some meaning
from the raw, structured, and unstructured data. It uses various tools and techniques to extract meaningful
data from raw data. Data Science is also known as the Future of Artificial Intelligence.

Real-world Applications of Data Science

1. In Transport:

Data Science is also entered in real-time such as the Transport field like Driverless Cars. With the
help of Driverless Cars, it is easy to reduce the number of Accidents.
Foundations of Data Science Unit1 -5-
2. In Search Engines:

The most useful application of Data Science is Search Engines. As we know when we want to
search for something on the internet, we mostly use Search engines like Google, Yahoo, and Bing, etc. So
Data Science is used to get Searches faster.

3. In Finance:

Data Science plays a key role in Financial Industries. Financial Industries always have an issue of
fraud and risk of losses. Thus, Financial Industries needs to automate risk of loss analysis in order to carry
out strategic decisions for the company. Also, Financial Industries uses Data Science Analytics tools in
order to predict the future. It allows the companies to predict customer lifetime value and their stock
market moves.

4. In E-Commerce:

E-Commerce Websites like Amazon, Flipkart, etc. uses data Science to make a better user
experience with personalized recommendations.

5. In Health Care:

In the Healthcare Industry data science act as a boon. Data Science is used for:

 Detecting Tumor.

 Drug discoveries.

 Medical Image Analysis.

 Virtual Medical Bots.

 Genetics and Genomics.

 Predictive Modeling for Diagnosis etc.

6. Image Recognition:

Currently, Data Science is also used in Image Recognition. For Example, When we upload our
image with our friend on Facebook, Facebook gives suggestions Tagging who is in the picture. This is done
with the help of machine learning and Data Science.
7. Targeting Recommendation:

Targeting Recommendation is the most important application of Data Science. Whatever the user
searches on the Internet, he/she will see numerous posts everywhere. For example: Suppose I want a
mobile phone, so I just Google search it and after that, I changed my mind to buy offline. In Real -World
Data Science helps those companies who are paying for Advertisements for their mobile. So everywhere
on the internet in the social media, in the websites, in the apps everywhere I will see the
recommendation of that mobile phone which I searched for. So this will force me to buy online.

8. Airline Routing Planning:

With the help of Data Science, Airline Sector is also growing like with the help of it, it becomes
easy to predict flight delays.
Foundations of Data Science Unit1 -6-
9. Data Science in Gaming:

In most of the games where a user will play with an opponent i.e. a Computer Opponent, data
science concepts are used with machine learning where with the help of past data the Computer will
improve its performance. There are many games like Chess, EA Sports, etc. will use Data Science concepts.

10. Medicine and Drug Development:

The process of creating medicine is very difficult and time-consuming and has to be done with full
disciplined because it is a matter of Someone's life. Without Data Science, it takes lots of time, resources,
and finance or developing new Medicine or drug but with the help of Data Science, it becomes easy
because the prediction of success rate can be easily determined based on biological data or factors.

Q8. Data Security Issues

Data science is vulnerable to various security issues, including data breaches, privacy violations,
and the potential for misuse of AI-generated synthetic data. Data security threats in data science include
unauthorized access, data breaches, and non-compliance with regulations. Effective data security in data
science requires addressing these issues through robust access controls, encryption, regular data backups,
and employee training.

The following are the security issues in data science:

1. Data Breaches and Unauthorized Access:
 Data breaches involve the unauthorized access, disclosure, or theft of sensitive data, which can lead
to significant financial and reputational damage.
 Unauthorized access can occur due to weak passwords, inadequate access controls, or
compromised insider credentials.
2. Data Privacy Concerns:
 Data privacy is a major concern in data science, especially when dealing with large datasets
containing sensitive personal information.
 Balancing the need for data analysis with individual privacy rights is a critical challenge.
3. Data Poisoning and Manipulation:
 Data poisoning: involves injecting malicious data into a dataset to compromise the integrity of
machine learning models.
 Manipulated data: can lead to inaccurate predictions, flawed insights, and biased decision-making.
 Detecting and preventing data poisoning attacks: is a significant challenge in data science.
4. Insecure APIs and Systems:
 APIs (Application Programming Interfaces) are often used to access and share data, but they can be
vulnerable to attacks if not properly secured.
 Insecure APIs can be exploited to gain unauthorized access to data, manipulate data, or disrupt
systems.
5. Evolving Threats and Advanced Persistent Threats (APTs):
 Cyber threats are constantly evolving, requiring ongoing adaptation of security measures and data
science models.
 APTs are sophisticated and persistent attacks that can evade traditional security measures.
Foundations of Data Science Unit1 -7-
6. Challenges in Access Control and Authentication:
 Controlling access to data is crucial for maintaining confidentiality and integrity.
 Poorly defined access control policies can lead to unauthorized access and data breaches.
 Effective authentication mechanisms are essential to verify the identity of users and systems
accessing data.
7. Lack of Security Audits and Patch Management:
 Regular security audits are essential to identify vulnerabilities and ensure compliance with security
policies.
 Lack of centralized patch management can leave systems vulnerable to known security flaws.
8. Risks Associated with AI-Generated Synthetic Data:
 AI-generated synthetic data can be used to simulate real-world data for training and testing machine
learning models.
 If not properly secured, synthetic data can be misused for malicious purposes, such as creating fake
accounts, generating fraudulent information, or launching attacks.
9. Non-Compliance with Data Protection Regulations:
 Organizations must comply with various data protection regulations, such as GDPR, CCPA, and
others.
 Non-compliance can result in significant fines, legal penalties, and reputational damage.

Q9. Data Collection Strategies

Data collection strategies involve various methods for gathering information to support research,
analysis, or decision-making. These methods can be broadly categorized into primary and secondary data
collection techniques. Common primary methods include surveys, interviews, observations, and focus
groups, while secondary data collection relies on existing information sources like market research
reports or government publications.

Primary Data Collection Methods:

 Surveys: Gather information from a large group of individuals through questionnaires, often online
or in person.
 Interviews: Involve one-on-one conversations to gather detailed information, either structured
(pre-defined questions) or unstructured (flexible, conversational).
 Observations: Involve systematically watching and recording behaviors, events, or situations.
 Focus Groups: Facilitated discussions with small groups of people to explore opinions, attitudes,
and experiences.
 Experiments: Involve manipulating variables to observe cause-and-effect relationships.

Secondary Data Collection Methods:

 Document Reviews: Analyzing existing documents like reports, articles, or case studies.
 Database Analysis: Using existing databases to extract relevant information.
 Literature Reviews: Summarizing and synthesizing findings from published research.
 Online Tracking: Monitoring online behavior through website analytics or social media monitoring.
Foundations of Data Science Unit1 -8-
 Transactional Tracking: Analyzing data generated from transactions, such as sales records.

When choosing a data collection strategy, consider factors like the type of data needed
(qualitative or quantitative), the research questions, the resources available, and the target population.

Q10. Data pre-processing overview in Data Science

Data pre-processing is a critical step in any data science project, essential for transforming raw,
messy data into a clean, structured, and usable format for analysis and machine learning models.

Why is data preprocessing important?

Raw data often contains errors, inconsistencies, and irrelevant information that can significantly
impact the accuracy and reliability of your analysis and model predictions. Proper data preprocessing
addresses these issues, leading to:

 Improved data quality: It ensures the data is accurate, complete, and consistent.
 Enhanced model performance: Models trained on high-quality data learn better patterns, leading
to more accurate predictions and reduced over fitting.
 More efficient analysis: Clean and structured data is easier to process and analyze, streamlining the
entire data science workflow.
 Better interpretability: Clear and consistent features make it easier to understand how models are
making decisions.
Key steps in data pre-processing
1. Data cleaning
 Handling missing values: Missing data can occur due to various reasons, such as collection errors or
incomplete records. Common strategies include:
o Deletion: Removing rows or columns with missing values. This is suitable for small amounts
of missing data but can lead to information loss.
o Imputation: Replacing missing values with estimated values, like the mean, median, or
mode of the column. More advanced techniques include regression imputation and K-
nearest neighbors imputation.
 Removing duplicates: Identifying and eliminating redundant entries to prevent bias and ensure
accuracy.
 Correcting errors and inconsistencies: Fixing typos, standardizing formats (e.g., date formats, string
cases), and resolving structural discrepancies.

2. Data integration
 Merging data from multiple sources: Combining datasets from different databases, spreadsheets,
or APIs into a unified format.
 Schema matching: Aligning fields and data structures across different sources to ensure
consistency.
 Data deduplication: Identifying and removing duplicate entries across multiple datasets to maintain
data integrity.
3. Data transformation
Foundations of Data Science Unit1 -9-
 Scaling and normalization: Adjusting numerical feature values to a similar range (e.g., 0-1) or
standardizing them to have zero mean and unit variance. This is crucial for algorithms sensitive to
feature scales.
 Encoding categorical variables: Converting categorical data into numerical representations that
machine learning algorithms can process. Techniques include one-hot encoding, label encoding,
and binary encoding.
 Feature engineering: Creating new features or modifying existing ones to enhance the model's
performance and capture more complex relationships.
 Binning and discretization: Grouping continuous data into discrete categories or intervals. This can
simplify data and handle noisy or outlier values.
 Data smoothing: Reducing noise from the dataset to identify underlying trends and patterns more
clearly.
4. Data reduction
 Feature selection: Choosing the most relevant features to improve model performance and reduce
overfitting.
 Dimensionality reduction: Transforming data into a lower-dimensional space while preserving
essential information. Techniques like Principal Component Analysis (PCA) are often used.
 Sampling: Selecting a representative subset of data when dealing with very large datasets.
5. Data discretization
Data discretization is the process of converting continuous numerical values into discrete categories or
bins. It simplifies complex data, improves model performance, and reduces noise.
 Equal-width binning: Dividing the range of data into intervals of equal size.
 Equal-frequency binning: Dividing data into bins, so each contains roughly the same number of
samples.
6. Data Munging
While often used interchangeably with data wrangling and sometimes incorporated as part of other
pre-processing tasks, it is important to understand the distinctions between data munging and data
filtering.
 Data munging: Data munging, also referred to as data wrangling, focuses on transforming and
mapping data from one format into another that is more appropriate and valuable for downstream
purposes like analytics. It prepares raw, unstructured data into analysis-ready information.
7. Date Filtering
 Data filtering: Data filtering is the process of selecting a subset of data based on specific criteria or
conditions to focus on the most relevant information