Introduction to Data Science

Introduction to Data Science

Data Science is an interdisciplinary field that focuses on extracting meaningful knowledge,
insights, and patterns from data. It combines techniques from statistics, computer science,
and domain expertise to analyze both structured and unstructured data. With the rise of big
data, artificial intelligence (AI), and machine learning (ML), data science has become one of
the most essential areas in technology, business, and research.

Key Components of Data Science

1. Data Collection – Gathering raw data from various sources such as databases, sensors,
social media, and surveys.

2. Data Cleaning & Preprocessing – Removing errors, duplicates, and inconsistencies to

prepare high-quality data.

3. Exploratory Data Analysis (EDA) – Using statistical methods and visualization to

understand patterns and trends.

4. Model Building – Applying machine learning algorithms to make predictions or

classifications.

5. Interpretation & Decision-Making – Converting results into actionable insights for

businesses or scientific research.

Core Disciplines Involved

Mathematics & Statistics → foundation for data analysis and probability modeling.

Computer Science → programming (Python, R, SQL), databases, and algorithms.

Domain Knowledge → applying insights to specific industries like healthcare, finance,

marketing, etc.

Applications of Data Science

Business → Customer behavior analysis, recommendation systems (like Netflix, Amazon).

Healthcare → Disease prediction, medical imaging analysis.

Finance → Fraud detection, stock market prediction.

Government & Policy → Smart cities, crime prediction, public services optimization.
Social Media & Marketing → Sentiment analysis, targeted advertising.

Why Data Science is Important?

It helps organizations make data-driven decisions instead of relying only on intuition.

It enables automation through AI and ML.

It uncovers hidden patterns in massive datasets that humans cannot detect manually.

what is data science lifecycle/process?

The data science process is an iterative cycle that transforms raw data into useful data
products (like recommendation systems, fraud detection tools, or predictive dashboards).

Data Science Life Cycle

[Link] Definition

– The first step is to understand the problem clearly. A business or researcher asks a
question such as: “Why are customers leaving?” or “Can we predict sales next month?”
Without a clear question, the project has no direction Obtaining Data – Data is collected
from databases, APIs, IoT devices, surveys, etc.

2. Collecting Data

Once the question is clear, the next step is to gather data. Data can come from databases,
websites, surveys, sensors, or devices. Sometimes the data is in neat tables (structured),
but other times it may be messy text, videos, or images (unstructured).

3. Preparing Data

Raw data is usually messy, so it must be cleaned and organized. This means fixing
missing values, removing duplicates, and correcting errors. Exploratory Data Analysis
(EDA) is also done here to find patterns, trends, or unusual values. Feature engineering
(creating new useful columns like turning “date of birth” into “age”) also happens in this
step.

4. Building Models

In this step, machine learning is used to build predictive models. Different algorithms are
applied depending on the problem. For example, regression predicts numbers,
classification divides data into categories, clustering groups similar items, and deep
learning works with images, text, and speech. The models are trained with past data and
tested to make sure they work well.
5. Visualizing and Explaining Results

Data insights must be explained in a simple way so that others can understand. Charts,
graphs, dashboards, and reports are created to show the results. Tools like Tableau,
Power BI, or Python libraries (Matplotlib, Plotly) are used. This helps decision-makers
use the insights in real life.

6. Deployment (Data Products)

The final step is to put the model into real use. This is called deployment. The model
becomes a data product that can work inside apps or systems. Examples include Netflix
recommendations, fraud detection in banks, or predictive maintenance in factories. Once
deployed, the model must be monitored and improved regularly.

Introduction to Data
Data is a collection of raw facts, figures, and symbols that may not have meaning on their
own. When processed, it becomes information and knowledge. Data is considered the 'new
oil' in today's world.

Types of Data

1. Based on Nature
Structured Data → Organized in tables, easy to analyze (e.g., student records).

Unstructured Data → No predefined format (e.g., videos, text, social media posts).

Semi-structured Data → Not fully organized but has tags (e.g., XML, JSON).
2. Based on Measurement (Statistical Perspective)
Qualitative Data → Describes qualities; can be Nominal or Ordinal.

 Nominal → Labels without order (e.g., Gender, Religion, Country).

 Ordinal → Ordered categories (e.g., Education level: Primary, Secondary, Higher).

Quantitative Data → Numerical; can be Discrete or Continuous.

Discrete → Countable (e.g., Number of students in a class).

  Continuous → Measurable values (e.g., Height, Temperature, Weight).

Datasets
A dataset is a collection of related data organized for analysis. It usually comes in tabular
form (rows & columns) or other structured formats.

Types include Training, Testing, Validation, and Open Datasets.

Types of Datasets
1. Training Dataset – Used to train machine learning models.

2. Testing Dataset – Used to evaluate model performance.

3. Validation Dataset – Fine-tunes the model to avoid overfitting.

4. Open Datasets – Publicly available (e.g., Kaggle datasets, UCI Machine Learning
Repository)

Data Quality
Data quality refers to the degree to which data is accurate, complete, reliable, and useful for
analysis or decision-making. Poor data quality leads to wrong insights, wasted resources,
and bad decisions.
Measurements of Data Quality
1. Accuracy

Data must correctly represent the real-world values.

Example: A student’s age recorded as 20 when they are actually 21 shows inaccuracy.

2. Completeness

Data should not have missing values or gaps.

Example: Missing email addresses in a customer database.

3. Consistency

Data should be uniform across different sources and formats.

Example: A customer’s name spelled “Ali” in one database and “Alee” in another.

4. Timeliness
Data must be up-to-date and available when needed.

Example: Using last year’s stock market data for real-time trading decisions is not timely
6. Uniqueness

Data should not have duplicates.

Example: A patient registered multiple times in a hospital system.

Data Collection Issues

When collecting data, several challenges may reduce its quality:

1. Human Errors

Mistakes in data entry (typos, wrong formats).

Example: Entering salary as “$50000” vs “50000$”.

2. Incomplete Data

Respondents skipping survey questions or sensors failing to capture readings.

3. Biased Data Collection

Data that does not represent the whole population.

Example: Conducting a survey only in cities, ignoring rural areas.

4. Inconsistent Data Sources

Different systems using different formats or coding standards.

Example: Date recorded as DD/MM/YYYY in one place and MM/DD/YYYY in another.

5. Outdated Data
Old data that no longer reflects reality.

Example: Using customer addresses from 5 years ago.

Data Processing Stages

Data processing is the systematic series of steps used to transform raw data into meaningful
information. It is widely used in statistics, research, business intelligence, and data science.

1. Data Collection

Gathering raw data from various sources: surveys, databases, sensors, social media, IoT
devices, etc.

Must ensure reliability, relevance, and accuracy of sources.

2. Data Preparation (Cleaning & Organizing)

Correcting errors, handling missing values, and removing duplicates.

Standardizing formats (e.g., dates, currency).

Example: Changing 01/09/25 and 2025-09-01 into the same format.

3. Data Input

Entering prepared data into storage systems like databases, spreadsheets, or data
warehouses.

Ensures proper indexing for quick retrieval.

4. Data Processing

Applying algorithms, statistical methods, or software tools to analyze the data.

Techniques include sorting, filtering, classification, regression, clustering, etc.

5. Data Aggregation (Important Step)

Aggregation means combining data from multiple records to produce summary

information.

Example:

Total sales per month (instead of individual transactions).