Module 4: Understanding Data under Python
1. Introduction to Data
Definition of Data
Data refers to raw facts, observations, or measurements collected from various sources that can be processed to
produce meaningful information.
Examples:
Temperature readings
Student marks
Sensor outputs
Website traffic data
When data is analyzed and interpreted, it becomes information that helps in decision-making.
Types of Data
1. Structured Data
Structured data is organized in a tabular format, such as rows and columns.
Example:
ID Name Marks
1 Rahul 85
2 Anita 90
Python libraries used:
Pandas
NumPy
2. Unstructured Data
Data that does not follow a specific structure.
Examples:
Images
Videos
Audio files
Text documents
3. Semi-Structured Data
Data that has some structure but not strictly tabular.
Examples:
JSON files
XML files
Importance of Data in Python
Python is widely used in data analysis, machine learning, and scientific computing because it provides powerful
libraries for handling and processing data.
Common libraries:
NumPy → numerical computation
Pandas → data analysis
Matplotlib → visualization
SciPy → scientific computing
Example of simple data handling:
data = [10, 20, 30, 40]
print("Data:", data)
2. Data Collection
Definition
Data collection is the process of gathering information from various sources for analysis and decision-making.
In Python-based systems, data can be collected from multiple sources.
Sources of Data
1. Sensors and IoT Devices
Examples:
Temperature sensors
Soil moisture sensors
Environmental sensors
Example:
temperature = 28
humidity = 65
print("Temperature:", temperature)
print("Humidity:", humidity)
2. Databases
Data can be collected from databases such as:
MySQL
PostgreSQL
MongoDB
Example:
import sqlite3
conn = [Link]("[Link]")
cursor = [Link]()
[Link]("SELECT * FROM students")
data = [Link]()
print(data)
3. Web Data
Data can be collected from websites using web scraping.
Example libraries:
BeautifulSoup
Requests
Example:
import requests
response = [Link]("[Link]
print([Link])
4. Surveys and Forms
Data collected from:
Online forms
Questionnaires
Feedback systems
3. Data Storage
Definition
Data storage refers to the process of saving collected data in a structured format so it can be retrieved and
processed later.
Types of Data Storage
1. File Storage
Data stored in files such as:
CSV files
Excel files
JSON files
Text files
Example: Writing data to a file
file = open("[Link]", "w")
[Link]("Python Data Processing")
[Link]()
Reading data:
file = open("[Link]", "r")
print([Link]())
[Link]()
2. CSV File Storage
CSV (Comma-Separated Values) files are commonly used for storing tabular data.
Example:
import csv
with open("[Link]", "w") as file:
writer = [Link](file)
[Link](["Name", "Marks"])
[Link](["Rahul", 85])
3. Database Storage
Databases allow efficient storage and retrieval of large datasets.
Example using SQLite:
import sqlite3
conn = [Link]("[Link]")
cursor = [Link]()
[Link]("CREATE TABLE IF NOT EXISTS students(name TEXT, marks
INTEGER)")
[Link]("INSERT INTO students VALUES('Rahul',85)")
[Link]()
[Link]()
4. Data Processing
Definition
Data processing is the method of converting raw data into meaningful information.
This involves several steps such as:
1. Data cleaning
2. Data transformation
3. Data analysis
4. Data visualization
Example of Data Processing
numbers = [10, 20, 30, 40]
total = sum(numbers)
average = total / len(numbers)
print("Total:", total)
print("Average:", average)
Output:
Total: 100
Average: 25
Steps in Data Processing
1. Data Cleaning
Removing errors or missing values.
Example:
data = [10, None, 20, 30]
clean_data = [x for x in data if x is not None]
print(clean_data)
2. Data Transformation
Changing data format or structure.
Example:
numbers = [1,2,3]
squared = [x**2 for x in numbers]
print(squared)
5. Statistical Techniques for Data Processing
Statistical techniques help in summarizing, analyzing, and interpreting data.
Python provides libraries such as:
NumPy
Pandas
SciPy
1. Mean (Average)
Mean represents the central value of a dataset.
Formula:
Mean = Sum of values / Number of values
Example:
import numpy as np
data = [10, 20, 30, 40]
mean = [Link](data)
print("Mean:", mean)
2. Median
Median is the middle value of a sorted dataset.
Example:
import numpy as np
data = [10, 20, 30, 40]
median = [Link](data)
print("Median:", median)
3. Mode
Mode is the value that appears most frequently in a dataset.
Example:
from statistics import mode
data = [1,2,2,3,4]
print(mode(data))
4. Standard Deviation
Standard deviation measures how much the data varies from the mean.
Example:
import numpy as np
data = [10,20,30,40]
std = [Link](data)
print("Standard Deviation:", std)
5. Variance
Variance measures the spread of the data.
Example:
import numpy as np
data = [10,20,30,40]
var = [Link](data)
print("Variance:", var)
Example: Complete Statistical Analysis
import numpy as np
data = [12, 15, 18, 20, 22]
print("Mean:", [Link](data))
print("Median:", [Link](data))
print("Standard Deviation:", [Link](data))
print("Variance:", [Link](data))
Summary
Topic Description
Data Raw facts and observations
Data Collection Gathering data from sources
Data Storage Saving data for future use
Data Processing Converting raw data into useful information
Statistical Techniques Methods like mean, median, variance for analyzing data