Exercise for 3 sessions with banking data
Target topics:
Data Structure
Program and Control Statements
Details of Data Handling Variables
Data Handling Program Examples
[Link]
[Link]
[Link]
happy_hour1.py
happy_hour2.py
[Link]
[Link]
[Link]
[Link]
[Link]
Summary of Python Variable Types and Their
Usages:
Variable
Description Usage Example
Type
int Whole numbers Counting, indexing age = 25
Measurements,
float Numbers with decimals temperature = 36.6
percentages
str Text strings Names, messages name = "Alice"
bool True/False values Logical conditions is_raining = True
fruits = ['apple',
list Mutable sequence Grouping related items 'banana']
tuple Immutable sequence Fixed data grouping coordinates = (10, 20)
student = {'name':
dict Key-value pairs Mapping data 'Alice'}
Unordered unique
set Unique collections unique_numbers = {1, 2, 3}
values
NoneType Absence of value Initializing variables result = None
bytes Immutable binary data Binary file handling data = b"hello"
bytearray Mutable binary data Modifying binary data data = bytearray(b"hello")
range Immutable number Loops and iterations range(5)
Variable
Description Usage Example
Type
sequence
Mathematical
complex Complex numbers z = 3 + 4j
computations
Sample programs
Class syllabus program – 5
Program 1 - Variable Understanding
# Variables are just nicknames
# they're plain, lowercase words
name = "Mattan Griffel"
orphan_fee = 200
teddy_bear_fee = 121.80
total = orphan_fee + teddy_bear_fee
print(name, "the total will be", total)
# This will produce an error unless you've already defined subtotal
# print(subtotal)
# You can use an f-string to put a variable directly into a string
print(f"{name}, the total will be ${total:,.2f}")
Data engineering program example [Execute]
Program 2 – Variable understanding cont….
# Strings are text surrounded by quotes
# Both single (' ') and double (" ") can be used
kanye_quote = ('My greatest pain in life is that I will never '
'be able to see myself perform live.')
print(kanye_quote)
# Switch to single quotes if your string uses double-quotes
hamilton_quote = 'Well, the word got around, they said, "This kid is insane,
man"'
print(hamilton_quote)
# A few string functions
print(kanye_quote.upper())
print(kanye_quote.lower())
print("ok fine".replace("fine", "great").upper())
# F-Strings let you write Python code directly inside of a string
# inside of curly brackets ({ })
print(f"1 + 1 equals {1 + 1}")
name = "mattan griffel"
print(f"{[Link]()}, the total will be ${200 + 121.8 :,.2f}")
Program 3 – Understanding If statement
answer = input("Do you want to hear a joke? ")
if [Link]() in ["yes", "y"]:
print("What's loud and sounds like an apple?")
print("AN APPLE")
elif [Link]() in ["no", "n"]:
print("Fine.")
else:
print("I don't understand.")
Program 4: understanding loops
# Use for to loop over a list
numbers = [1, 2, 3]
for number in numbers:
print(number)
stocks = ["fb", "aapl", "nflx", "goog"]
for stock in stocks:
print([Link]())
# Looping Challenge
for number in range(1, 11):
print(number, "squared is", number * number)
squares = []
for number in range(1, 11):
[Link](number * number)
print(squares)
Program 5: Understanding function
# Functions are little snippets of code
# that we can reuse over and over
def average(numbers):
return sum(numbers) / len(numbers)
grades = [90, 85, 74]
prices = [12.99, 9.99, 5.49, 7.50]
print(average(grades))
print(average(prices))
print(average([0, 1, -1]))
def get_city(address):
return [Link](', ')[1]
columbia = "3022 Broadway, New York, NY 10027, USA"
city = get_city(columbia)
print(city)
# Function Challenge #1
def get_state(address):
return [Link](', ')[2].split()[0]
state = get_state(columbia)
print(state)
def divisible_by(number, divisor):
return number % divisor == 0
print(divisible_by(number=15, divisor=3))
print(divisible_by(number=20, divisor=3))
# Function Challenge #2
def uppercase_and_reverse(text):
return [Link]()[::-1]
print(uppercase_and_reverse('Banana'))
6. Write a simple program to load the given csv fine and workout the below
mathematical computations: [from Banking data]
Data structure:
Variable type
Customer_ID Str
Age Numeric
Qualification String
Geo_Location String
State Staring
Personal_Score Numeric
Profession String
Tax_Payer Logical
Income Numeric
AVG_Trans Numeric
Loan_Paid Logical
Customer_Nam
e string
a. Extract the converted excel file [.csv] format to into data frame
b. Prepare the data profile report
c. List out Geo specific customer volume & display them in Bar-chart
d. Convert the gio specific customer quality into % & display them in pie
chart
e. Consolidate customer list against the Personal score & visualize the
results in pie chart?
….Cont..
Mini Project:
DATA engineering example program: [Assignment]
Here are a few basic Python programs that demonstrate common tasks in data engineering:
1. Reading Data from a CSV File
In data engineering, a common task is to read data from CSV files for further processing.
python
Copy code
import pandas as pd
# Read data from a CSV file
df = pd.read_csv('[Link]')
# Display the first 5 rows of the dataset
print([Link]())
2. Handling Missing Data
Cleaning data often involves handling missing or null values.
python
Copy code
import pandas as pd
# Read data from a CSV file
df = pd.read_csv('data_with_missing_values.csv')
# Check for missing values
print([Link]().sum())
# Fill missing values with the mean of the column
df_filled = [Link]([Link]())
# Drop rows with missing values
df_dropped = [Link]()
# Print the cleaned data
print(df_filled.head())
print(df_dropped.head())
3. Data Aggregation (GroupBy)
In data engineering, you might need to aggregate data based on certain features (e.g., calculating
the average sales by product).
python
Copy code
import pandas as pd
# Sample data
data = {'Product': ['A', 'B', 'A', 'B', 'A', 'B'],
'Sales': [100, 200, 150, 250, 130, 300]}
df = [Link](data)
# Group by 'Product' and calculate the total and average sales
grouped_df = [Link]('Product').agg({'Sales': ['sum', 'mean']})
print(grouped_df)
4. Data Transformation (Normalizing Data)
Transforming data for machine learning models often involves scaling or normalizing features.
python
Copy code
import pandas as pd
from [Link] import MinMaxScaler
# Sample data
data = {'Feature1': [10, 20, 30, 40, 50],
'Feature2': [100, 200, 300, 400, 500]}
df = [Link](data)
# Normalize data to be between 0 and 1
scaler = MinMaxScaler()
df_scaled = [Link](scaler.fit_transform(df), columns=[Link])
print(df_scaled)
5. Connecting to a Database (PostgreSQL)
Data engineers often work with databases. Here’s an example of how to connect to a PostgreSQL
database and query data.
python
Copy code
import psycopg2
# Establish a connection to the PostgreSQL database
conn = [Link](
dbname="your_db_name",
user="your_username",
password="your_password",
host="your_host",
port="5432"
)
# Create a cursor to interact with the database
cur = [Link]()
# Execute a query
[Link]("SELECT * FROM your_table LIMIT 5")
# Fetch the results
rows = [Link]()
# Print the results
for row in rows:
print(row)
# Close the connection
[Link]()
[Link]()
6. ETL (Extract, Transform, Load) Workflow
ETL is a critical part of data engineering. Here’s an example of reading data, transforming it, and
saving it to a new file.
python
Copy code
import pandas as pd
# Extract: Read data from CSV
df = pd.read_csv('raw_data.csv')
# Transform: Clean and process the data
df['new_column'] = df['existing_column'] * 2 # Example transformation
# Load: Save the transformed data to a new CSV
df.to_csv('transformed_data.csv', index=False)
print("ETL process complete!")
7. Data Pipeline with Airflow (Basic Example)
Airflow is used to schedule and monitor workflows. Here’s a simple DAG (Directed Acyclic
Graph) that defines a pipeline.
python
Copy code
from airflow import DAG
from [Link].dummy_operator import DummyOperator
from [Link].python_operator import PythonOperator
from datetime import datetime
def my_function():
print("This is a simple ETL function")
# Define the DAG
dag = DAG('simple_pipeline', description='Simple ETL DAG',
schedule_interval='@daily',
start_date=datetime(2023, 1, 1), catchup=False)
# Define the tasks
start_task = DummyOperator(task_id='start', dag=dag)
etl_task = PythonOperator(task_id='etl_task', python_callable=my_function,
dag=dag)
end_task = DummyOperator(task_id='end', dag=dag)
# Set up the pipeline
start_task >> etl_task >> end_task