Git Repository CI/CD Pipeline
Jupyter ML Training ML Inference Data and model
Notebooks Pipeline Pipeline monitoring
Development Environment
Historical data Production Environment
for analysis
Historical + new Save trained ML
data for model model
training ML Training
Pipeline
New data for
inference
Model
Data Storage
Storage ML Model
Data and ML Prediction
Predictions ML Inference
Pipeline
UI Application Load trained
ML model
Data Producer
Data and Model
Predictions
Performance Data and model
metrics monitoring service
ML Training Data Scientist
Pipeline
ML Inference
Pipeline
Model
Storage
Pipeline Running
Orchestration ML Engineer
Data Manager
Module
Data
Storage
Data Engineer
Inference Pipeline Dashboard App Docker with Training Pipeline
Docker Container ML Predictions & History Data Docker Container
Inference & History
Preprocessing history data data Preprocessing
Pipeline Pipeline
Feature Eng. Feature Eng.
Pipeline Pipeline
Data Data
Manager UI Application Manager
Module Module
Inference Training
Pipeline Pipeline
ML ML Predictions
Predictions History data History
Postprocessing data Postprocessing
Pipeline Docker Volumes Pipeline
with Data & Models
ML Predictions
Historical Data
Trained
Inference data Storage Model Storage ML Model
Trained ML Model
Forecasting Rental Bike Count for the next hour for dynamic pricing optimization
Problem Description
A bike-sharing platform operating across an entire city wants to optimize its dynamic pricing system.
To do so, it must accurately forecast the total number of bike rentals expected in the next hour city-wide.
These forecasts will drive real-time pricing adjustments to:
Maximize revenue during surges
Stimulate demand during slow periods
Business objective
Develop a real-time machine learning system that forecasts hour-ahead rental demand across the city,
enabling:
Dynamic pricing based on predicted aggregate demand
Incentive campaigns during forecasted low-demand hours
Improved planning for supply-demand balance and fleet utilization
Application Development Steps
Install Structure the EDA ML Model
the tools repo Analysis Building
Create UI app in Developing Developing training
Dash inference pipelines pipelines
Creating Docker & Creating a Git Repo Deploying
Docker compose files and pushing the code application on cloud
Training and Production Data Split
Rented
Bike
Count
t = 2012-08-07 11:00:00 t = 2012-12-31 23:00:00
2011-01-01
00:00:00
Raw Data Database Real-Time Prod Data
(raw_data/[Link]) (prod_data/real_time_data_prod.parquet)
Training Pipeline
History
data Preprocessing
Pipeline
Feature Eng.
Pipeline
Write and execute the code from
loading raw data to saving the model
Training
Pipeline
Postprocessing
Pipeline
Trained
ML Model
Local Environment Production Environment
2011-01-01 00:00:00
80% ....
....
2012-08-07 11:00:00
Raw Data Database Database in production
(raw_data/[Link]) (prod_data/database_prod.parquet)
DataManager.initialize_prod_database()
Raw Data
(raw_data/raw_data_full.parquet)
20% 2012-08-07 12:00:00
....
....
2012-12-31 23:00:00
Data for inference (from the future)
(prod_data/real_time_data_prod.parquet)
Step 1
Create a Production Database based on the historical data available before timestamp
t = 2012-08-07 11:00:00
Rented
Bike
Count We do it to start with the same data available when:
t = 2012-08-07 11:00:00
we initialize the application
our entire data (train + forecasting) is finished, so we
can re-run the application again
2011-01-01
00:00:00 Assume we live right now at 2012-08-07 11:00:00.
So, this data is the only data that is “physically”
available to us for model training and testing offline
Raw Data Database
(raw_data/[Link])
2011-01-01 00:00:00
....
DataManager.initialize_prod_database() ....
2012-08-07 11:00:00
Database in production
(prod_data/database_prod.parquet)
Step 2
We run the training pipeline
Rented
Bike
Count Run training pipeline Preprocessing
t = 2012-08-07 11:00:00
Pipeline
Feature Eng.
Pipeline
2011-01-01
00:00:00
Assume we live right now at 2012-08-07 11:00:00.
Here, we split into
So, this data is the only data that is “physically” Training
available to us for model training and testing offline the train and
Pipeline test datasets
Raw Data Database
(raw_data/[Link])
DataManager.initialize_prod_database() Postprocessing
Pipeline
Database in production
(prod_data/database_prod.parquet)
2011-01-01 00:00:00
....
....
2012-08-07 11:00:00
We save the trained model
models/prod
Inference Pipeline
Streaming &
history data
Preprocessing
Pipeline
Feature Eng.
Pipeline
Write and execute the code from loading new data &
a trained model and predict the future values
Inference
Pipeline
Postprocessing
Pipeline
Real time ML
Predictions
Load Trained ML Model
Step 3
We run the inference pipeline
Rented
Bike
Count t = 2012-08-07 11:00:00
ML Model Predictions
(prod_data/real_time_predictions.parquet)
We save prediction
2011-01-01
00:00:00
Postprocessing
Pipeline
Data comes to the ML App when UI button is clicked.
Database in production Data comes from
(prod_data/database_prod.parquet) Append data (prod_data/real_time_data_prod.parquet)
Inference
2011-01-01 00:00:00 at every pipeline iteration 2012-08-07 12:00:00
(e.g. 2012-08-07 12:00:00) Pipeline
.... ....
.... ....
2012-08-07 11:00:00 2012-12-31 23:00:00
Feature Eng.
Pipeline
2011-01-01 00:00:00 Slice the data batch of N latest points 2012-08-06 06:00:00
Run inference Pipeline Preprocessing
.... required for feature generation (e.g. 30) ....
.... .... Pipeline
2012-08-07 11:00:00 2012-08-07 11:00:00
2012-08-07 12:00:00 2012-08-07 12:00:00
Load the model
models/prod
ML Project Blueprint Mac OS
Linux
You with Mac OS Windows
1. Install conda
2. Create environment
3. Install dependencies
4. Set up data folders
5. Configure paths
6. Handle OS-specific issues
7. Troubleshoot common problems
You with Mac OS Separate environment with it’s
Docker Docker Docker
own libraries, Python version &
other dependences Container Container Container
ML ML User
Training Inference Interface
App App App
ML Project Blueprint
Docker Engine
Dockerfiles
Docker-compose file
Operating System
Mac OS
Linux
Windows
Docker
Image
docker push
Registry
docker build
ml-app Docker image
docker pull
docker run
Dockerfile
Docker Docker
container container
Your Laptop
Docker container Docker container Docker container
ml-app ml-app
(training) (inference) ml-ui
Docker Volumes
What is next?
Use this template to create another project
Develop more complex apps with pipeline managers/orchestrators (Kedro,
Prefect, Airflow)
Model more complex problems, make deeper analyses, engineer more complex
features, and solve other problems
Add model / experiment tracking and model registry
Add tests with Pytest
Add CI/CD pipeline
Add data and model monitoring services
Create (near) real-time ML apps
Add databases and ETL to load and store data and models
Use code formatting to clean up, lint, and check our code
And more...
Get 2 Bonus Rewards by leaving a
testimonial HERE