Overview
This project implements a complete monitoring and alerting stack for a Recommendation
System API. It consists of three main components:
1. FastAPI Application - A recommendation engine with built-in Prometheus metrics
2. Prometheus - Time-series database for metrics collection and storage
3. Grafana - Visualization and alerting platform
Project Structure
monitoring-stack/
├── [Link] # Orchestration configuration
├── app/
│ ├── [Link] # FastAPI application
│ └── __pycache__/ # Python cache directory
├── prometheus/
│ └── [Link] # Prometheus scrape configuration
├── grafana/
│ └── provisioning/
│ ├── alerting/
│ │ └── [Link] # Alert rules configuration
│ └── datasources/
│ └── [Link] # Prometheus datasource setup
Component Details
1. FastAPI Application (Recommendation Engine)
Location: [Link]
A high-performance recommendation API built with FastAPI that exposes
a /recommend endpoint with built-in Prometheus metrics.
Key Features:
• Endpoint: GET /recommend - Returns a list of recommended posts
• Metrics Exposed:
• recommendation_requests_total (Counter) - Total requests received
• recommendation_latency_seconds (Histogram) - Request processing time with
buckets
• recommendation_errors_total (Counter) - Count of model failures
• Metrics Endpoint: GET /metrics - Prometheus-compatible metrics output
Behavior:
• Adds latency simulation (0.1 to 1.0 seconds random delay)
• Simulates 20% failure rate for error tracking
• Returns error response on model failure, or recommendation data on success
Configuration:
• Container: python:3.10-slim
• Port: 8000 (exposed to host)
• Runtime: Uvicorn ASGI server
• Dependencies: fastapi, uvicorn, prometheus_client
2. Prometheus Monitoring System
Location: [Link]
Prometheus is a pull-based monitoring system that scrapes metrics from target applications
and stores them in a time-series database.
Configuration:
• Global Settings:
• Scrape interval: 5 seconds
• Metrics are collected every 5 seconds
• Scrape Configuration:
• Job name: fastapi
• Target: app:8000 (Docker internal hostname)
• Metrics endpoint: [Link]
Data Flow:
1. Prometheus queries the /metrics endpoint every 5 seconds
2. FastAPI application returns Prometheus-formatted metrics
3. Prometheus stores data points with timestamps
4. Data is queried by Grafana for visualization and alerting
Default Port: 9090
3. Grafana Visualization & Alerting
Location: provisioning
Grafana is the visualization and alerting layer that connects to Prometheus and provides
dashboards and alert notifications.
3.1 Datasource Configuration
File: [Link]
Automatically provisions Prometheus as a datasource:
• Name: Prometheus
• Type: Prometheus
• URL: [Link] (Docker internal)
• Default: Yes (set as primary datasource)
3.2 Alert Rules
File: [Link]
Defines alerting rules that trigger based on metric thresholds:
Alert: High Recommendation Latency
• Condition: 95th percentile of recommendation latency > 0.3 seconds
• Evaluation Window: 1 minute
• For Duration: Alert fires after condition is true for 2 minutes
• Description: Triggers when API response time is too slow
• Rule Expression:
• histogram_quantile(0.95, rate(recommendation_latency_seconds_bucket[1m])) > 0.3
Alert Configuration:
• SMTP Enabled: Yes
• SMTP Host: [Link]
• From Address: miraclinvinnarasi@[Link]
• Alert Name: Grafana Alerts
Default Port: 3000
Docker Compose Orchestration
File: [Link]
Services Defined:
Service Image Container Port Volume Mounts
Name
app python:3.10-slim recsys_app 8000 ./app:/app
prometheus prom/prometheus prometheus 9090 [Link]
grafana grafana/grafana grafana 3000 ./grafana/provisioning:/etc/grafana/
provisioning
Networking:
• All services connected via Docker bridge network: monitoring-stack_default
• Services communicate using container names as hostnames
Setup & Deployment Status
Current Status: Running
Deployment Date: February 4, 2026
Last Command:
docker compose up -d
Deployment Result:
[+] up 4/4 ✔ Network monitoring-stack_default Created
✔ Container recsys_app Created
✔ Container prometheus Created
✔ Container grafana Created
All services are successfully running.
Accessing the Stack
1. FastAPI Application
• URL: [Link]
• Endpoints:
• GET /recommend - Get recommendations
• GET /metrics - View Prometheus metrics
2. Prometheus
• URL: [Link]
• Features:
• Query interface for metrics
• Target status viewer
• Alert visualization
3. Grafana
• URL: [Link]
• Features:
• Dashboard creation and visualization
• Alert management and notification setup
• Provisioned Prometheus datasource
• Pre-configured alert rules
Key Metrics & Monitoring
Application Metrics
1. recommendation_requests_total
• Type: Counter
• Measures: Total number of recommendation requests received
• Usage: Track request volume and API usage patterns
2. recommendation_latency_seconds
• Type: Histogram
• Measures: Request processing time with percentile buckets
• Usage: Monitor performance and identify slow requests
• Alert Threshold: P95 > 0.3 seconds
3. recommendation_errors_total
• Type: Counter
• Measures: Total number of failed recommendations
• Usage: Track error rate and model failures
Current Alert State
Alert Rule: High Recommendation Latency
• Status: Provisioned and configured
• Evaluation: Every 1 minute
• Fire Condition: P95 latency > 1.5s for 2+ minutes
• Recovery Condition: P95 latency < 1.0s
• Notification: Email to miraclinvinnarasi@[Link] via Gmail SMTP
• Expected Behavior: Alert fires only when genuine slowdowns occur above 1.5s
threshold
PromQL Query Examples
Common queries for monitoring:
# Request rate per second
rate(recommendation_requests_total[1m])
# 95th percentile latency
histogram_quantile(0.95, rate(recommendation_latency_seconds_bucket[1m]))
# Error rate percentage
(rate(recommendation_errors_total[1m]) / rate(recommendation_requests_total[1m])) * 100
# Total requests
recommendation_requests_total
Alert Notification Configuration
SMTP Settings:
• Host: [Link]
• User: xxx@[Link]
• From: Grafana Alerts xxx@[Link]
• Status: Enabled
The stack is configured to send email notifications when alerts trigger. When the "High
Recommendation Latency" alert activates, Grafana will send an alert notification to the
configured email.
Technology Stack Summary
Component Technology Version Purpose
Container Runtime Docker Latest Container orchestration
API Framework FastAPI Latest RESTful API server
ASGI Server Uvicorn Latest Application server
Metrics Library prometheus_client Latest Metrics instrumentation
Time-Series DB Prometheus Latest Metrics storage & querying
Visualization Grafana Latest Dashboards & alerts
Python Python 3.10 Application runtime
Troubleshooting
View Logs:
docker logs recsys_app
docker logs prometheus
docker logs grafana
Restart Services:
docker compose restart
Stop Stack:
docker compose down
View Running Containers:
docker compose ps
Summary
This monitoring stack provides a complete end-to-end solution for monitoring a
recommendation system API. It captures performance metrics, provides visual insights
through Grafana dashboards, and implements automated alerting for performance
degradation. The infrastructure is containerized and easily deployable, with automatic
provisioning of datasources and alert rules.