Languages: English | فارسی

A complete end-to-end recommendation engine providing both real-time rankings and historical analytical insights.

• Streaming: Spark Structured Streaming 4.0, Kafka (Redpanda)
• Storage: PostgreSQL (Hot Data), ClickHouse (OLAP/Analytics), Elasticsearch (Observability)
• Orchestration: Airflow 3.0
• Serving: FastAPI
• Infrastructure: Docker & Docker Compose

├── airflow/            # DAGs for aggregations, maintenance, and monitoring
├── api/                # FastAPI application (Real-time & Analytics routes)
├── docker/             # Dockerfiles and specialized compose files
├── kafka/              # Producer scripts and Kafka Connect configurations
├── postgres/           # Database initialization scripts
├── spark/              # Spark Streaming application and notebooks
├── config.py           # Centralized configuration management
└── .env                # Environment variables (Credentials & Hosts)

• DeepWiki: https://deepwiki.com/AmirJlr/RealTimeRecSys

[User Events] → [Redpanda]
      │
      ├─► [Spark Streaming] ──► [PostgreSQL]  ───┐
      │     (Real-time Ranks)                    │
      │                                          │
      ├─► [Kafka Connect]   ──► [ClickHouse]  ───├────► [FastAPI] ──► [End User]
      │     (Raw Event Log)      (Analytics)     │
      │                                          │
      └─► [Kafka Connect] ──► [Elasticsearch] ───┘
            (Observability)      (Dashboard)

• Dual-Path Serving: FastAPI serves real-time recommendations from PostgreSQL and batch analytics (top items/categories) from ClickHouse.
• Automated Operations: Airflow manages data lifecycle, from daily aggregations to system health monitoring.

1. Configure Environment Create a .env file in the root directory based on project credentials.

2. Start Infrastructure To ensure all environment variables are correctly loaded from the root .env file, use the --env-file flag:

   From Project Root:

   docker compose --env-file .env -f docker/compose-clickhouse.yml -f docker/compose-elastic.yml -f docker/compose-redpanda.yml -f docker/compose-spark.yml -f docker/compose-airflow.yml up -d

   From docker/ folder:

   cd docker
   docker compose --env-file ../.env -f compose-redpanda.yml -f compose-spark.yml -f compose-airflow.yml -f compose-clickhouse.yml -f compose-elastic.yml up -d

3. Access Services

   • FastAPI Dashboard: http://localhost:8181
   • API Docs: http://localhost:8181/docs
   • Airflow: http://localhost:8080 (airflow/airflow)
   • Redpanda Console: http://localhost:9100
   • Spark Master: http://localhost:9080

4. Configure Airflow Connections To enable maintenance DAGs like recommender_maintenance.py, you must create a Postgres connection in the Airflow UI (Admin -> Connections):

   • Connection Id: postgres_rec
   • Connection Type: Postgres
   • Host: postgres (from POSTGRES_HOST)
   • Database: recommendation_db (from POSTGRES_DB)
   • Login: (from POSTGRES_USER)
   • Password: (from POSTGRES_PASSWORD)
   • Port: 5432 (from POSTGRES_PORT)

• GET /recommendations: Real-time personalized top items (Postgres).
• GET /analytics/top-items: Historical top performers (ClickHouse).
• GET /analytics/top-categories: Popular categories over time (ClickHouse).

├── airflow/            # DAGها برای تجمیع‌ها، نگهداری و مانیتورینگ
├── api/                # برنامه FastAPI (مسیرهای بلادرنگ و تحلیلی)
├── docker/             # Dockerfileها و فایل‌های compose اختصاصی
├── kafka/              # اسکریپت‌های تولیدکننده و تنظیمات Kafka Connect
├── postgres/           # اسکریپت‌های مقداردهی اولیه دیتابیس
├── spark/              # برنامه Spark Streaming و نوت‌بوک‌ها
├── config.py           # مدیریت متمرکز تنظیمات
└── .env                # متغیرهای محیطی (اعتبارنامه‌ها و میزبان‌ها)

[رویدادهای کاربر] → [Redpanda]
      │
      ├─► [Spark Streaming] ──► [PostgreSQL]  ───┐
      │     (رتبه‌بندی بلادرنگ)                  │
      │                                          │
      ├─► [Kafka Connect]   ──► [ClickHouse]  ───├────► [FastAPI] ──► [کاربر نهایی]
      │     (لاگ خام رویدادها)   (تحلیل‌ها)      │
      │                                          │
      └─► [Kafka Connect] ──► [Elasticsearch] ───┘
            (رصدپذیری)          (داشبورد)