Skip to content
#

real-time-data-pipeline

Star

Here are 12 public repositories matching this topic...

Language: All
Filter by language
All 12 Python 6 Jupyter Notebook 3 Java 1 PLpgSQL 1 TypeScript 1
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

kishlayjeet / Stock-Market-Real-Time-Data-Pipeline-with-Apache-Kafka-and-Cassandra

Star 29

A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.

python aws real-time kafka cassandra ec2 pipeline etl data-engineering stock-market aws-ec2 ec2-instance apache-kafka kafka-streams kafka-streaming data-pipeline etl-pipeline data-engineering-pipeline stock-market-data-pipeline real-time-data-pipeline
  • Updated Jun 7, 2023
  • Python

aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-Streaming

Star 16

This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.

notifications python java docker web-development kafka spring-boot dashboard reactjs spark-streaming data-analytics mysql-database postgresql-database data-processing change-data-capture debezium real-time-systems real-time-data-pipeline
  • Updated Oct 24, 2024
  • Java

aymane-maghouti / Real-Time-Data-Pipeline-Using-Kafka

Star 15

This project implements a real-time data pipeline using Apache Kafka, Python's psutil library for metric collection, and SQL Server for data storage. The pipeline collects metrics data from the local computer, processes it through Kafka brokers, and loads it into a SQL Server database. Additionally, a real-time dashboard is created using Power BI.

python real-time data-visualization data-collection apache-kafka powerbi data-streaming real-time-data-pipeline
  • Updated Oct 31, 2023
  • Python

quixio / community-highlights

Star 1

Collecting highlights from the Quix community and social media in the form of interesting questions, comments, challenges, solutions and insights

python kafka etl stream-processing data-engineering kafka-consumer kafka-producer python-kafka data-pipeline streaming-etl stream-processor realtime-data event-driven-architecture kafka-python kafka-producer-consumer streaming-pipeline data-intensive-applications realtime-etl real-time-data-pipeline real-time-data-processing
  • Updated Aug 30, 2024
  • Python

wiktorbielski / precious-metals-rag-airflow-pipeline

Star 0

An end-to-end data engineering and AI pipeline that orchestrates real-time precious metals market analysis. The system ingests spot prices via Apache Kafka, stores time-series data in Google BigQuery, and utilizes Retrieval-Augmented Generation (RAG) with local LLMs (Ollama) to automate financial sentiment analysis and trend detection.

python bigquery airflow kafka etl gcp data-engineering rag llm real-time-data-pipeline
  • Updated Mar 22, 2026
  • Python

stereosky / community-highlights

Star 0

Collecting highlights from the Quix community and social media in the form of interesting questions, comments, challenges, solutions and insights

python kafka etl stream-processing data-engineering kafka-consumer kafka-producer python-kafka data-pipeline streaming-etl stream-processor realtime-data event-driven-architecture kafka-python kafka-producer-consumer streaming-pipeline data-intensive-applications realtime-etl real-time-data-pipeline real-time-data-processing
  • Updated Feb 4, 2025
  • Python

Improve this page

Add a description, image, and links to the real-time-data-pipeline topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the real-time-data-pipeline topic, visit your repo's landing page and select "manage topics."

Learn more