Scribd
Upload
5 views96 pages

De With Python

The document outlines a comprehensive guide on data engineering, covering foundational concepts, infrastructure setup, and practical applications. It includes chapters on file handling, database interaction, data cleaning, and building data pipelines, with a focus on tools like Apache Kafka and Spark. Additionally, it addresses version control, monitoring, and deployment strategies for production data pipelines.

Uploaded by

kiran kin
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views96 pages

De With Python

The document outlines a comprehensive guide on data engineering, covering foundational concepts, infrastructure setup, and practical applications. It includes chapters on file handling, database interaction, data cleaning, and building data pipelines, with a focus on tools like Apache Kafka and Spark. Additionally, it addresses version control, monitoring, and deployment strategies for production data pipelines.

Uploaded by

kiran kin
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Chapter 1: What Is Data Engineering

Chapter 2: Building Our Data Engineering Infrastructure


Chapter 3: Reading and Writing Files
Chapter 4: Working with Databases
Chapter 5: Cleaning and Transforming Data
Chapter 6: Building a 311 Data Pipeline
Chapter 7: Features of a Production Data Pipeline
Chapter 8: Version Control Using the NiFi Registry
Chapter 9: Monitoring and Logging Data Pipelines
Chapter 10: Deploying Your Data Pipelines
Chapter 11: Building a Production Data Pipeline
Chapter 12: Building an Apache Kafka Cluster
Chapter 13: Streaming Data with Kafka
Chapter 14: Data Processing with Apache Spark
Chapter 15: Real-Time Edge Data – Kafka, Spark, and MiNiFi
Appendix

You might also like