Scribd
Upload
212 views4 pages

Data Engineering Complete Notes

The document outlines key concepts in data engineering, focusing on creating scalable infrastructure and components, data ingestion, and processing methods like ETL and ELT. It emphasizes the importance of data cleaning, structuring, and enriching, as well as the use of modern storage solutions such as data lakes and warehouses. Additionally, it covers stream processing and the need for secure data storage practices.

Uploaded by

bansarinimbalkarofficial221
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
212 views4 pages

Data Engineering Complete Notes

The document outlines key concepts in data engineering, focusing on creating scalable infrastructure and components, data ingestion, and processing methods like ETL and ELT. It emphasizes the importance of data cleaning, structuring, and enriching, as well as the use of modern storage solutions such as data lakes and warehouses. Additionally, it covers stream processing and the need for secure data storage practices.

Uploaded by

bansarinimbalkarofficial221
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

DATA ENGINEERING NOTES

1. Creating a Scalable Infrastructure

- Use distributed systems for handling increasing data.

- Add more servers (horizontal scaling) instead of upgrading a single one.

- Use cloud services to auto-scale.

- Real-life example: Netflix scaling to serve millions of users.

2. Creating Scalable Components

- Break big systems into smaller parts (microservices).

- Each component scales independently.

- Example: Login service scales separately during peak hours.

3. Ingesting and Preparing Data

- Data comes from apps, logs, sensors.

- Preparation includes cleaning, structuring, validation.

4. ETL vs ELT

ETL: Extract → Transform → Load (older systems)

ELT: Extract → Load → Transform (modern cloud)

Example: Amazon Redshift uses ELT.

5. Data Wrangling

- Converting raw data into usable format.

- Example: Fixing inconsistent date formats.

6. Data Discovery

- Understanding what data exists and where.

- Example: Analyst searches product sales dataset.


7. Data Structuring

- Organizing unstructured data into tables.

- Example: JSON logs → database tables.

8. Data Cleaning

- Removing errors, duplicates.

- Example: Detecting invalid phone numbers.

9. Data Enriching

- Adding more information to existing data.

- Example: Adding location info from PIN code.

10. Data Validating

- Ensuring data meets rules.

- Example: Checking age is between 0–120.

11. Data Publishing

- Making prepared data available for users or dashboards.

12. Batch Ingestion Processing

- Loading data at intervals (hourly/daily).

- Good for large historical datasets.

13. Purpose-built Data Ingestion Tools

- Tools like AWS Glue, Apache NiFi, Talend.

14. Scaling Considerations for Batch

- Use distributed computing.

- Use compression.

- Schedule jobs during low-traffic hours.


15. Stream Processing

- Real-time data flow processing.

- Example: Fraud detection in banks.

16. Scaling Stream Processing

- Use partitioning and replication.

- Tools: Kafka, Spark Streaming.

17. Ingesting IoT Data by Stream

- IoT devices send thousands of messages per second.

- Use MQTT, Kafka, AWS IoT Core.

18. Storage in Modern Data Architecture

- Includes data lakes, warehouses, databases.

19. Storing and Organizing Data

- Use folder structures, metadata, catalogs.

20. Data Lake Storage

- Stores raw data cheaply.

- Example: AWS S3.

21. Data Warehouse Storage

- Stores processed, analytical data.

- Example: Snowflake, BigQuery.

22. Purpose-built Databases

- Key-value (Redis)

- Document (MongoDB)
- Graph (Neo4j)

23. Storage in Support of Pipeline

- Raw → Cleaned → Enriched → Published layers.

- Ensures smooth data flow.

24. Securing Storage

- Encryption at rest & in transit.

- Access control policies.

- Backups and versioning.

You might also like