DATA ENGINEERING (BCSG0034) - DETAILED

EXAM STUDY GUIDE

Prepared from the GLA [Link] CSE (AIML) Syllabus
How to Use This Guide
This guide is designed for a 15-day preparation window. Focus first on: 1. Data Engineering
Lifecycle 2. ETL 3. Data Modeling & Normalization 4. OLTP vs OLAP 5. Data Warehouse vs Data
Lake 6. Data Pipelines & Airflow 7. Data Security & Privacy These topics frequently appear as
5-mark and 10-mark questions.
Introduction to Data Engineering

Data Engineering is the discipline of designing, building and maintaining systems that collect,
store, process and deliver data. It acts as the backbone of analytics, business intelligence,
data science and AI systems.

Importance:
• Makes data available in usable form.
• Supports business decisions.
• Enables machine learning and analytics.
• Handles large-scale data efficiently.

Applications:
E-commerce, healthcare, banking, logistics, social media, IoT, fraud detection and recommendation
systems.

Data Engineering Lifecycle

1. Data Generation – data produced by applications, websites, sensors and users.

2. Data Collection – gathering data using APIs, databases, scraping and logs.
3. Data Storage – storing data in databases, lakes or warehouses.
4. Data Processing – cleaning, transforming and enriching data.
5. Data Analysis – extracting insights through reports and dashboards.

Role of a Data Engineer:

• Build data pipelines.
• Maintain data quality.
• Design storage systems.
• Ensure scalability and security.

Data Collection

Sources of Data:
Internal Sources: ERP, CRM, transactions, inventory systems.
External Sources: APIs, social media, public datasets, partners.

Web Scraping:
Technique used to extract information from web pages.
Popular tools: BeautifulSoup, Selenium, Scrapy.

APIs:
Application Programming Interfaces allow software systems to exchange data.
Advantages: Structured data, reliable access, automation.

Batch Processing vs Stream Processing

Batch Processing:
Data processed at intervals.
Example: Monthly salary generation.

Advantages:
• Cost efficient
• Simpler implementation

Stream Processing:
Data processed continuously.
Example: UPI notifications, stock market feeds.

Advantages:
• Real-time insights
• Immediate action

Exam Table:
Batch = delayed results, grouped processing.
Stream = instant results, continuous processing.

Data Modeling and ER Diagrams

Data Modeling organizes information for storage and retrieval.

Entity: Real-world object (Student).

Attribute: Property (Roll Number).
Relationship: Association (Student enrolls in Course).

Benefits:
• Better database design
• Reduced redundancy
• Improved consistency

Normalization

Normalization reduces duplicate data.

1NF:
• Atomic values only.
• No repeating groups.

2NF:
• Must be in 1NF.
• Remove partial dependency.

3NF:
• Must be in 2NF.
• Remove transitive dependency.

Advantages:
• Better consistency
• Reduced redundancy

Denormalization:
Intentional duplication to improve query speed.

OLTP and OLAP

OLTP (Online Transaction Processing):

• Supports daily operations.
• Fast inserts and updates.
• Example: ATM, Railway booking.

OLAP (Online Analytical Processing):

• Supports reporting and analytics.
• Complex queries.
• Historical data.

Common Exam Difference:

OLTP = operational systems.
OLAP = decision support systems.

Relational Databases and NoSQL

Relational Databases:
MySQL, PostgreSQL, Oracle.
Characteristics:
• Tables and rows
• Structured schema
• ACID compliance

NoSQL Databases:
MongoDB (Document)
Redis (Key-Value)
Cassandra (Column)
Neo4j (Graph)

Advantages:
• Scalability
• Flexible schema
• High availability

Data Warehouse and Data Lake

Data Warehouse:
Stores processed and structured data.
Optimized for analytics.

Benefits:
• Faster reporting
• Historical analysis

Data Lake:
Stores raw data in original format.
Supports structured, semi-structured and unstructured data.

Warehouse = Processed.
Lake = Raw.

ETL Process

ETL = Extract, Transform, Load.

Extract:
Gather data from databases, APIs and files.

Transform:
Clean, standardize and enrich data.

Load:
Store transformed data into warehouse.

Benefits:
• Data consistency
• Better analytics
• Centralized storage

Most important long-answer topic.

Data Cleaning and Integration

Data Cleaning Techniques:

• Remove duplicates
• Handle missing values
• Correct formatting issues
• Validate records

Data Integration:
Combining data from multiple sources into a unified view.

Challenges:
• Different formats
• Duplicate records
• Inconsistent naming

Data Quality

Key Dimensions:
Accuracy – correct values.
Completeness – no missing values.
Consistency – same meaning everywhere.
Validity – follows rules.
Timeliness – up-to-date.
High quality data improves decision making.

Data Pipelines and Airflow

Data Pipeline:
Automated flow of data from source to destination.

Stages:
Ingestion → Processing → Storage → Analysis

Apache Airflow:
Workflow orchestration platform.
Uses DAGs (Directed Acyclic Graphs).

Functions:
• Scheduling
• Monitoring
• Dependency management

Pipeline Design Principles:

Scalability, Reliability, Monitoring, Security.

Monitoring and Maintenance

Metrics:
• Throughput
• Latency
• Error Rate
• Availability

Benefits:
• Detect failures early
• Maintain reliability
• Improve performance

Data Security and Privacy

Goals:
Confidentiality, Integrity and Availability (CIA Triad).

Security Measures:
• Authentication
• Authorization
• Encryption
• Access Control

Symmetric Encryption:
Same key for encryption and decryption.
Example: AES
Asymmetric Encryption:
Public and private keys.
Example: RSA

Data Privacy Regulations

Privacy protects personal information such as emails, phone numbers and IDs.

GDPR:
European privacy regulation.

DPDP Act:
India's Digital Personal Data Protection framework.

Importance:
• Protect users
• Reduce misuse
• Ensure legal compliance

Important Exam Questions

5 Marks:
• ETL
• Data Lake vs Warehouse
• OLTP vs OLAP
• Batch vs Stream Processing
• NoSQL Databases
• Airflow and DAG

10 Marks:
• Data Engineering Lifecycle
• ETL Architecture
• Normalization
• Data Pipelines
• Security and Privacy
• Relational vs NoSQL Databases
One-Day Before Exam Revision Sheet
Remember: • Lifecycle = Generation → Collection → Storage → Processing → Analysis • ETL =
Extract → Transform → Load • 1NF → 2NF → 3NF • OLTP = Transactions • OLAP = Analytics •
Warehouse = Processed Data • Lake = Raw Data • Airflow uses DAGs • AES = Symmetric
Encryption • RSA = Asymmetric Encryption • Data Quality = Accuracy, Completeness, Consistency,
Validity, Timeliness