Data Engineering, Data Science, and Machine Learning
Course content
Module 1: Data Ingestion & Integration
• Introduction to Data Ingestion
• Understanding Data Pipelines
• Batch and Real-Time Data Ingestion
• Overview of Data Integration Tools (Apache Nifi, Kafka, Sqoop, etc.)
• Ingest data from relational and non-relational data sources
• Implement real-time data streaming with Kafka
Module 2: Data Storage (HDFS, MinIO, Cassandra)
• Introduction to Distributed Data Storage
• Apache HDFS Overview
• MinIO Overview
• Apache Cassandra Overview
• Store and retrieve large datasets using HDFS, MinIO, and Cassandra
• Comparison of storage architectures
Module 3: Lakehouse Platform (Delta Lake & Apache Hudi)
• Introduction to Data Lakehouse Architecture
• What is a Data Lakehouse?
• Delta Lake Overview
• Apache Hudi Overview
• Build a Lakehouse with Delta Lake
• Data versioning and time travel with Apache Hudi
Module 4: Data Catalog (Apache Atlas / DataHub)
• Introduction to Data Governance and Data Cataloging
• Importance of Data Governance
• Overview of Apache Atlas
• Overview of DataHub
• Catalog and tag data in a distributed system
• Integrate metadata management with Apache Atlas/DataHub
Module 5: Data Preparation
• Introduction to Data Preparation
• Data Cleaning, Transformation, and Feature Engineering
• Handling Missing Data, Outliers, and Duplicates
• Normalization, Scaling, and Encoding
• Data pre-processing using Python (Pandas, NumPy)
• Feature engineering for machine learning models
Module 6: Data Science / Machine Learning
Part A: Python Intermediate
• Key Concepts:
o Functions, Data Structures, Error Handling, Object-Oriented Programming
o NumPy, Pandas, Matplotlib, Seaborn for Data Science
• Intermediate Python exercises focused on data handling and visualization
Part B: Exploratory Data Analysis (EDA)
• Key Concepts:
o Descriptive Statistics, Data Visualization, Trend and Pattern Analysis
o Correlation, Heatmaps, and Multivariate Analysis
• Perform EDA on a real-world dataset using Python (Pandas, Matplotlib, Seaborn)
Part C: Statistical Analysis & Hypothesis Testing
• Key Concepts:
o Probability Distributions, Statistical Inference
o Hypothesis Testing (t-test, chi-square, ANOVA)
o Confidence Intervals
• Apply statistical tests to analyze a dataset and draw conclusions
Part D: Machine Learning - Regression
• Key Concepts:
o Linear Regression, Polynomial Regression, Regularization (Lasso, Ridge)
o Model Evaluation Metrics (RMSE, R²)
• Build and evaluate regression models using Python (Scikit-learn)
Part E: Machine Learning - Classification
• Key Concepts:
o Logistic Regression, Decision Trees, Random Forest, Support Vector Machines
o Model Evaluation Metrics (Accuracy, Precision, Recall, F1-Score, ROC-AUC)
• Develop classification models and assess their performance
Part F: Machine Learning - Clustering & Dimensionality Reduction
• Key Concepts:
o K-means, Hierarchical Clustering
o Dimensionality Reduction Techniques (PCA, t-SNE)
• Apply clustering algorithms and visualize the clusters
• Implement PCA to reduce dataset dimensions
Part G: Deep Learning - Artificial Neural Networks (ANN)
• Key Concepts:
o Introduction to Neural Networks, Activation Functions, Backpropagation
o Multi-Layer Perceptrons (MLP)
• Build and train a basic ANN using TensorFlow/Keras
Part H: Deep Learning - Convolutional Neural Networks (CNN)
• Key Concepts:
o Convolutional Layers, Pooling, Flattening, Fully Connected Layers
o Applications of CNNs in Image Recognition
• Develop and train a CNN for image classification using TensorFlow/Keras
Part I: Deep Learning - Large Language Models (LLM)
• Key Concepts:
o Transformer Architecture, Attention Mechanisms
o Pre-trained Models (BERT, GPT, etc.)
o Fine-Tuning Language Models for NLP Tasks
• Build and fine-tune a language model using Hugging Face or OpenAI GPT