Unified Problem Statement: Census
Income Intelligence Platform (Detailed)
Source Data - Census Source Data
1. Business Context
Build an intelligent census analytics platform that transforms raw citizen demographic and
income data into trusted insights and real-time predictions, enabling organizations to
improve policy planning, financial eligibility assessment, customer segmentation, and data-
driven decision-making.
2. Problem Objective
The platform should:
- Ingest and standardize data from multiple sources
- Ensure high data quality and consistency
- Generate ML-powered income predictions
- Provide real-time APIs and analytical dashboards
- Enable business users to explore, monitor, and act on reliable data insights
3. Data Engineering Layer (Detailed)
3.1 Ingestion Contract
- Read CSV with headers
- Validate schema consistency
- Capture row count and column metadata
- Identify data quality risks (invalid tokens, inconsistent formats, nulls, duplicates)
3.2 Column Standardization
- Convert all column names to lowercase
- Trim whitespace
- Ensure deterministic schema ordering
3.3 Data Cleaning Rules
- Age: remove ERROR_* tokens, strip non-digits, cast to numeric
- Workclass: replace nulls with 'Unknown'
- Trim all categorical columns
- Remove duplicates based on business keys
3.4 Event Time Parsing
- Parse multiple timestamp formats into event_time_std
- Maintain parsing order (ISO, epoch, custom formats)
- Track parse failure counts
3.5 Label Normalization
- Normalize income into two values: <=80k and >80k
- Map all variations to canonical labels
- Validate no drift in label vocabulary
3.6 Numeric Casting
- Clean numeric columns (remove $, %, commas, k suffix)
- Cast to double
- Validate no cast failures
3.7 Data Quality Metrics
- Null counts
- Invalid record counts
- Timestamp parse failures
- Label consistency checks
3.8 Publish Layer
- Write to Delta table: [Link].donation_data_v1
- Use overwrite mode (idempotent)
- Validate schema and row count
4. Machine Learning Layer (Detailed)
4.1 Data Preparation
- Load curated dataset
- Identify categorical and numerical features
- Handle missing values
- Encode categorical variables
- Scale numerical features
4.2 Modeling
- Train baseline models
- Use algorithms such as Logistic Regression, Decision Trees
- Observe underfitting/overfitting
4.3 Manual Tuning
- Adjust hyperparameters manually
- Track performance changes across runs
4.4 Evaluation
- Metrics: Accuracy, Precision, Recall, F1-score, ROC-AUC
- Confusion matrix analysis
- Feature importance evaluation
4.5 Experiment Tracking
- Log parameters, metrics, timestamps
- Store in MLflow
- Compare experiments
4.6 Deployment
- Register best model
- Deploy via model serving endpoint
- Enable real-time inference
5. Application Layer (Detailed)
5.1 Frontend
- ReactJS UI
- Data grid with pagination, filtering, sorting
- Visualization dashboards
5.2 Backend APIs
- Fetch dataset (paginated)
- Fetch record details
- Prediction API
- Experiment tracking API
- Data quality metrics API
5.3 Database
- Store metadata and logs
- Use PostgreSQL/MySQL
5.4 Core Features
- Data exploration UI
- Record-level drill-down
- Data quality dashboard
- Prediction interface
- Experiment comparison
6. Success Criteria
Data Engineering:
- Idempotent pipeline
- Clean schema
- Stable labels
Machine Learning:
- Reproducible models
- Improved performance
- Full experiment tracking
Application:
- Functional UI
- Real-time predictions
- Integrated system