Deep Learning Beyond Accuracy - CSC 591/791 ECE 591

Project Abstract - Comprehensive Version

Beyond Naive Quantization: A Comprehensive Study of

Fairness-Aware Model Compression Across Architectures and
Demographics

PROBLEM STATEMENT

While quantization significantly reduces the computational and memory footprint of deep neural networks (DNNs),
enabling deployment on edge devices, its impact on model fairness across demographic groups remains poorly
understood. Existing research shows contradictory findings: some studies report consistent bias amplification,
while others find inconsistent patterns. Furthermore, most fairness-aware quantization methods incur additional
training costs. This project comprehensively investigates: (1) How do different quantization techniques (PTQ vs.
QAT) affect fairness differently? (2) Which model architectures are more resilient to quantization-induced fairness
degradation? (3) Can we develop lightweight mitigation strategies that preserve both efficiency gains and fairness
without expensive retraining? (4) What is the relationship between quantization bit-width, model efficiency,
accuracy, and fairness metrics?

MOTIVATION

1. Practical Deployment Gap: While quantization is standard in production ML systems (TensorFlow Lite,
PyTorch Mobile), practitioners lack guidance on fairness-preserving compression strategies. Our work provides
actionable insights.

2. Equity at Scale: With billions of edge devices running quantized models for sensitive applications (face
recognition, hiring, healthcare), even small fairness degradations can affect millions.

3. Contradictory Literature: Recent 2024 papers show conflicting results. We systematically resolve this by
testing multiple architectures, quantization methods, and datasets with statistical rigor.

4. Efficiency-Fairness Pareto Frontier: No existing work comprehensively maps the three-way trade-off between
model size, accuracy, and fairness across quantization strategies.

5. Lightweight Mitigation: Existing fairness-aware methods require expensive retraining. We explore post-hoc
strategies.

REPRESENTATIVE FIGURE

Multi-panel visualization showing:

Panel A: Heatmap showing accuracy degradation across 3 models × 3 demographic groups × 4 quantization
levels
Panel B: 3D scatter plot: Model Size Reduction (%), Overall Accuracy (%), Fairness Gap (∆DP)
Panel C: Line plots showing Pareto frontiers for PTQ vs. QAT vs. Fairness-aware quantization
COMPREHENSIVE IMPLEMENTATION PLAN

Phase 1 - Infrastructure & Baseline (Weeks 1-2)

Models (5 diverse architectures): ResNet-50, MobileNetV2, EfficientNet-B0, ViT-Small, SqueezeNet
Datasets: CelebA (~200K, gender/age), UTKFace (~23K, age/gender/race), FairFace (validation)
Baseline Metrics: Overall & per-group accuracy, model size, FLOPs, latency, DP, EO, PE

Phase 2 - Quantization Method Comparison (Weeks 3-4)

Techniques:
1. Post-Training Quantization (PTQ): Static/Dynamic (INT8, INT4), Calibration strategies
2. Quantization-Aware Training (QAT): Fine-tune with simulated quantization (1-5 epochs)
3. Mixed Precision: Sensitivity-based and group-aware bit allocation

Evaluation Matrix: Test 5 models × 4 methods × 3 bit-widths = 60 configurations

Phase 3 - Fairness-Aware Quantization & Mitigation (Weeks 5-6)

Mitigation Strategies:
1. Calibration-Based: Bias-aware calibration with demographic oversampling (no retraining)
2. Lightweight Fine-tuning: Fairness-constrained fine-tuning (1-2 epochs) with DP regularization
3. FairQuanti-Inspired: Identify bias-sensitive neurons, allocate higher precision
4. Hybrid Approaches: FP32 classifier + INT4 backbone

Ablation Studies: Calibration dataset size, demographic balance, layer-wise sensitivity, bit-width sensitivity

Phase 4 - Comprehensive Analysis (Weeks 7-9)

Multi-dimensional Analysis:
• Architecture comparison: Which are most robust to fairness degradation?
• Quantization method comparison: PTQ vs. QAT vs. Mixed Precision cost-benefit
• Demographic analysis: Which groups most vulnerable? Intersectional effects?
• Pareto frontier analysis: 3D efficiency-accuracy-fairness trade-off space

Statistical Rigor: Bootstrap confidence intervals (1000 iterations), paired t-tests, Bonferroni correction, Cohen's d
effect sizes

Phase 5 - Documentation (Week 10)

Comprehensive visualizations, detailed report, practical guidelines, 13-minute presentation
EXPECTED RESULTS AND HYPOTHESES

H1: Architecture-Dependent Fairness Resilience

Expected: Larger models (ResNet-50, ViT) preserve fairness better than lightweight models (MobileNetV2,
SqueezeNet) by 2-3% ∆DP at INT4.
WHY: Redundant capacity in large models absorbs quantization noise. Lightweight models operate near minimal
capacity.
Test: Compare ∆DP across architectures at each quantization level with statistical significance.

H2: QAT Provides Modest Gains at High Cost

Expected: QAT improves fairness by 1-2% over PTQ but requires 5-10× more computation.
WHY: QAT adapts to quantization but doesn't explicitly optimize fairness without fairness-aware losses.
Test: Measure 'fairness improvement per GPU-hour' efficiency metric.

H3: Calibration Dataset Composition Critically Affects Fairness

Expected: Balanced demographic calibration improves fairness by 3-5% vs. random sampling, no training needed.
WHY: Calibration sets activation ranges. Unrepresented groups get suboptimal scaling.
Test: Ablation comparing random vs. stratified demographic calibration.

H4: Layer-wise Sensitivity Varies; Classifiers Most Critical

Expected: Mixed-precision with FP32 classifier + INT4 backbone preserves 80%+ fairness with 70%+
compression.
WHY: Early layers learn general features; late layers learn discriminative features where bias manifests.
Test: Layer freezing experiments and gradient-based sensitivity analysis.

H5: INT8 is the 'Sweet Spot'

Expected: INT8 shows <1% ∆DP while INT4 causes 3-5% degradation. Size reduction: INT8=4×, INT4=8×.
WHY: INT8 has hardware support and sufficient dynamic range. INT4 hits 'quantization cliff'.
Test: Bit-width sweep to identify fairness degradation inflection point.

H6: Contradictory Literature Explained

Expected: We'll reproduce both consistent bias (CNNs) and inconsistent bias (ViTs), showing architecture predicts
behavior.
WHY: Convolution and attention have different numerical sensitivities.
Test: Direct CNN vs. Transformer comparison on same dataset.
LITERATURE SURVEY

Paper 1: "You Never Know: Quantization Induces Inconsistent Biases in Vision-Language

Foundation Models" (Oct 2024, arXiv)
Contribution: Tests ViL models (CLIP, BLIP) across demographics. NO consistent bias amplification -
model-specific patterns.
Key Finding: CLIP-ViT minimal change at INT8, 5% drop on elderly at INT4. BLIP shows opposite.
Relevance: Tests if finding extends to unimodal vision or is multimodal-specific. We hypothesize architecture
matters more.

Paper 2: "FairQuanti: Enhancing Fairness via Neuron Role Contribution" (ACM TOPS 2024)
Contribution: Identifies biased neurons, uses mixed-precision. Bayesian optimization for bit allocation.
Key Finding: 1.03× DP improvement, 1.51× DPR improvement, <7.5% accuracy loss. Works structured &
unstructured data.
Relevance: Provides baseline metrics. We compare naive quantization vs. these benchmarks. Implement
simplified version.

Paper 3: "Uncovering the Hidden Cost of Model Compression" (CVPR 2024, Misra et al.)
Contribution: Shows compression harms underrepresented classes. Top-1 accuracy misleading - per-class
reveals disparities.
Key Finding: INT8 causes 12% drop on rare classes vs. 2% on common. Proposes 'fairness tax' concept.
Relevance: Motivates per-group metrics over overall accuracy. We adopt 'fairness tax' to quantify cost of fairness.

Paper 4: "Model Compression in Biometrics: A Survey" (ScienceDirect Aug 2024)

Contribution: Survey of quantization/pruning/distillation in biometrics. Addresses bias in identity systems.
Key Finding: Biometrics lags in compression-fairness. Calls for fairness-first methods. Notes EU AI Act pressure.
Relevance: Domain context for why fairness matters. Validates face recognition choice.

RESOURCES & TOOLS

Software: PyTorch 2.0+, [Link], TFLite, fairlearn, AIF360, [Link], matplotlib, seaborn, plotly
Datasets: CelebA (~202K), UTKFace (~23K), FairFace (~108K)
Models: [Link] (ResNet, MobileNet, EfficientNet), timm (ViT, SqueezeNet)
Compute: ~15-20 GPU hours total, NCSU HPC accessible
Metrics: Accuracy (overall, per-group, worst-group), Fairness (DP, EO, PE, ∆DP), Efficiency (size, FLOPs,
latency, memory), Statistical (CI, p-values, effect sizes)

TIMELINE

Week Phase Deliverables

1-2 Infrastructure 5 models fine-tuned, datasets prepared, baselines

3-4 Quantization 60 configs tested, PTQ vs QAT compared

5-6 Mitigation 4 techniques implemented, ablations done

7-8 Analysis Statistical analysis, 10+ figures, Pareto frontiers

9-10 Documentation Report, presentation, code documented

EXPECTED CONTRIBUTIONS
1. Empirical Evidence: First comprehensive quantization-fairness comparison across architectures
2. Practical Guidelines: Actionable recommendations for fair compressed model deployment
3. Theoretical Insights: Understanding of architecture-specific resilience to quantization bias
4. Lightweight Mitigations: Cost-effective strategies without expensive retraining
5. Literature Resolution: Reconciliation of contradictory 2024 findings
6. Open Source: Code and analysis scripts for reproducibility

FEASIBILITY & RISK MITIGATION

Executable because: Pre-trained models, PTQ minimal compute, QAT optional, modular design, clear stopping
criteria

Risk Mitigation:
• Compute limited → Reduce to 3 models
• Time limited → Focus PTQ only, skip QAT ablations
• Dataset issues → Use CelebA only
• Weekly checkpoints to identify issues early

This comprehensive project addresses critical gaps in fairness-aware model compression. By systematically
exploring quantization methods, architectures, and mitigations, we provide actionable insights for responsible
deployment of compressed models in high-stakes applications. The multi-dimensional efficiency-accuracy-fairness
trade-off analysis informs both research and practice.