🧠 Methodology

The Elaké – AI & AR Fashion Recommendation and Virtual Try-On System is designed
to provide users with a personalized, interactive, and sustainable fashion experience. The
methodology involves the step-by-step development of an intelligent system that detects
body type and skin tone, recommends suitable outfits based on occasion and color
compatibility, and enables both photo-based and live AR try-on experiences.

The overall development process consists of six major phases:

1. Data Collection and Preparation

2. Data Preprocessing and Feature Extraction
3. Model Development and Training
4. Recommendation Engine Design
5. Augmented Reality (AR) Integration
6. System Integration, Evaluation, and Deployment

1. Data Collection and Preparation

Data collection is the foundation of the Elaké project. The system utilizes both curated
datasets from Lence Studio and publicly available datasets for training and testing.

1.1 Data Sources

 Lence Studio Assets:

A collection of 2D and 3D fashion assets, including:
o Garment PNGs with transparent backgrounds.
o 3D models (GLB/GLTF formats).
o Metadata such as garment category, occasion, color palette, and recommended
body shape.
 Public Datasets:
o DeepFashion and DeepFashion2 – for clothing item segmentation and
attribute extraction.
o ModaNet – for labeled fashion items and body segmentation.
o COCO (Common Objects in Context) – for person keypoints and pose
estimation.

1.2 Data Annotation

Each image and asset in the dataset is annotated with:

 Body Shape Category: Hourglass, Pear, Apple, Rectangle, Triangle, or Round.

 Key Body Measurements: Shoulder width, chest, waist, hip, and height ratio.
 Skin Tone Category: Very Light, Light, Medium, Tan, Brown, or Dark.
 Garment Metadata: Category, color, size, and suitable occasion (Casual, Formal,
Party, etc.).
Annotation tools such as CVAT and Label Studio are used to mark body keypoints,
segmentation masks, and garment anchor points.

2. Data Preprocessing and Feature Extraction

Preprocessing ensures that all images and assets are consistent, clean, and model-ready.

2.1 Image Preprocessing

 Normalization: All images are resized to 224×224 pixels and converted to RGB
color space.
 Data Augmentation: Techniques such as random rotation, flipping, scaling,
brightness variation, and cropping are used to enhance dataset diversity and reduce
overfitting.
 Segmentation: Person segmentation is performed using DeepLabV3 to isolate the
body region from the background.
 Pose Estimation: Key body landmarks (shoulders, hips, knees, etc.) are extracted
using MediaPipe Pose or OpenPose frameworks.

2.2 Feature Extraction

From the segmented and normalized images, important features are extracted:

 Body shape features: Ratios between key body points (shoulder-to-hip, waist-to-hip,
etc.).
 Color features: Mean color values in LAB color space from the detected skin region.
 Texture features: Extracted from garments to assist in matching fabric types and
styles.

These features form the input for the deep learning models.

3. Model Development and Training

The Elaké system uses multiple AI models integrated into a single inference pipeline. Each
model has a distinct task and operates sequentially.

3.1 Body Shape Detection Model

 Architecture:
The model uses MobileNetV3-Large as a lightweight CNN backbone. The final
layers are modified to classify images into six body shape categories.
Additionally, a regression head estimates measurements such as shoulder, waist, and
hip width for fine-grained personalization.
 Loss Function:
 o CrossEntropy Loss for body type classification.
o Smooth L1 Loss for body measurement regression.
o Total Loss = (1 × CE) + (0.5 × L1)
 Training Parameters:
o Optimizer: AdamW
o Learning rate: 0.0001
o Batch size: 32
o Epochs: 50
o Accuracy Target: ≥ 90% on validation data.

3.2 Skin Tone Detection Model

 Goal: Classify user’s skin tone and estimate average LAB color embedding.
 Pipeline:
1. Detect face and exposed skin area (neck, hand) using RetinaFace.
2. Crop the region and convert to LAB color space.
3. Feed into a CNN classifier (MobileNetV2) for tone classification.
 Classes: Very Light, Light, Medium, Tan, Brown, and Dark.
 Evaluation Metric: F1-score (target ≥ 0.88 per class).
 Bias Reduction: Dataset balancing across ethnic and gender diversity to ensure
fairness.

3.3 Model Serving

Both models are exported in ONNX format and integrated into a FastAPI microservice.
Endpoints include:

 /detect-body-shape – returns body type and confidence.

 /detect-skin-tone – returns tone category and LAB values.
 /analyze – combines both responses and sends them to the recommendation module.

4. Recommendation Engine Design

The recommendation engine bridges AI predictions with real-world outfit selection.

4.1 Input Parameters

 Body Shape and Measurements

 Skin Tone (LAB and categorical)
 Selected Occasion (Casual, Formal, Party, Ethnic, etc.)
 Garment Metadata (from Lence Studio assets)
4.2 Algorithm

1. Filtering: Selects outfits matching the selected occasion and compatible body shape.
2. Color Matching:
o Calculates color harmony between garment color and user’s skin tone.
o Uses color difference formula in LAB space (ΔE).
3. Ranking: Combines weighted scores from:
o Fit Compatibility (40%)
o Color Harmony (40%)
o Popularity / User Feedback (20%)
4. Output: Returns top 5–10 outfit recommendations with confidence scores.

4.3 Recommendation Output

The output is provided as JSON and visualized in the web dashboard as cards displaying:

 Outfit image
 Recommended occasion
 Confidence score
 “Try Now” button for AR preview.

5. Augmented Reality (AR) Integration

The AR component enables users to virtually try on the recommended outfits.

5.1 Static Try-On (Photo-Based)

 Input: Uploaded full-body image and selected garment PNG.

 Method:
1. Compute body keypoints using MediaPipe Pose.
2. Align garment anchor points (neck, shoulder, waist) with body keypoints.
3. Warp the garment using Thin Plate Spline (TPS) transformation.
4. Blend the garment with user image using alpha compositing and lighting
correction.
 Output: Rendered image with the user “wearing” the outfit.

5.2 Live Try-On (Camera-Based)

 Technology Used:
o MediaPipe (real-time pose tracking)
o [Link] / ModelViewer (3D rendering)
o WebGL / ARCore / ARKit (mobile AR support)
 Process:

1. Capture video stream from webcam or smartphone camera.

2. Detect body joints in real-time.
 3. Dynamically overlay garment model on detected body using 3D coordinate
mapping.
4. Adjust scale and rotation per frame for accurate tracking.
 Performance Target:

o Minimum 15 frames per second (FPS).

o Latency under 200ms for smooth real-time feedback.

6. System Integration, Evaluation, and Deployment

6.1 System Architecture
Layer Technology Purpose

Frontend [Link] / [Link] + Tailwind CSS User Interface and AR view

Backend [Link] / Express API Gateway and logic control

AI Engine FastAPI + PyTorch Model inference services

Database MongoDB / Firebase User profiles, outfit data, analysis results

Storage AWS S3 / Firebase Storage Stores user images and 3D assets

AR Module [Link] / WebAR / ARCore Renders virtual garments

Deployment Vercel / Render / AWS EC2 Cloud hosting and scalability

6.2 Evaluation Metrics

Component Metric Target Value

Body Type Classification Accuracy ≥ 90%

Skin Tone Detection F1-Score ≥ 88%

Recommendation Relevance User Feedback ≥ 4/5

AR Try-On Alignment IoU (Overlay Accuracy) ≥ 0.65

Live Try-On Performance FPS ≥ 15 FPS

6.3 Deployment and Scalability

 The system is containerized using Docker for reproducibility.

 Each microservice (AI, Recommendation, AR, Frontend) is deployed separately.
 APIs are secured with JWT Authentication.
 Future scalability can be achieved using cloud GPUs or serverless inference
endpoints.
7. Workflow Summary
Step Process Description

1️⃣ User uploads a photo or opens live camera stream

2️⃣ FastAPI backend analyzes image (body type + skin tone)

3️⃣ Recommendation engine selects best-fitting outfits

4️⃣ Outfits displayed with confidence and color match scores

5️⃣ User selects outfit → AR overlay for static or live try-on

6️⃣ User feedback stored to improve future recommendations

8. Expected Outcomes
 Personalized clothing suggestions for both men and women.
 Real-time AR try-on experience powered by AI-based body analysis.
 Improved user confidence in online shopping.
 Reduced product returns due to accurate fit visualization.
 Contribution to sustainable fashion technology by minimizing waste.