ShopSense: AI-Powered Visual Search for

E-Commerce in Pakistan
Prepared By:

• Hasnain Morani (MTN-SP_000073)

• Maria Zubair (MTN-FL_000084)
• Muhammad Majid (MTN-FL_000093)

Supervised By: Nosheen Fatima

Institution: National University of Modern Languages (NUML), Multan Campus
Submission Date: April 2026

Table of Contents
• Introduction
• Literature Review
• Problem Statement
• Objectives
• Proposed Solution and Methodology
• System Workflow
• Tools and Technologies
• Innovation and Contribution
• Project Scope and Limitations
• Social and Professional Benefits
• Expected Outcomes
• Project Timeline
• References

• Introduction
Pakistan's digital economy is growing steadily, and this growth is clearly visible in online
shopping activity. According to PCMI, Pakistan's e-commerce sector reached approximately
USD
7.7 billion in 2024 and is expected to continue growing at a compound annual growth rate of
17% through 2027. At the same time, the country’s cellular subscriptions crossed 200 million by
the end of 2025, showing how strongly mobile devices shape digital behavior.
This expansion is closely linked with social media usage. DataReportal reported that Pakistan
had nearly 80 million social media users in 2026. Among these platforms, TikTok had 79.9
million
users aged 18 and above, while Instagram reached 22.4 million users. These platforms have
made product discovery increasingly visual, as people often notice products in short videos,
screenshots, posts, and shared images before they ever know the product name.
However, this creates a practical search problem. A user may see a handbag in an Instagram
reel or a pair of shoes in a TikTok clip, but local e-commerce platforms such as Daraz do not
provide a
simple image-based search option. As a result, users must guess keywords, search manually, and
browse many unrelated results. This process becomes even more difficult when users want to
compare prices across multiple local sellers.

ShopSense is proposed as a simple and realistic solution to this problem. It is an Android mobile
application that allows users to upload a product image or enter a text query in English or
Roman Urdu. The system then returns visually similar product listings along with side-by-side
prices from Daraz and one selected Shopify-based Pakistani store. To support this process, the
project uses CLIP, a pre-trained model that converts both images and text into comparable
numeric
representations so that similar products can be matched efficiently.

This project is designed as a minimum viable product (MVP) for a final-year undergraduate
project. Its goal is to build a working prototype that addresses a real local problem in a
practical way. It is not intended to be a large commercial system or a high-scale production
platform.

• Literature Review
• Existing Commercial Tools
Google Lens is one of the most widely known visual search tools. It performs general image
recognition effectively and can identify many everyday objects and products. However, it does
not align well with Pakistani e-commerce inventories. In many cases, it directs users to
international websites or sellers, which limits its usefulness for buyers who specifically want
results from Daraz or local online stores.
PriceOye is a known Pakistani price comparison platform, but its search process is based on
text input and manual category browsing. It does not allow users to search through an image.
This
makes it less useful when a shopper discovers a product visually and does not know the
correct product name or keywords.
Daraz Internal Search mainly depends on keyword-based queries within its own platform.
Users cannot upload an image to search for visually similar products. In addition, its search is
limited to Daraz’s own listings, which means it does not support cross-platform comparison.

• Academic Foundations
Radford et al. (2021) introduced CLIP, a vision-language model trained on a very large set of
image-text pairs. CLIP maps images and text into a shared embedding space, where similarity
can be measured using cosine similarity. This is useful for product search because the system
can
compare an uploaded image with product images, or compare text with product images,
without training a new model from scratch. Its zero-shot capability makes it especially suitable
for a student project with limited time and computing resources.
Malkov and Yashunin (2020) proposed HNSW, a method for fast approximate nearest neighbor
search in high-dimensional data. This idea is useful when product embeddings must be searched
quickly. In practical terms, HNSW can be used through the pgvector extension in PostgreSQL to
search thousands of stored vectors efficiently without setting up a separate vector database
service.

Tóth et al. (2024) showed that CLIP-based approaches can outperform alternatives such as
DINO in multimodal product matching tasks in fashion e-commerce. Their findings support
the use of CLIP for visual product retrieval, especially in settings where both images and text
descriptions are relevant.

Hendriksen et al. (2022) extended CLIP for category-to-image retrieval in e-commerce.

Their work demonstrated that vision-language pretraining can support practical retrieval tasks
in shopping environments, strengthening the case for using CLIP in product discovery
systems.
Butt et al. (2025) highlighted that Roman Urdu remains a low-resource language in information
retrieval research. Their work supports the idea that even simple Roman Urdu handling can
provide meaningful local value. For a student-level prototype, a dictionary-based Roman Urdu
mapping is a realistic way to address this issue without building a full language processing
system.

• Identified Gap
Existing tools and prior research show that important pieces of the problem have been
addressed separately, but not together in a Pakistan-focused mobile application. There is still
no simple local tool that combines image-based product search, cross-platform price
comparison from Pakistani e-commerce sources, and Roman Urdu query support in one
system. ShopSense is proposed to
address this exact gap with a practical and limited implementation.

• Problem Statement
• Visual-to-Text Mismatch
Many Pakistani shoppers discover products visually through social media posts, short videos,
screenshots, and forwarded images. However, local e-commerce platforms usually require
text-based search. This creates a mismatch between how users find products and how
platforms expect them to search.

• Scattered Pricing
The same or similar product may appear at different prices across Daraz and independent
Shopify-based stores. Users must check each source separately, which is slow and
inconvenient. In most cases, they stop after viewing only a few results and may miss a better
option.

• Language Gap
A large amount of informal online communication in Pakistan takes place in Roman Urdu.
Yet e-commerce search systems usually expect English keywords. Users who search with
terms such as “laal jora” or “kala bag” may not receive useful results, even when relevant
products exist.
Together, these problems make online shopping less efficient and less accessible. They increase
search time, reduce confidence in purchase decisions, and create frustration for mobile-first
Pakistani users who often begin product discovery through visual content.

• Objectives
 • Develop an Android mobile application that supports product search through both
image upload and text input.
• Implement visual similarity matching using the pre-trained CLIP ViT-B/32 model and
target a Precision@5 score of at least 70% on the test catalog.
• Aggregate product listings and prices from Daraz and one Shopify-based Pakistani store
and present them side by side.
• Provide basic Roman Urdu query support using a manually curated dictionary of
200+ common shopping terms.
• Achieve an average end-to-end search response time of under 5 seconds for a
catalog of 5,000 to 10,000 products.

• Proposed Solution and Methodology

• Solution Overview
ShopSense is a mobile application supported by a Python-based backend. A user uploads an
image or enters a text query, and the system searches for similar products in a local database.
After finding the closest matches, it retrieves current prices from selected sources and shows
the results in a ranked and easy-to-read form.
The proposed system focuses on practical functionality rather than complex infrastructure. Its
main purpose is to demonstrate that a working prototype can solve the image-to-product search
problem in a local Pakistani context.

• Step-by-Step System Flow

Step 1 — User Input
The user either uploads or captures a product image, or types a text query. If the text is
written in Roman Urdu, the system first converts common Roman Urdu terms into English
using a built-in dictionary.

Step 2 — Embedding Generation

The system uses the CLIP ViT-B/32 model to convert the query into a numeric vector. For an

image query, the image encoder is used. For a text query, the text encoder is used. These vectors
represent the meaning and appearance of the query in a form that can be compared
mathematically.

Step 3 — Similarity Search

The generated query vector is compared with precomputed product vectors stored in
PostgreSQL using the pgvector extension. HNSW indexing helps retrieve the top matching
products quickly, even when the catalog contains thousands of items.

Step 4 — Price Lookup

After candidate products are identified, the backend checks pricing from Daraz and the selected
Shopify-based store. Daraz data is collected through web scraping of public listings, while
Shopify data is accessed through its public API where available. A local cache is refreshed
every 6 to 12
hours to reduce repeated requests and improve response time.

Step 5 — Result Display

The matched products are ranked based on similarity and shown to the user with product
images, titles, prices, source labels, and purchase links. Users can also sort or filter the results,
for example by choosing lower-priced options.

• Architecture (Three Layers)

• Frontend: Flutter Android app
• Provides the user interface for image upload, camera capture, text entry, and
result display.
• Supports simple interaction flow suitable for an MVP.

• Backend: FastAPI on AWS EC2

• Handles API requests from the mobile app.

• Performs CLIP inference, similarity search, Roman Urdu term mapping, and
price aggregation.
• Data layer: PostgreSQL + pgvector, AWS S3 for images, Redis cache

• PostgreSQL stores product details and embeddings.

• AWS S3 stores product images used in the catalog.
• Redis stores recent price lookup results to reduce repeated processing.

• Data Collection
The initial product catalog will be built using public product listings from Daraz and one
selected Shopify-based Pakistani store. The collection process will focus on categories such as
electronics, fashion, and home items, as these are common in online shopping and visually
distinguishable.
For each product, the system will store:

• Title
• Image URL
• Price
• Source platform
• Category

After collection, CLIP embeddings will be generated for the product images and stored in
the database. The target catalog size is 5,000 to 10,000 products across 5 to 8 categories.
Refresh scripts will run every 6 to 12 hours to keep the local catalog reasonably updated.

• System Workflow
The following diagram summarizes the complete workflow of the proposed system.

User Opens App

│
▼
┌─────────────────┐
│ Upload Image │──── OR ────┐
│ or Capture Photo│ │
└────────┬────────┘ │
│ ▼
│ ┌──────────────┐
│ │ Type Text │
│ │ Query │
│ └──────┬───────┘
│ │
│ ┌──────▼───────┐

│ → Translate │
│ │ Roman Urdu? │
│
│ └──────┬───────┘
│ │
▼ ▼
┌─────────────────────────────────┐

(Image or Text → 512-d vector)│

│ CLIP Embedding Generation │
│
└────────────┬────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Similarity Search (pgvector) │
│ Top-N matched products │
└────────────┬────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Price Aggregation │
│ Daraz + Shopify Store │
└────────────┬────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Ranked Results Display │
│ Images, Titles, Prices, │
│ Purchase Links, Filters │
└─────────────────────────────────┘

• Tools and Technologies

Component Technology Why This Choice
Mobile App Flutter (Android) Single codebase, fast prototyping, good for MVP
Backend API FastAPI (Python) Lightweight, fast, native Python ML integration
Pre-trained, no custom training needed, proven for
AI Model OpenAI CLIP (ViT-B/32) product search
Vector No separate infrastructure, HNSW indexing for fast
PostgreSQL + pgvector similarity search
Database
Cache Redis Reduces repeated price lookups, simple to set up
Reliable, cheap storage within Free Tier (5 GB),
Image Storage AWS S3 easy integration
Backend AWS EC2 ([Link] or 750 free hours/month under AWS Free Tier,
Hosting [Link]) sufficient for MVP
Python (BeautifulSoup,
Data Collection Scrapy) Standard tools for web scraping
Scheduled
Cron jobs (Linux) Simple, reliable, no extra infrastructure
Tasks

Component Technology Why This Choice

Version Control Git + GitHub Industry standard for team collaboration

AWS Usage Strategy

• EC2 will host the FastAPI backend and CLIP inference service. The 750 free hours
per month are sufficient for development and testing during the project period.
• S3 will store product images collected for the catalog. The free tier of 5 GB storage
and 20,000 GET requests per month is adequate for the planned MVP dataset.
• Lambda is optional and will only be considered for lightweight scheduled tasks if cron
jobs are not sufficient during implementation.
• Estimated AWS cost is expected to remain between $0 and $15 per month under free-
tier-oriented usage.
• No complex cloud architecture is required, as the project is intentionally limited
to a practical student-level setup.

• Innovation and Contribution

ShopSense does not claim to introduce a new AI model or a new search algorithm. Its
contribution lies in combining existing technologies in a practical way to solve a specific local
problem. The
focus is on useful integration rather than technical novelty for its own sake.

• Image-to-Product Search for Pakistani PlatformsShopSense enables users to start a

product search with an image and receive results linked to Pakistani e-commerce
sources, especially Daraz and a local Shopify-based store.
• Cross-Platform Price ComparisonThe system combines product matching with side-
by-side price comparison, helping users make better choices without switching
between
multiple apps or websites.
• Basic Roman Urdu SupportA small but purposeful Roman Urdu dictionary improves
accessibility for users who search with common local terms such as “neela kurta” or
“safed shoes.”
• Reproducible Student Architecture
The project uses open-source tools and a simple AWS setup. This makes the
implementation easier to understand, document, and reproduce in future student projects.
• Project Scope and Limitations
• In Scope
• Development of a functional Android MVP with image-based and text-based search
• Use of the pre-trained CLIP model without custom model training
• Product catalog of 5,000 to 10,000 items
• Price comparison from two sources: Daraz and one Shopify-based Pakistani store
• Roman Urdu dictionary-based query support with 200+ terms
• Hosting and storage through AWS EC2 and AWS S3

• Out of Scope
• iOS deployment
• Real-time price history and historical price graphs
• Advanced Roman Urdu natural language processing
• User accounts, wishlists, shopping carts, or in-app purchasing
• Scaling to more than 2 to 3 external sources
• Load testing beyond 100 concurrent users
• Payment gateway integration

• Known Limitations
• Product prices may be delayed by 6 to 12 hours depending on the scraping and refresh
schedule.
• CLIP may perform less effectively on niche local products or culturally specific designs
not strongly represented in its training data.
• The Roman Urdu dictionary can handle common terms but not all misspellings,
spelling variations, or slang.
• Web scraping may fail temporarily if the structure of source pages changes significantly.
• A small EC2 instance has limited computing power, so inference speed may be slower
than GPU-based systems. This is acceptable for an MVP.

• Social and Professional Benefits

For Consumers
ShopSense can reduce the effort required to search for products discovered visually on social
media. Instead of guessing product names and browsing manually, users can move from image
to relevant results more directly. Roman Urdu support also makes the search process easier for
users who are less comfortable with English keywords.

For the Local E-Commerce Ecosystem

The project encourages better price visibility by showing options from more than one local
source. This can support more informed buying decisions and promote fairer competition
among sellers. It also demonstrates a practical model for cross-platform product discovery in
Pakistan.
For Academia
ShopSense offers a documented example of how a modern vision-language model can be
applied to a real local problem within realistic student constraints. It also contributes a small
Roman Urdu shopping vocabulary resource that may be useful for future research in local
information retrieval.

• Expected Outcomes
• A working Android application that supports both image-based and text-based
product search.
• A CLIP-based matching system that achieves Precision@5 of at least 70% on the
test catalog.
• A price comparison feature using Daraz and at least one Shopify-based Pakistani store.
• A Roman Urdu dictionary containing 200+ shopping terms integrated into the
search pipeline.
• An average search response time of under 5 seconds for a catalog of 5,000 to 10,000
items.
• A final project report covering system design, implementation, testing, and lessons
learned.

• Project Timeline
Phase Duration Key Activities
Literature review, requirements gathering, system
Phase 1: Research & Design Weeks 1–3 architecture, UI wireframes
Build scraping scripts, collect product catalog, generate
Phase 2: Data Collection Weeks 3–5 CLIP embeddings
Phase 3: Backend Set up FastAPI on EC2, CLIP inference endpoint,
Development Weeks 4–8 pgvector integration, price aggregation
Phase 4: Mobile App Weeks 6– Flutter UI, API integration, search flow, result display
Development 10 with price cards
Phase 5: Roman Urdu Weeks 8– Dictionary curation, integration with search pipeline
Module 10

Phase Duration Key Activities

Phase 6: Testing & Weeks 10– Functional testing, Precision@5 evaluation, response
Evaluation 13 time benchmarking
Phase 7: Documentation & Weeks 13– Final report writing, demo preparation, presentation
Submission 15

• References
• A. Radford, J. W. Kim, C. Hallacy et al., "Learning Transferable Visual Models from
Natural Language Supervision," in Proc. 38th Int. Conf. Machine Learning (ICML), PMLR,
vol. 139, pp. 8748–8763, 2021.
• Y. A. Malkov and D. A. Yashunin, "Efficient and Robust Approximate Nearest
Neighbor Search Using Hierarchical Navigable Small World Graphs," IEEE Trans.
Pattern Anal. Mach. Intell., vol. 42, no. 4, pp. 824–836, 2020.
• S. Tóth, S. Wilson, A. Tsoukara, E. Moreu et al., "End-to-End Multi-Modal Product
Matching in Fashion E-Commerce," arXiv preprint arXiv:2403.11593, 2024.
• M. Hendriksen, M. Bleeker, S. Vakulenko et al., "Extending CLIP for Category-to-
Image Retrieval in E-Commerce," in Proc. European Conf. Information Retrieval
(ECIR), Springer, 2022.
• U. Butt, A. Zahid, and M. Usman, "Roman Urdu as a Low-Resource Language: Building
the First IR Dataset and Benchmarks," in Proc. EMNLP Workshop on Low-Resource NLP,
ACL Anthology, 2025.
• J. Johnson, M. Douze, and H. Jégou, "Billion-Scale Similarity Search with GPUs,"
IEEE Trans. Big Data, vol. 7, no. 3, pp. 535–547, 2021.
• A. Dagan, I. Guy, and S. Novgorodov, "Shop by Image: Characterizing Visual Search
in E-Commerce," Information Retrieval Journal, Springer, 2023.

• J. Li, Z. Dou, Y. Zhu, X. Zuo, and J.-R. Wen, "Deep Cross-Platform Product Matching
in E-Commerce," Information Retrieval Journal, vol. 23, no. 2, pp. 136–158, 2020.
• DataReportal, "Digital 2026: Pakistan," 2026.
Available: [Link]
pakistan
• Payments & Commerce Market Intelligence (PCMI), "E-commerce Projections for
Pakistan (2024–2027)," 2024. Available: [Link]
ecommerce-market- HYPERLINK "[Link]
market-data/"data/
• ResearchAndMarkets, "Pakistan B2C Ecommerce Market Size & Forecast," 2025.
Available: [Link]
• PostgreSQL Global Development Group, "pgvector: Open-Source Vector Similarity
Search for PostgreSQL," 2025. Available: [Link]