AgriSearch: A Farmer-Friendly

Smart Search Engine

Course : CSD358 – Information Retrieval

Nilaansh Mathur - 2310110537

Manasvi Vedanta - 2210110385
Shivam Doriya - 2310110598

Instructor Name-Dr. Sonia Khetarpaul

GITHUB -[Link]
Shiv Nadar University Dr. Sonia Khetarpaul

Contents
A. Abstract ......................................................................................................................................................................................................... 3
B. Introduction ................................................................................................................................................................................................ 3
1.1 Importance of the Topic ................................................................................................................................................................... 4
1.2 Motivation ................................................................................................................................................................................................ 4
1.3 Contributions.......................................................................................................................................................................................... 4
C. Literature Review / Related Work .................................................................................................................................................... 4
2.1 Deep Learning for Plant Disease Detection........................................................................................................................... 4
2.2 Speech Recognition for Agriculture........................................................................................................................................... 5
2.3 LLM-Based Agricultural Advisory Systems............................................................................................................................ 6
2.4 Novelty of Our Project ....................................................................................................................................................................... 6
D. Objectives ............................................................................................................................................................................................... 7
3.1 Primary Objectives .............................................................................................................................................................................. 7
3.2 Technical Objectives........................................................................................................................................................................... 7
3.3 Usability & Impact Objectives ....................................................................................................................................................... 8
E. Proposed Model ................................................................................................................................................................................... 8
4.1 Overall Architecture ........................................................................................................................................................................... 8
4.2 Vision Engine (MobileNetV2) ........................................................................................................................................................ 9
4.3 Audio Engine (Whisper Speech-to-Text) ............................................................................................................................. 10
4.4 Reasoning Engine (Gemini LLM) .............................................................................................................................................. 11
F. Methodology.............................................................................................................................................................................................. 11
5.1 Dataset and Data Preparation ................................................................................................................................................... 11
5.2 Data Augmentation .......................................................................................................................................................................... 12
5.3 Image Classification Model (MobileNetV2) ........................................................................................................................ 13
5.4 Training Strategy............................................................................................................................................................................... 13
G. Experimentation and Results ...................................................................................................................................................... 15
6.1 Experimental Setup.......................................................................................................................................................................... 15
6.3 Evaluation Metrics ............................................................................................................................................................................ 15
6.4 Model Performance.......................................................................................................................................................................... 16
H. Conclusion ........................................................................................................................................................................................... 17
I. Limitations ................................................................................................................................................................................................. 18
J. Future Improvements ........................................................................................................................................................................... 18
References .......................................................................................................................................................................................................... 18

2
Shiv Nadar University Dr. Sonia Khetarpaul

A. Abstract
The AgriSearch for Farmers is a multimodal system designed to help farmers identify plant
diseases and provide them with relevant and clear agricultural advice. Therefore, the system
accepts input from all three modalities (voice, image, and text) and can be used by farmers with
low levels of literacy or limited technological ability.
The solution includes three components using artificial intelligence (AI):
• A MobileNet V2 Convolutional Neural Network (CNN) to classify plant diseases
• Whisper as the speech-to-text converter
• Google Gemini: This model provides treatment and prevention advice based on the identified
disease.
Our model utilized the PlantDoc dataset, which contains images taken from actual fields versus
lab-captured leaves. Utilizing real world field images is a more challenging but realistic task than
utilizing lab captured images. Our final model reached a 54.39% validation accuracy, which aligns
with existing research regarding PlantDoc and accurately represents the challenges faced when
dealing with actual farm photos.
The system will also provide:
• Confidence in the disease prediction
• Low-confidence warnings
• Symptoms, cause, treatment process, and prevention measures in a structured format for advice.
By providing these features, the system will assist farmers in obtaining quick, easily accessible and
understandable information to aid in their farming decisions. Additionally, this project exemplifies
the potential for integrating computer vision, speech recognition and language models to improve
the functionality of digital agricultural advisory services.

B. Introduction
Agriculture is the backbone of the Indian economy, and identification of disease in plants still
depends upon conventional techniques among a majority of farmers. The timely guidance
required during the spread of a disease is inaccessible to farmers either due to distances,
expenses, or nonavailability of agricultural officers.
Today, even though mobile phones and basic internet access have penetrated most rural areas,
many of the digital tools are far from being user-friendly for farmers. Text-based applications
assume literacy, while image-based applications mostly fail as most models are trained on clean
laboratory datasets. This creates a gap between available technology and practical farmers' needs.

3
Shiv Nadar University Dr. Sonia Khetarpaul

The project presented here tries to fill this gap by developing a Smart Search Engine for Farmers.
It will be capable of understanding images, voice inputs, and text
queries with which it will then provide clear and structured agricultural advice..

1.1 Importance of the Topic

Plant diseases have been causing immense losses to crops every year. Even a small infection can
reduce the yields and directly affect the farmer's income. A tool which can identify the disease as
soon as possible and recommend possible treatments can:
• Prevent crop damage at an early stage
• Reduce dependency on field experts
• Support remote area farmers
• Improve the awareness of symptoms and prevention

1.2 Motivation
The project is motivated by three real challenges:
• Real-world images are noisy: Most research uses PlantVillage, which contains clean leaf
images. But farmers take photos in the field, with shadows, soil, and multiple leaves.
• Literacy and language barriers: Many farmers prefer speaking instead of typing.
• Lack of guidance after diagnosis: Most apps only give a disease name, not treatment
steps.
We wanted to build a system that solves all three problems.

1.3 Contributions

C. Literature Review / Related Work

Our project is related to three lines of research: Deep Learning for Plant Disease Detection, Speech-Enabled
Agricultural Assistant, Large Language Model (LLM) Driven Advisory System.

The below papers influenced our design decisions and demonstrate the gap our project aims to fill

2.1 Deep Learning for Plant Disease Detection

(a) PlantVillage & the “Clean Data Problem”
Many early plant-disease studies used the PlantVillage dataset, which contains clean and centered leaf
images.
Research by Mohaned (2024) and JISEM (2024) reported >97–99% accuracy using MobileNetV2,

4
Shiv Nadar University Dr. Sonia Khetarpaul

EfficientNet, and attention-based CNNs.

However:

• These models performed extremely well only on sanitized lab-style images.

• When tested on field images, accuracy dropped sharply.

This helped us understand that high accuracy on PlantVillage does not reflect real-world
capability, and motivated our shift to PlantDoc.

(b) PlantDoc: Real-World Benchmark

The paper PlantDoc: A Dataset for Visual Plant Disease Detection by Singh et al., 2020 showed that
accuracy drops to 30–60% on real field images due to shadows, soil, clutter, and multiple leaves. Later
studies confirmations.

(c) Why We Chose MobileNetV2

MobileNetV2 (Sandler et al., 2018) is widely recommended for:
• Mobile/low-power devices
• Fast inference
• Low model size (important for rural connectivity)
Several papers showed MobileNetV2 achieving competitive results with much lower memory cost.
This aligns perfectly with our aim: a lightweight model suitable for future mobile deployment.

2.2 Speech Recognition for Agriculture

Whisper and Voice-First Systems
The Whisper model (Radford et al., 2023)
Agriculture-based voice research such as:
• AI-Powered Voice Assistant for Farmers (IndiaAI, 2024)

Study Model(s) Used PlantDoc Accuracy

Singh et al. (2020) ResNet, DenseNet 31–53%

MDPI (2024) Ensemble (InceptionResNetV2 + MobileNetV2) ~60%

PeerJ (2024) Improved MobileNetV2 ~58%

Kumar et al. (2025) Lightweight CNNs 45–62%

5
Shiv Nadar University Dr. Sonia Khetarpaul

• KrishokBondhu Voice Advisory System (2025)

showed that farmers are more likely to use voice than text, particularly in rural
communities with low literacy rates.
These papers inspired us to include Whisper for speech-to-text functionality to
make the system more accessible for farmers that are more comfortable speaking
than typing

2.3 LLM-Based Agricultural Advisory Systems

(a) Rise of Agri-LLMs
Recent studies such as AgroLLM (2025), AgriGPT (2024), and [Link] (2024) explore how
large language models can provide:
• Explanations
• Step-by-step treatments
• Preventive measures
• Context-aware recommendations
These works show that an LLM can act like a digital agricultural assistant when given structured
prompts.

(b) Multi-Modal LLM Literature

Papers like Multi-Modal LLMs in Agriculture (2024) and Agri-LLaVA (2024) highlight the
importance of combining vision + language for disease identification.
However:
• Most systems use large GPUs and server-level infrastructure.
• Very few combine CV + ASR + LLM in a single tool.
This gap directly supports the novelty of our work.

2.4 Novelty of Our Project

The main novelty of our project lies in how different AI components are combined and designed
specifically for practical use by farmers. The key novel contributions are:
• Multi-modal design that supports image, text, and voice inputs in a single system.
• Whisper-based audio recognition that accepts 90+ languages and accents, making the
system usable for farmers who prefer speaking instead of typing.
• Integration of three AI modules — MobileNetV2 (CV), Whisper (ASR), and Gemini (LLM)
— into one unified pipeline, which is uncommon in existing agricultural tools.

6
Shiv Nadar University Dr. Sonia Khetarpaul

• Training on the real-world PlantDoc dataset, not the usual clean PlantVillage dataset,
making the system more suitable for field conditions.
• Structured advisory output (symptoms, causes, treatment, prevention) instead of only
giving a disease name, making the tool function like a digital agricultural officer.
• Low-confidence safety mechanism that warns farmers when predictions are uncertain,
improving reliability.
• Lightweight model (~19MB) designed for future mobile deployment,
supporting areas with limited connectivity.
• Farmer-centric interface, simple enough for rural users and supportive of
low literacy levels

D. Objectives
The main objective of this project is to build a Smart Search Engine for
Farmers that can identify plant diseases and provide clear agricultural advice
through image, voice, and text inputs.

3.1 Primary Objectives

• To develop a multi-modal system that supports image, audio, and text
queries.
• To train a MobileNetV2-based CNN on the PlantDoc dataset for real-world
plant disease detection.
• To integrate Whisper for accurate speech-to-text conversion across multiple
languages and accents used by farmers.
• To use Google Gemini for generating structured agricultural guidance
(symptoms, causes, treatment, prevention).
• To build a simple and farmer-friendly interface using frontend.

3.2 Technical Objectives

• Implement preprocessing and augmentation to improve model generalization.
• Achieve a practical and consistent validation accuracy on PlantDoc.
• Add automated low-confidence warnings to avoid incorrect disease suggestions.
• Ensure smooth communication between CV, ASR, and LLM modules.

7
Shiv Nadar University Dr. Sonia Khetarpaul

3.3 Usability & Impact Objectives

• Reduce dependency on human agricultural experts.
• Provide quick and understandable advice for farmers in rural areas.
• Support voice-based queries for farmers who face literacy challenges.
• Design a tool that can be extended to future mobile-based offline deployment.

E. Proposed Model
Our system follows a modular multi-modal architecture that connects three separate AI models
into one pipeline:
(1) a Vision Engine for leaf images,
(2) an Audio Engine for farmer speech, and
(3) a Reasoning Engine for generating treatment advice.

The architecture is designed so that farmers can start with any input mode—image, voice, or
text—and still receive structured and understandable agricultural support.

4.1 Overall Architecture

The system is built using a hub-and-spoke design, where frontend acts as the central controller and
routes the input to the correct module.

Flow of the System

1. Input Selection (Image / Audio / Text)

2. Preprocessing based on the selected mode
3. AI Engine Processing:
o MobileNetV2 → Image classification
o Whisper → Speech-to-text transcription
o Gemini → Final advisory generation
4. Output Display with confidence score, warnings, and treatment suggestions

8
Shiv Nadar University Dr. Sonia Khetarpaul

Fig: System Architecture of the Multi-Modal Smart Search Engine

4.2 Vision Engine (MobileNetV2)

The image-based disease detection module uses MobileNetV2, chosen for being:

• Lightweight (≈ 19 MB)
• Fast enough for real-time use
• Suitable for rural devices and future mobile deployment

Model Modifications

• Removed original classification head

• Added custom Dense layers for 28 PlantDoc classes
• Used Global Average Pooling + Dropout for regularization

9
Shiv Nadar University Dr. Sonia Khetarpaul

Output
Layer / Component Purpose
Shape

Feature extraction from raw images using

MobileNetV2 Base (Frozen) 7 ✕7✕1280
pre-trained weights (ImageNet).

Dimension reduction (flattens 3D feature

GlobalAveragePooling2D 1280
map to 1D vector).

Dense (512,
512 Non-linear learning of high-level patterns.
Activation='relu')

Prevents overfitting by randomly ignoring

Dropout (0.5) 512
50% of neurons during training.

Dense (256, Feature refinement and further

256
Activation='relu') dimensionality reduction.

Regularization (secondary check against

Dropout (0.3) 256
overfitting).

Dense (28, Output layer; generates probability scores

28
Activation='softmax') for the 28 disease classes.

4.3 Audio Engine (Whisper Speech-to-Text)

The system uses Whisper Base to convert farmer speech into text.

Advantages of Whisper:

• Works well with Hindi and regional Indian accents

• Handles farm noise like wind, birds, etc.
• Requires no manual phoneme training

This makes the system accessible for farmers who cannot type or prefer speaking in their own
language

10
Shiv Nadar University Dr. Sonia Khetarpaul

4.4 Reasoning Engine (Gemini LLM)

Gemini Responsibilities

• Generate a structured advisory report

• Explain causes, symptoms, treatments, and prevention
• Provide organic and chemical options

Fig: System Workflow

F. Methodology
This section explains how the dataset was prepared, how the model was trained, how the audio and
text components were processed, and how all modules were integrated to form the final system .

5.1 Dataset and Data Preparation

Dataset Used

We used the PlantDoc dataset, which contains real field images with natural variations
such as shadows, soil, multiple leaves, and non-uniform lighting. This makes it more
suitable for real farm usage compared to lab datasets like PlantVillage.

Data Split

11
Shiv Nadar University Dr. Sonia Khetarpaul

• Training Split: 80%

• Validation Split: 20%

Metric Value

Total Images 2,569

Number of Classes 28

Average Images per Class ~67

Input Image Size 224✕224

5.2 Data Augmentation

To reduce overfitting and improve generalization on noisy field images, we applied several
augmentation techniques:

• Random horizontal flip

• Random rotation (up to 20°)
• Random zoom
• Random translation
• Normalization using MobileNetV2’s preprocess_input()

12
Shiv Nadar University Dr. Sonia Khetarpaul

5.3 Image Classification Model (MobileNetV2)

We selected MobileNetV2 because it is lightweight (≈19 MB), fast, and suitable for deployment on
low-power devices.

Model Architecture

• Base model: MobileNetV2 (ImageNet weights)

• Custom layers added:
o GlobalAveragePooling2D
o Dense (512, ReLU)
o Dropout (0.5)
o Dense (256, ReLU)
o Dropout (0.3)
o Dense (28, Softmax)

5.4 Training Strategy

Phase 1: Feature Extraction

• Freeze all MobileNetV2 layers

• Train only Dense layers
• Epochs: 30
• Optimizer: Adam (1e-3)

Phase 2: Fine-Tuning

• Unfreeze top layers of MobileNetV2

• Epochs: 10
• Optimizer: Adam (1e-4)
• Callbacks used:
o ModelCheckpoint (auto-save best model)
o EarlyStopping (avoid overfitting)
o ReduceLROnPlateau (improve convergence)

13
Shiv Nadar University Dr. Sonia Khetarpaul

5.5 Audio Processing (Whisper STT)

Whisper converts farmer audio (Hindi, English, regional accents) into text.

Whisper STT Pipeline

1. Audio → Log-Mel Spectrogram

2. Encoder extracts acoustic features
3. Decoder generates text sequence

5.6 Advice Generation (Gemini)

After receiving either:

• Disease prediction from MobileNetV2

• Transcribed text from Whisper
• Manual text query

Gemini generates structured agricultural advice.

Prompt Structure Used

• Overview
• Main causes
• Common symptoms
• Treatment steps
• Prevention tips

Low-Confidence Alert Logic

If model confidence < 60%:

1, if 𝑝 < 0.60
Warning Flag = {
0, otherwise

This triggers Gemini to add a caution message to the advisory.

5.7 End-to-End Integration

Frontend acts as the glue layer.

14
Shiv Nadar University Dr. Sonia Khetarpaul

Workflow

1. User uploads image / records audio / types text

2. Preprocessing
3. MobileNetV2 / Whisper / Text Pipeline
4. Gemini reasoning
5. Output shown with confidence score

G. Experimentation and Results

This section describes how the model was evaluated, the metrics we used, and the outcomes of the
experiments conducted on the PlantDoc dataset.

6.1 Experimental Setup

Hardware

• CPU: Standard laptop CPU (dedicated GPU)

• RAM: 16 GB
• OS: Windows 10/11
• Development Environment: Python 3.12, TensorFlow
• Audio STT: Whisper Base
• LLM: Google Gemini API

This setup reflects a realistic student-level environment and demonstrates that the system is
lightweight and easy to deploy.

6.2 Dataset Split

The dataset was divided as follows:

6.3 Evaluation Metrics

We used standard metrics for multi-class classification:

(a) Accuracy

Correct Predictions
Accuracy =
Total Predictions

15
Shiv Nadar University Dr. Sonia Khetarpaul

(b) Cross-Entropy Loss

𝐿 = −∑ 𝑦𝑖 log⁡(𝑦̂𝑖 )
𝑖=1

Where:

• 𝐶 = 28 classes
• 𝑦𝑖 = true class
• 𝑦̂𝑖 = predicted probability

6.4 Model Performance

Final Model Results on Validation Set, these results are consistent with published benchmarks on the
PlantDoc dataset, which typically range between 45–60% for lightweight models like MobileNetV2.

6.5 Training Curves

FIG: Actual Training History

The model was trained for approximately 33 epochs.

Figure shows the actual Training vs Validation Accuracy and Training vs Validation Loss curves
generated during our experiments.

16
Shiv Nadar University Dr. Sonia Khetarpaul

6.6 Comparison with Existing Work

Model / Study Dataset Accuracy

Singh et al. (2020) PlantDoc 31–53%

MDPI Ensemble (2024) PlantDoc ~60%

PeerJ MobileNetV2 (2024) PlantDoc ~58%

Our Model (2025) PlantDoc 54.39%

SAMPLE OUTPUTS RESULTS

H. Conclusion
• The project successfully builds a multi-modal Smart Search Engine that supports image, voice,
and text inputs.
• MobileNetV2 identifies plant diseases, Whisper handles speech in multiple languages, and
Gemini generates structured agricultural advice.
• The model achieves -54% validation accuracy on PlantDoc, which is realistic for noisy field
datasets.
• The system delivers practical outputs (symptoms, causes, treatment, prevention) instead of just
class labels.

17
Shiv Nadar University Dr. Sonia Khetarpaul

• Overall, the project meets its goals of accessibility, usability, and real-world relevance for
farmers.

I. Limitations
• PlantDoc dataset is small, imbalanced, and noisy, reducing accuracy.
• Similar-looking diseases cause misclassification, especially for lightweight models.
• System depends on internet connectivity for Whisper and Gemini.
• Gemini outputs English responses even when farmers speak Hindi/regional languages.
• Advice is generic, not adapted to local weather, soil, or pest conditions.

J. Future Improvements
• Collect more field images and expand the dataset for higher accuracy.
• Try stronger models (EfficientNet, MobileNetV3, ViT) or ensemble methods.
• Add multilingual output so the advisory matches the language spoken by the farmer.
• Deploy model offline using TensorFlow Lite for rural areas.
• Integrate local agricultural data (weather, soil health, regional pests).
• Use RAG (Retrieval-Augmented Generation) to provide more accurate and grounded advice.
• Build a dedicated mobile or PWA app for easier farmer access.

ACKNOWLEDGEMENT
We were greatly aided by the guidance and encouragement of

Dr. Sonia Khetarpaul, to whom we extend our sincere gratitude, when

working on this project. We are thankful to Shiv Nadar University for
the resources they provided and the supportive environment in which
we were able to complete this project and for being able to present this
information to the user. Our peers and our families have given us
motivation and constant support in return, and we could feel it made
the difference and brought this research to completion.

18
Shiv Nadar University Dr. Sonia Khetarpaul

References
1. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C., 2018, June.
MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4510–
4520).
2. Singh, D., Jain, N., Jain, P. and Kayal, P., 2020, November. PlantDoc: A dataset
for visual plant disease detection. In Proceedings of the 7th ACM IKDD CODS and
25th COMAD (pp. 249–253).
3. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C. and Sutskever, I.,
2023, December. Robust speech recognition via large-scale weak supervision. In
Advances in Neural Information Processing Systems (NeurIPS).
4. Google DeepMind, 2023, December. Gemini: A family of highly capable
multimodal models. In arXiv preprint arXiv:2312.11805.
5. Kumar, A., Sharma, R. and Patel, S., 2025, August. Mobile-friendly deep learning
for plant disease detection: A lightweight CNN benchmark. In arXiv preprint
arXiv:2508.10817.
6. Samuel, D.J., Banerjee, S. and Rao, M., 2025, March. AgroLLM: Connecting
farmers and agricultural practices through large language models. In arXiv
preprint arXiv:2503.04788.
7. Sapkota, R., Gautam, D. and Shakya, S., 2024, May. Multi-modal LLMs in
agriculture: A comprehensive review. In ResearchGate Preprint / IJFMR Special
Issue on AI for Agriculture.
8. Jha, K., Doshi, A., Patel, P. and Shah, M., 2024, July. Plant disease detection
using MobileNetV2 and explainable AI. In Journal of Information Systems
Engineering and Management (JISEM).
9. Li, X., Chen, Y. and Wu, H., 2024, April. A robust ensemble model for plant
disease detection using deep learning architectures. In MDPI Agriculture.
[Link], M., Das, S. and Roy, S., 2023, December. KrishokBondhu: A retrieval-
augmented voice-based agricultural advisory system for Bengali farmers. In
arXiv preprint arXiv:2312.08916.

19