A Project Report

On

AI-Based Food Recognition and

Nutrition Analysis System

Submitted in partial fulfillment of the

requirement for the award of the
degree of

BACHELOR OF TECHNOLOGY

DEGREE
Session 2025-26
in
Computer Science and Engineering
By
Naushin Fatima (2241233)
Sakshi Singh
Jyoti Kumari
Farman Sheikh
Under the guidance of Mrs. Surabhi Purwar
SCHOOL OF COMPUTER SCIENCE AND ENGINEERING
IILM UNIVERSITY, GREATER
May, 2026
 SCHOOL OF COMPUTER SCI. AND ENGG.
IILM UNIVERSITY, GREATER NOIDA

CANDIDATE’S DECLARATION

We hereby certify that the work which is being presented in the project, entitled “"AI Based
Food Recognition and Nutrition Analysis System”, in partial fulfillment of the requirements
degree of Bachelor of Technology in the School of Computing Science and Engineering of
IILM University, Greater Noida is an original work carried out from Aug 2025 to May 2026,
under the supervision of Mrs. Surabhi Purwar, School of Computing Science and
Engineering, IILM University, Greater Noida.

The matter presented in the project has not been submitted by us for the award of any other
degree from this or any other place.
Naushin Fatima (CS-2241233), Sakshi Singh, Jyoti Kumari, Farman

This is to certify that the above statement made by the candidates is correct to the best of my
knowledge.

Signature of Guide
Mrs. Surabhi Purwar
Assistant Professor
School of Computer Science and Engineering
IILM University, Gr Noida.

ii
 CERTIFICATE

This is to certify that the Project Report entitled “Hybrid AI-Based Food Recognition
and Nutrition Analysis System Using Vision Language Models and USDA
Nutritional Retrieval” which is submitted by by Naushin Fatima (CS-2241233),
Sakshi Singh, Jyoti Kumari and Farman Sheikh in partial fulfillment of the requirement
for the degree of Bachelor of Technology in the School of Computer Science and
Engineering of IILM University, Greater Noida is a record of the candidate's own work
carried out by them under my supervision. The matter embodied in this project is original
and has not been submitted for the award of any other degree.

Signature of HoD Signature of Guide

Dr. Anil Saroliya Mrs. Surabhi Purwar
Professor Assistant Professor
School of Comp. Sci. & Engg School of Comp. Sci. & Engg
IILM University, Gr Noida IILM University, Gr Noida

iii
 ACKNOWLEDGEMENT

It gives us a great sense of pleasure to present the report of the Project undertaken during B.
Tech. Final Year. We owe a special debt of gratitude to Ass. Mrs. Surabhi Purwar, School
of Computer Science & Engineering, IILM University, Greater Noida, India, for his constant
support and guidance throughout the course of our work. His sincerity, thoroughness, and
perseverance have been a constant source of inspiration for us. It is only his cognizant efforts
that our endeavors have seen the light of day.

We also take the opportunity to acknowledge the contribution of Professor Dr. Anil Saroliya,
HoD, School of Computer Science & Engineering, IILM University, Greater Noida, for his
full support and assistance during the development of the project.

We also do not like to miss the opportunity to acknowledge the contribution of all faculty
members of the department for their kind assistance and cooperation during the development
of our project. Last but not least, we acknowledge our friends for their contribution to the
completion of the project.

Naushin Fatima (CS-2241233)

Sakshi Singh

Jyoti Kumari

Farman Sheikh

iv
 ABSTRACT
The increasing demand for automated healthcare solutions has created a need for intelligent dietary
monitoring systems. Traditional nutrition applications require manual food logging and often fail to
provide accurate nutritional analysis for complex meals.

This project proposes a AI Based Food Recognition and Nutrition Analysis System.

The system uses Qwen2-VL-7B for food classification, ingredient extraction, and meal recognition
from uploaded food images. The identified food items are matched with the USDA Food Data
Central database to retrieve nutritional information such as calories, carbohydrates, protein, fats,
fiber, sugar, and sodium.

A custom re-ranking algorithm improves matching accuracy by considering ingredient similarity,

meal relevance, food freshness, and dietary preferences. The system also includes a health scoring
engine that evaluates food quality based on WHO and ICMR nutritional guidelines. Additionally,
Google Gemini is used for benchmarking purposes to compare performance.

The final system generates detailed nutritional reports, health scores, recommendations, and
comparison outputs through an interactive dashboard.

v
 LIST OF TABLES

Table No. Table Name

Table 1 Literature Comparison
Table 2 Dataset Information
Table 3 Nutritional Parameters
Table 4 Model Comparison

vi
 LIST OF FIGURES

Figure No. Figure Name

Figure 1 System Architecture

Figure 2 Input Interface Figure

Figure 3 Food Classification Output

Figure 4 Model Comparison Output

vii
 LIST OF ABBREVIATIONS

Abbreviation Full Form

AI Artificial Intelligence
VLM Vision Language Model
USDA United States Department of Agriculture
API Application Programming Interface
GUI Graphical User Interface
WHO World Health Organization
ICMR Indian Council of Medical Research

viii
 TABLE OF CONTENTS

Page
DECLARATION ii
CERTIFICATE iii
ACKNOWLEDGEMENTS iv
ABSTRACT v

List of Tables vi
List of Figures vii
List of Abbreviations viii
CHAPTER 1: INTRODUCTION
1.1 Introduction 10
1.2 Problem statement 10
1.3 Scope of research 11
1.4 Research hypothesis 14
1.5 Objectives 15
CHAPTER 2: LITERATURE REVIEW 17
2.1 Background 17
2.2 Summary of literature review and research gap 20
CHAPTER 3: METHODOLOGY 23
3.1 Materials 23
3.2 Summary of methodology 27
CHAPTER 4: RESULTS AND DISCUSSION 30
4.1 Summary of results and discussion 30
CHAPTER 5: CONCLUSION AND RECOMMENDATIONS 33
5.1 Conclusion 33
5.2 Recommendations 34
REFERENCES 37
APPENDICES 38

9
 CHAPTER 1

INTRODUCTION

1.1 Introduction

With the rapid growth of health awareness and increasing cases of obesity, diabetes,
cardiovascular diseases, and lifestyle-related disorders, people are becoming more conscious
about maintaining proper nutrition. Monitoring daily food intake is an essential part of
maintaining a healthy lifestyle, as nutrients such as calories, proteins, carbohydrates, fats,
fiber, sugar, and sodium directly impact human health. However, tracking nutritional intake
manually remains a difficult and time-consuming process for most individuals.

Traditional nutrition tracking applications such as MyFitnessPal and CalorieMama

require users to manually search for food items and enter meal details. This process often
becomes repetitive and inconvenient, leading to inaccurate tracking or complete
discontinuation of usage. In addition, traditional food recognition systems based on CNNs
and object detection models often struggle with mixed meals, varying food presentations,
ingredient identification, and accurate nutritional mapping.

Recent advancements in Artificial Intelligence, particularly Vision Language Models

(VLMs), have created new opportunities for automating nutrition analysis. These models
are capable of understanding food images more effectively by identifying meal types, visible
ingredients, and contextual food information. This helps overcome many limitations of
traditional image classification systems.

Key Challenges in Existing Systems

Manual food logging requires significant user effort

Difficulty in identifying mixed meals
Limited food category recognition
Inaccurate nutrition estimation
Lack of personalized health recommendations
Poor handling of regional food variations

The proposed project, “Hybrid AI-Based Food Recognition and Nutrition Analysis System Using Vision
Language Models and USDA Nutritional Retrieval,” aims to automate the entire nutrition tracking
process. The system uses Qwen2-VL-7B for food image classification, USDA FoodData Central for
retrieving nutritional values, and a custom health scoring engine based on WHO and ICMR
guidelines. The project also uses Google Gemini as a benchmark model for
performance comparison.

1.2 Problem Statement

10
In today’s fast-paced lifestyle, maintaining proper nutrition has become increasingly difficult
for individuals due to unhealthy eating habits, irregular meal schedules, and lack of
awareness about nutritional intake. Many people consume food without understanding its
nutritional composition, which can lead to health problems such as obesity, diabetes, high
blood pressure, heart diseases, and nutritional deficiencies.

Existing nutrition tracking applications require users to manually enter food details, search
for meal names, and estimate portion sizes. This process is often time-consuming, repetitive,
and prone to human error. Users may forget to log meals regularly, enter incorrect food items,
or provide inaccurate serving sizes, which reduces the reliability of such applications.

Traditional AI-based food recognition systems using CNNs and object detection models also
face several limitations. These systems often struggle in real-world scenarios because:

The same food can appear differently based on preparation style and presentation
Mixed meals contain multiple ingredients that are difficult to identify accurately
Regional dishes are often not included in predefined datasets
Nutritional mapping after food detection remains inaccurate
Most systems do not provide health recommendations or dietary insights

For example, a food item such as biryani may have different ingredients, preparation
methods, and nutritional values depending on the region or cooking style. Traditional systems
may incorrectly classify such meals or provide inaccurate nutritional information.

Another major issue is the lack of integration between food recognition systems and reliable
nutritional databases. Many applications provide only basic calorie estimates without
detailed information about protein, carbohydrates, fats, sugar, sodium, and fiber content.

Therefore, there is a need for an intelligent automated system that can accurately recognize food
from images, extract ingredients, retrieve reliable nutritional data, and provide health
recommendations with minimal manual effort from users.

The proposed project addresses these challenges by developing a hybrid AI-based nutrition
analysis system that combines vision language models, nutritional databases, health scoring
mechanisms, and comparative AI analysis to improve the efficiency and accuracy of nutrition
tracking.

1.3 Scope of Research

The scope of this research focuses on developing an intelligent food recognition and nutrition
analysis system that automates the process of identifying food items and estimating their
nutritional composition using artificial intelligence. The research explores the integration of
vision language models, nutritional databases, and health analytics to create a complete
11
nutrition tracking solution.

The proposed system primarily focuses on analyzing food images uploaded by users and
generating detailed nutritional reports. It identifies food items, extracts visible ingredients,
retrieves nutritional values, and provides health recommendations based on dietary
guidelines. The research emphasizes improving automation in nutrition tracking by reducing
manual user effort.

The scope of this research includes the following areas:

1. Food Image Recognition

Food Image Recognition The system uses Qwen2-VL as the primary vision-language model to
analyze food images and identify important attributes such as meal name, visible ingredients, food
category, vegetarian or non-vegetarian classification, and confidence scores. This enables the
system to understand both simple and complex meals more effectively than traditional image
classification models.

2. Nutritional Data Retrieval

After identifying the food item, the system integrates USDA FoodData Central to retrieve accurate
nutritional information. The retrieved data includes important nutritional parameters such as
calories, protein, carbohydrates, fats, fiber, sugar, sodium, and saturated fats. This allows the system
to generate structured nutritional reports for users.

3. Health Analysis

The project includes a health scoring engine based on WHO and ICMR dietary guidelines to
evaluate the overall quality of food. The system generates health scores, identifies potential health
benefits and concerns, and provides dietary recommendations to help users make healthier food
choices.

4. Comparative AI Analysis

The research also includes a comparative analysis between the proposed hybrid model and Google
Gemini. This comparison evaluates food recognition accuracy, response quality, and nutritional
estimation performance to validate the effectiveness of the proposed approach.

5. Real-World Applications

The proposed system can be applied in multiple domains including healthcare monitoring systems,
fitness applications, diet planning platforms, hospitals, nutrition clinics, corporate wellness
programs, and smart healthcare environments. These applications demonstrate the practical
usability of the research.

Limitations of Current Research

The current research is limited to image-based food analysis and does not perform precise portion
estimation. The system also depends on external APIs for certain functionalities and may face
difficulties in detecting hidden ingredients in highly complex meals.

12
1.4 Research Hypothesis

The primary hypothesis of this research is that a hybrid artificial intelligence system
integrating vision language models, nutritional databases, and health analytics can provide
more accurate, efficient, and automated nutrition analysis compared to traditional manual
and object detection-based nutrition tracking systems.

Traditional nutrition tracking applications rely heavily on manual food logging, which
often leads to inaccurate nutritional tracking and poor user experience. Similarly,
conventional food recognition systems based on basic image classification and object
detection models face challenges in identifying mixed meals, extracting ingredients, and
generating reliable nutritional insights.

This research hypothesizes that combining Qwen2-VL for contextual food recognition,
USDA FoodData Central for nutritional retrieval, health scoring mechanisms based on
WHO and ICMR dietary guidelines, and Google Gemini benchmarking for comparative
validation can significantly improve food recognition accuracy, nutritional reliability, and
overall automation in dietary analysis systems.

The study further assumes that such an integrated framework can reduce manual effort,
improve personalized health recommendations, and offer practical applications in
healthcare, fitness, and smart nutrition monitoring systems.

1.5 Objectives

The primary objective of this project is to develop an intelligent and automated nutrition
analysis system that can accurately identify food items from images, retrieve nutritional
information, and provide health recommendations with minimal manual effort from users.

Traditional nutrition tracking systems often require manual food logging and fail to provide
accurate analysis for complex meals. The proposed project aims to overcome these limitations
by integrating artificial intelligence, nutritional databases, and health analytics into a single
automated platform.
The specific objectives of this research are explained below:

Primary Objective

13
The project aims to develop an AI-based system capable of automatically recognizing food
items from uploaded images and reducing dependency on manual food logging methods.
Another major objective is to improve nutritional awareness by generating automated dietary
insights for users.

Technical Objectives
A major technical objective of this project is to utilize Qwen2-VL for food classification and
ingredient extraction. The system is designed to identify meal names, visible ingredients, food
categories, vegetarian or non-vegetarian classifications, and confidence scores from uploaded
food images. This improves food recognition performance for both simple and complex
meals.

Another important objective is to integrate intelligent processing techniques that improve

recognition reliability and allow better contextual understanding compared to traditional
image classification models.

Nutritional Objectives
The project aims to integrate USDA FoodData Central for accurate nutritional retrieval. The
system retrieves important nutritional parameters including calories, protein, carbohydrates,
fats, fiber, sugar, sodium, and other dietary values required for nutritional analysis. The
objective is to provide users with structured nutritional reports that help them understand their
food consumption patterns.

Health Analysis Objectives

Another major objective is to design a health scoring engine based on WHO and IIMR dietary
guidelines to evaluate food quality. The system generates health scores, identifies benefits and
concerns associated with meals, and provides personalized dietary recommendations. This
helps users make healthier food choices based on nutritional insights.

Research Objectives
The project also aims to compare the proposed hybrid system with Google Gemini for
benchmarking purposes. The research evaluates system performance based on accuracy,
efficiency, response quality, and nutritional output reliability. Another objective is to identify
existing limitations and propose future improvements for AI-based nutrition tracking systems.
The successful achievement of these objectives will help create a smart, scalable, and practical
nutrition monitoring system that can be implemented in healthcare, fitness, and wellness
industries.

14
 CHAPTER 2
LITERATURE REVIEW

2.1 Background

Nutrition monitoring has become an important area of research due to the growing health
concerns associated with unhealthy eating habits and lifestyle diseases such as obesity,
diabetes, cardiovascular disorders, and hypertension. Proper nutritional tracking helps
individuals maintain balanced diets and make informed food choices. However, traditional
methods of nutrition tracking have largely depended on manual calculations and user inputs,
which often reduce efficiency and accuracy.

Earlier nutrition tracking systems were primarily based on manual food logging applications
where users were required to enter meal names, portion sizes, and food quantities manually.
Applications such as MyFitnessPal, CalorieMama, and similar diet-tracking platforms
became popular because they provided users with nutritional information and calorie
estimates. However, these applications created several practical challenges including time-
consuming food logging processes, inaccurate portion estimation, continuous user
dependency for manual input, and reduced long-term user engagement.

These limitations encouraged researchers to explore automation using artificial intelligence

and computer vision technologies. The first major advancement in automated food
recognition involved the use of computer vision techniques. Traditional image processing
methods focused on extracting visual features such as color, texture, and shape from food
images. These handcrafted feature extraction methods showed limited performance due to
variations in food appearance, lighting conditions, and image quality.

With the advancement of deep learning, researchers began using Convolutional Neural
Networks (CNNs) for food classification tasks. Models such as ResNet, MobileNet,
VGGNet, and EfficientNet were trained on food datasets such as Food-101 and UEC Food-
100 to classify food items based on images. These models significantly improved
classification accuracy compared to traditional image processing methods.

Later, object detection models such as YOLO were introduced to identify multiple food
items within a single image. These models helped improve recognition of meals containing
15
multiple food components. However, object detection systems still faced several limitations
including dependence on predefined food categories, difficulty in handling mixed meals,
poor ingredient identification, limited contextual understanding, and inability to generate
health recommendations.

Another major challenge in automated nutrition systems is nutritional mapping. Identifying

food is only one part of the problem; calculating accurate nutritional values requires reliable
databases and intelligent mapping systems. Many existing systems provide only calorie
estimates and fail to offer complete nutritional analysis.

Recent advancements in Vision Language Models (VLMs) and Generative AI have

introduced new possibilities in food analysis. These models can understand visual
information along with contextual relationships, enabling better recognition of complex
meals and ingredient-level analysis. Models such as Qwen-VL and Google Gemini have
demonstrated strong performance in multimodal tasks involving image understanding and
text generation.

The proposed research builds on these advancements by integrating multiple technologies

into a single hybrid system. The project combines Qwen2-VL for food recognition, USDA
FoodData Central for nutritional retrieval, intelligent re-ranking mechanisms for improved
matching accuracy, health scoring systems for dietary evaluation, and Google Gemini for
comparative benchmarking.

This combination helps overcome the limitations of traditional systems and provides a more
accurate, automated, and intelligent solution for nutrition analysis.

As healthcare technology continues to evolve, automated nutrition analysis systems are

expected to play a significant role in fitness monitoring, hospital nutrition planning, smart
healthcare ecosystems, and personalized wellness applications. This research contributes to
that growing field by developing a practical and scalable AI-driven nutrition tracking
system.

16
Existing
Technique Used Strengths Limitations
System/Model
Requires
MyFitnessPal Manual food logging Large food database
manual input
Limited
CalorieMama Image recognition Easy food scanning nutrition
depth
Cannot detect
Good for single food
CNN Models Image classification complex
recognition
meals
Limited
Detects multiple food
YOLO Model Object detection contextual
items
understanding
High
Vision Language Image-text Better contextual
computational
Models understanding reasoning
cost
Portion
Qwen2-VL + USDA + Automated complete
Proposed System estimation
Gemini nutrition pipeline
limitation

Table 1: Literature Comparison

17
2.2 Summary of Literature Review and Research Gap

The literature review highlights that significant research has been conducted in the fields of
food recognition, nutritional analysis, computer vision, and healthcare monitoring systems.
Earlier systems primarily relied on manual food logging applications such as MyFitnessPal
and CalorieMama, which allowed users to manually enter food details and track calories.
While these systems provided basic nutritional information, they required continuous user
input and were often inconvenient for long-term usage.

To improve automation, researchers introduced machine learning and deep learning models
for food image classification. Convolutional Neural Networks (CNNs) such as ResNet,
MobileNet, VGGNet, and EfficientNet improved food recognition accuracy for single-item
meals. Later, object detection models such as YOLO enabled identification of multiple food
items within a single image. These approaches represented major improvements over manual
nutrition tracking systems.

However, the literature review identified several limitations in existing systems:

Heavy dependency on manual food logging

Limited ability to analyze mixed or complex meals

Poor ingredient-level identification

Difficulty in handling regional food variations

Inaccurate nutritional mapping after food detection

Lack of personalized health recommendations

Limited contextual understanding of food images

Most existing systems focus only on food classification or calorie estimation and fail to
provide complete nutritional intelligence. Very few systems integrate food recognition,
nutritional databases, health scoring mechanisms, and recommendation systems into a single
automated platform.

Additionally, traditional deep learning models require large labeled datasets and are restricted
to predefined food categories. These models often fail when presented with new dishes, mixed
meals, or foods with similar visual appearances.

Recent advancements in Vision Language Models have created opportunities to overcome these
18
limitations. However, research combining VLMs with nutritional retrieval systems and health
analytics remains limited.

Research Gap

Based on the literature review, the following research gaps were identified:
Lack of fully automated nutrition tracking systems
Limited integration of food recognition with reliable nutritional databases

Absence of intelligent ingredient-based food matching systems

Lack of health scoring mechanisms based on medical guidelines

Limited comparative analysis between modern AI models

Poor adaptability to real-world food variations ow the Proposed System Addresses the Gap
The proposed project addresses these gaps by developing a Hybrid AI-Based Food
Recognition and Nutrition Analysis System that integrates:

Qwen2-VL for food classification and ingredient extraction

USDA FoodData Central for nutritional retrieval
Intelligent re-ranking algorithms for better food matching

Health scoring based on WHO and ICMR guidelines

Google Gemini benchmarking for comparative analysis

This integrated approach provides a more accurate, automated, and scalable solution
compared to traditional nutrition tracking systems and existing research approaches.

Thus, the proposed system contributes toward the development of smarter healthcare and
nutrition monitoring applications.

19
 CHAPTER 3
METHODOLOGY

3.1 Materials

The successful implementation of the proposed Hybrid AI-Based Food Recognition and Nutrition
Analysis System required multiple software tools, datasets, machine learning models, APIs, and
computational resources. These materials were selected to ensure accurate food recognition,
nutritional retrieval, health analysis, and system benchmarking. The project integrates advanced
Vision Language Models, external nutritional databases, and health scoring algorithms to create an
automated nutrition tracking system.

1. Food Image Datasets

Food datasets were required for testing and validating the food recognition capabilities of the
system. Food-101 was used because it contains 101 food categories and thousands of food images,
making it suitable for evaluating general food classification performance. UEC Food-100 was used
for testing multi-food image recognition capabilities because it contains complex meal images with
multiple food items. In addition to publicly available datasets, a custom dataset containing Indian
food images was created because many standard datasets have limited representation of regional
Indian meals. The custom dataset included food categories such as biryani, Indian thali, curry
dishes, rice meals, fast food items, fruits, and salads. This helped improve testing on real-world
food variations.

2. Vision Language Model

The proposed system uses Qwen2-VL-7B-Instruct as the primary Vision Language Model for food
image analysis. This model performs multiple tasks including food classification, meal
identification, ingredient extraction, vegetarian and non-vegetarian classification, and confidence
score generation. Unlike traditional object detection models, Qwen2-VL provides contextual
understanding of complex food images. It can recognize mixed meals, identify visible ingredients,
and generate structured outputs, making it more suitable for nutritional analysis tasks.

3. Google Gemini API

Google Gemini 2.5 Flash was used as a benchmark model for comparative analysis. It was
integrated to compare food recognition outputs, validate nutritional estimates, and evaluate
20
response quality. This benchmarking process helped measure the performance of the proposed
hybrid system against another advanced AI model and demonstrated the strengths and limitations
of both systems.

4. USDA FoodData Central

The system integrates USDA FoodData Central API for retrieving nutritional information after food
identification. The database provides standardized nutritional values including calories, protein,
carbohydrates, fats, fiber, sugar, sodium, and saturated fats. This database was selected because it
offers reliable and structured nutritional information that improves the accuracy of the generated
nutrition reports.

5. Programming Language

The entire project was implemented using Python because of its strong ecosystem for machine
learning, API integration, automation, and data processing. Python allowed smooth integration
between the AI model, nutritional APIs, and evaluation modules.

6. Libraries and Frameworks

Several Python libraries were used during system development. PyTorch was used for deep learning
model execution, while Transformers enabled Qwen model integration. PIL was used for image
processing tasks and Requests handled API communication. NumPy and Pandas were used for
numerical computation and data handling. Matplotlib and Seaborn were used for performance
visualization and graph generation during result analysis.

7. Development Platform

The project was developed and tested using Google Colab and Jupyter Notebook environments.
Google Colab provided GPU resources required for running large AI models efficiently, while
Jupyter Notebook supported experimentation and debugging during development.

8. Hardware Requirements

The system required a GPU/TPU-enabled environment for efficient model execution. A stable
internet connection was necessary for API communication, and standard computing devices were
used for testing and result validation.

21
9. Health Guideline References

The health scoring module was developed using WHO nutritional recommendations and ICMR
dietary guidelines. These standards helped the system generate health scores, dietary
recommendations, benefits, and health concerns based on nutritional intake.

To build the health scoring system, the following standards were used:
WHO nutritional recommendations
ICMR dietary guidelines

These standards helped generate health scores, recommendations, concerns, and benefits.
The combination of these materials enabled the development of a complete AI-driven
nutrition analysis pipeline capable of performing food recognition, nutritional estimation,
health analysis, and benchmarking.

Dataset Name Number of Classes Purpose

Food-101 101 food categories Food classification testing
UEC Food-100 100 food categories Multi-food recognition
Custom Indian Dataset 50+ food categories Regional food testing
Real-time User Images Variable System validation

Table 2: Dataset information

22
3.2 Summary of Methodology

The methodology of this project was designed to create a complete automated pipeline for
food recognition, nutritional analysis, and health evaluation. The proposed system
integrates multiple artificial intelligence models, nutritional databases, and health analytics
techniques to overcome the limitations of traditional nutrition tracking systems.

The process begins when a user uploads a food image into the system. The uploaded image is
first processed using image handling techniques such as resizing, formatting, and conversion
to ensure compatibility with the AI models.

The processed image is then analyzed using Qwen2-VL-7B, which acts as the primary food
classification model. This model identifies important details from the image, including:

Meal name
Visible ingredients
Food category
Vegetarian/non-vegetarian classification
Confidence score

Once the food item is identified, the extracted meal information is sent to the USDA
FoodData Central database to retrieve detailed nutritional values. The system fetches
important nutritional parameters such as:

Calories
Protein
Carbohydrates
Fats
Sugar
Fiber
Sodium
Saturated fats

23
 Parameter Description
Calories Total energy intake
Protein Muscle growth and repair
Carbohydrates Energy source
Fats Energy storage
Fiber Digestive health
Sugar Blood glucose impact
Sodium Blood pressure impact
Saturated Fat Heart health indicator
Table 3 Nutritional Parameters

To improve nutritional accuracy, a custom re-ranking mechanism was implemented. This

mechanism evaluates multiple food matches and selects the most relevant result by
considering:

Meal similarity
Ingredient matching
Freshness preference
Processed food penalties
Dietary restrictions

After nutritional values are retrieved, the system applies a Health Scoring Engine based on WHO
and I CMR dietary guidelines. This module analyzes the nutritional composition and generates:

Health score
Food verdict
Health benefits
Dietary concerns
Personalized recommendations

For comparative evaluation, Google Gemini was integrated as a benchmark model to compare food
recognition and nutrition estimation performance with the proposed system.

The final output is displayed through an interactive dashboard where users can view
complete nutritional reports, health analysis, and recommendations in a structured format.

The overall methodology can be summarized in the following workflow:

24
 Figure 3.1

This methodology enabled the development of a scalable, automated, and intelligent

nutrition analysis system capable of providing better contextual understanding and
nutritional insights compared to traditional systems.

25
 CHAPTER 4
RESULTS AND DISCUSSION

4.1 Summary of Results and Discussions

The results obtained from the implementation and testing of the proposed Hybrid AI-Based
Food Recognition and Nutrition Analysis System demonstrate that the system successfully
achieved the major objectives defined at the beginning of the project. The developed model
was able to automate food recognition, nutritional retrieval, health scoring, and
recommendation generation with minimal user intervention.

The system was tested using multiple food images collected from standard datasets such as
Food-101, UEC Food-100, and custom food images containing Indian meals. The
experimental results showed that the system was capable of identifying various food items,
including complex meals such as biryani, Indian thali, rice dishes, fruits, salads, and fast-
food items.

The Qwen2-VL model performed effectively in recognizing meal names, identifying visible
ingredients, classifying food as vegetarian or non-vegetarian, and generating confidence
scores for predictions. Its contextual understanding capabilities enabled better performance
compared to traditional image classification models, particularly when processing mixed
meals containing multiple food components.

The integration of USDA FoodData Central significantly improved nutritional retrieval by

providing structured nutritional information such as calories, protein, carbohydrates, fats,
fiber, sugar, sodium, and saturated fats. This enabled the system to generate more accurate
nutritional reports compared to traditional calorie-tracking applications that often provide
limited nutritional details.

The custom re-ranking algorithm further enhanced nutritional retrieval performance by

selecting the most appropriate food match based on ingredient similarity, food relevance,
freshness preference, and processed food penalties. This helped reduce incorrect food
mapping and improved the reliability of nutritional outputs.

26
The health scoring engine successfully generated health scores, food verdicts, nutritional
benefits, dietary concerns, and personalized recommendations. These outputs provided
users with deeper nutritional insights and made the system more practical for healthcare and
fitness-related applications.

Comparative analysis with Google Gemini demonstrated that the proposed hybrid model
produced more structured and reliable nutritional outputs due to its direct USDA integration
and deterministic scoring mechanism. While Gemini generated faster responses in some
cases, its outputs were less consistent for detailed nutritional breakdown tasks.

Several strengths were observed during system evaluation. The proposed model
significantly reduced manual effort for users by automating food analysis. It demonstrated
strong contextual understanding of food images, delivered accurate nutritional retrieval,
generated health recommendations, and handled complex food combinations more
effectively than traditional systems.

Despite strong performance, certain challenges were identified during testing. Exact portion
size estimation remains difficult because nutritional values can vary depending on serving
quantity. Hidden ingredients were not always detected in highly complex dishes. The
system also depends on API availability for nutritional retrieval and requires relatively high
computational resources for running large Vision Language Models efficiently.

Overall, the experimental findings demonstrate that the proposed hybrid system offers a
practical, scalable, and intelligent solution for automated food recognition and nutrition
analysis while also identifying areas for future improvement.

The results validate the effectiveness of integrating Vision Language Models, nutritional
databases, and health analytics into a single platform. This chapter demonstrates that the
proposed system has strong potential for practical applications in healthcare, fitness
monitoring, and smart nutrition management systems.

27
Parameter Qwen2-VL Google Gemini
Food Recognition Accuracy High Moderate
Ingredient Detection High Moderate
Structured Output High Moderate
Nutrition Retrieval High (USDA integrated) Limited
Response Speed Moderate Fast
Health Recommendation Available Limited

Table 4 Model Comparison

28
 CHAPTER 5
CONCLUSION AND RECOMMENDATIONS

5.1 Conclusion

This project successfully developed a Hybrid AI-Based Food Recognition and Nutrition Analysis
System that automates food identification, nutritional estimation, and health evaluation
using artificial intelligence. The primary goal of the project was to reduce the
dependency on manual food logging systems and provide users with a smarter and more
efficient method for tracking their nutritional intake.

Traditional nutrition tracking applications often require users to manually enter food details,
which can be time-consuming and inaccurate. Similarly, conventional food recognition
systems face limitations in handling mixed meals, identifying ingredients, and
generating meaningful health insights. The proposed system was designed to overcome
these limitations by integrating advanced technologies into a single automated platform.

The developed system uses Qwen2-VL-7B for food image recognition and ingredient
extraction, allowing it to identify meal names, visible ingredients, food categories, and dietary
classifications from uploaded food images. The integration of USDA FoodData Central enabled
accurate retrieval of nutritional values such as calories, proteins, carbohydrates, fats, sugar,
sodium, and fiber.

A custom re-ranking algorithm was implemented to improve food matching accuracy

by evaluating meal similarity, ingredient relevance, freshness preferences, and dietary
restrictions. This significantly improved the reliability of nutritional retrieval.

One of the major contributions of this project is the implementation of a Health Scoring
Engine based on WHO and ICMR dietary guidelines. This module allowed the system to
generate:

Health scores
Food verdicts
Benefits
Concerns
Personalized recommendations
29
 The project also incorporated Google Gemini as a benchmark model to compare the
performance of the proposed system and validate nutritional outputs.

The experimental results demonstrated that the proposed hybrid system performed
effectively in recognizing food items, analyzing nutritional content, and generating useful
health insights. The system was able to handle both simple and complex meals while reducing
user effort and improving automation.

The major objectives of the project were successfully achieved:

Automated food recognition

Ingredient extraction
Nutritional retrieval
Health analysis
Recommendation generation
Comparative AI benchmarking

This project highlights the growing potential of artificial intelligence in healthcare and
nutrition management. The proposed system can be applied in fitness applications, hospitals,
healthcare systems, diet planning platforms, and smart wellness ecosystems.

In conclusion, the project demonstrates how combining Vision Language Models, nutritional
databases, and health analytics can create a more intelligent, scalable, and practical solution
for future nutrition tracking systems.

5.2 Recommendations

Although the proposed Hybrid AI-Based Food Recognition and Nutrition Analysis System
successfully achieved its primary objectives, several opportunities exist for future improvement and
expansion. These recommendations can further enhance the system’s accuracy, scalability, efficiency,
and real-world usability.

One of the major limitations identified during system testing was accurate portion size estimation. The
current system generates nutritional values based on approximate serving sizes, which may affect
precision in calorie and nutrient calculations. Future research can improve this limitation by integrating
advanced portion estimation techniques such as depth sensing, three-dimensional image processing,

30
reference object detection, and computer vision-based volume estimation. These improvements can
significantly enhance nutritional accuracy.

Currently, the developed system functions as a research prototype. Future development can focus on
building a mobile application that allows users to capture food images directly through smartphone
cameras for real-time nutritional analysis. A mobile-based implementation would improve
accessibility, user convenience, and large-scale adoption of the system.

The proposed model currently processes static food images. Future versions can be extended to support
real-time video-based food tracking, where users can continuously monitor meals in restaurants,
cafeterias, or home environments. This would make the system more interactive and suitable for
continuous dietary monitoring.

Another potential enhancement involves integration with wearable devices such as smartwatches,
fitness bands, and health trackers. By combining food consumption analysis with physical activity
tracking, sleep monitoring, calorie expenditure, and personalized health analytics, the system can
evolve into a complete health management ecosystem.

A major challenge observed during implementation was the limited availability of regional food
datasets, particularly for Indian meals and local cuisines. Many publicly available datasets do not
sufficiently represent regional dishes, street foods, and traditional meals. Future research should focus
on developing larger and more diverse datasets that include Indian cuisines, regional food varieties,
traditional meals, and street food items to improve food recognition performance across diverse
populations.

Further improvements may also include reducing model response time, optimizing computational
requirements, improving hidden ingredient detection, and enhancing personalization through user-
specific dietary preferences and medical conditions.

By implementing these recommendations, the proposed system can evolve from a research prototype
into a scalable commercial nutrition platform capable of supporting healthcare institutions, fitness
industries, wellness platforms, and individual users worldwide.

31
 REFERENCES
[1] Bossard, L., Guillaumin, M., and Van Gool, L., “Food-101 – Mining Discriminative
Components with Random Forests,” European Conference on Computer Vision (ECCV),
pp. 446–461, 2014.

[2] Matsuda, Y., Hoashi, H., and Yanai, K., “Recognition of Multiple-Food Images by
Detecting Candidate Regions,” IEEE International Conference on Multimedia and Expo,
2012.

[3] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A., “You Only Look Once: Unified,
Real-Time Object Detection,” Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2016.

[4] He, K., Zhang, X., Ren, S., and Sun, J., “Deep Residual Learning for Image Recognition,”
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[5] Howard, A. G. et al., “MobileNets: Efficient Convolutional Neural Networks for

Mobile Vision Applications,” arXiv preprint arXiv:1704.04861, 2017.

[6] Tan, M., and Le, Q., “EfficientNet: Rethinking Model Scaling for Convolutional Neural
Networks,” International Conference on Machine Learning (ICML), 2019.

[7] Bai, J. et al., “Qwen2-VL: Enhancing Vision-Language Model Capabilities in Visual

Understanding,” Alibaba Cloud Research, 2024.
[8] Google, “Google Gemini API Documentation,” Google Developers Documentation,
2025.
Available: [Link]

[9] United States Department of Agriculture, “USDA FoodData Central Database,”

Available: [Link]

[10] World Health Organization (WHO), “Healthy Diet Guidelines,”

WHO Official Documentation, 2023.

[11] Indian Council of Medical Research (ICMR), “Dietary Guidelines for Indians,”
National Institute of Nutrition, 2024.
32
 APPENDICES

The appendices section includes additional supporting material related to the development,
implementation, testing, and evaluation of the proposed Hybrid AI-Based Food Recognition and
Nutrition Analysis System. These supplementary materials provide detailed evidence of system
development and experimental validation.

Appendix A: System Architecture Diagram

This appendix contains the complete architecture diagram of the proposed system showing
the flow of operations from image input to final nutritional output.

Workflow included:

User Input Image

Qwen2-VL Classification
Ingredient Extraction
USDA Nutritional Retrieval
Re-ranking Module
Health Scoring Engine
Gemini Benchmark Comparison
Final Dashboard Output

Figure 1
33
Appendix B: Sample Input Images

This appendix includes sample food images used for testing the system.

Examples include:

Apple
Biryani
Indian Thali
Rice dishes
Salads
Fast food items

Figure2

Appendix C: Sample Output Screenshots

This appendix contains screenshots of the final output dashboard generated by the system.

Output includes:
34
 Meal Name
Ingredients
Calories
Protein
Carbohydrates
Fats
Health Score
Recommendations

Appendix D: Model Comparison Results

This appendix includes comparison outputs between:

Qwen2-VL
Google Gemini

Metrics Compared:
Accuracy
Nutritional estimation
Response time
Output quality

Figure 3

35
Figure 4

Figure 5

Appendix E: Dataset Details

This appendix provides detailed information about datasets used:

Food-101
UEC Food-100
Custom Indian food dataset

It includes sample images and dataset descriptions.

Appendix F: Testing and Validation Results

This appendix includes additional testing outputs used during project evaluation.
Examples:

36
 Accuracy reports
Nutritional validation results

Error handling outputs

Figure 6

Figure 7

The appendices strengthen the report by providing practical implementation evidence and
additional technical documentation for evaluators.

37