Proceedings of the 5th International Conference on Inventive Research in Computing Applications (ICIRCA 2023)

IEEE Xplore Part Number: CFP23N67-ART; ISBN: 979-8-3503-2142-5

Recipe Recommendation using Image Classification

Ms. Anitha E Ms. S. Nandhini, Mr. Hari Kumar Palani
Asst. Professor, Department of IT Assistant Professor, Professor,
Loyola-ICAM College of Engineering and Department of Computer Science and School of Engineering
Technology Engineering, Ajeenkya D Y Patil University,
Chennai, India Sathyabama Institute of Science and Pune
anitha.e@[Link] technology,
Chennai
Mr. Marshal Mano C,
Assistant Professor, Department of IT
2023 5th International Conference on Inventive Research in Computing Applications (ICIRCA) | 979-8-3503-2142-5/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICIRCA57980.2023.10220811

Loyola-ICAM College of Engineering and

Technology
Chennai, India

Abstract - As the digital era develops, smart food by combining machine learning algorithms and the user's
technologies are more in demand because they make meal input data. The goal of recipe recommendation is to help
planning and preparation simpler and more enjoyable. Recipe
recommendation systems have become widespread in order to users discover new and intriguing recipes that they are likely
provide clients with customized recipe recommendations based to appreciate and assist them with meal preparation and
on their dietary preferences, level of cooking proficiency, and grocery shopping.
availability of ingredients. On the other hand, the majority of
recipe recommendation systems rely on textual inputs or pre-
defined categories of ingredients, which might not accurately In this paper, a recipe recommendation system is
reflect what the user has in their pantry. This research study proposed that makes use of convolutional neural networks
proposes a recipe suggestion system based on user-provided (CNNs) and image classification to generate recipes based
images of readily available ingredients that combine image
on the user-provided image of the ingredient. The method
recognition and convolutional neural networks (CNNs). It will
recommend recipes based on the image provided. This will attempts to address the issue of scarce ingredient supply and
make finding recipes more quick and simple than it would assist users in knowing the preparation of a recipe. The
otherwise be. In this instance, text-to-speech guided technology suggested system is made up of three primary parts: pre-
is used so that the user can just replay the procedure as needed
rather than reading it repeatedly. This research work will
processing of the images, feature extraction using a CNN
include additional elements like redirecting to a grocery cart model that has already been trained, and recipe
by clicking a link. This concept will be very helpful to those recommendation using content-based filtering. It also
who don't know what to cook. Thanks to this initiative, users provides a text-to-speech feature for the procedure while
won't have to waste time thinking about the recipes.
giving recommendations.
Keywords - Recipe Recommendation, Convolutional Neural
Network(CNN), Image Classification, Machine Learning, Food
Recommendation. II. INTRODUCTION TO DOMAIN
The field of machine learning has experienced
I. INTRODUCTION
tremendous expansion and enhancement in recent years.
In the modern world, machine learning is becoming more Convolutional Neural Networks (CNNs) are a well-known
and more significant since it has the tendency to automate and effective type of machine learning method.
and optimize a variety of tasks across numerous industries. Convolutional Neural Networks have seen great growth and
improvement in recent years. CNNs have completely
Convolutional neural networks (CNNs) have radically
changed the field of computer vision and are especially
altered how computer vision and machine learning are well-suited to processing and interpreting images. CNNs
practiced. They are a special type of neural network that has have been employed in a variety of applications. CNNs are
excelled at handling multidimensional data, including gaining importance across a wide range of industries,
images. CNNs learn hierarchical representations of the input including healthcare, banking, and transportation, because
of their capacity to understand intricate patterns and
data by combining convolutional layers, pooling layers, and
relationships in data.
fully connected layers. They have been applied to a variety
of tasks, such as object identification, facial recognition, and
image categorization. A. Computer Vision
Computer Vision is a field of artificial intelligence.
A recipe recommendation is a process of providing It is used for allowing computers to derive information from
recommendations of different recipes based on the user's images, videos, and other inputs. This is used for scene
preferences and customization. This is often accomplished

979-8-3503-2142-5/23/$31.00 ©2023 IEEE 292

Authorized licensed use limited to: Zhejiang University. Downloaded on September 14,2024 at 16:59:31 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Inventive Research in Computing Applications (ICIRCA 2023)
IEEE Xplore Part Number: CFP23N67-ART; ISBN: 979-8-3503-2142-5

reconstruction, object detection, and audio and video The following are some of the related works regarding
analysis. recipe recommendation and image classification.

Convolutional Neural Networks (CNNs) are

mainly used in computer vision for image processing. The In a recent study, a paper [1] proposed a Convolutional
process starts with feeding the input data. Then the data is Neural Network model for recognizing the 11 different
broken down into pixels for processing. Different states of 20 different food items. Foods and their states are
dimensions of the image are noted for classification. Then discovered based on the deep features available in the
the image is classified based on the identified features and images of the food. The generalization capability of the
attributes. CNN performed well with a publicly available dataset. But
some food states are hard to recognize which occurs as a
challenge.
The convolutional neural network needs to be
trained first in order to use it for image classification. The Another study [3] has used NOVA classification model
filters that classify the images are learned during training. to segregate the different foods available in an image and
The model is trained with a large dataset of images with their healthiness level. The convolutional layer here uses 32
labels. The features are compared within a given dataset and filters with 19 residual bottleneck layers. It can accurately
classification is extracted. After training, the model is fed classify foods due to its multi-label classification model.
with unknown images to test. It does feature extraction and The limitation here is the image needs to be taken from a
classification to identify the image. specific angle to know the healthiness of the food.

CNNs are an effective method for image

classification in computer vision. They have been used, For recognizing fruits based on the image, a paper [5]
among other things, for object detection, facial recognition, proposed a two-stage feature selection approach which uses
Densnet121 for feature extraction. Then the recognition is
and hygienic inspection of processed and raw foods.
done through the Adaptive particle Grey Wolf Optimization
method. The accuracy is high and flexible to apply in
B. Tensorflow: different datasets. Feature selection should be made
carefully else it may be over fitting to the datasets.
TensorFlow is an open-source machine learning
framework. It is used for various tasks ranging from
building a machine-learning model to deploying it. The
In another paper [8], recipes are recommended based on
main advantage is that it can handle large amounts of data. It the text input of ingredients and cooking tags given by the
also facilitates complex computational tasks. It was user. It uses a set transformer model that accepts variable-
developed by the Google Brain team. TensorFlow gives size inputs. Yet the limitation here is if the training data is
both novices and experts a collection of procedures with insufficient, then the model gives poor results.
simple, high-level APIs to build machine learning models in
a variety of languages. It has escalated tools for visualizing
and debugging machine learning models. In recent research [10], filter-based methods and
statistical methods are used for classifying fruits and
C. OpenCV: vegetables. The limitations were insufficient feature
descriptors because of the different nature of the data.
OpenCV (Open Source Computer Vision) is a
popular open-source computer vision and machine learning
software library. It is mainly used for image and video Overall, these studies provide different ways to use
[Link] is used to train and deploy machine-learning image classification that helped in this paper.
models for image classification. It can be done using a wide
variety of programming languages. The pattern of the image
is identified using vector space and performs mathematical IV. METHODOLOGY
operations on them. The main advantage of OpenCV is its
potential to work on a wide range of input sources and The Recipe Recommendation using image classification
support various video formats. OpenCV is a powerful and of user-provided ingredient input, which is proposed in this
paper, has 5 modules. They are Data Acquisition, image
versatile tool for computer vision.
pre-processing, feature extraction, image recognition and
recommendation, and text-to-speech module.
III. RELATED WORK
The research field of recipe recommendation using Convolutional Neural Networks (CNN) are a cutting-
image classification is one that is expanding quickly and has edge technique that is frequently used to train computers to
caught the interest of many researchers. In recent years, a recognize and classify images in order to obtain a trained
number of studies have been done on this subject. dataset. To recognize the ingredient’s images in real-time,
Researchers in this area are working to create computer OpenCV is employed as the library.
vision-based algorithms that can suggest recipes to
consumers based on the food images they have.

979-8-3503-2142-5/23/$31.00 ©2023 IEEE 293

Authorized licensed use limited to: Zhejiang University. Downloaded on September 14,2024 at 16:59:31 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Inventive Research in Computing Applications (ICIRCA 2023)
IEEE Xplore Part Number: CFP23N67-ART; ISBN: 979-8-3503-2142-5

A. Data Acquisition
The data required for recipe recommendation using
image classification must be gathered and prepared by the
data acquisition module. In order to collect photos and
recipe metadata, including ingredients and preparation
directions, this module may include browsing recipe
websites. Also, it can entail gathering and maintaining
collections of pictures of food and the associated recipe
data. The data acquisition module, in general, is crucial in
ensuring that the recommendation system has access to
high-quality, pertinent data that can be utilized to produce
precise recipe suggestions. The test dataset is obtained from Fig. 1. System Architecture.
Kaggle that contains 15000 images for training, 3000
images for testing, and 3000 images for validation. E. Text To Speech Module
B. Image Preprocessing Text to Speech (TTS) module is used for providing
A crucial part of computer vision applications, the image accessibility to the user. By clicking a button, the TTS
preprocessing module involves cleaning and processing raw module will convert the recommended recipe instructions
picture data before it is input into a machine learning model text to audible voice. The engine used for the TTS module
for analysis. The preprocessing module seeks to improve the for Google Text To Speech API. The user can change or
input image's quality and to extract helpful features that aid adjust the volume. It makes it easier for users to listen to the
in precisely identifying and classifying the object of interest. instructions rather than read them. There is another button
The process of rescaling can be done by a deep learning that will redirect the user to buy other necessary ingredients,
library called TensorFlow. in case they are in need.

V. RESULTS AND MODEL ACCURACY

C. Feature Extraction
A key part of the image classification pipeline is the
feature extraction module, which transforms raw image data
into a set of numerical characteristics that machine learning
models may use to classify images. This module extracts
features from food images for use in recipe recommendation
systems that categorize images. Shapes, colors, textures, and
other qualities that are important for classifying food images
can be included in these attributes. This module produces a
set of feature vectors that numerically represent the image
and can be used as input for machine learning models.

D. Image Recognition and Recommendation

A central element of the recipe recommendation system
that makes use of computer vision techniques to identify and
suggest recipes based on input images is the image
recognition and recommendation module. This module
employs machine learning algorithms to match the feature Fig. 2. User Interface – I.
vectors produced by the feature extraction module with
recipes that have features in common. This module produces
a list of suggested recipes that closely resemble the inputted
image. The user interface can display this list of
recommendations. Overall, using the users' input images as
a basis, machine learning will present accurate and pertinent
recommendations to them.

Fig. 3. User Interface – II.

979-8-3503-2142-5/23/$31.00 ©2023 IEEE 294

Authorized licensed use limited to: Zhejiang University. Downloaded on September 14,2024 at 16:59:31 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the 5th International Conference on Inventive Research in Computing Applications (ICIRCA 2023)
IEEE Xplore Part Number: CFP23N67-ART; ISBN: 979-8-3503-2142-5

classification model using public food image dataset and deep

learning methods”. IEEE Access, 10, 128732 - 128741.
[5] Tri Tran Minh Huynh, Tuan Minh Le, Long Ton That, Ly Van Tran
& Son Vu Truong Dao (2022). “A Two-stage feature selection
approach for fruit recognition using camera images with various
machine learning classifiers”. IEEE Access, 10, 132260 - 132270.
[6] Guofan Shao, Lina Tang & Hao Zhang (2021). “Introducing image
classification efficacies”. IEEE Access, 9, 134809 - 134816.
[7] Christoph Trattner , Dominik Moesslang & David Elsweiler (2018).
“On the predictability of the popularity of online recipes”. EPJ Data
Science, 7:20, 35370 - 35381.
[8] Mogan Gim, Donghyeon Park, Michael Spranger, Kana Maruyama &
Jaewoo Kang (2021) “RecipeBowl: A Cooking recommender for
ingredients and recipes using set transformer''. IEEE Access, 9,
143623 - 143633.
[9] Mohammed Ahmed Subhi, Sawal Hamid Ali & Mohammed
Abulameer Mohammed (2019). “Vision-based approaches for
automatic food recognition and dietary assessment: A Survey”. IEEE
Fig. 4. Model Accuracy.
Access, 7, 35370 - 35381.
[10] Khurram Hameed, Douglas Chai & Alexander Rassau. (2018). “A
comprehensive review of fruit and vegetable classification
VI. CONCLUSION techniques”. Image and Vision Computing, 80, 24-44.
In conclusion, the systems for recommending recipes [11] Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu, Yong Rui
& Shuqiang Jiang (2017). “You are what you eat: exploring rich
that make use of image classification techniques based on recipe information for cross-region food analysis”. IEEE Transactions
Convolutional Neural Networks are gaining popularity. It on Multimedia ( Volume: 20, Issue: 4), 950 - 964.
can offer individualized recipe suggestions that are tailored [12] Shota Horiguchi, Sosuke Amano, Makoto Ogawa & Kiyoharu
to a user's particular requirements and preferences by Aizawa (2018). “Personalized classifier for food image recognition”.
assessing and categorizing images of the materials. The use IEEE Transactions on Multimedia ( Volume: 20, Issue: 10, October
2018), 2836 - 2848.
of CNNs provides high accuracy in identifying ingredients.
[13] Xianan Qin & Xiaoming John Zhang (2021). “An Industrial dyeing
This not only reduces the time and effort required for meal recipe recommendation system for textile fabrics based on data-
preparation, but it also enables people with dietary needs mining and modular architecture design”. IEEE Access, 9, 136105 -
and preferences to learn about new and interesting foods. 136110.
[14] Jayme Garcia Arnal Barbedo (2013). “Digital image processing
The inclusion of a text-to-speech module in addition to techniques for detecting, quantifying and classifying plant diseases''.
SpringerPlus, 2:660.
image classification improves the accessibility of recipe
[15] Javier Marín, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia
instructions for users with visual impairments or those who Salvador, Yusuf Aytar, Ingmar Weber & Antonio Torralba (2019).
prefer audio instructions. Users can listen to the recipe “Recipe1M+: A Dataset for learning cross-modal embeddings for
directions while cooking with the push of a button, making cooking recipes and food images”. IEEE Transactions on Pattern
the process easier and more fun. It also has a link that Analysis and Machine Intelligence ( Volume: 43, Issue: 1), 187 - 203.
redirects to a shopping site, where users can buy some small
ingredients required for cooking or others ingredients that
are included in a recipe.

Ultimately, meal preparation and planning for people of

all backgrounds and abilities could be revolutionized by
recipe recommendation systems that include image
recognition and text-to-speech modules. We can anticipate
more developments in this area as well as more
individualized and precise recipe recommendations thanks
to advances in deep learning and natural language
processing.
VII. REFERENCES
[1] Gianluigi Ciocca, Giovanni Micali & Paolo Napoletano (2020). “State
recognition of food images using deep features”. IEEE Access, 8,
32003 - 32017.
[2] Frank Po Wen Lo, Yingnan Sun, Jianing Qiu & Benny Lo (2020).
“Image-based food classification and volume estimation for dietary
assessment: a review”. IEEE Journal of Biomedical and Health
Informatics ( Vol: 24, Issue: 7, July 2020), 1926 - 1939.
[3] Shady Elbassuoni, Hala Ghattas, Jalila El Ati, Zoulfikar Shmayssani,
Sarah Katerji, Yorgo Zoughbi, Aline Semaan, Christelle Akl, Houda
Ben Gharbia & Sonia Sassi (2022). “DeepNOVA: A Deep learning
NOVA classifier for food images”. IEEE Access, 10, 128523 -
128535.
[4] Minki Chun, Hyeonhak Jeong, Hyunmin Lee, Taewon Yoo &
Hyunggu Jung (2022). “Development of Korean food image

979-8-3503-2142-5/23/$31.00 ©2023 IEEE 295

Authorized licensed use limited to: Zhejiang University. Downloaded on September 14,2024 at 16:59:31 UTC from IEEE Xplore. Restrictions apply.