An Automated Computer Vision Based System for

Bottle Cap Fitting Inspection

Rutwik Kulkarni Sourav Kulkarni Shardul Dabhane
Department of Computer Engineering: Department of Computer Engineering: Department of Computer Engineering:
Pune Institute of Computer Technology Pune Institute of Computer Technology Pune Institute of Computer Technology
Pune, India Pune, India Pune, India
rutwikkulkarni02@[Link] souruly@[Link] [Link]@[Link]

Nachiket Lele R.S. Paswan

Department of Computer Engineering: Department of Computer Engineering:
Pune Institute of Computer Technology Pune Institute of Computer Technology
Pune, India Pune, India
lelenachiket991@[Link] rspaswan@[Link]

Abstract— Inspection of the cap is one of the most crucial is suitable to have a cap fit on it. During the fitting, many
phases of packaging bottles. Defects like loosely fitted cap, defects can occur, like loose fitting of cap, scratches on the
scratches/broken cap may occur. It is important to detect these cap, the cap ring might break and many more. An inspection
errors as soon as possible. This paper proposes an automated system is able to carry out various tasks of quality control
system by which the bottle cap defects can be identified. The automatically detecting cap fit, liquid fill level, etc. Different
bottles with these defects will be rejected by the system. The types of inspection systems are used for different inspection
methods used in this paper are based on computer vision. The needs. Online-automated inspection for quality and process
system comprises of four methods, which are utilized for bottle control is becoming an important element of automated
cap defect detection: Pattern recognition, Clustering, Object
manufacturing systems. Object detection is able to identify
Detection and Line Detection. This paper also presents a
the bottle as an object and further, the cap on the bottle. The
comprehensive analysis and a comparison of all the four
methods on various parameters. The system has an extensive
fitting of the cap can be judged by various other vision
social and practical value with increasing the productivity, techniques and then the accuracy of each technique can be
improve the quality of inspection and profitability. compared. A system to carry out specific inspection of bottle
caps has been proposed in this paper.
Keywords— Packaging, Fault detection, Pattern Recognition,
Object Detection, Line Detection, Computer Vision, Clustering II. LITERATURE REVIEW
Packaging defects is one of the open problems in the field of
computer vision and food quality assurance. Many
I. INTRODUCTION researchers have tried to tackle this problem. Some of the
Computer Vision provides the computer the ability to important work in this field is noted below:
have desirable understanding of objects from a singular
image or from an array of images. It also removes the human The authors of [1] gave variety of methods for inspection of
factor from the process, which is laborious and time bottle. They have used Automated Visual Inspection System
consuming. An automatic bottle cap defect detection system (AVIS) to carry out their research. Their paper describes the
is likely to be more efficient and that makes it an attractive various mechanisms like top and bottom detection, cap
option. placement and fill level inspection, empty bottle detection,
etc. To carry out the cap placement detection, they have taken
The handling of bottles on an assembly line is a very
image of the bottle and converted it to grayscale. Canny edge
delicate job. During the packaging of these bottles, due to a
variety of reasons, the bottle’s cap might not be properly detection algorithm, which convolves the image with a
installed. A bottle with a loose cap is not suitable for sale. horizontal mask, is used. After extraction of horizontal lines,
Before automated vision systems, human inspectors used to morphological opening is applied to remove noise reaching
carry out quality control. Human inspection is slower and the to the edges and lines. Using the horizontal mask as a
efficiency of quality control is affected by human reference line, distances are measured vertically until the
shortcomings. An inspection system using computer vision height of the cap and this is the standard height. Any distance
allows us overcome these pitfalls. It allows us to acquire, less or more than that means the cap is improperly fit.
process, analyze and understand digital images and also allow
us to extract high-dimensional data. We are also able to The work [2] proposes a system which uses a microcontroller.
produce numerical or symbolic information, e.g., in the forms The two units of this system are the inspection system and the
of decisions. The understanding of these images implies the rejection system. The language used by them is MATLAB.
transformation of visual images into descriptive data that can The bottles are detected using an IR sensor, which is part of
be interfaced with other processes to carry out various types the inspection system. The rejection system consists of a
of actions. This explicit understanding of images can be seen robotic arm connected to a DC motor, which will throw away
as the separation of the symbolic data from the image data. the bottle with the improperly fit cap. The bottle’s image is
The fitting of the cap takes place after the liquid is poured first captured in the inspection system and the [R, G, B]
into the bottle. It is essential in this phase to ensure that bottle

978-1-7281-3591-5/19/$31.00 ©2019 IEEE

matrix of the image is separated. Then edge detection is used
and image clustering index takes place.
Shen, Yehu et. al. [3] propose a pattern matching method in
which the scratches on bottles are detected. The method used
is a template based edge comparison method. In this method,
the edge is first smoothened based on a guided filter
technique. The edge information is extracted. Then modified
Hough transform is combined with a hierarchical orientation
searching method to locate the position and orientation of the
candidate bottle cap in the test image. At the end, rotation
compensated edge image is compared with the template edge
image in order to determine whether there exist scratches or
not in the test image.
In [4] the authors propose developing a machine inspection
system, which detects the defects of PET bottles with faulty
support rings. Serious defects such as surface foreign matters
and link ring broken of the PET caps are identified. For
detecting the defect of surface foreign matters, the algorithm
employs mean filtering to estimate the background, and then
compare the inspected image with the background image by Fig1: Proposed System
using a dynamic threshold. As a result, the crack region is
extracted. To detect a broken link ring, HSV color space A. Inspection Module
transform is used. The gap between the link ring and cap is
better obtained here.
While all of the above techniques try to approach the problem
in their own ways, none of them employ an ensemble based
system which combines the advantages of different
techniques thus giving higher accuracy. In our work, we have
introduced such a system and tried to make the individual
detection algorithms as generalized as possible which can be
applied real-time to various products with a similar use-case,
which includes bottles of different shapes, sizes, beverages,
etc.
III. PROPOSED SYSTEM
In this paper, we propose an automated system based on
machine vision for bottle cap packaging fault identification.
The system consists of:
● Assembly Line: Assembly line on which bottles
(object of interest) are present.
● Proximity Sensor: Proximity Sensors are used to
detect the presence of bottles on the assembly line
and trigger the camera module. Fig2: Inspection System
● Camera Module: 5MP (Industrial camera/ Webcam)
is used to capture the images of the bottles on the The inspection module rigorously inspects the captured
trigger given by proximity sensor. image of the bottle. It uses various algorithms like clustering,
● Inspection Module: It inspects and identifies the line detection, Convolutional Neural Networks to detect the
packaging fault (if present) in the bottle cap fitting defects in the bottle packaging. Majority voting is used to
by using various computer vision algorithms. The ensemble the results of the techniques. The combined result
detailed explanation of this module is given later in will depend on all the results of individual techniques,
this section. whether fault is detected or not detected. If fault is detected
● Result module: The decision given by the inspection by even one of the techniques, it is transmitted to the
module is forwarded to the result module which will consequent result module, where the bottle can be checked
display the output with bottle identification number. manually. The system concentrates on maximizing the
specificity of the model and helps the industry to minimize 3) Segment the cluster from image having the
the faults in packaging. color same as that of the bottle cap.
The advantages of this system are :-
● Non-intrusive: The high-speed assembly line is not
affected by the inspection of the bottles in real-time.

● Capturing images on the fly: The image-capturing

device must be fast enough to capture images of
individual objects as they go past the camera on the
assembly line.
● Synchronization of computation time and actual
speed of Production Line:- The system manages to
minimize computation time and also synchronize Fig 4: a) Bottle Cap of Correctly Packaged Image
the speed with that of the assembly line. b) Separation between Ring and Cap seen in
Incorrectly Packaged Bottle.
IV. METHODOLOGIES
4) Convert the obtained image to gray scale and draw
In this section, we will discuss various algorithms, which contour lines to identify the maximal connected cluster. (If
we developed for fault detection. We will also present a the bottle is packed well this cluster also consists of the ring
comparative study of all the algorithms used. The Line below the cap else this ring is absent in the cluster)
Detection and Clustering algorithms were implemented in
Python programming language by using the OpenCV library.
The object detection algorithm was implemented using the
tensor flow object detection library while the pattern
recognition algorithm was built by using convolutional neural
networks in Keras Library.
(a) (b)
Fig 5: Rectangular bounded box on countoured bottle cap
A. Clustering images.
(a) Correctly Packaged – Cap and Ring
In this method K - Means Clustering is used which (b) Incorrectly Packaged – Cap Only
separates the captured image of the bottle into numerous
clusters. The unsupervised learning algorithm detects the cap 5) The shape of the contour lines and the subsequent
by identifying it in a distinct cluster. bounding box enables us to detect the flaws in the cap fitting.
The steps in the algorithm are:
1) Read the image in [R, G, B] format. 6) As we can see from the Fig 5, the area of cap of a well-
2) Apply K Means clustering to separate image into packaged bottle is more than that of a defective bottle.
clusters. If the color of liquid and the cap is same then
aggressive clustering (more no of clusters, i.e. K>20) is
B. Line Detection preceded by Object Detection.
required.
Object detection consists of a neural network based
supervised learning algorithm which is used to detect any
particular object in an image. In our system, we use object
detection algorithms to segment the bottle cap from the
image. We use SSD (Single Shot multibox Detector) for cap
detection. SSD extracts the feature maps (VGG 16 is used for
this purpose) and applies convolutional filters to detect the
bottle cap in the image. We used SSD because it gave better
performance than Faster- RCNN that is commonly used for
object detection.

Fig3: Clusters in Image obtained by K-Means Clustering.

 Fig 6: Bottle Cap Detection

Once the bottle cap has been identified, it is passed on to Fig 8: Edge Detection and Line Detection for Input Image with
the line detection module where it can be determined whether Packaging Fault
the cap is fitted correctly or not, thus identifying the presence
of a packaging fault, if any.
In the line detection module, the following steps are taken: C. Pattern Recognition.
1. Get isolated Bottle Cap Image from Object This method uses Convolutional Neural Network (CNN) for
Detection module detection of patterns in the well packaged and the defective
2. Apply Gaussian Blur to the image bottles. CNN applies various filters on the labelled data to
3. Then, apply Canny Edge Detection to the image extract features from it. In order to better identify the rings
4. Then, apply probabilistic Hough Line Detection to present on the bottle, which are seen clearly in the case of
the output image received after Edge Detection defective bottles, the adaptive thresholding of the images was
5. Discard all lines that have slopes greater than 0.05 done. The adaptive thresholding also makes the algorithm
or less than -0.05 insensitive of the color differences and helps it focus on the
6. Get the y-coordinates of the centres of the lines. differences in the position of the bottle cap. The CNN was
7. Apply natural partitioning to divide the lines in 3 trained on the metrics of accuracy and specificity to gain the
groups. better results.
8. Based on the differences between the groups, adjust
group boundaries.
9. If any line has y-coordinate, of its center in the
middle group, then the bottle cap is fit incorrectly
and therefore packaging fault is present.

Fig 9: Adaptive Thresholding of the Bottle Image

Fig 7: Edge detection and Line Detection for input Image with no
Packaging Fault
The confusion matrix is as follows: VI. FUTURE SCOPE
As the system is fault detection, system measure indicates
that the fault is not detected and negative measure indicates This system is built specifically for detection of the cap of a
that fault is detected. The system was trained on 400 bottle bottle. Further a similar system can be used to track other
images and tested on 170 thresholded images. defects such as fill detection, distortion detection, labelling
faults etc.
This is a part of the work where the system will be
TABLE I. CONFUSION MATRIX modified to inspect for contents of the liquid. Foreign object
detection inside the liquid can be carried out by modifying
170 Predicted Values the system and pattern recognition algorithms. Thus, we plan
to build an all in one quality detection and assurance system
for the beverage industry, which will not only help to capture
Actual Values Negative Positive the packaging faults but also help detect the suspended
components in the liquid thereby ensuring that only the best
Negative 110 0 products reach the consumer.

Positive 10 50 ACKNOWLEDGMENT

We would like to take this opportunity to thank Mr. Pravin

Waghmare(CEO - Prescient Technologies) and the entire
team of Prescient Technologies who guided us throughout the
D. Comparison of algorithms project and provided us with all necessary resources. We
thank them for their valuable support and suggestions..
TABLE II. COMPARISON

Algorithms Robustness Sensitivity Determinist

REFERENCES
to camera to Color of ic/ Non
position the liquid Determinist
and cap ic (N/D) [1] Shubhi Sharma, K. Vinaya Krupa, Roshni Gandhi, Aditi Jain, Nehal
Shah, “Empty and Filled Bottle Inspection System,” ICTACT Journal
on Image and Video Processing, Volume 6, Number 2, 2016, pp. 1122-
Clustering( NO YES D 1126.
K-means) [2] Pruthvi Kumar S, Dr H.V Ramakrishna, “Automated Bottle Cap
Inspection using Machine Vision System,” IJIRT ,Volume 2, Issue 2,
Line YES NO N 2015, pp. 2349-6002.
Detection [3] S Y. Shen, R. Mo, L. Wei, Y. Zhu, Z. Peng and Y. Zhang, "Bottle cap
scratches detection with computer vision techniques," 2013 Ninth
International Conference on Natural Computation (ICNC), Shenyang,
Pattern YES NO N
2013, pp. 1314-1318.
Recognition
[4] L. Kong, Q. Wu, K. Lin and B. Chen, "Scratch detection of round
(CNN) buttons based on machine vision," 2017 10th International Congress
on Image and Signal Processing, BioMedical Engineering and
Informatics (CISP-BMEI), Shanghai, 2017, pp. 1-6.
[5] M. Campos, M. Ferreira, T. Martins and C. Santos, "Inspection of
bottles crates in the beer industry through computer vision," IECON
V. CONCLUSION 2010 - 36th Annual Conference on IEEE Industrial Electronics Society,
Glendale, AZ, 2010, pp. 1138-1143.
The above work presents a vision-based inspection [6] X.X. Jiang, Z.J. Zhang, Y.N. Wang, et al, “PET full bottle fast detection
system, which is capable of inspecting the bottle cap and based on gradient diffusion of gray projection,” Journal of Electronic
identifying any present defects on it. The work is carried out Measurement and Instrumentation, vol.30, no.8, 2016, pp.1152-1159.
by using image-processing techniques like pattern [7] Slim Abdelhedi, Khaled Taouil, Bassem Hadjkacem, “Design of
recognition, object detection, line detection and clustering. Automatic Vision-based Inspection System for Monitoring in an Olive
Oil Bottling Line,” International Journal of Computer Applications
Our system was able to detect the smallest of displacement of (0975 – 8887), Volume 51– No.21, August 2012.
bottle cap from its intended position and help in discarding [8] L. Yazdi, A. S. Prabuwono and E. Golkar, "Feature extraction
the bottle. The focus was on discarding as many ill-fitted algorithm for fill level and cap inspection in bottling machine," 2011
bottles as possible. The system worked fast in real-time by International Conference on Pattern Analysis and Intelligence
comparing the image of the bottle with the trained data to Robotics, Putrajaya, 2011, pp. 47-52.
make a decision about the bottle. All the techniques used are [9] Riza Sulaiman, Anton S. Prabuwono, “Intelligent Visual Inspection of
Bottling Production Line Through Neural Network,” The Journal of the
compared based on robustness, sensitivity and whether the
Institution of Engineers, Malaysia (IEM Journal). Vol. 69, No. 4,
method is deterministic or non-deterministic. December 2008, pp. 56-61.