Received: 3 August 2021

| Revised: 29 November 2021

| Accepted: 17 December 2021

DOI: 10.1002/ese3.1056

ORIGINAL ARTICLE

Defect object detection algorithm for electroluminescence

image defects of photovoltaic modules based on deep
learning

Ziyao Meng1,2,3 | Shengzhi Xu1,2,3 | Lichao Wang1,2,3 | Youkang Gong1,2,3 |

Xiaodan Zhang1,2,3 | Ying Zhao1,2,3

1
Engineering Research Center of Thin
Film Photoelectronic Technology, Abstract
Ministry of Education, Tianjin, China Visual inspection of photovoltaic modules using electroluminescence (EL) im-
2
Institute of Photoelectronic Thin ages is a common method of quality inspection. Because human inspection re-
Film Devices and Technology, Nankai
University, Tianjin, China
quires a lot of time, object detection algorithm to replace human inspection is a
3
Key Laboratory of Photoelectronic popular research direction in recent years. To solve the problem of low accuracy
Thin Film Devices and Technology of and slow speed in EL image detection, we propose a YOLO-­based object detec-
Tianjin, Nankai University, Tianjin,
tion algorithm YOLO-­PV, which achieves 94.55% of AP (average precision) on
China
the photovoltaic module EL image data set, and the interference speed exceeds
Correspondence 35 fps. The improvement of speed and accuracy benefits from the targeted design
Shengzhi Xu, Engineering Research
of the network architecture according to the characteristics of EL image. First,
Center of Thin Film Photoelectronic
Technology, Ministry of Education, we weaken the backbone's ability to extract deep-­level information so that it can
Tianjin 300350, China. focus on extracting the low-­level defect information. Second, the PAN network
Email: xusz@[Link]
is used for feature fusion in the Neck part. But, only the single-­size feature map
output is retained, which significantly reduces the amount of calculation. Also,
we analyze the impact of data enhancement methods on model overfitting and
performance. Finally, we give effective data enhancement methods. The results
show that the object detection algorithm in this paper can meet the requirements
for high-­precision and real-­time processing on the PV module production line.

KEYWORDS
deep learning, electroluminescence image, object detection, photovoltaic modules, YOLO
algorithm

1 | I N T RO DU CT ION the manufacturer's reputation will be affected, and the

power station will also be under tremendous pressure on
Solar power is currently one of the most important forms operation and maintenance.
of new energy power generation. In the photovoltaic Some relevant studies1,2 show that 41% of the fail-
power station, photovoltaic modules undertake the critical ures in photovoltaic power plants are caused by panels.
energy conversion function, which is the core component. According to the quality report of photovoltaic mod-
If the quality of photovoltaic modules is not guaranteed, ules given by German Rhine and Hamelin Solar Energy
This is an open access article under the terms of the Creat​ive Commo​ns Attri​bution License, which permits use, distribution and reproduction in any medium, provided
the original work is properly cited.
© 2022 The Authors. Energy Science & Engineering published by Society of Chemical Industry and John Wiley & Sons Ltd.

Energy Sci Eng. 2022;00:1–14.  [Link]/journal/ese3 | 1

2 |    MENG et al.

Research Institute, about 6% of photovoltaic modules were discussed. Finally, concluding observations and the direc-
damaged during production and transportation before in- tion of the future work are provided in Section 6.
stallation, which may be related to impact and vibration
during transportation. Module defects will seriously affect
the stable and efficient operation of the power plant, so 2 | RELATED WORK
it is necessary to carry out quality inspection during the
production and power generation.3 As far as we know, many researches on EL image defect de-
Methods as current–­voltage curve scanning, electrolu- tection revolve around image classification: Deitsch et al.5
minescence imaging, infrared radiation test, ultraviolet uses an improved VGG-­19 model to predict the defect
fluorescence, etc., are widely used methods for PV module probability of cell EL images. By rounding the predicted
failure detection.4 Among them, EL image has high reso- continuous probability to the nearest neighbor of the four
lution, and can detect cracks and other defects that cannot original classes, they directly compare convolution neural
be found by naked eyes, so it has become the usual detec- network (CNN) decisions against the ground truth labels.
tion method at present in both production line and onsite The results in public data set of solar cells show that the
of plant. The EL images inspection manually is time-­ method using CNN is more accurate than the support vec-
consuming and laborious, besides that subjective factor tor machine (SVM), reaching 88.42%. Akram et al.6 regard
affects the whole inspection process, which makes it dif- the identification of cell defects as a two-­category prob-
ficult to achieve standardized defect detection. Therefore, lem. They propose a neural network classification model
a fast and accurate detection algorithm is needed to auto- with only four layers of convolution. Also, they compress
matically detect defects in the PV modules. the input image size to save processing time and use data
The object detection algorithm requires not only clas- enhancement to improve the robustness of the network.
sifying the object category but also using one bounding In public solar cell data set, the process speed of each pic-
box to mark the location. Nowadays, object detection al- ture is 8.07 ms per picture, and the accuracy is 93.02%.
gorithms based on deep learning have become the main- Tang et al.7 divide the cell into defect-­free, microcrack,
stream. In many cases, it has replaced the traditional finger-­interruption, break in private data set, regarding
object detection algorithm. Object detection with artificial the defect identification problem as a four-­category prob-
neural network (ANN) has been applied to many fields in- lem. Given the lack of data sets, they use one generative
cluding EL image. adversarial networks (GAN) to fake additional images for
In this paper, we propose an object detection YOLO-­PV data enhancement, which greatly improves the accuracy
based on YOLO. The YOLO-­PV network structure is op- of classification. GAN enhancement is also proved to be
timized specially for PV modules, to rapidly and accu- one good method for solar cell classification in our previ-
rately detect defects. The contributions of this paper are ous work.8
as follows: For the research of object detection tasks in EL im-
ages, Liu et al.9 improved the feature extraction part
1. To propose a standard for detecting defects in EL and RPN part in Faster-­RCNN and proposed GA-­Faster-­
images of PV modules and establish a complete PV RCNN. The model can identify the defects on the cell
module defect detection data set. and mark its location. This work uses GA-­RPN to greatly
2. The YOLO-­ PV network structure is proposed com- reduce the number of candidate frames. The detection
bined with the actual situation of the photovoltaic speed is improved in self-­made data set. Zhang et al.10 de-
module defect detection task. Through experiments on signed a detection algorithm of surface defects on solar
the PV module data set, we verify the effectiveness of cells, which combines the results of Faster R-­CNN and
the network. R-­FCN to improve detection precision and position ac-
3. We use various data augment methods to solve the curacy. This method creatively integrates the detection
overfitting problem and give an effective data augment results of two mature network models, and the mAP of
suggestion. three defects on the self-­made data set reaches 85.7%.
Otamendi et al.11 perfectly combine three deep learning
The remainder of this work is organized as follows: technologies: Faster-­RNN, EfficientNet, and autoencoder
Section 2 introduces related work. Section 3 introduces to build an end-­to-­end deep learning pipeline that de-
the EL image and object detection data set. Section 4 in- tects, locates, and segments cell-­level anomalies from the
cludes data preprocessing, network structure design, net- entire photovoltaic modules via EL images. In the object
work loss function, and evaluation criteria. In Section 5, detection part, this work improves the Faster-­RNN to bet-
the performance of the proposed model is evaluated by ter perform the solar cell detection task. This work not
experiments, and the experimental results are given and only does the object detection of PV module defects, but
MENG et al.    | 3

also uses autoencoder to complete the task of anomaly the image will be dark. The defects of solar cells can be
segmentation module. classified into two categories5: the first one is called the
While briefly summarizing the results of the existing intrinsic deficiencies which arise due to material proper-
research, we find that the current research has the fol- ties, such as black core and broken grid. The second one is
lowing problems: low detection accuracy, slow detection the process-­induced extrinsic defects, such as microcracks
speed, non-­uniform defect detection standards. Although and breaks. According to the International Energy Agency
the existing methods have greatly saved the inspection standards,15 these defects will have a certain impact on the
time, it still cannot meet the real-­time processing of the safety and power generation capacity.
EL images. It is still challenging to apply in some envi- As shown in Figure 1, we can observe the following
ronments with sensitive hardware resources and high defects through EL images:
real-­time requirements. At the same time, a large number
of false detections and missed detections caused by low 1. Black area: An irregular black area in the EL image,
accuracy seriously affect subsequent analysis and the re- often caused by material defects or poor contact caused
liability of algorithms; hence, further research is needed. by inappropriate welding.
2. Cracks: Potential cracks cannot be found directly by
naked eyes. It can only be detected by EL technology.
3 | DE F ECT AN D DATA SET This type of cracks may cause damage to the cell or not.
They are generally caused by the weak external stress
Electroluminescence, abbreviated as EL, refers to the from manufacturing, transportation, and installation.
luminescence phenomenon in which materials directly 3. Break: Severe cracks cause part of the cell completely
convert electrical energy into light energy under external be electrically isolated from the others, represents as
electric field. Injection electroluminescence is commonly a black area in the EL image. Severe external thermal
used in the photovoltaic industry. This method was first stress and mechanical stress may cause break cell.
proposed by Y. Takahashi.12 The specific step is to apply 4. Finger failure: Finger failure is a black line perpendic-
a forward bias to the solar cell. The forward current will ular to the bus bar, which may be caused by too low
inject many unbalanced carriers into the solar cell, which printing height, too low printing pressure, or problems
will produce injection electroluminescence. Since the in- with the bus bar welding.
trinsic silicon band gap is about 1.12 eV, we can calculate 5. Low cell: Low cell is significantly darker than other cells
that the peak of infrared light generated by the electrolu- in the module's EL image. Cells with different efficien-
minescence should be around 1150 nm, which is beyond cies are welded on the same modules, and the lifetime
the strong response range of the CMOS sensor. So, when of the cell's minority carrier is quite different, which is
using the SLR camera to shoot EL image, it needs in a dark the main reason for the low cell.
room to avoid the adverse effects of ambient light. In other 6. Scratch: Scratch is the appearance of single or multiple
way, the SWIR camera with InGaAs sensor will be good at smooth and uneven black lines on the cell. The reason
imaging the EL of solar cell and make it possible to do fast of scratch may be that sharp objects cause nicks on the
imaging which leads to lower cost in PV plant mapping cells during transportation on the production line.
when cooperated with drone. 7. Black cell: Black cell appears on the EL image as one
or more cells are completely black. The reasons for the
black cell include short-­circuit of the cell, low-­quality
3.1 | Failure detected in silicon chip material, or cracks caused by improper
electroluminescence images welding process.
Besides, the cell may also have defects, such as broken
Takashi Fuyuki et al.13 point out that for the detailed corner, black edges. Some abnormalities may also occur
inspection of cell performance, the most essential mate- during the EL image shooting.
rial parameter is the minority carrier diffusion length
(or lifetime), which determines the collection efficiency.
They also point out that the brightness intensity of the EL 3.2 | EL image object detection data set
image has a one-­to-­one correspondence with the minor-
ity carrier diffusion length. Würfel et al.14 also consider For supervised learning, a complete training process re-
that the minority carrier diffusion length can be uniquely quires labeled data set. However, the detection standards
determined by the brightness of the EL image. This is for defects in production lines and power stations are often
because the minority carrier diffusion will be low in the inconsistent. For power station operators, they hope the
defective area of the cell, and the corresponding area in defective cells that impact power generation efficiency and
4 |    MENG et al.

F I G U R E 1 Photovoltaic cell defects observed in EL images. (A) Black area. (B) Cracks. (C) Break. (D) Finger failure. (E) Low cell. (F)
Scratch. (G) Black cell. (H) Broken corner

safety can be detected in time. As for production line, not 4. For all other defects, such as break, low cell, and bro-
only defects as mentioned above must be found, but also ken corner. they are marked as defects in this work.
finger failure, black corner, etc., because these defects re-
flect the deficiencies in raw materials or manufacturing pro- These pictures are taken by different types of equip-
cesses. Through defect detection, these deficiencies can be ment, including a professional EL camera (NXL-­100) and
found in time to avoid affecting module quality. It is worth a modified SLR camera (Nikon D5300), which also makes
mentioning that different types of defects are defined in this the collected images more diverse. The parameters for
work in order to better mark the defective cell. However, in shooting are focal length 26 mm, aperture f/4 and expo-
the following work, these defects are uniformly marked and sure time 4–­10 s. An inspection expert and two graduate
detected as the same type, because the strategy we adopt students mark these pictures. We use labeling software to
on the production line is to use neural network to identify make labels. Mark the defects in the EL image with a rect-
defects first, and then use manual classification if necessary. angular box and save the marking information (including
Based on the demands of manufacturers, in this paper, we the size and position of the box and the object type) in an
establish a set of defect detection standards that can be ap- XML file. We take 360 of 2144 pictures as the test set, and
plied on the production line. The main points are as follows: the remaining 1784 pictures are divided into training set
and verification set according to 8:2.
1. For defects that may reflect material defects and weld- The collected images have a resolution of
ing errors, such as black area, finger failure, we must 7380 × 3838. To save computing resources and re-
mark them as defects. move redundant information, we first resize the image
2. Cracks include cross-­cracks, parallel cracks, reticu- to a suitable size. In this work, the image is scaled to
lar cracks, etc. As long as the crack causes the fail- 960 × 512 pixels, and the aspect ratio is retained to re-
ure area of the cell, we need to mark the crack as a duce image distortion. In this process, the choice of
defect. For the slight crack, we ignore the V-­shaped image scaling algorithm is very important, because
crack which is <6 mm and the cross-­crack which is object detection tasks often require fine-­grained infor-
<14 mm. mation of the image. But some resize algorithms may
3. For the scratch, we ignore the single scratch <8 mm. lose some defects such as slight cracks. Therefore, we
For the single scratch >8 mm and multiple scratches, choose the pixel area relationship interpolation algo-
we mark them as defects. rithm, as shown in Figure 2.
MENG et al.    | 5

F I G U R E 2 In the comparison before

and after image resize, some minor
defects are still preserved

FIGURE 3 YOLO algorithm flowchart

4 | M ET H ODOLOGY directly from the image. This makes the YOLO algorithm
extremely fast and can process streaming video in real
4.1 | YOLO-­PV time. After continuous iteration, the accuracy and speed
of the YOLO algorithm have made great progress.
Object detection algorithm based on deep learning can As mentioned above, the YOLO algorithm implements
be divided into two categories: one-­step and two-­step.16 A end-­to-­end detection, and the process is shown in Figure 3.
typical two-­step object detector is R-­cnn17 series. The algo- The idea of the algorithm is to divide the image into S*S
rithm first obtains candidate regions through region selec- grids. If the center of an object is located in this grid, then
tion methods, then resizes the candidate regions to a fixed the grid is responsible for predicting that object. Each grid
size and sends them to the CNN network to extract image predicts B boxes and a confidence score. The confidence
features. Finally, SVM classifier is used to classify the ex- score means whether there is an object in the grid. So, the
tracted features, then get the classification results of each predicted bounding box has five parameters (x, y, w, h, c),
region. The two-­step method has a very complex work- where (x, y) represent the coordinates of the center point
flow. Model training is divided into several stages, and the of the bounding box, and (w, h) are the relative values of
steps are cumbersome: fine tuning CNN Network, training the width and height of the bounding box for the whole
SVM, and training frame regressor. The Faster-­RCNN18 picture, and c is the confidence scores of each bounding
proposed later solves the above problems to a certain ex- box. At the same time, each grid also has a category score
tent, but there are still many redundant calculations. C, and C represents the conditional class probabilities.
Common one-­stage detectors include YOLO,19 SSD,20 Using one-­hot code, if the data set has 20 categories, then
etc. In large production lines, processing speed is often as C is a 20-­dimensional vector. The output of the network is
important as accuracy. In addition, with the application a vector of S*S*(B*5 + C).
of UAV patrol inspection in power station, if the detection
video can be processed in real time, the detection work
will be more efficient. To match the production rate of 4.1.1 | YOLO-­PV structure
production line and meet the needs of real-­time detection,
we give priority to speed and choose the YOLO algorithm. YOLO-­PV is based on the YOLO v421 algorithm, an ob-
YOLO algorithm is the first one-­stage detector in the field ject detection algorithm designed to detect defects in PV
of deep learning. It reframes the object detection as a re- modules. Before designing the network, we first analyze
gression problem. It uses a single convolutional neural the characteristics of the PV module defect detection task,
network to predict the position and category of the object which is summarized as follows:
6 |    MENG et al.

1. Since the PV module is composed of cells arranged in

an orderly manner, the defective cells to be detected
will not overlap and block each other. Each cell is
independent of the other, and the defective cell is
often not related to the surrounding cells. So, there
is no need for complicated semantic understanding
and excessive global information.
2. In reality, the size of the cell is constant, so the size of
the bounding box we predict is relatively fixed, which
greatly reduces the error caused by the bounding box in
the interference. F I G U R E 4 Illustration of PANet. PANet introduces a top-­
3. EL images are grayscale images, and the defects in the down pathway and a bottom-­up pathway to fuse multiscale features
EL images are often low-­level features (such as color, from level 3 to 5 (P3–­P5)
shape, and texture) without semantic information.
This can reduce the acquisition of high-­level semantic important features extracted by the backbone, which is
when designing the network structure. conducive to the specific learning of the Head in the next
step.
On the basis of analysis of the above characteristics In the Neck part, we use the method of path aggre-
and application requirements, we present the YOLO-­PV gation. Common methods of path fusion are: FPN,25
network structure, which may also be suitable for many PANet,26 and Bi-­FPN.27 Compared with FPN, PANet and
industrial inspection tasks with relatively fixed targets Bi-­FPN have better performance, but the computational
(such as crack detection of ceramic tiles). Our network complexity increases. The Bi-­FPN model is more com-
framework is the same as other mainstream object detec- plex than PANet and is difficult to model simplification.
tors and is divided into three parts: Backbone, Neck, and Considering that only one size of the feature map output
Head. is retained in the following work, we use PANet to fuse the
semantic information in the multilayer feature map. The
Backbone structure of PANnet is shown in Figure 4. It takes level
In the object detection algorithm, the backbone part is 3–­5 input features, where Pi represents a feature level with
used to extract the input features, which is the cornerstone resolution of 1/2i of the input images. First, different fea-
of subsequent tasks. The processing time of backbone part ture map inputs are obtained from backbone, and then the
accounts for more than half of the whole algorithm pro- information of the feature maps is fused with each other.
cess to achieve high accuracy and fast detection. The back- Finally, there are feature map outputs of different sizes in
bone part must be redesigned and simplified. The major PANet, in which a small feature map size is used to detect
components of the backbone include CBM, Res_unit and large objects, and a large feature map size is used to de-
CSPn22 modules. tect small objects. Multiscale feature map output is very
CBM: This module contains a convolutional layer, a effective when the size of the detection target is not con-
batch normalization layer, and a Mish23 activation func- stant. However, the size and aspect ratio of the defective
tion layer. The batch standardization layer is to alleviate cell in this task are fixed. Therefore, we simplify the PAN
overfitting during the training process. structure and propose SPAN, which only retains one-­size
Res_Unit: This module is a residual block, which is feature map output. It greatly reduces the amount of cal-
composed of two CBM and an addition module. It can culation of the PAN network.
maintain low complexity in deeper network layers and The Neck of YOLO-­PV includes CBM, CBL (including
improve feature extraction ability.24 a convolution layer, a BN layer and a Leaky Relu activa-
CSPn: CY Wang et al.22 point out that CSPNet can tion function layer), and SPAN. In addition, to improve
reduce the amount of calculation by 20% while enhanc- the model's attention to spatial, we introduce the attention
ing the learning ability of CNN, effectively reducing the mechanism SAM28 layer.
memory usage. Therefore, referring to this study, we apply
CSPNet to ResNet to form a CSPn module, where n rep- Head
resents n Res_units in the module. The head part uses the same head layer as YOLO V3.
We use this layer to predict both class probabilities and
Neck bounding box coordinates.
The Neck part, as a connecting link between the Backbone Figure 5 shows the structure of YOLO-­PV. Since the
and the Head plays an important role. It integrates the defect detection of PV modules does not involve complex
MENG et al.    | 7

FIGURE 5 Architecture of YOLO-­PV

semantic understanding and the complex relationship be- loss, and classification loss. The loss function calculation
tween target and background, we use four CSPN modules formula is given by Equation (1), where w, h, B are the
to extract image feature information in the backbone. The width and height of the feature map and the number of
input image size is 960 × 512. After each CSPN module, boxes predicted by each point, C represents the confidence
the size of the feature map will be reduced by half. In the level, and p(c) represents the category prediction result.
obj
last three CSPN modules, 8, 16, and 32 times of downs- 1ij means that if the box at i, j has an object, its value is 1,
ampling are extracted. Three feature maps of different otherwise, it is 0. 1ij
noobj
means that if the box at i and j has
sizes represent the feature information of different levels.
no object, its value is 1, otherwise it is 0. 𝜆iou is iou_nor-
Through the Neck part, the understanding and learning of
malizer, 𝜆cls is cls_normalizer, and 𝜆c is classes_multipli-
feature information are strengthened. The learned feature
ers. They are all adjustable hyperparameters. The
information is a single-­size feature map. Then, the output
confidence loss adopts MSE, and the classification loss
can be obtained through the YOLO layer.
part adopts the cross-­entropy loss function. Positioning
To verify the effectiveness of the YOLO-­PV network
loss uses CIOU (Complete IoU) 29 loss instead of MSE loss
structure on the PV module data set, in Section 5, we
for bounding box regression. CIOU considers three more
compare the effect of using different backbones to extract
factors than IOU: overlap area, center point distance, and
feature information and different Neck to fuse the feature
aspect ratio. Using Lciou makes the bounding box predic-
information on the network performance.
tion more accurate.

|
∑w∗h ∑B2
4.1.2 Loss function obj
∑
L = 𝜆iou L1 + 𝜆cls L2 − 1ij 𝜆c L3 (1)
i=0 j=0
c∈class

In this work, we use the loss function of YOLO v4, which

∑w∗h ∑B
includes three parts of positioning loss, object confidence L1 =
obj
1 Lciou (2)
i=0 j=0 ij

∑w∗h ∑B obj
∑w∗h ∑B noobj
L2 = ̂ i )2 +
1ij 𝜆c (Ci − C 1 ̂ i )2
𝜆c (Ci − C (3)
i=0 j=0 i=0 j=0 ij
8 |    MENG et al.

3. Take the average of the obtained 11 precision rates to

L3 = ̂
pi (c)log(pi (c)) + (1 − ̂
pi (c))log(1 − pi (c)) (4)
get the value of AP.

4.1.3 | Evaluation criteria

4.2 | Solving overfitting
The most important evaluation index in the field of tar-
get detection is AP.30 The calculation of AP requires When analyzing the actual experimental results, we find
IOU, precision, recall rate, and other indicators. How that the validation set loss will increase slightly during
to calculate these indicators will be introduced one by the later training. The performance AP of the model will
one. IOU is a standard to measure the accuracy between decrease (see Section 5.1). We think that the model may
the predicted box and the ground truth, and the calcula- have overfitting. It may be due to the difference of dis-
tion of IOU is given by Equation (5). The larger the IOU tribution between training set and validation set, which
value, the closer the prediction box and the ground truth makes the model better in training set, but poor in valida-
are. When the IOUGP (IOU between the predicted and tion set.
the ground truth) is greater than a certain threshold, The methods to solve the problem include adding BN
we believe that the network has correctly detected the layer31 and Dropout layer,32 early stopping, and data aug-
object. The calculation method of Precision and Recall mentations. For the first two methods, we add BN layer and
is given by Equations (7) and (8). Among them, TP rep- Dropout layer to the network during network design. We
resents a correct detection of a ground-­truth bounding, save the model weight in time during the training process,
which is detection IOUGP ≥ threshold. FP represents a check the AP of the model in the training set, and use the
wrong detection (an incorrect detection of a nonexistent best weight to interference. But these two methods change
object, or a misplaced detection of an existing object), the symptoms instead of addressing the core of the prob-
which is the detection IOUGP < threshold. FN represents lem. The essence of the overfitting problem is that the gap
the number of targets that have not been detected by the between the training set and the validation set. Therefore,
network. NTP, NFP, and NFN represent TP, FP,FN re- collecting more data is the most effective solution, but we
spective statistics. cannot obtain unlimited data due to condition. Increasing
data diversity through data enhancement has been an ef-
A∩B fective way to make limited data generate greater value.
IOU = (5)
A∪B Commonly used data enhancement methods include flip,
rotate, adjust color temperature, exposure, etc. Unlike the
Ground Truth ∩ Prediction box image classification task, the object detection data set may
IOUGP = (6)
Ground Truth ∪ Prediction box cause image displacement during data enhancement, so
the corresponding label must be adjusted. We choose to
NTP rotate the picture randomly and adjust the exposure ran-
Precision = (7)
NTP + NFP domly to enhance the data set, and we also use the mosaic
method. Figure 6 shows the original image and the image
NTP after data enhancement. The impact of different data en-
Recall rate = (8)
NTP + NFN hancement methods on network performance will be dis-
cussed in Section 5.
Another important evaluation criterion called AP is
calculated on the basis of Precision and Recall rate. This
work uses F11-­point interpolated average precision to cal- 5 | EX PERIMENTAL RESULTS
culate the AP. The specific steps are given as follows: AND DISCUSSION

1. First, the algorithm outputs all the defect prediction This part introduces the environment, some hyperpa-
boxes and sorts them according to the confidence rameters and the data enhancement methods used in the
level. experiment. This work uses deep learning frameworks
2. Set different values of K, select the top k prediction TensorFlow and Darknet. Experimental operating system
boxes, calculate FP and TP, so that recall rate R is equal Ubuntu includes 18.04, CPU model i7-­4790, GPU model
to 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9,1.0 respec- GeForce RTX 2080 Ti. We need to set the learning rate,
tively, and calculate the precision rate under different epochs, etc., at the start of the experiment. The values of
recall rates. these hyperparameters are given in Table 1.
MENG et al.    | 9

F I G U R E 6 Image after data

enhancement: (A) original image,
(B) random adjustment of exposure,
(C) random rotation, (D) new image
generated by mosaic

TABLE 1 Training hyperparameters of defect detection model This shows that using CSP-­PV as the backbone can reduce
the depth and complexity of the model. So, it improves the
Description Value
performance and accelerates the convergence, thus saving
Epoch 4000 the training time.
Batch size 32 When VGG-­16 is used as a feature extraction network,
Momentum 0.949 the loss is the largest, and the value of loss appears to get
Weight decay 0.0005 unstable the change with sharp fluctuations. This demon-
Learning rate 0.0001 strates that using VGG-­ 16 may not be able to extract
enough feature information to fulfill the requirements
for object detection, and the robustness of the network
is insufficient. However, the remaining five groups of
5.1 | Comparison of different backbones experiments will have the problem that the loss on the
validation set will increase in the later stage of the train-
In Section 4.1.1, we propose CSP-­PV as the backbone to ing. The AP will drop significantly. We suggest that this is
extract the feature information of EL image. In order to probably because the model has been overfitted. The over-
verify the effectiveness of the proposed model, we com- fitting phenomenon may be due to too little data to learn
pare the performance of the object detection algorithm enough information, or the network model is too compli-
using VGG-­16,33 Resnet50,24 Resnet101, Darknet53,34 cated to learn some useless information (such as noise)
CSPDacknet,21 and CSP-­PV as backbone. To avoid the in the training set. Comparing the experimental results of
interference of other parts on the experimental results, Resnet50 and Resnet101 shows that the more complex the
neck and head in this experiment use the corresponding model, the more serious the overfitting phenomenon. To
parts in YOLO v4. The experimental results are shown in overcome the problem of overfitting, we use different data
Figure 7. The first six lines in Table 2 show the precision, enhancement methods in Section 5.3.
recall, and the AP when using different backbones.
It can be seen from the experimental results that the
best AP on the validation set is 90.7% when using CSP-­PV 5.2 | Comparison of different necks
as Backbone, which is an increase of 1.88% compared with
the AP on the validation set by CSPDarknet. This may be The PAN has multiple sizes of feature map output, respon-
because CSPDark is in-­depth, and the feature extraction sible for predicting objects of different sizes. In SPAN, we
ability is stronger, so it can learn more deep information only keep one-­ size feature map output, which greatly
and global information. This information may be redun- reduces the computation cost. In addition, we introduce
dant for this task and may eventually affect the accuracy. spatial attention layer in the Neck part of YOLO-­PV to
Therefore, the network structure of deep learning is not as enhance the model's acquisition of spatial information.
complicated as possible. It should be designed reasonably In this section, we use YOLO-­PV as the object detection
according to practical requirements. We also find that the algorithm to experiment on the data set, and the experi-
model converges faster when CSP-­PV is used as the back- mental results are shown in Figure 8. We compare it with
bone, and achieves the best AP at about the 2700th epoch. the loss and AP of CSP-­PV as the backbone in Section 5.1
10 |    MENG et al.

F I G U R E 7 Validation loss and AP

of different backbone during the training
process

TABLE 2 Defect detection results table of different model fixed. Using a single-­size feature map can slightly improve
AP, and at the same time, greatly reduce the number of
Backbone Precision Recall AP
output feature maps to avoid the waste of computing re-
Darknet19 73.81% 73.42% 73.74% sources. YOLO-­PV makes the model faster by reducing the
Resnet50 73.42% 73.95% 74.10% complexity of the feature extraction network and remov-
Resnet101 79.45% 79.04% 79.15% ing the redundant feature map output. So, it takes only
Darknet53 84.67% 83.56% 84.38% 28 ms to process an EL image, which reduces the process-
CSPDarknet 89.31% 88.30% 88.82% ing speed by 36.36% compared with 44 ms for YOLO v4.
CSP-­PV 91.15% 90.41% 90.7%
MODEL Precision Recall AP
5.3 | Comparison of different data
YOLO-­PV 91.56% 91.03% 91.34%
augmentation

In the above experiment, we find that the model has an

(the two differ only in the Neck part). Through contrastive overfitting problem. In response to this problem, three dif-
analysis, we can conclude that the AP using the YOLO-­PV ferent data enhancement methods are used to solve the
structure reaches 91.34%, which is an increase of 0.64% overfitting phenomenon in the training process in this
compared with the CSP-­PV group in Section 5.1. This re- section. The three methods are random rotation, random
sult shows that the effect of multiscale prediction will not adjustment of exposure, and mosaic method. To verify
be qualitatively improved when the object size is relatively whether the three methods are effective, we first apply the
MENG et al.    | 11

F I G U R E 8 Validation loss and

AP of YOLO-­PV and using the data
augmentation method during the training
process

T A B L E 3 AP of validation set before and after data stable. Using the mosaic data enhancement method can
augmentation increase the AP of the model the most, thus increasing
the AP by 2.26%. The mosaic data enhancement method is
After
to splice four pictures together to generate a new picture.
Data augmentation Without operation
operation operation (%) (%)
This method can greatly enrich the background of the tar-
get in the object detection task. But in this experiment, the
Angle 91.34 89.74
EL image's background is the same, so we think this effect
Exposure 91.34 92.86 can be produced because the mosaic method synthesizes
Mosaic 91.34 93.60 the new EL images. These new EL images are equivalent
All positive operations 91.34 94.76 to expanding the number of EL images in the training set.
More images are involved in the training, which can im-
prove the performance of the model.
three methods separately, and the experimental results Therefore, we apply random exposure adjustment and
are shown in Figure 8. After using the data enhancement mosaic to the training process at the same time. Through
method, the overfitting phenomenon of the model is al- the results of Figure 8, we can see that after using the ran-
leviated, but not all data enhancement methods will bring dom adjustment of exposure and the mosaic method, the
positive effects. convergence speed of the model becomes slower, but there
As shown in Table 3, when we randomly rotate the is no obvious overfitting phenomenon in the training pro-
image, the AP of the model drops by 1.6%. This may be be- cess. The value of the loss function decreases with the in-
cause there are no rotated module images in the validation crease of training times and finally stabilizes. AP increases
set. Adding this situation to the training set will make the by 3.42%–­94.76%.
gap between the training set and the validation set larger.
For the model, although more features are learned, these
features do not belong to the validation set, thus affecting 5.4 | Model performance on test set
the model's acquisition of effective information. However,
the method of rotating images may be helpful to some data To verify the performance of the model, we test the per-
sets that contain pictures from various shooting angles. formance of the model on the test set. Table 4 shows the
The AP of the model will increase by 1.52% when the ex- AP of the model on the training set, verification set, and
posure is adjusted randomly. This may be because, when test set. The effect of YOLO-­PV on the test set decreased
the exposure is increased, some unobvious defects, such slightly within an acceptable range to 94.55% compared
as cracks and broken grids may be more obvious, which with the verification set. The model still maintains its
is conducive to the defect feature extraction of the model. characteristics of high accuracy. Figure 9 shows the test set
What is more, adjusting the exposure can also simulate results of object detection after using YOLO-­PV and data
the situation of inconsistent picture brightness caused by enhancement methods. The model not only accurately
the unfixed input current and exposure time. It greatly en- identify the defects, such as break cell and black cell, but
riches the diversity of data sets andmakes the model more also identify some very subtle defects such as cracks less
12 |    MENG et al.

than 10mm. These defects are very difficult to be detected accuracy are equally important on PV module production
with the naked eyes. However, the model is not robust in line. Therefore, in this paper, a fast object detection algo-
identifying some subtle finger failure and slight low cell. rithm for the EL image of PV modules is proposed based
In practice, missed inspections may occur. For finger fail- on YOLO v4 algorithm, which can quickly and accurately
ure, some of finger failure features are not obvious and are identify the defective cells in PV modules. After the com-
easily mixed with the background (such as regular black pletion of the network design, we have carried out experi-
lines). For low cells, some slight low cells are not signifi- ments on EL image data sets. The experimental results
cantly different from normal cells in the modules, causing show that compared with the YOLO v4 algorithm, the
missed inspection. Precision, Recall rate and AP are improved. The process-
ing time reduces 36.36%, which meets the requirements
of industrial detection speed and accuracy. The algorithm
6 | CO N C LUSION can be better applied to the working scene with only CPU
and mobile terminal. In addition, the structure of the algo-
Fast and high-­accuracy object detection algorithm is an rithm can be used for other industrial detection tasks with
important work because detection speed and detection relatively fixed size and independent objects.
This paper uses three different data enhancement
TABLE 4 AP of training set, verification set and test set methods, analyzes the actual effect on the EL image, and
Data set AP (%)
explains the performance improvement and possible ap-
plication scenarios. In the next, we can extend this work
Training set 95.31
in several directions. A more concise backbone structure
Validation set 94.76 and more efficient feature fusion method to further im-
Test set 94.55 prove the speed and efficiency must be used. In addition,

FIGURE 9 Defect prediction results by the best model on test set. (A) Correct detections for defects. (B) Missed defects marked by green
boxes
MENG et al.    | 13

on Cognitive Computing and Hybrid Intelligence (CCHI),

the defect classification task is not involved in this work. Cognitive Computing and Hybrid Intelligence (CCHI), 2019
We can use a more novel backbone extraction network to 2nd China Symposium On; 2019:292-­ 297. doi:10.1109/
extract deeper network information without increasing CCHI.2019.8901952
the complexity of the model or use the currently extracted 10. Zhang X, Hao Y, Shangguan H, et al. Detection of surface de-
feature information to complete a more complex object fects on solar cells by fusing multi-­channel convolution neural
detection task. At the same time, for the data set, it can networks. Infrared Phys Technol. 2020;108:103334.
be seen from the experimental results that data enhance- 11. Otamendi U, Martinez I, Quartulli M, et al. Segmentation of
cell-­level anomalies in electroluminescence images of photo-
ment can well improve the performance of the network
voltaic modules. Sol Energy. 2021;220:914-­926.
model in the case of limited data. However, this work only 12. Takahashi Y, Kaji Y, Ogane A, Uraoka Y, Fuyuki T.
explores three data enhancement methods, which can ex- "Luminoscopy"-­novel tool for the diagnosis of crystalline sil-
plore more effective data enhancement methods in future icon solar cells and modules utilizing electroluminescence.
work. 2006 IEEE 4th World Conference on Photovoltaic Energy
There are still many problems in the research of EL Conference, Photovoltaic Energy Conversion, Conference Record
image, such as the secondary use of the results in this of the 2006 IEEE 4th World Conference On, Vol. 1; 2006:924-­927.
doi:10.1109/WCPEC.2006.279607
work. In addition, there is still no clear relationship be-
13. Fuyuki T, Kaji Y, Ogane A, Takahashi Y. Analtic findings in the
tween the defect problems reflected in EL images and
photographic characterization of crystalline silicon solar cells
the actual performance. The algorithm for quantitative using electroluminescence. 2006 IEEE 4th World Conference on
analysis of EL defects has not been deeply studied. Future Photovoltaic Energy Conference, Photovoltaic Energy Conversion,
research should focus on more accurate and faster identi- Conference Record of the 2006 IEEE 4th World Conference On,
fication of defective solar cells. As well as the algorithm Vol. 1; 2006:905-­907. doi:10.1109/WCPEC.2006.279602
detection results should be effectively used to quantify the 14. Würfel P, Trupke T, Rüdiger M, et al. Diffusion lengths of
performance of solar cells. silicon solar cells from luminescence images. J Appl Phys.
2007;101(12):1650. doi:10.1063/1.2749201
15. Kntges M, Kurtz S, Packard C, et al. Review of failures of pho-
ORCID
tovoltaic modules; 2014.
Ziyao Meng [Link] 16. Zou Z, Shi Z, Guo Y, Ye J. Object detection in 20 years: a survey;
2019.
REFERENCES 17. Girshick R, Donahue J, Darrell T, Malik J. Rich feature hier-
1. Lillo-­Bravo I, González-­Martínez P, Larrañeta M, Guasumba-­ archies for accurate object detection and semantic segmenta-
Codena J. Impact of energy losses due to failures on photovol- tion. 2014 IEEE Conference on Computer Vision and Pattern
taic plant energy balance. Energies. 2018;11(2):363. doi:10.3390/ Recognition, Computer Vision and Pattern Recognition (CVPR),
en110​20363 2014 IEEE Conference on, Computer Vision and Pattern
2. Köntges M, Kajari-­Schröder S, Kunze I, Jahn U. Crack sta- Recognition (CVPR), 2013 IEEE Conference On; 2014:580-­587.
tistic of crystalline silicon photovoltaic modules. European doi:10.1109/CVPR.2014.81
Photovoltaic Solar Energy Conference; 2011. 18. Girshick R. Fast R-­CNN. Proceedings of the IEEE International
3. Paggi M, Berardone I, Infuso A, Corrado M. Fatigue degrada- Conference on Computer Vision, 2015 International Conference
tion and electric recovery in Silicon solar cells embedded in on Computer Vision, ICCV 2015; n.d.:1440-­1448. doi:10.1109/
photovoltaic modules. Sci Rep. 2015;4(1):1-­7. doi:10.1038/srep0​ ICCV.2015.169
4506 19. Redmon J, Divvala S, Girshick R, Farhadi A. You only look
4. Kntges M, Kurtz S, Packard C, Jahn U, Friesen G. Review of once: unified, real-­time object detection. 2016 IEEE Conference
failures of photovoltaic modules; 2014. on Computer Vision and Pattern Recognition (CVPR), Computer
5. Deitsch S, Christlein V, Berger S, et al. Automatic classifica- Vision and Pattern Recognition (CVPR), 2016 IEEE Conference
tion of defective photovoltaic module cells in electrolumines- On; 2016:779-­788. doi:10.1109/CVPR.2016.91
cence images. Sol Energy. 2019;185:455-­468. doi:10.1016/[Link]​ 20. Liu W, Anguelov D, Erhan D, et al. SSD: single shot multibox
er.2019.02.067 detector[J]. European Conference on Computer Vision; 2016.
6. Akram MW, Li G, Jin Y, et al. CNN based automatic detection 21. Bochkovskiy A, Wang CY, Liao HYM. Yolov4: optimal speed and
of photovoltaic cell defects in electroluminescence images. accuracy of object detection. arXiv preprint arXiv:2004.10934;
Energy. 2019;189:116319. doi:10.1016/[Link].2019.116319 2020.
7. Tang W, Yang Q, Xiong K, Yan W. Deep learning based au- 22. Wang C-­Y, Mark Liao H-­Y, Wu Y-­H, Chen P-­Y, Hsieh J-­W, Yeh
tomatic defect identification of photovoltaic module using I-­H. CSPNet: a new backbone that can enhance learning capa-
electroluminescence images. Sol Energy. 2020;201:453-­460. bility of CNN. 2020 IEEE/CVF Conference on Computer Vision
doi:10.1016/[Link]​er.2020.03.049 and Pattern Recognition Workshops (CVPRW), Computer Vision
8. Yiping Z. EL image recognition of solar cell module defects and Pattern Recognition Workshops (CVPRW),2020 IEEE/
based on deep learning; 2020. CVF Conference On; 2020:1571-­ 1580. doi:10.1109/CVPRW​
9. Liu L, Zhu Y, Ur Rahman MR, Zhao P, Chen H. Surface de- 50498.2020.0020323
fect detection of solar cells based on feature pyramid net- 23. Misra D. Mish: a self regularized non-­monotonic activation
work and GA-­faster-­RCNN. 2019 2nd China Symposium function. arXiv preprint arXiv:1908.08681; 2019.
14 |    MENG et al.

24. He K, Zhang X, Ren S, Sun J. Deep residual learning for image 30. Padilla R, Netto SL, da Silva EAB. A survey on performance met-
recognition. 2016 IEEE Conference on Computer Vision and rics for object-­detection algorithms. 2020 International Conference
Pattern Recognition (CVPR), Computer Vision and Pattern on Systems, Signals and Image Processing (IWSSIP) ; 2020. ISBN
Recognition (CVPR), 2016 IEEE Conference On; 2016:770-­778. 9781728175393. doi:10.1109/iwssi​p48289.2020.9145130
doi:10.1109/CVPR.2016.90 31. Ioffe S, Szegedy C. Batch normalization: accelerating deep net-
25. Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S. work training by reducing internal covariate shift. Proceedings
Feature pyramid networks for object detection. In: Proceedings of the 32nd International Conference on Machine Learning, in
of the IEEE conference on computer vision and pattern recogni- Proceedings of Machine Learning Research, Vol. 37; 2015:448-­
tion; 2017:2117-­2125. 456. [Link]
26. Liu S, Qi L, Qin H, Shi J, Jia J. Path aggregation network for in- 32. Srivastava N, Hinton G, Krizhevsky A, Sutskever I,
stance segmentation. 2018 IEEE/CVF Conference on Computer Salakhutdinov R. Dropout: a simple way to prevent neural net-
Vision and Pattern Recognition, Computer Vision and Pattern works from overfitting. J Mach Learn Res. 2014;15(6):1929-­1958.
Recognition (CVPR), 2018 IEEE/CVF Conference on, CVPR; 33. Simonyan K, Zisserman A. Very deep convolutional net-
2018:8759-­8768. doi:10.1109/CVPR.2018.00913 works for large-­ scale image recognition. arXiv preprint
27. Tan M, Pang R, Le QV. EfficientDet: scalable and efficient ob- arXiv:1409.1556; 2014.
ject detection. 2020 IEEE/CVF Conference on Computer Vision 34. Redmon J, Farhadi A. Yolov3: an incremental improvement.
and Pattern Recognition (CVPR), Computer Vision and Pattern arXiv preprint arXiv:1804.02767; 2018.
Recognition (CVPR), 2020 IEEE/CVF Conference on, CVPR;
2020:10778-­10787. doi:10.1109/CVPR4​2600.2020.01079
28. Woo S, Park J, Lee J-­ Y, Kweon IS. CBAM: convolutional How to cite this article: Meng Z, Xu S, Wang L,
block attention module. Computer Vision –­ECCV 2018 : 15th Gong Y, Zhang X, Zhao Y. Defect object detection
European Conference, Munich, Germany, September 8–­14, 2018, algorithm for electroluminescence image defects of
Proceedings, Part VII, 3; 2018. doi:10.1007/978-­3-­030-­01234​-­2_1
photovoltaic modules based on deep learning.
29. Zheng Z, Wang P, Liu W, Li J, Ye R, Ren D. Distance-­iou loss:
faster and better learning for bounding box regression. arXiv;
Energy Sci Eng. 2022;00:1–­14. doi:10.1002/ese3.1056
2019. doi:10.1609/aaai.v34i07.6999