Nonlinear Unsharp Masking Algorithm

Dat Ngo, Seungmin Lee and Bongsoon Kang

Department of Electronics Engineering
Dong-A University
Busan, Korea
bongsoon@[Link]

Abstract—Image properties such as contrast, brightness, to the input image, therein lies the cause of its lack of
sharpness, and colorfulness exert a great influence on human generality. Histogram equalization (HE) could be considered as
perception of image quality. Thus, image enhancement has been an implicit image enhancement method, since the desired effect
assumed an important role in image analysis and computer vision. is achieved through rearranging the histogram of image
We propose an efficient algorithm for improving the image luminance. Even though HE exhibits good performance with
sharpness that automatically resolves the well-known out-of- low computational complexity, it suffers from one noticeable
range problem by the means of nonlinear transformation. shortcoming that large smooth image areas are usually over-
Adding to that, an optional contrast enhancement step could be enhanced. Several refinement approaches have been proposed,
applied concurrently to further enhance the image. The proposed
e.g. adaptive gamma correction with weighting distribution [4]
algorithm also reduces the halo effect by utilizing the modified
hybrid median filter in an iterative manner. A comparative study
and contrast-limited adaptive histogram equalization with dual
against other benchmarking methods and experimental results gamma correction [5]. Regarding the image enhancement
validate the performance of the proposed method. method carried out in the frequency domain, homomorphic
filtering (HF) [6] is a good example. Unlike conventional
Keywords—image enhancement; unsharp masking; contrast denoising/enhancement techniques that assume an additive
enhancement; nonlinear transformation; modified hybrid median noise model, HF works with the multiplicative model, also
filter known as the illumination-reflectance model of image
formation. Then, by simply applying the normalized high-pass
I. INTRODUCTION filter, the high-frequency noise could be easily removed.
Additionally, by further linearly transforming the high-pass
Image degradation rooted in a variety of external factors filter, the detail enhancement effect could also be achieved.
like weather phenomena, poor lighting conditions, and man- However, all the above-mentioned image processing methods
made aerosols (e.g. dust from industrialization) is inevitable in share a common out-of-range problem. They must be either
computer vision applications. For instance, a lack of incoming carefully parameterized or truncated to keep image values in
light to image sensors or the attenuation of light by the proper range, leading to the lack of generality and possibly
transmission medium is a common source of minute details unwanted color artifacts.
fading. This kind of image degradation is highly likely to make
an adverse impact on image processing algorithms designed for In this paper, we propose an algorithm that automatically
ideal environmental conditions. A prime example of this is the prevents the out-of-range problem by making the clever use of
sharp decrease of recognition rate of deep learning systems due the logarithm function. In addition, an optional contrast
to weather conditions such as fog, snow, and rain. Hence, enhancement technique could be applied in a concurrent
image enhancement algorithms focusing on the sharpness and manner to improve both sharpness and contrast of images. We
contrast of images have many practical applications. There has also reduce the possible halo artifact by using the iterative
been unceasing scientific effort put into the development of modified hybrid median filter. The rest of the paper is
new algorithms. organized as follows. Section II describes the theories of
sharpness enhancement and nonlinear approach to unsharp
Image enhancement algorithms are generally classified into masking. Section III details the proposed algorithm. Section IV
three main categories: nonlinear transfer function-based, provides a thorough evaluation, while Section V concludes the
histogram-based, and frequency domain methods [1]. Due to paper.
their low computational complexity and easy adjustment,
nonlinear transfer functions are widely used for image contrast
enhancement. Low-light image enhancement and gamma II. FUNDAMENTALS
correction are cases in point. The simple light stretch algorithm
[2] solely invokes simple arithmetic operations, giving rise to A. Sharpness enhancement
the compact and fast hardware implementation [3] that is Equation (1) describes the equation for unsharp masking
highly appropriate to real-time surveillance systems. However, algorithm, where z is the enhanced image, y refers to the
this algorithm requires to be manually parameterized according background of the input image x, and d denotes the detail

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.
information. In general, the background y is obtained using a III. PROPOSED ALGORITHM
low-pass filter, and then detail information d is calculated as We consider an N-bit image x, where the pixel gray scale
the subtraction of the background from the input image, d = x – value is within [0, 2N-1]. The input image must be normalized
y. The positive gain α is employed as a scaling factor to control to the range (-1, 1) of the proposed nonlinear function shown in
the amplitude of detail information. As the signal d may (5). This is achieved by first scale the image by 1/(2N-1) such
contain noise together with high frequency information, α must that it is in the range [0, 1]. Then the two extreme values are
be carefully adjusted so that unsharp masking techniques only replaced by their rightmost and leftmost values to change the
enhance the desired image details. range to (0, 1). Finally, the linear transformation (x ≔ 2x – 1)
is applied so that input image x is now in the desired range (-1,
z=y+d () 1).

Another problem possibly arising in unsharp masking Φ(x) = log[(1 + x)/(1 – x)], x ∈ (-1, 1) ()
techniques is the occurrence of visually unpleasant halo
artifacts. This is traced back to the background extraction from The proposed unsharp masking algorithm follows (4) to
input images. The use of a low-pass filter may create an adaptively enhance the image details. The block diagram of the
unwanted by-product of smoothing image edges, resulting in entire proposed approach is depicted in Fig. 2, and each step is
over-shoots and under-shoots in areas of sharp edges in the thoroughly described in following subsections.
signal d. Thus, the enhancement of over-shoots and under-
shoots creates halo artifacts. To resolve these two shortcomings
of unsharp masking techniques, edge-preserving filters and
adaptive gain control have been taken into consideration.

B. Nonlinear approach to unsharp masking

Following the idea presented in [7], usual operations like
addition and multiplication could be generalized to tackle the
out-of-range problem. Fig. 1 shows this kind of generalized
system, where Φ denotes a nonlinear function. The generalized
addition and scalar multiplication operations denoted by ⊕
and ⊗ are defined in (2) and (3), where x and y are signal
samples, and α is a real scalar.

x ⊕ y = Φ-1[Φ(x) + Φ(y)] () Fig. 2. Block diagram of the proposed nonlinear unsharp masking
algorithm.
 ⊗ x = Φ-1[Φ(x)] ()
A. Background signal extraction
The above-mentioned sharpness enhancement could also be Deng utilized the standard median filter (SMF) in an
expressed in terms of generalized operations, as shown in (4). iterative manner to extract the background signal [7]. This
To maintain a high level of generality, the function h(y) could process terminates when the mean squared difference of
be either linear or nonlinear, indicating that it is possible to images between two successive iterations falls below a pre-
perform an optional operation on the background signal y. The determined threshold. SMF functions properly in the smooth
adaptive gain α(d) is a function of the detail signal d for image areas but it may smooth down the sharp edges in image
selectively enhancing the image edges. areas containing abrupt changes like objects’ outline. Applying
SMF iteratively could result in halo artifacts as discussed in
z = h(y) ⊕ [(d) ⊗ d] () Section II-A. Hence, we proposed using the modified hybrid
median filter (mHMF) in place of SMF. mHMF was proposed
Deng discovered a connection between the Bregman in [9] for the accuracy estimation of haze distribution in the
divergence and the generalized system, providing useful hazy image. mHMF first computes the medians of three
insights into the development of such a system [7]. Therefore, a windows: square, cross, and diagonal. Then, the median of
general rule for selecting the nonlinear function Φ was these three is selected as the final result that would replace the
presented in [8], which stated that Φ must be strictly convex center pixel of filtering window. Accordingly, in the smooth
and differentiable. image areas, mHMF behaves in a similar manner to SMF, but
in the abrupt image areas, the information from cross and
diagonal windows helps mHMF to better preserve image edges.
In order to assess the performance of mHMF and SMF, we
apply these filters to the same input image 20 times. The
window size and threshold are set to 5×5 and 1.5×10-4,
Fig. 1. Block diagram of a generalized system. respectively. The experiment was conducted in MATLAB

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.
R2019a on a Core i7-6300 CPU (3.4GHz) with 32GB RAM. α(1) = α(–1) = αMIN, and η is a parameter controlling the
The result in Fig. 3 shows that using mHMF makes the mean decreasing rate of α(d).
squared difference fall below the threshold more quickly.
Adding to that, image data at line 185 of the input image were
also plotted in Fig. 4 to prove that the mHMF preserves edges α(d) = β + γ×exp(– |d|η) ()
better than SMF. Thus, the iterative mHMF is used in the
proposed algorithm to extract the background information from To enhance the image details, the gain must be greater than
the input image. or equal to one. Accordingly, αMIN is set to one. Regarding the
values of αMAX, we show the mapping function αMAX⊗d by
B. Adaptive gain control setting αMIN = 1 in Fig. 5. It can be seen that the large value of
αMAX leads to over-enhancement of small details and saturation
To develop an adaptive gain function, it is necessary to of sharp edges. Thus, αMAX is set to five in this paper. By setting
understand which component in the image details d must be αMIN = 1 and αMAX = 5, we plot the mapping function α(d)⊗d
enhanced. As mentioned in Section II-A, the detail signal d with η varies from 0 to 3. In order to avoid saturation, η could
contains: 1) image edges, 2) noise, and 3) over-shoots and be set to 0.5 or 1.
under-shoots. The use of the iterative mHMF presented in the
preceding section reduces the number of over-shoots and
under-shoots. Additionally, since out-of-range problem is C. Optional contrast enhancement
automatically resolved by the means of nonlinear In traditional image processing algorithms, sharpness and
transformation, the effect of high-frequency noise is contrast enhancements could not achieved simultaneously. In
insignificant. Hence, a simple form of adaptive gain function generalized schemes, since the input image is separated into
could be adopted. In this paper, we utilize an exponential detail and background signals, unsharp masking and contrast
function that gradually decrease from its maximum value αMAX enhancement algorithms can be applied to corresponding
to its minimum value αMIN. The mathematical equation of components concurrently. This is motivated by Deng’s work in
adaptive gain α(d) is shown in (6), where β and γ are two [7]. In this paper, we utilize the well-known contrast-limited
parameters obtained by solving the equations: α(0) = αMAX and adaptive histogram equalization (CLAHE).

Fig. 3. Mean squared difference of images between two consecutive Fig. 5. Illustrations of mapping function αMAX⊗d for different values of
iterations for two cases: using SMF and using mHMF. αMAX.

Fig. 4. Plot of image line 185 data to compare edge-preserving Fig. 6. Illustrations of mapping function α(d)⊗d for different values of
characteristics of SMF and mHMF. η.

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.
 Fig. 7. Illustration of the proposed algorithm’s automatic out-of-range prevention characteristic. Top left: input image line data, top right: result of traditional
unsharp masking with constant gain, bottom left: result of recently-proposed algorithm using staircase gain, and bottom right: result of the proposed method
with simple constant gain.

determining the improvement of visibility level. Thus, higher

IV. EVALUATION values of e and r are desired in image enhancement approaches.
A. Automatic out-of-range prevention
TABLE I. AVERAGE QUANTITATIVE SCORES ON IVC IMAGE DATASET
To illustrate that the proposed algorithm is capable of
automatically preventing out-of-range problem, Fig. 7 shows Image Constant gain Staircase gain Proposed method
an input image line data and its corresponding enhanced results No. e r e r e r
of the traditional unsharp masking with constant gain, the 1 0.026 1.926 0.053 2.232 0.569 2.556
recently-proposed algorithm with staircase gain [10], and the 2 0.427 1.738 0.539 2.046 0.899 2.546
proposed method with simple constant gain in lieu of adaptive 3 0.639 1.819 0.749 2.214 1.005 3.515
gain. For fair assessment, the gain is set to 3 for methods with 4 0.467 1.872 0.553 2.230 0.675 2.648
5 0.078 1.851 0.027 1.970 0.283 2.041
constant gain. In case of the recent method with staircase gain,
6 0.672 1.765 0.791 2.163 1.266 3.100
three values of gain are set to 1, 2, and 3. It can be seen that the
7 1.045 1.742 1.302 2.071 1.318 2.724
out-of-range problem is most severe in the top right result of 8 0.984 1.495 1.446 1.794 2.686 2.959
Fig. 7, since a constant gain is applied equally to all image 9 0.408 1.695 0.600 1.904 1.017 3.043
details. In the bottom left result, this problem is alleviated to a 10 1.718 1.941 2.083 2.394 2.464 3.098
certain extent, since three values of gain are assigned according 11 0.926 1.851 1.649 2.290 2.512 3.790
to the local variance of image. In the bottom right result, the 12 0.676 1.825 0.726 2.222 1.001 2.150
proposed algorithm effectively enhances the input data while 13 0.345 1.760 0.482 2.020 0.630 2.664
preventing it varying outside the normalized range. 14 0.022 2.009 0.075 2.354 0.953 2.591
15 0.690 2.029 0.928 2.483 1.527 4.329
B. Quantitative assessment 16 0.759 1.663 1.073 1.979 1.430 2.790
17 0.038 2.039 0.006 2.287 0.512 2.622
The rate of new visible edges (e) and the quality of the 18 0.356 1.862 0.451 2.201 0.638 2.464
contrast restoration (r) [11] are used to evaluate the 19 0.237 1.857 0.258 2.215 0.539 3.066
performance of three methods mentioned in the previous 20 3.586 1.985 4.249 2.480 5.263 2.879
section, where the proposed algorithm now utilizes the 21 1.635 2.037 1.893 2.485 1.920 2.630
adaptive gain instead of simple constant gain. In both 22 1.541 1.600 2.185 1.916 2.159 2.796
subsections, we do not apply the optional CLAHE. Metric e 23 1.162 1.826 1.694 2.227 1.420 2.726
assesses the ability of an algorithm to recover edges that are 24 0.486 1.564 0.658 1.770 0.728 2.190
invisible in the original image, but that are visible in the 25 0.635 1.495 1.170 1.791 2.954 3.817
restored image. In addition, metric r denotes the ratio Avg. 0.771 1.810 1.023 2.150 1.455 2.870

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.
 Fig. 8. Visual assessment of the proposed algorithm with benchmarking methods. From left to right: input image, results and their corresponding zoomed-up
regions of the traditional unsharp masking with constant gain, the recent-proposed method with staircase gain, and the proposed nonlinear approach.

Table I shows these two metrics for 25 images in the IVC

image dataset [12]. The best result is marked bold. It is evident REFERENCES
that the proposed algorithm possesses the best enhancement [1] L. Artur, R. B. David, R. H. Paul, and M. A. Alin, “Automatic contrast
power under e and r metrics. Fig. 8 presents a scene faded by enhancement of low-light images based on local statistics of wavelet
coefficients,” Digital Signal Processing, vol. 23, no. 6, pp. 1856–1866,
thin haze to visually assess the performance of three algorithms. Dec. 2013.
The red cropped regions show that the proposed nonlinear [2] D. Ngo, S. Lee, and B. Kang, “Light Stretch Algorithm for Image
unsharp masking method produces the most visually- Quality Enhancement,” ICVR 2018 Proceedings of the 4th International
satisfactory result in comparison to the other two Conference on Virtual Reality, pp. 56–60.
benchmarking algorithms. [3] D. Ngo and B. Kang, “Preprocessing for High Quality Real-time
Imaging Systems by Low-light Stretch Algorithm,”
[Link]., vol. 22, no. 3, pp. 585–589, Sep. 2018.
V. CONCLUSIONS
[4] S.-C. Huang, F.-C. Cheng, and Y.-S. Chiu, “Efficient Contrast
In this paper, a nonlinear unsharp masking algorithm is Enhancement Using Adaptive Gamma Correction With Weighting
presented. Image enhancement approaches that operate directly Distribution,” IEEE Trans. on Image Process., vol. 22, no. 3, pp. 1032–
1041, Mar. 2013.
on image gray scale value may suffer from out-of-range
[5] Y. Chang, C. Jung, P. Ke, H. Song, and J. Hwang, “Automatic Contrast-
problem and produce visually unpleasant halo artifacts. The Limited Adaptive Histogram Equalization With Dual Gamma
proposed algorithm automatically gets rid of out-of-range issue Correction,” IEEE Access, vol. 6, pp. 11782–11792, 2018.
by the means of nonlinear transformation. Adding to that, halo [6] W. A. Mustafa, W. Khairunizam, H. Yazid, Z. Ibrahim, A. Shahriman,
artifacts are also reduced by clever use of modified hybrid and Z. M. Razlan, “Image Correction Based on Homomorphic Filtering
median filter in an iterative manner to decompose the input Approaches: A Study,” in 2018 International Conference on
image into detail and background signals. Experimental results Computational Approach in Smart Systems Design and Applications
(ICASSDA), Kuching, 2018, pp. 1–5.
shown that the proposed method is superior to benchmarking
algorithms in terms of both visual quality and quantitative [7] Guang Deng, “A Generalized Unsharp Masking Algorithm,” IEEE
Trans. on Image Process., vol. 20, no. 5, pp. 1249–1261, May 2011.
metrics.
[8] F. Nielsen and R. Nock, “Sided and Symmetrized Bregman Centroids,”
IEEE Trans. Inform. Theory, vol. 55, no. 6, pp. 2882–2904, Jun. 2009.
ACKNOWLEDGMENT [9] G. Kim, S. Lee, and B. Kang, “Single Image Haze Removal using Hazy
Particle Maps,” IEICE Transactions on Fundamentals of Electronics,
This research was supported by the BK21 Plus Program Communications and Computer Sciences, vol. E101-A, no. 11, pp.
(Future-oriented innovative brain raising type, 1999–2002, Nov. 2018.
22A20130000047) funded by the Ministry of Education (MOE, [10] D. Ngo and B. Kang, “Image Detail Enhancement via Constant-Time
Korea) and National Research Foundation of Korea (NRF). Unsharp Masking,” unpublished.

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.
[11] N. Hautière, J.-P. Tarel, D. Aubert, and É. Dumont, “BLIND [12] K. Ma, W. Liu, and Z. Wang, “Perceptual evaluation of single image
CONTRAST ENHANCEMENT ASSESSMENT BY GRADIENT dehazing algorithms,” in 2015 IEEE International Conference on Image
RATIOING AT VISIBLE EDGES,” Image Analysis & Stereology, vol. Processing (ICIP), 2015, pp. 3600–3604.
27, no. 2, pp. 87–95, May 2011.

Authorized licensed use limited to: DONG A UNIVERSITY. Downloaded on December 22,2023 at 01:30:27 UTC from IEEE Xplore. Restrictions apply.