2013 Seventh International Conference on Image and Graphics

A Real-Time Full HD 2D-to-3D Video Conversion System Based on FPGA

Shao-Jun Yao1, 2, Liang-Hao Wang1, 2, Dong-Xiao Li1, 2, Ming Zhang1, 2

1
Institute of Information and Communication Engineering,
Zhejiang University, Hangzhou 310027, China
2
Zhejiang Provincial Key Laboratory of Information Network Technology,
Hangzhou 310027, China
E-mail: y22412@[Link]

In a typical 2D-to-3D conversion system, 2D video is

Abstract converted to 3D video by using the produced depth maps,
which is automatically generated from 2D video with depth
The 2D to 3D conversion technique plays an important cues. Numbers of automatic 2D to 3D conversion
role in the development and promotion of algorithms have been proposed so far [1]-[7]. However,
three-dimensional television (3DTV), because it can accurate depth maps are difficult to recover even with high
provide adequate supply of high-quality 3D program complexity algorithms because generating depth maps
content. This Paper focuses on the real-time from 2D video is an ill-posed problem.
implementation of 2D-3D system, of which the major parts For 3D consumer electronics devices, real-time
are source depth-map generation and depth image-based conversion is required. However, either the depth
rendering (DIBR). The depth-map is generated by generation with multi-depth cues, or DIBR is
multi-depth cues, according to the types of the 2D scene, time-consuming algorithm. They both need to be
including luminance and color, visual saliency and accelerated in some way to meet the real-time requirement.
adaptive depth models. Then the original 2D image or In this paper, a real-time full HD 2D-to-3D system is
video is converted to stereoscopic one by DIBR with the proposed, which is accelerated with new hardware
depth map. The proposed system is accelerated by being architecture on FPGA.
designed in new hardware architecture on FPGA. Our This paper will firstly take an overview of our proposed
method is verified on various sequences and experiment system in Section 2. Section 3 describes our algorithms and
results show that, the system can achieve 30 frames per Section 4 analyzes the hardware implementation. Finally,
second for processing full HD (1920×1080) video on Xilinx experiment results will be shown in Section 5 and a
Virtex IV XC4VSX25 while the resulting image or video is conclusion will be given in Section 6.
both realistic and visual pleasing.
2. Overview of our system
As shown in Figure 1, the proposed system contains
1. Introduction three components, source depth-map generation, virtual
In recent years, 3DTV has been a hot topic because it depth-map rendering and virtual view rendering.
provides viewers with feeling of immersion. But as the For source depth-map generation, firstly, which depth
development of 3D display technology, the growing model to choose is determined according to the types of the
demand for 3D content becomes a problem. As we know, 2D scene. Then the data value of luminance and color is
there exits several ways to make 3D content, like active converted to a factor of 0-1, which is used to refine depth
depth sensing, stereoscopic cameras recording and 3D model. Last we fuse the refined depth model and visual
graphics rendering. However, these methods are both saliency with appropriate weights to generate the source
costly and time-consuming and only suitable for new depth-map.
content production, so the shortage of 3D content becomes In the process of virtual view rendering, we use the
one of severe bottlenecks for 3D industry. In this case, an method of reverse DIBR to render virtual view images,
effective automatic 2D-to-3D conversion system is which can avoid the problem of hole-filling. So the virtual
necessary, because it not only can produce 3D content with depth-map should be generated from the source one before
lower cost and less time, but also makes full use of large reverse DIBR executes. Then the original 2D image or
amount of existing 2D content. video is converted to stereoscopic one, which can be

978-0-7695-5050-3/13 $26.00 © 2013 IEEE 787

786
774
DOI 10.1109/ICIG.2013.157
presented on a 3D display. visual saliency and depth models. Depth-image-based
For real-time consideration, the proposed system rendering (DIBR) is used to generate the virtual views of a
described above is designed in new hardware architecture 3D scene from color images and associated per-pixel depth
on FPGA, where the pipeline architecture is used. Besides, information.
to avoid floating-point operation, all floating-point
parameters are rounding after appropriate amplification 3.1. Source depth-map generation
during the computation.
In our system, image data is processed row by row and The method of source depth-map generation consists of
pixels within a row are processed one by one instead of a the following three parts:
parallel processing. Since the clock sequence of the system 1) Which depth model to choose is determined by the
ensures the requirement of real-time, this approach can types of the 2D scene. As shown in Figure 2, three depth
reduce the consumption of system resources. models can be used [8]. According to the human visual
perception feature, we apply near-to-far depth model as the
major cue in most cases.
2) For each pixel, data value of luminance and color is
Decoded 2D video
converted to a factor of 0-1, which is used to refine depth
model.
The conversion function in [9] is described as follows in
(1):
Depth model DR [1D(Y 128) /128]u[1 E(Cr 128) /128]
determined
Visual saliency (1)
computition
u[J (128 Cb) /128 1]
Depth model Where the value of and are about 0.3-0.1, and is about
refinement 0.3-0.5 in empirical rule.
This conversion is based on two characteristics. Firstly,
the edge in the depth map is usually the edge of the input
Depth fusion image. Secondly, in human visual perception warm color is
likely to be nearer than cold color. Moreover, objects with
Source depth-map generation higher luminance are likely to be nearer than those with
lower luminance. The detail about this part can be viewed
in paper [9].
3) We fuse the refined depth model and visual saliency
Depth Image Based Rendering with appropriate weights to generate the source depth-map.
In [6], a good measurement of visual saliency is
presented. Its measurement of visual saliency combines the
Virtual depth-map rendering color saliency and discrete moment transform (DMT)
together to get a final saliency map. The calculation of
color saliency is shown in (2).
1 1 (2)
Reverse Depth Image Based Rendering sc (i, j ) ˜
s (i, j ) v(i, j )
cs  exp(  ) cv  exp(  )
ms mv
Virtual image rendering Where cs and cv are two constants.
The DMT at location (i,j) of image f is defined as:
k k
3D display device dmt x , y (i, j ) ¦ ¦ q(i  r , j  s)(i  r ) ( j  s)
r ks k
x y (3)

According to the implementation of visual saliency in [6],

Figure 1: Framework of the proposed 2D-3D system. data value of visual saliency range from 0 to 165. As the
refined depth model range from 0 to 255, we amplify the
visual saliency to 0-255 before fusing them. Weights can be
3. Algorithm adjusted to meet the requirement of source depth-map. As
The core algorithms of our 2D-3D system are source shown in Figure 3, source depth-map becomes sharper with
depth-map generation and DIBR. The former one detects a greater weight for visual saliency, otherwise becomes
three main depth cues, including luminance and color, smoother.

775
787
788
 3.2. DIBR
Depth-image-based rendering (DIBR) is used to
generate the virtual views of a 3D scene from color images
and associated per-pixel depth information [10].
Shift-sensor camera model is used in our system, which
means there only exits horizontal disparity among images
of various views. Then as shown in Figure 4, for example, p
(a) Depth model one
is a point of interest in 3D scene, cs and cv are the source
view and virtual view in our system, xs and xv are the
horizontal pixel coordinates of related views.

(b) Depth model two

xs xv

cs cv
(c) Depth model three
Figure 2: Three depth models. Figure 4: Top-view of shift-sensor camera model.
The projection of DIBR can be expressed as follows in
(4):
fb
xs  xv (4)
z
Where f specify the focus of camera, b specify the
baseline distance between two views, z specify the distance
between point p and cameras.
Reverse depth-image-based rendering (R-DIBR) is the
(a)
same as DIBR except it is based on virtual view depth-map.
Since each pixel in virtual view image is mapped to one in
source view, no hole will exit in the virtual view image.
In our system, DIBR is used to render the virtual
depth-map while R-DIBR is used to render the virtual view
image.

4. Hardware architecture
(b)
As shown in Figure 5, the proposed hardware
architecture contains three major work module, one control
module and four blocks of RAM. Computation speed and
resource cost are the major concern.
The entire flow is started from loading one source image
from video-in interface. Because source depth-map
generation is single-pixel based and no vertical disparity
(c) exits in DIBR, source image data is processed row by row
Figure 3: Comparison between two source depth-maps of image with the method mentioned in section 2. Afterwards, the
(a): (b) Weight is 0.1 for refined depth model and 0.9 for visual stereoscopic image is output to show on 3D display devices
saliency. (c) Weight is 0.9 for refined depth model and 0.1 for by video-out interface.
visual saliency.

776
788
789
 Dmax  3

Dmax  2

Dmax  1

Figure 6: Pipeline work of 2D-3D system.

S0 S Dmax SnDmax Sn Sn Dmax

V0 Vn
Figure 5: Overall proposed hardware architecture.
Figure 7: Description of rendering process.
To meet real-time requirement, pipeline architecture is
used in system. As shown in Figure 6, the circuit within a 5. Experimental results
row starts at processing cycle 1, where SDMG (source The proposed hardware architecture of 2D-3D system is
depth-map generation) is executed for pixel 0. Continuing implemented by using Verilog. The implementation is
at cycle 2, VDMR (virtual depth-map rendering) starts verified with RTL simulations using ModelSim Xilinx. The
working from the beginning pixel. Then at cycle Verilog code is synthesized to a VPM6467 Xilinx Virtex
( Dmax  3 ), VVR (virtual view rendering) starts since the IV FPGA. The resulting netlist is placed and routed to the
first depth value of virtual depth-map is determined. This same FPGA at 95.515MHz using Xilinx ISE 10.1. Table 1
pipeline architecture ensures one pixel data out per clock shows the result from the Xilinx synthesis tool of what
cycle in our system. Since our system clock frequency is resources is used for a device. The number of Slice is 1537,
fast enough, we can achieve real-time requirement. and the maximum frequency is 95.515MHz. The number of
on-chip block RAMs is 10 with 180Kbs in total. The
The Dmax mentioned above specify the Maximum
system can achieve real-time requirement for full HD
disparity between source and virtual images. Figure 7 (1920×1080) video with a 30fps at 74.25MHz.
explains the reason why VVR has a Dmax  1 cycles delay
Table 1: FPGA device utilization specifications.
than VDMG. Here Sn is the pixel of source depth-map and
Vn is that of virtual depth-map. We suppose Sn is mapped
to Vn ' , then | n  n ' | is the disparity which must not be
bigger than Dmax . So for Vn , the possible pixels which
can be mapped to it are S n  Dmax to S n  Dmax . But for n <
The 2D-3D system is programmed in C language by
Dmax , like n = 0, only S0 to S Dmax exist. Therefore, the
using Visual Studio 2008 in order to compare the
data value of V0 cannot be determined until VVR is performance with the proposed hardware architecture. We
use Intel core2 duo 3.16GHz with 2GB RAM. The
already processing on S Dmax 1 . comparing result is shown in Table 2, the proposed
Meanwhile we can get another conclusion from Figure 7: hardware architecture is 15x faster than CPU by frame rate.
at one cycle, the number of effective pixels for DIBR or
R-DIBR is 2 Dmax  1 . So the memory depth of related Table 2: Performance comparison.

data RAM is only need to be bigger than 2 Dmax  1 ,

which can save a lot of memory resources.

777
789
790
 The result of stereoscopic images are shown in Figure 8, and Technology Innovation Team of Zhejiang Province,
where Figure 8(a) are source 2D images in the 1920×1080 China (Grant No. 2009R50003) and China Postdoctoral
resolution and Figure 8(b) are the stereoscopic images of Science Foundation (Grant No. 20110491804,
resolution 1920×1080 generated by the hardware method 2012T50545).
are shown in Figure 8(c).
References
[1] J. Ens and P. Lawrence, “An investigation of methods of
determining depth from focus,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 15, no. 2, pp. 523–531, 1993.
[2] S. A. Valencia and R. M. Rodriguez- Dagnino,
“Synthesizing stereo 3D views from focus cues in
monoscopic 2D images,” in Proc. SPIE, 2003, vol. 5006, pp.
377–388.
[3] S. Battiato, S. Curti, M. La Cascia, M. Tortora, and E.
Scordato, “Depth map generation by image classification,”
in Proc. SPIE, Apr. 2004, vol. 5302, pp. 95–104.
[4] X. Huang, [Link], J. Huang, D. Li, and M. Zhang, “A depth
(a) extraction method based on motion and geometry for 2D to
3D conversion”, in 3rd Int. Symp. Intell. Inf. Technol. Appl.,
2009, pp. 294–298.
[5] K. Yamada and Y. Suzuki, “Real-time 2D-to-3D conversion
at full HD1080P resolution”, the 13th IEEE International
Symposium on Consumer Electronics, 2009, pp.103-107.
[6] C. Huang, Q. Liu and S. Yu, “Regions of interest extraction
from color image based on visual saliency”, Springer
Science Business Media, 2010.
[7] E. Imre, S. Knorr, A. A. Alatan, and T. Sikora, “Prioritized
sequential 3D reconstruction in video sequences of dynamic
scenes,” in IEEE [Link]. Image Process. (ICIP), Atlanta,
GA, 2006.
(b) [8] Pan Ji, Dong-Xiao Li, Ming Zhang, “An Automatic 2D to 3D
Figure 9: Result of the proposed system Conversion Algorithm Using Multi-depth Cues”, Audio,
(a)Source 2D images Language and Image Processing (ICALIP), 2012 3rd
(b)Stereoscopic images International Conference on, pp.546-550.
[9] Chao-Chung Cheng, Chung-Te Li, Liang-Gee Chen, “Video
6. Conclusion 2-Dütoü3-D conversion based on hybrid depth cueing”,
Journal of the Society for Information Display, 2010,
This paper proposed an end-to-end 2D-3D system with
pp.704-716.
real-time performance, in which source depth-map [10] Fehn, C., “A 3D-TV system based on video plus depth
generation and DIBR are accelerated by FPGA. Compared information”, Thirty-Seventh Asilomar Conference on
with the CPU-C based platform, the performance of this Signals, Systems and Computers, 2003, pp.1529-1533.
hardware architecture can achieve 30fps HD which is 15x
faster.
Future work will focus on further accelerating this
end-to-end system. For FPGA implementation, timing
optimization will be done to improve the frequency of
system clock.

Acknowledgment
This work was supported in part by the National Natural
Science Foundation of China (Grant No. 61072081,
61271338), the National High Technology Research and
Development Program (863) of China (Grant No.
2012AA011505), the National Science and Technology
Major Project of the Ministry of Science and Technology
of China (Grant No. 2009ZX01033-001-007), Key Science

778
790
791