Real-time 3D Video System Based on FPGA

Zan Li1,a , Xue Song Ye1,b and Hong Zhang1,c

Ling Lu1,d ,Chen Lu1,e and Li Cheng Cheng1,f
1
Department of Biomedical Engineering, Zhejiang University, Hang Zhou
a
lizangstc@[Link], byexuesong@[Link], czhangh@[Link]
d
geniuslu6@[Link], eluchen26@[Link], fclc15021206029@[Link]

Abstract—With the higher demand of the three dimensional chip. CMOS image sensors generate two-channel 8-bit video
(3D) imaging, a high definition real-time 3D video system data and control signals. DDR2 and FIFO are used to buffer
based on FPGA is proposed. The system is made up of CMOS large amount of video streaming. DDR2 accesses as time-
image sensors, DDR2 SDRAM, High Definition Multimedia sharing mode to match the different-frequency input and
Interface (HDMI) transmitter and Field Programmable Gate output data. How to realize 3D format by FPGA is the key of
Array (FPGA). CMOS image sensor produces digital video the 3D imaging technology. There are four kinds of common
streaming. DDR2 SDRAM buffers large amount of video data. 3D data format: Frame Sequential, Frame Packing, Side-by-
FPGA processes the video streaming and realizes 3D data Side (half/full) and Top-and-Bottom (half). Frame Sequential
format conversion. HDMI transmitter is utilized to transmit
and Frame Packing are introduced due to their popularization.
3D format data. Using the active 3D display device and shutter
glasses, the system can achieve the living effect of real-time 3D
HDMI transmitter receives the 3D format video data and
high definition imaging. The resolution of the system is transmits them to 3D display device. Qsys builder in Quartus
720p@60Hz in 3D mode. II is in charge of setting and controlling configuration of
CMOS image sensors and HDMI transmitter. The resolution
Keywords—CMOS image sensor; FPGA; HDMI; 3D HD of 3D monitor is 1920h1080. The system structure is shown
imaging in Fig.1.

I. INTRODUCTION
With the help of well-developed electronic technique, the
3D technology has got great development in the past few
decades. Compared with 2D imaging, 3D technology
provides depth information. In the field of display, 3D
technology can truly reproduce the original appearance of
world, giving us the immersive visual feast [1]. By
introducing 3D imaging technology into laparoscopy system,
the implement of surgical operation will be more accurate.
The surgical success rate and efficiency will be greatly
improved [2-4]. In addition to increasing the maximum
resolution and expanding support for color spaces, HDMI
1.4 standard adds 3D formats and even more features. With
high performance in embedded design domain, FPGA is a
powerful tool to process real-time video data [5-6]. The
Fig. 1. 3D video system structure
combination of FPGA and HDMI 1.4 standard gives us a
platform to design 3D HD video system.
This paper proposes a HD real-time 3D video system III. DESIGN OF SYSTEM MODULE
based on FPGA and HDMI interface, focusing on
introducing the design idea of FPGA control modules and A. CMOS/HDMI Settings Controller
HDMI interface circuit. Compared with traditional 3D video The controller core is established via the Qsys Builder in
system (PC + Video acquisition card), 3D video system Fig.1. It is built with a combination of the Nios II processor
based on FPGA is more flexible, reusable and scalable. The and I2C controller, which not only sets the CMOS image
system can be a feasible solution for embedded design of 3D sensor’s output (DVP mode), but also configures HDMI
video system. transmitter’s 3D mode. Nios II processor works as a
microcontroller. Using PIO of FPGA to simulate IIC’s
II. SYSTEM ARCHITECTURE timing, Nios II processor can finish easy controlling
The 3D video system consists of an ALTERA’s Cyclone functions. By setting different register parameters, CMOS
IV GX FPGA device, two OMNIVISION’s CMOS image image sensor and HDMI transmitter will work in several
sensors, a DDR2 chip and a HDMI 1.4 standard transmitter modes.

978-1-4799-2860-6/13/$31.00 ©2013 IEEE

469
B. DDR2 SDRAM and FIFO Control Module
Compared with DDR SDRAM, DDR2 SDRAM has the
ability of double data transmission. With FBGA package,
DDR2 owns better electrical properties and heat dissipation.
DDR2’s stable working frequency reached more than
200MHz. With high speed and large capacity, DDR2 is used
to buffer the real-time high-definition video streaming [7].
The model of DDR2 is MT47H32M16HR-25E. Its
capacity is 128MB and data width is 32-bit. DDR2 has 32
banks and each bank own 4MB capacity which can storage
1280 h 720 32-bit data. We use DDR2 controller to
accomplish power-on initialization, refreshing control signals,
transforming state machine, and producing read and write Fig. 3. DDR2 controller’s read-write state machine
control signals. The pixel clock of CMOS image sensor is
42MHz, while the DDR2 controller’s write frequency is
83.3MHz in half-rate mode. HDMI interface chip’s pixel C. Data Width and Color Adjustments
clock in 720p/3D mode is 148.5MHz while the DDR2 The data width of CMOS image sensor is 8-bit. The data
controller’s read frequency is 83.3MHz. It’s necessary to use width of HDMI interface is 24-bit. The data width of DDR2
FIFO to match different clock domains. FPGA generates is 32-bit. It’s necessary to use data width adjustment module
related control signals to operate in-out FIFO. In-out FIFO is to match different data widths. This module combines two-
used as the first level buffer device. Data stream will be channel 8-bit data and 16-bit empty data into 32-bit data. The
buffered by FIFO before DDR2. 0-7 bit data carries first channel signal while the 16-23 bit
Working in half-rate mode, the read and write data rate of data carries second channel signal. The 8-15 bit and 24-31
DDR2 controller is 1332.8MB/s. The total video data bit data is reserved for other uses. The aim of combining
throughput of the system is 760MB/s (<1332.8MB/s). The two-channel data together is to ensure the synchronization of
data throughput of DDR2 controller completely meets the two-channel signals when they are processed by FPGA in
requirement of the system’s data transmission. To complete following module.
the real-time data processing, DDR2 controller should To ensure the stability, trueness and smoothness of the
implement time-sharing read and write operations. CMOS 3D imaging effect, we realize Bayer conversion and Color
horizontal reference (CMOS_HREF) signal is divided into Look Up Table (CLUT) with the hardware description
two parts. The data read speed of HDMI interface is much language [8]. The 32-bit data from the output FIFO is
faster than the data write speed of CMOS image sensor. separated into two-channel 8-bit data and 16-bit empty data.
Therefore one-third of CMOS_HREF signal cycle is The 8-bit data is converted into 24-bit RGB data via Bayer
provided for DDR2 controller to complete write operation conversion. Each of R, G, B is 8-bit. The value ranges from 0
and two-thirds of CMOS_HREF signal cycle is provided for to 255. Color Look Up Table (CLUT) is added to process
DDR2 controller to complete read operation. The video RGB data. We can make color enhancement of RGB’s each
streaming flows from input FIFO to DDR2 when DDR2 component. The CLUT is carried out by dual-port RAM
controller enters into write mode and the input FIFO is not which is equipped with two independent sets of address, data
empty. It’s able to transmit data from DDR2 to output FIFO and control buses. How to access to the same storage cell by
when DDR2 controller enters read mode while the output different address buses is the key to complete the processing
FIFO is not full. DDR2 controller’s time-sharing reading and operation. The control module is shown in Fig.4:
writing operations buffer large amount of real-time video
data. The control diagram is shown in Fig.2. The DDR2
controller’s read-write state machine is described in Fig.3.

Fig. 4. Data width and color adjustments

D. Data pre-process and post-process

Data pre-process module finishes the synchronization of
Fig. 2. DDR2 and FIFO controller two-channel data and control signals. Due to different

470
transmission delay of two CMOS image sensors, the pixel in 3D format accurately, vertical and horizontal shift module
clocks of which are not exactly the same. Two-channel data is necessary. The relative position of two images determines
stream must be processed at the same time when control the quality of 3D imaging. By regulating one-channel data
signal is active. Several levels of D flip-flop register are enable signal and using Hsync, the position of first pixel in
utilized to put off the faster channel signal. each frame image can be adjusted conveniently. If the first
Data post-process module is used to make the imaging pixel arrives earlier, the image shifts left. If the first pixel
effect better [9]. It is implemented by FPGA and NIOSII arrives later, the image shifts right. To shift image at the
platform. Under different intensity of illumination light, the position of first pixel is the key to ensure the stability of the
image will lose its original color. To implement white image. The two 3D formats will be introduced in detail in the
following section:
balance process for data stream after Bayer conversion will
make the color of image more rich. The hardware is based 1) Frame sequential
on single-port RAM in FPGA. As a typical white balance HDMI interface module transmits two-channel data by
algorithm, grey world method is used to process RGB data. doubling the frame rate. The original 30fps speeds up to
The specific method is showed as follow: 60fps. This means that when vertical sync counts from 1 to
1ǃ Assign a mean value˄0̚255˅ K . 60, the odd number frame contains the image for the left eye
2ǃ Calculate the average of RGB each component : while the even number frame carries the image for the right
AveR , AveB and AveG , then calculate conversion eye. The left and right images are displayed alternately
when they transmitted to 3D monitor.
coefficient: 2) Frame packing
Q(rǃgǃb)=K / Ave( RǃGǃB )    HDMI interface module generates horizontal sync signal
3ǃ Calculate the transformed image˖ by counting pixel clock and vertical sync by counting
 imgਾ (rǃgǃb)= Q(rǃgǃb) ∗ imgࡽ (rǃgǃb)    horizontal sync signal. The left and right frame data will be
put into appropriate location according to vertical sync. The
first-channel data is transmitted to HDMI transmitter when
horizontal sync counts from 1 to 720 while the second-
channel data is transmitted when horizontal sync counts
from 751 to 1470. There are 30 lines for blanking period
which carries no video data. Fig.5 describes frame
sequential and frame packing 3D format. Fig.6 shows
HDMI interface module.

Fig. 5. Data pre-process and post-process module

E. HDMI Interface Module

The HDMI interface is a digital audio/video interface
technology with maximum data rate of 5Gbps, which can
transmit digital audio and video signals simultaneously. It
avoids signal attenuation in the transmission process without
Fig. 6. Frame sequential and Frame packing 3D format
A/D or D/A conversion before signal transmission [10].
Compared with the DVI interface, HDMI interface has
smaller volume, longer transmission distance (15meters) and
faster transmission speed [11]. We utilize sil9136 which
supports HDMI 1.4 standard as HDMI transmitter. The
HDMI interface module offers HDMI transmitter 3D format
data and control signals like pixel clock, horizontal sync,
vertical sync and data enable signals.
One of the primary 3D formats is Frame Sequential which
consists of a sequence of alternating frames wherein each
successive frame carries the image for the left or the right
eye. As for Frame packing, it’s the default format of the
HDMI1.4 3D video format structures which is composed of
two stereoscopic pictures: Left and Right. Compared to Side-
By-Side (Half) and Top-and-Bottom 3D format, the
advantage of frame Sequential and frame packing is that they
retain the content's original resolution for the best 3D Fig. 7. HDMI interface module
pictures [12]. In order to calibrate the position of video data

471
 IV. THE SYSTEM VERIFICATION V. CONCLUSION
The 3D video system is realized with CMOS image This paper outlines a HD real-time 3D video system
sensor㸪FPGA and HDMI interface circuit. We use Verilog based on FPGA and HDMI interface. It briefly introduces
HDL to design hardware module. We implement gate level the storage and transmission of video streaming, several
simulation of DDR2 controller timesharing operation with control modules in FPGA and two kinds of 3D format based
Modelsim 6.5, as shown in Fig.7. Using SignalTap to on the HDMI interface. The data stream generated by two
simulate the acquisition of two-channel video data and 3D CMOS image sensors is processed by FPGA and buffered by
display format. From fig.8, we can see that when line_count DDR2 SDRAM while HDMI interface module finishes the
is 520, HDMI module receives the first-channel data. When combination of two-channel data. The display results suggest
line_count is 1057, HDMI module receives the second- that the system achieves 720p60Hz display in 3D mode. This
channel data. It’s completely in conformity with the format system structure simplifies the circuit structure and achieves
of Frame packing. Fig.9 shows the result of 3D imaging. We the good imaging effect. According to the performance of
can see the two-channel video data is displayed in 3D mode FPGA, DDR2 and HDMI interface, 1080p in 3D mode can
clearly. The imaging effect of this system is also very good. also be realized. Compared with traditional video system,
The system occupies 20,067 LEs, 130 pins and this system has the advantage of lower cost, lower power
2,034,144 memory bits. The usage allowance of system consumption and smaller size. With high definition, real-
resource is still large, which will be easy to achieve 1080p in time and good 3D imaging effect, the system can be used in
3D mode. industrial and medical fields.

ACKNOWLEDGMENT
This work is supported by the National Key
Technology Research and Development Program of the
Ministry of Science and Technology of China under the
grant of NO. 2012BAI14B06.

REFERENCES
[1] Du Lei , Tang Wenlong , Ye Peng , “Autostereoscopy Image Display
System”, Hubei University Wuhan 430062, The 1st Asia-Pacific
Workshop on FPGA Applications,Xiamen,China,2012.
[2] Ameet Singh, Rupali Saraiya. Three-dimensional endoscopy in sinus
Fig. 8. The simulation of DDR2 controller surgery [J].Current Opinion in Otolaryngology & Head and Neck
Surgery, 2013, 21(1):3-10.
[3] Nicolas C. Buchs, Francesco Volonte, François Pugin, et al. Three-
dimensional laparoscopy: a step toward advanced surgical navigation
[J]. Surgical Endoscopy and Other Interventional Techniques, 2013,
27(2):692-693.
[4] Honeck PatrickWendt-Nordahl Gunnar Rassweiler Jenset al
Three-Dimensional Laparoscopic Imaging Improves Surgical
Performance on Standardized Ex-Vivo Laparoscopic Tasks >J@
Journal of Endourology2012, 26(8):1085-1088
[5] Mayer-Lindenberg, F, “High-Level FPGA Programming through
Mapping Process Networks to FPGA Resources”, International
 Conference on Reconfigurable Computing and FPGAs, 2009,pp. 302-
307.
Fig. 9. Timing simulation of two-channel data acquisition and [6] Liu Jin , Chen Liang , Liu Ying , Xie Yizhuang, “Design of
display spaceborne SAR imaging processing and fast Verification Based on
FPGA”, IET, Radar Conference 2013,pp.1-5.
[7] Xuzhi Wang , Yanru Ma , Wanggen Wan , “Design of DDR2
SDRAM controller for video post processing pipeline”, IET, 2009, pp.
117-120.
[8] Wang Chunping , Gao Xijun , Fu Qiang , “A design of video capture-
output and pre-processing system based on FPGA”, ICECC, 2011, pp.
21-24.
[9] Zhao Wenge , He Huiming , “FPGA-based video image processing
system research”, ICCSIT,2010,pp. 680-682.
[10] Lei Wu , Yang Eryan , Ronghui Cheng , “HDMI interface based vidio
post process platform design”, ICALIP, 2010, pp. 1724-1728.
[11] Kraemer, J.G., Shaw, J.A., “Achieving EMI compliance with DVI
and HDMI on defense/aerospace platforms”, EMC, 2009, IEEE, 200,
pp. 213-218.
[12] Polzlbauer, F. , Bate, I. , Brenner, E. , “Optimized Frame Packing for
Embedded Systems”, IEEE,2012, pp. 65-68.
Fig. 10. The result of 3D imaging

472