400V DC Distribution for Data Centers
400V DC Distribution for Data Centers
Abstract- In a typical data center, less than half the energy 400V and 550V. Power delivery efficiency is evaluated over a
consumed is delivered to the compute load, with the rest lost in range of loads. This is important because converters in the
power conversion, distribution and cooling. Traditionally power
distribution is at 400/480V AC in data centers and at 48V DC in power delivery train typically operate at less than half of their
telco facilities. Higher voltage DC has been proposed as an rated load, where conversion efficiency is much lower than at
energy efficient distribution option for both types of facilities, and heavy loads.
this paper presents an analytical evaluation of several data center Through an analysis of several power delivery architectures,
power delivery architectures over a range of loads, showing that a this paper shows that facility-level 400V DC distribution
400V facility-level DC distribution option is the most efficient. The provides increased energy efficiency for data and telco centers
equipment required to support such an architecture is discussed
and results from a small scale demonstration are included. over a wide load range. This is supported by measurements
from a small scale demonstration of AC and 400V DC
I. INTRODUCTION distribution.
Data and telco centers are major energy consumers, with ener- II. POWER DISTRIBUTION OPTIONS
gy consumption estimated at 40 TWh in 2005 in the United Existing and proposed power distribution architectures may
States alone, and 120 TWh world wide [1]. A very large data be divided into three broad categories: AC, rack-level DC and
center requires on the order of 10 MW of power to support the facility-level DC [9].
computing infrastructure and this is expected to increase to A. AC
50MW in the future. In a typical data center, less than half of In the United States, the medium voltage (MV) from the
this power is delivered to the compute load, which includes mi- utility is stepped down to 480Vac at the building entrance, as
croprocessors, memory and disk drives. The rest of the power is shown in Figure 1. Typically a double conversion
lost in power conversion, distribution, and cooling, resulting in
Uninterruptible Power Supply (UPS) is used which supplies
high utility bills, a large environmental footprint and the inabi-
power to a Power Distribution Unit (PDU) where the voltage
lity to fill equipment racks.
is stepped down to 208Vac. This step-down is required
It has also been shown that the capital cost of the power delive-
because the Power Supply Units (PSUs) in the servers can
ry and cooling infrastructure required for a data center already
accept a universal input range of 90 – 264Vac and the line-to-
exceeds the purchase price of the servers it will support, and it
neutral voltage in a 480Vac distribution system is 277Vac,
is expected that the life time energy costs of a server will in the
which exceeds the input range of the PSU.
future also exceed its purchase price [2]. This will bring about a
This conventional power delivery architecture is burdened
significant change in the economics of operating a data centers,
with many conversion stages. Inside of the double conversion
as server hardware is no longer the primary cost driver in a data
UPS, the AC input is converted to a DC voltage at which point
center.
it is connected to an energy storage system, typically a battery,
Traditionally power distribution is at 400/480V AC in data
and then it is inverted to AC again. Oftentimes an isolation
centers and at 48V DC in telco facilities. The use of higher
transformer is included at the input or output of the UPS.
voltage DC as an energy efficient distribution option for both
Within the PSU, the AC input is converted to a DC voltage
types of facilities, which sacrifices little in the way of
ranging from 380V to 410V. This DC voltage is stepped down
reliability compared to low voltage DC distribution, has been
with an isolated DC/DC converter to a Safe Extra Low
discussed for almost a decade [3, 4, 5, 6, 7]. Its potential to
Voltage (SELV), typically 12Vdc or 48Vdc. Distribution at
improve utilization of distributed renewable generation [8] and
12V is assumed for this paper. Some loads, such as hard
its lower component cost [4] has also been pointed out.
drives, can accept 12V directly while others, such as
Expanding upon previously published studies, this paper will
microprocessors and memory, need voltage regulators (VRs)
present an evaluation of AC power distribution with different
to step the voltage down further. Server fans also run directly
Uninterruptible Power Supplies (UPSs) as well as rack and fa-
off 12V.
cility-level DC distribution at voltage levels of 48V,
Bypass
12V
480V 208V 400V DC/DC
AC/DC DC/AC
MV AC AC AC/DC DC VR
PSU
Server
UPS PDU
Rack
Bypass
12V
480V 400V DC/DC
AC/DC DC/AC 230V
MV (400V) AC/DC DC VR
AC
AC
480V PSU
only Server
UPS
PDU Rack
Figure 2 400Vac : Conventional in Europe with 400Vac input; proposed for US with 480Vac input
12V
480V 208V 400V DC/DC
MV (400V) (230V) AC/DC DC VR
AC 480V AC
AC/DC DC/AC PSU
only Server
PDU Rack
UPS
Figure 3 Specialized (Delta-conversion) AC UPS; with transformer in PDU in US (480Vac input) and without transformer in Europe (400Vac input)
33
C. Facility-level DC This has prompted proposals for a 550V/48V DC
Facility-level DC distribution offers higher efficiency by distribution architecture, shown in Figure 6, which alleviates
eliminating the inverter (DC/AC conversion stage) in the UPS, the copper requirement within the facility by limiting the
the AC/DC converter in the PSU, as well as the transformer in 48Vdc distribution to within close proximity of the racks [7].
the PDU. DC Distribution at 48Vdc, as shown in Figure 5, is It is not as efficient as facility-level DC distribution at 48Vdc
common in telco facilities. While it provides high efficiency, because of the additional Power Conversion Unit (PCU)
its applicability to large data and telco centers is limited by the which steps down 550Vdc to 48Vdc, which needs to be
high currents which result when distributing power exceeding isolated because of the SELV output.
1MW at such a low voltage. Currently facility-level DC distribution directly to the PSU
The amount of copper required to limit cable losses to a is limited to 48Vdc as servers are only manufactured with a
specified level increases with decreasing voltage, and is in universal AC input range or 48Vdc input. Since an AC PSU
theory inversely proportional to the square of the distribution contains a DC bus voltage in the range of 400Vdc, it can be
voltage. The ratio by which the amount of copper in the cables seen that with simple modifications to the PSU, a ~400Vdc
would have to increase in order to achieve the same cable compatible input PSU may be developed, which would lend
losses at 48Vdc as at 480Vac (three phase) theoretically itself to facility level distribution at ~400Vdc, as shown in
exceeds 100, but when real cable sizes are determined for a Figure 7. Voltage levels ranging from 300Vdc to 400Vdc have
specific design, the ratio is closer to 20. In addition, the cost of been proposed [5,12,13], but a single worldwide standard is
copper has increased significantly over the past five years. desirable to ease implementation.
12V
480V 208V 48V/ DC/DC
MV (400V) (230V) AC/DC 400V VR
AC AC DC
480V
AC/DC DC/AC PSU
only Server
PDU Rack
UPS
Figure 4 Rack-level DC
480V/ 12V
400V AC / DC -48V -48V DC/DC
MV AC DC DC VR
PSU
Server
UPS PDU
Rack
480V/ 12V
400V 550V DC/DC 48V DC/DC
AC/DC
MV AC DC DC VR
PCU PSU
Server
UPS PDU
Rack
480V/ 12V
400V 400V 400V DC/DC
AC/DC
MV AC DC DC VR
PSU
Server
UPS PDU
Rack
34
Distribution at these higher voltages result in a slightly servers per rack, 10 racks per PDU, and three PDUs per each
higher efficiency than the 48Vdc distribution, since the UPS of 7 UPSs. The load per server is varied from 30W to 300W,
does not have to be isolated. resulting in maximum power consumption per rack of 17 kW
for a high efficiency 480Vac distribution architecture.
III. POWER DELIVERY EFFICIENCY MODELING
The component rating Po,rated is determined as shown below,
Previously published comparisons of power delivery
where Pmax is the power delivered by the component when the
efficiency for different architectures consider only a single
maximum load of 300W is applied in the server and kd is a
operating point for all equipment, often at heavy load where
derating factor applied to each component, as shown in Table
efficiency is maximized [4,6,9]. However, converters in the
I.
power delivery train often operate at less than half of their
rated load, where conversion efficiency is much lower than at Pmax
Po,rated
heavy loads. This is because many server work loads include kd
significant idle time, power conversion equipment is often de-
rated and high availability requirements drive the use of
2
redundant power delivery architectures. It is therefore y = 0.0617x + 0.0353x + 0.0241
0.14 95%
necessary to evaluate power delivery efficiency over a range
of loads. 0.12 93%
Each component in the power delivery chain is modeled as
having a combination of three types of losses: 0.10 91%
1) Losses which are independent of the output power,
Losses [p.u.]
0.08 89%
such as those associated with generating the
housekeeping power. These dominate at light loads. 0.06 87%
2) Losses which vary linearly with output power, such as
switching losses in the power semiconductors, gate 0.04 Ploss 85%
drive loss and core loss in magnetics. Avg efficiency
3) Losses which vary with the square of the output power, 0.02 83%
Poly. (Ploss)
a result of resistance in the circuits, including on-state
0.00 81%
resistance of power semiconductors and winding losses 0.0 0.2 0.4 0.6 0.8 1.0
in magnetics. These dominate at heavy loads. Output power [p.u.]
On a per unit basis, where the per unit output power Po,pu
and per unit losses Ploss,pu are related to the power rating of the Figure 8 Conversion efficiency and losses for a baseline AC UPS
conversion stage Po,rated as follows,
Po Ploss TABLE I
Po, pu and Ploss , pu , PARAMETERS USED IN EFFICIENCY MODELING
Po,rated Po,rated a0 a1 a2 K100 kd
Baseline AC UPS 0.0241 0.0353 0.0617 89.2 0.9
where the losses can be expressed as High efficiency AC
Ploss , pu a0 a1Po, pu a2 Po2, pu UPS
Specialized AC UPS
0.0345
0.0086
-0.0162
0.0241
0.0455
-0.0027
94.0
97.1
The parameters a0, a1 and a2 may be determined through a 48V DC UPS 0.0245 0.0000 0.0524 92.9
400V/550V DC UPS 0.0095 0.0188 0.0208 95.3
second order polynomial curve fit from a plot of losses versus
PDU isolation 0.9
output power, as shown in Figure 8 for a baseline AC UPS. transformer 0.0026 -0.0054 0.0343 96.9
The losses may be determined from published efficiency data Rack converter 48V 0.9
as follows DC 0.0222 0.0135 0.0468 92.4 5
Rack converter 400V
Ploss , pu 1 K 1 Po , pu DC 0.0154 0.0258 -0.0008 96.1
Baseline AC PSU 0.0518 0.1787 0.0947 75.5 0.8
where K is the conversion efficiency, and the full load High efficiency AC
efficiency K 100 is related to the loss parameters as follows PSU (208Vac input) 0.0403 0.0113 0.0906 87.6
High efficiency AC
K100 1 / 1 a0 a1 a2 PSU (230Vac input) 0.0418 -0.0033 0.0985 87.9
This general modeling technique [14] is applied to the 400V DC PSU 0.0398 -0.0098 0.0930 89.0
power delivery architectures introduced above and the 48V DC PSU 0.0300 0.0215 0.0409 91.5
Baseline VR (fct of 0.9
parameters used are summarized in Table I. For each power Iout,pu) 0.0547 -0.0291 0.2650 77.5
delivery architecture, the loss parameters of a power High efficiency VR
conversion stage were determined from either published data (fct of Iout,pu) 0.0295 0.0474 0.1036 84.7
from manufacturers, or from prototype measurements. The
comparison was done for a data center with 210 racks, with 42
35
For more efficient power distribution architectures, the A 6.5% energy savings is calculated when operating this
losses in components are reduced and therefore the ratings of facility at full load with 400Vdc facility-level distribution
upstream components are reduced. This introduces some error, compared to high efficiency 480Vac distribution, an
since there will be a coarser granularity in equipment rating in approximate yearly savings of $200,000 at the US national
a real system. Nonetheless, this allows comparison at equal average of 6.2 c/kWh.
loading across all architectures. Redundancy is not considered Facility 400Vdc
for this comparison, but could be built into the model by 75
proper selection of the derating factor kd. Facility 48Vdc
The models also include cable losses, but these are
compares facility input power, which includes cooling power, 50.0 55.0 60.0 65.0 70.0 75.0
at full load for the different architectures. Power delivery efficiency [%]
A well designed cooling system is assumed with an HVAC-
effectiveness, defined as the ratio of UPS input power to the Figure 10 Comparison of calculated full load efficiencies
total power used to cool the facility, of 3.5 at full load.
TABLE II
CALCULATED EFFICIENCY COMPARISON AT FULL LOAD
AC Rack-level DC Facility-level DC
Baseline High 400Vac Specialized Rack-level Rack-level Facility- Facility- level Facility-
480Vac efficiency AC UPS 48Vdc 400Vdc level 550V/48V level
480Vac 480Vac (480Vac) (480Vac) 48Vdc DC 400Vdc
UPS 89.20 94.00 94.00 97.09 97.09 97.09 92.86 95.32 95.32
PDU or PCU 93.20 94.03 96.11 94.03 93.82 93.92 96.11 91.44 96.78
(including cable losses)
Rack level converter N/A N/A N/A N/A 92.38 96.12 N/A N/A N/A
(including cable losses)
Server PSU 75.46 87.56 87.95 87.56 91.54 89.05 91.54 91.54 89.05
VR stage 81.56 87.69 87.69 87.69 87.69 87.69 87.69 87.69 87.69
Total 51.64 67.97 69.86 70.01 67.37 68.23 71.85 69.98 72.70
36
losses associated with the boost conversion stage are
Facility 400Vdc eliminated, resulting in a higher efficiency power supply. An
Facility 550V/48Vdc
800W (23W/in3) prototype DC input power supply was
developed and its efficiency and losses are compared in Figure
Facility 48Vdc 14 to a 650W (15W/in3) high efficiency AC input PSU
Rack-level 400Vdc operated at 208Vac input.
A 400Vdc rated input connector is also required and a four
Rack-level 48Vdc
pin configuration with a positive, negative and ground pin as
Specialized AC UPS well as an additional signal pin which would allow
400Vac implementation of an enable circuit to prevent arcing upon
disconnection is proposed, as shown in Figure 13. A Cenelec
High efficiency AC
work group was established recently to start standards
Baseline AC development of appropriate DC connectors, and IEC is
expected to take over this effort. Some prototype commercial
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0
400Vdc connectors are available which are mechanically
Total facility input power [MW]
compatible with standard AC input (IEC320) connectors.
Similarly, a 400V DC output UPS can be developed from
Figure 11 Comparison of facility input power, including cooling
the AC/DC rectifier in a standard AC double conversion UPS.
Alternatively, with the same utility power as used by the high From 480Vac, a 400Vdc output can be generated from an
efficiency 480Vac system, the number of servers could be isolated converter, as shown in Figure 15, or a non-isolated
increased by over 600, or 7%. For all architectures, the converter, as shown in Figure 16 with a split bus from which
conversion efficiency falls off sharply at light loads since the two 400V distribution buses can be derived.
effects of light load losses are compounded by many stages.
400V
Techniques exist to improve light load efficiency, such as 90 – 264VAC AC Kss Lb DC
50/60 Hz fuse Db
building UPSs and power supplies with several smaller
EMI DC/DC
converters in parallel and enabling only as many of them as Filter 12V
(AC) CO
needed to supply the load with n+1 redundancy. While this Sb
may be applied to both AC and DC systems, it is easier to Chassis
implement in a DC distribution architecture because of the GND
absence of harmonics and phase synchronization requirements. Figure 12 Universal AC input PSU
V. EQUIPMENT FOR 400V DC DISTRIBUTION ~400Vdc DC fuse
400V
EMI
While this analysis shows that facility-level distribution at Filter
DC
400Vdc provides the highest system efficiency, it is not (DC)
DC/DC
currently used in industry as AC and 48V DC distributions 12V
are. A change in the infrastructure of a data center is required Enable
CO
Circuit
to implement it, therefore facility-level 400V DC distribution
Chassis
is primarily proposed for new data centers.
GND
The establishment of standards is required to facilitate new
product development, as well as the education of local Figure 13 DC input PSU
inspectors to ensure approval of new facilities built with 90 75
400Vdc distribution. ETSI has published a specification, ETSI
EE 300 132-3 [15], valid for AC and DC powered equipment 88 65
400Vdc
up to 400 V, and the intent is to achieve a uniform 208Vac
86 55
Efficiency [%]
37
It is proposed to connect the battery bank through a the boost switch Sb, and shorting out the boost diode Db.
converter to the distribution bus in order to achieve a well Neither the diode bridge at the input nor the boost inductor Lb
regulated bus. This allows optimization of the PSU efficiency was removed.
and power density. With an isolated DC UPS, it is possible to
connect the batteries directly to the distribution bus, as shown 480V 208V ~400V
in Figure 15, but this would require wide input range PSUs. AC AC
3
VI. SMALL SCALE DEMONSTRATION
In order to confirm the calculated energy savings, a small
scale demonstration of a high efficiency, conventional 480Vac
and a 400V DC facility-level distribution data center was Figure 15 Isolated DC output UPS with wide range output
assembled in collaboration with Lawrence Berkeley National
Laboratory and other industry partners. A photograph of the
demonstration is shown in Figure 17 and diagrams describing 480V SRC
~800V ‘A’
AC ~400V
the setup are in Figure 18.
The equipment used in the demonstration is listed in 3
Table III. For the AC setup, a 208Vac UPS was used, so a step ~500V
~400V SRC
down transformer was used upstream of the UPS to simulate ‘B’
the PDU transformer losses. Since a commercial DC UPS was
not available, a photovoltaic inverter was used, operated as a Figure 16 Non-isolated DC output UPS with split bus and narrow range
rectifier. Servers with 400Vdc input PSUs were not available, output
so their standard AC input PSUs were modified by disabling
DC AC
Figure 17 Photo of the small scale demonstration comparing conventional a high efficiency AC architecture (on right) with 380V DC facility-level distribution (on
left). Overhead lights operated on 380Vdc as well.
Pin,AC Pload,AC
AC Power
208V Standard AC
Strip
Pin,DC DC Pload,DC
AC Power
38
TABLE III
EQUIPMENT LIST TABLE
ACKNOWLEDGMENT
Manufacturer Part number (rating)
“PDU” transformer Powersmiths e-SAVER-C3L-75-480- The authors thank Kevin Bross, Modular Systems Architect
208 with Intel Corporation, for his contributions towards building
AC UPS Mitsubishi 2033A (75kVA) the models for the 48Vdc options, and Wilson Wu and Sen
DC UPS – Rectifier Satcon Power Systems PowerGate AE-75-60-PV Dou for the development of the 400Vdc prototype PSU.
(75kW) REFERENCES
DC UPS – Flywheel Pentadyne Power VSS 120
[1] J.G. Koomey, “Estimating total power consumption by servers in the
Corporation U.S. and the world,” February 20, 2007,
Busway (both AC Universal Electric Starline Track Busway [Link]
and DC) Corporation B225 [2] C.L. Belady, “In the Data Center, Power and Cooling Costs More Than
the IT Equipment it Supports,” Electronics Cooling, vol. 13, no. 1,
AC Circuit Breaker Square-D QOU [Link]
DC Circuit Breaker ABB TMAX T1ND30TL [3] Didier Marquet et al, “New power supply optimised for new telecom
networks and services,” IEEE Intelec ‘99, Copenhagen, 1999.
(20A)
[4] T. Yamashita, et al., “270 V DC – A Highly Efficient and Reliable Power
Square-D FHL3603013DC (30A) Supply System for Both Telecom and Datacom Systems,” IEEE Intelec ‘99,
Servers Intel Corporation Various models Copenhagen, 1999.
[5] U. Carlsson, et al., “Powering the Internet - broadband equipment in all
Sun Microsystems Various models facilities - the need for a 300V DC powering and universal current option,”
IEEE Intelec 2003, pp. 164-9.
In order to simulate real world operating conditions, [6] Marquet, D., et al., “New flexible powering architecture for integrated
service operators”, IEEE Intelec 2005
measurements were taken with the UPSs loaded at around 33 [7] P. Gross and K.L. Godrich, “Total DC integrated data centers,” IEEE
percent. Load banks were operated in parallel with the servers Intelec 2005, pp. 125-30.
and the AC and DC load banks were set to the same ratio as [8] W. Allen et al., “Securing Critical Information and Communication
Infrastructures through Electric Power Grid Independence,” IEEE Intelec
that of the input power to the server power supplies when 2003.
running their respective workloads, measured at 0.97. [9] A. Pratt and P. Kumar, “Evaluation of Direct Current Distribution in Data
The measured results are summarized in Table IV, showing Centers to Improve Energy Efficiency,” The Data Center Journal,
March 28, 2007,
a 7% input power savings of 400Vdc distribution compared to [Link]
an AC architecture with high efficiency components. This [10] N. Rasmussen, “Increasing Data Center Efficiency by Using Improved
compares well with the calculated savings of 6% in Figure 11. High Density Power Distribution,” APC White Paper #128,
[Link]
Further details are available in a published report [12]. [11] P. Singh, et al. “A power, packaging, and cooling overview of the IBM
eServer z900,” IBM J. Res. & Dev., Vol. 46, No. 6, November 2002, pp.
VII. CONCLUSIONS 711-38.
[12] M. Ton et al., High Performance Buildings: DC Power for Improved Data
Through an analysis of power delivery efficiencies of Center Efficiency, Lawrence Berkeley National Laboratory, December 2006
several power delivery architectures over a wide load range, ([Link] )
this paper showed that facility-level 400Vdc distribution [13] H. Ikebe, “ Power systems for telecommunications in the IT age,” IEEE
Intelec 2003, pp. 1-8.
provides increased energy efficiency for data and telco centers. [14] N. Rasmussen, “Electrical Efficiency Modeling of Data Centers,” APC Whi-
This is supported by measurements from a small scale te Paper #113, Revision 1, [Link]
demonstration of high efficiency 480Vac and 400Vdc 66CK3D_R1_EN.pdf
[15] ETSI Environmental Engineering (EE) EN 300 132-3 Standard: “Power sup-
distribution, which showed a 7% input power savings. While ply interface at the input to telecommunications equipment; Part 3: Operated
this paper focused on the power delivery efficiency by rectified current source, alternating current source or direct current source
improvements, these alone may not justify the change by itself. up to 400 V”, 2003.
However, the combination of high efficiency with the other
advantages of 400Vdc distribution, including higher reliability,
simplified implementation due to lack of phasing requirements
and harmonic mitigation, and reduced components cost,
makes for a compelling case in considering the adoption of
facility-level 400Vdc distribution.
TABLE IV
RESULTS FROM SMALL SCALE DEMONSTRATION
AC DC Ratio DC/AC
Load power (Pload) 23.3 kW 22.70 kW 0.974
UPS input power (Pin) 26.0 kW 24.10 kW 0.927
39