A Flash Storage Technology Primer

VMAX Engineering Technical Notes

March 2016

Abstract
Recent advancements with higher density, vertical, multi-layer cell
flash technology have led to the development of higher capacity
flash drives which now easily exceed 1 TB or more. The
introduction of these higher capacity flash drives has greatly
accelerated the inflection point where flash drives have the
equivalent economics of traditional hard drives functioning as the
primary storage media for enterprise applications in the data
center. This paper examines current flash drive technology trends
and takes a close look at the primary factors such as performance,
scalability, and endurance considerations which are driving the
industry toward an all flash data center.

EMC CONFIDENTIAL – INTERNAL USE ONLY

To learn more about how EMC products, services, and solutions can help solve your business and IT challenges,
contact your local representative or authorized reseller, visit [Link], or explore and compare products in
the EMC Store

Copyright © 2016 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on [Link].

2
EMC CONFIDENTIAL – INTERNAL USE ONLY
Table of Contents
Abstract ......................................................................................................................................................... 1
Executive Summary....................................................................................................................................... 5
Audience ....................................................................................................................................................... 5
Background ................................................................................................................................................... 6
Understanding Flash Storage Device Architecture ....................................................................................... 6
What is a NAND Flash Memory Cell .......................................................................................................... 6
Constructing a Basic Flash Device ............................................................................................................. 7
NAND Strings (Bit Line) ......................................................................................................................... 7
NAND Page (Word Line) ........................................................................................................................ 8
NAND BLOCK ......................................................................................................................................... 8
NAND Planes and Dies .......................................................................................................................... 9
Flash Controller Chip ........................................................................................................................... 11
Flash Cell Read Operations ..................................................................................................................... 12
Flash Cell Program and Erase (the P/E Cycle) ......................................................................................... 13
Types of NAND Flash Cells .......................................................................................................................... 14
Single Level Cell (SLC) NAND ................................................................................................................... 14
SLC Performance ................................................................................................................................. 15
SLC Scalability and Cost / GB ............................................................................................................... 15
Multi-Level Cell (MLC) NAND .................................................................................................................. 15
MLC Read and Write Operations ........................................................................................................ 16
MLC Performance ............................................................................................................................... 16
MLC Scalability and Cost/GB ............................................................................................................... 16
The Hypothetical Flash Dead-End ........................................................................................................... 17
Triple Level Cell (TLC or 3-bit MLC) NAND – Avoiding the Dead-End ..................................................... 17
TLC Read and Write Operations .......................................................................................................... 18
TLC Performance ................................................................................................................................. 18
TLC Scalability and Cost / GB............................................................................................................... 18
How TLC is Changing the Economics of Flash Storage ........................................................................ 19
Three-Dimensional NAND (3-D NAND) ................................................................................................... 20
3-D NAND Read and Write Operations ............................................................................................... 21

3
EMC CONFIDENTIAL – INTERNAL USE ONLY
 NAND Flash Cell Technology Summary ................................................................................................... 22
Understanding Flash Cell Endurance .......................................................................................................... 22
Flash drive endurance metrics ................................................................................................................ 23
Drive Writes per Day ........................................................................................................................... 26
Terabytes Written ............................................................................................................................... 27
Comparing Endurance of Flash Drives vs. Traditional HDD ................................................................ 27
Flash Endurance Conclusions .................................................................................................................. 28
Conclusion ................................................................................................................................................... 28
References .................................................................................................................................................. 29

4
EMC CONFIDENTIAL – INTERNAL USE ONLY
Executive Summary
The requirements of the modern enterprise data center with regard to storage capacity and storage
performance have increased dramatically in recent years with the need to support “millions” of virtual
devices and machines. Although traditional spinning disk media can still meet the storage capacity
requirements, it is having difficulty meeting the performance requirements (now measured in millions of
IOPS) for these environments.

Recent advancements in flash technology have led to a breakthrough in the capacity and economics of
flash storage. This breakthrough has greatly accelerated the inflection point of where flash storage has
the same economics of traditional spinning hard disk media (HDD). The incorporation of this new
technology into flash storage devices is now allowing the enterprise data center to meet the storage
capacity and performance requirements for highly virtualized environments at affordable economics.

This paper contains an overview of modern flash storage technology. It summarizes the various types of
flash devices and examines the current trends and considerations that are driving the enterprise to an
all flash data center.

This paper includes a detailed discussion of the following topics:

• Understanding flash storage device architecture

o Flash cell, strings, pages, blocks
• Types of Flash Cell Technologies
o SLC, MLC, TLC, and 3-D NAND
• Understanding Flash Cell Endurance and Endurance Metrics
o Drive Writes per Day (DWPD)
o Terabytes Written (TBW)
o Mean Time Between Failures (MTBF)

Audience
This paper is intended for internal EMC sales staff, partners, and support personnel wishing to gain a
deeper understanding of modern flash drive technology.

5
EMC CONFIDENTIAL – INTERNAL USE ONLY
Background
Modern flash storage technology was born the late 1980s by Toshiba when they were developing a
replacement for EEPROM - a low-cost type of non-volatile memory which could be erased and
reprogrammed, but needed a cumbersome process which relied on ultraviolet light to ensure a
complete erasure of the media. Toshiba came up with a new concept of using “Negative AND” logic
gates (NAND) as the primary component of a new non-volatile memory architecture called NAND Flash.
A NAND gate is an electronic component which produces an output of false only if all of its inputs are
true. Below is a traditional schematic of a NAND gate and a table of its inputs and outputs:

Figure 1. Standard NAND Gate Schematic and Input / Output Table

A
NAND
C
Gate
(A NAND B)
B

Inputs Output
A B 𝐶 = 𝐴 𝑁𝑁𝑁𝑁 𝐵
0 0 1
0 1 1
1 0 1
1 1 0

Toshiba’s new NAND architecture offered significant performance improvements, lower cost-per-bit,
and much larger capacities when compared to EEPROM. These improvements made NAND technology
suitable for storing data and laid the ground work for future generations of flash storage. Today’s flash
storage devices are the direct descendants of this original NAND technology.

Understanding Flash Storage Device Architecture

What is a NAND Flash Memory Cell

The NAND flash memory cell is the primary component of a flash storage device. Traditional NAND Flash
Cells store data by using groups of “floating-gate transistors” to store electric charge. A single floating
gate transistor (flash cell) is the smallest building block needed to store a single bit of data on a flash
storage device. The architecture and schematic of a flash cell is shown in the diagram below:

6
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 2. Flash Cell Structure and Schematic

Typical Flash Cell Diagram Typical Flash Cell Schematic

The basic operation behind a floating gate transistor is that when a charge is placed on the gate
terminal, the semiconducting silicon becomes conductive and the source and drain terminals are
connected thus establishing a current through the device (turning on the device). When the charge is
removed from the gate terminal, the silicon layer becomes insulating and the source and drain terminals
are disconnected (turning off the device). The floating gate layer is an electrode which is insulated by
from the rest of the structure by an additional insulating oxide layer. A floating gate transistor can store
charge by driving high enough current through the device so that charged electrons can “tunnel”
through the insulating oxide layer onto the floating gate where they can stay indefinitely even after the
device is switched off. This principle allows the floating gate transistor to remember its “on” state even
when it is switched off. The persistence of this charge in the floating gate is the primary enabling
technology behind the use of flash memory as storage.

Constructing a Basic Flash Device

A single flash cell by itself is relatively insignificant with respect to storing data; however, when they are
bundled together in the billions on a single chip, they can store significant amounts of data. The
following sections describe how flash cells are bundled together to form a flash storage device.

NAND Strings (Bit Line)

The first step in bundling flash cells together is connecting them into a string of cells called a NAND
string. Typically a NAND string consists of either 32 or 64 floating gate cells wired together in a series.
These strings are often referred to as a “bit line”. The amount of bytes contained in a bit line is the
minimum unit for a flash read operation. The cells in a NAND string are usually separated by nanometers
(nm). Flash drive manufacturers are continuously trying to find ways to shrink this spacing as a means of
improving flash storage density.

7
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 3. Flash Cell String (Bit-Line)

NAND Page (Word Line)

NAND Strings are connected into larger structures by placing the strings next to each other to form
columns. Individual flash cells from adjacent strings on the plane are connected horizontally to form
rows. These rows are called “word lines”. The word line stretches across all the columns in the row
(32768 cell columns is standard but newer flash devices have 64K or 128K columns). The amount of cells
in a word line is called a NAND Page. The amount of bits in a word line is considered the NAND page size.
The NAND page size is the minimum unit for writes in a flash device. Some typical flash page sizes are
4KB, 8KB, 16 KB, and 32 KB.

NAND BLOCK
NAND strings and pages form the columns and rows of a 2 dimensional array called a NAND block. The
total number of bits in the block is called the block size and it can be calculated by multiplying the
number of strings by the number of pages. The block size is the minimum unit for erases in a flash
device. Typical block sizes are reaching up to 8 Megabytes (8 MB).

8
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 4. Flash Block

NAND Planes and Dies

NAND blocks can be grouped together in a bank of blocks called a NAND Plane. Finally, one or many
planes can be grouped together to form a NAND die. NAND dies are relatively fragile and require
specialized equipment for placement and bonding. NAND dies are stacked together into a single
package with a standard pin out for soldering to printed circuit boards (PCBs) to form the flash chip. The
following diagram shows how many typical blocks form a plane, how many planes form a die, and then
how many dies form a chip (also referred to as a chip set).

9
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 5. Typical Flash Plane, Dies, and Chipset Configurations

Plane Die Chip / Chip Set

As previously stated, modern flash storage devices have block sizes approaching 8 MB. When 2048
blocks are used to create a plane, the plane will be 16 GB in size. Two planes form a 32 GB die, and four
32 GB dies form a 128 GB flash chip.

Note: Manufacturers usually publish plane and block capacity for specification documentation in terms
of Gigabits (Gb) where 8 Gb equal 1 Gigabyte (GB). Manufacturers will then publish the total capacity of
the device itself for consumer documentation in terms of GB. This paper expresses flash capacity in GB.

Chip sets are then placed onto a larger PCB along with a flash controller chip and other necessary
components to create the flash device. Below is a typical flash storage device:

Figure 6. Typical Flash Storage Device

10
EMC CONFIDENTIAL – INTERNAL USE ONLY
The flash device in the above figure uses ten chips. Continuing the previous example, if the above flash
device is using 128 GB chips, then the above device would have a raw capacity of approximately 1.2 TB
(10 x 128GB chips). The actual usable capacity (available to the user) would be less – perhaps around 1
TB. This difference is primarily due to flash device “over provisioning” which reserves a portion of raw
space on the device for wear leveling performed by the flash drive controller chip. This concept is
discussed in more detail later in the flash drive endurance section of this document.

Flash Controller Chip

The primary function of the flash controller is to receive commands from the host system that tell it
where to read or write a piece of data. For consumer level flash devices, controllers are usually designed
using cost and performance as the key criteria. In enterprise level devices, the controllers are not only
designed for performance, but are also designed to perform other key functions such as encryption,
endurance management, and error checking. Below is a table that highlights some of the more
important functions that an enterprise flash controller performs:

Table 1. Primary Flash Controller Functions

Flash Controller Function Description

SMART (Self-Monitoring The SMART function, available in some controllers, monitors and records
and Reporting Technology) data regarding many attributes of the flash device and memory. An
example is the ability to monitor the percentage of endurance cycles
remaining in the flash device since this is a key determining factor of the
life remaining.
Wear Leveling Wear Leveling is the ability to even out the number of write cycles
throughout the available NAND. Since each NAND block has a limited
number of erase/write cycles, if only one physical block is written
continuously, it will quickly deplete its endurance cycles. A controller’s
Wear Leveling algorithm monitors and spreads out the writes to different
physical NAND blocks.
Garbage Collection Garbage Collection is the process to reclaim previously written blocks of
data so they can be rewritten with new data. The reason for Garbage
Collection is NAND’s requirement to be erased prior to being written. As
said earlier a block is the smallest erasure unit of NAND flash. A page of
data can be written as long as it is already erased, but it is very inefficient
to erase an entire block just to reclaim a single page. Garbage Collection’s
main function is make this process more efficient by first copying and
moving any un-erased data which is not being written to a new NAND
block. Once this copying is completed, the entire original NAND block is
erased and it can be written to with fresh data.
Read and Program Disturb With the finer trace widths of the NAND flash, more issues arise to
maintain the data contents of the NAND cells. Read & Program Disturb
occur when cells are read or written causing cross coupling to adjacent
cells and occasionally changing their values. Controllers need algorithms
and in some cases circuitry to compensate for this phenomena.
Encrypt and Decrypt Engine For higher security applications, a hardware encryption and decryption
engine is generally built into the silicon of the controller. The encryption
11
EMC CONFIDENTIAL – INTERNAL USE ONLY
 engine is typically implemented in hardware to ensure speed for
encrypting/decrypting on the fly. The most popular encryption method
for SDDs today is AES256.
ECC Engine Error Checking & Correction are a key part of today’s flash device. ECC will
correct up to a certain number of bits per block of data. Without ECC,
many of the low cost consumer flash cards using very inexpensive
memory would not be possible.
Write Abort Write Abort is the when power to the flash device is lost during a write to
the NAND flash. Without a battery or SuperCap backed cache, it is likely
this data in transit will be lost. The more important aspect of this is to
ensure the flash device’s internal metadata and firmware remain
uncorrupted. This is the function of Write Abort circuitry mainly found in
Industrial Grade products.
Defect Management Every controller needs a method to deal with bad blocks of memory and
new defects. At the point a NAND block becomes unusable, some action
on the flash device controller’s part must happen. In some cases a spare
sector replaces the failed block. In a poor controller design, the flash
device fails. Each controller has its method to deal with defects.
Larrivee, Steve, “Solid State Drive Primer”, Cactus Technologies, June 8th, 2015

Flash Cell Read Operations

A read operation is performed by applying a voltage (vREAD) to the control gates of each cell in a bit
line. If current flows through the cell from the source terminal to the drain terminal, this means that
there is no charge stored in the floating gate of the cell, and the read operation returns an “erased” bit
value of 1. If current does not flow through the cell when the vREAD voltage is applied to the control
gate, this means that there is a charge stored in the floating gate, and the read operation returns a
“programmed” bit value of 0. The actual bit values returned by the read operation vary depending upon
the type of flash cell being used. As said earlier, the read operation is performed over the full bit line.
The amount of bits contained in the bit line is the minimum read unit for the device.

Figure 7. Flash Cell Read Operation

12
EMC CONFIDENTIAL – INTERNAL USE ONLY
Flash Cell Program and Erase (the P/E Cycle)
Writing (also known as programming) a flash storage is accomplished by applying a much higher voltage
(vWRITE) to the control gates of all the cells in a page (word line). This higher voltage is what allows the
electrons flowing from the source terminal to the drain terminal to “tunnel” through the insulating oxide
layer onto the floating gate – leaving the cells in the “programmed” state. Once the vWRITE voltage is
removed, the electrons on the floating gate are trapped until they are removed by an erase operation.
Again, a write operation is performed across an entire NAND page. The NAND page size is the minimum
write unit for the device. As said earlier, typical page sizes are 4K, 8K, and 32K.

An erase operation begins with another higher voltage (vERASE) being applied to the silicon substrate of
the entire NAND block. This higher voltage (vERASE) causes the electrons trapped on the floating gate to
tunnel back through the insulating oxide layer and into the current flowing from the source terminal to
the drain terminal – hence, “erasing” the charge from the floating gate. As stated earlier in the
document, the block size (typically around 8 MB) is the minimum unit for erase.

The storing and removing of electrons from the floating gate is called the Programming and Erase Cycle
(P/E Cycle) of a flash cell. Every time new data is to be written to the cell, a full P/E Cycle must be
performed, therefore, tunneling though the insulating oxide layer occurs twice for every new write to
the cell. Each write operation to a NAND page results in an erase operation being performed on the
entire block the page belongs to. The data on the block not being written to is moved to other locations
in the device. This moving of data is part of the device’s “Wear Leveling” process. Wear leveling is
discussed in more detail in the flash endurance section of this paper.

Figure 8. Flash Cell Program / Erase Cycle (P/E Cycle)

13
EMC CONFIDENTIAL – INTERNAL USE ONLY
Types of NAND Flash Cells

Single Level Cell (SLC) NAND

The simplest type of flash cell is called a Single Level Cell (SLC). When the SLC’s floating gate is charged,
it is considered “programmed” and has a binary value of 0. When the SLC’s floating gate has no charge,
it is considered “erased” and has a binary value of 1.

Table 2. SLC Charge States

Floating Gate State Voltage Applied to Cell Referred to as Bit Value

Charged High Voltage (> 4 V) Programmed 0
No Charge Low Voltage (< 4 V) Erased 1

The state of the flash cell (0 or 1 in SLC) can be manipulated by the voltage (Vt) applied to it. In an SLC, a
voltage of greater than 4.0V will result in the cell being marked as programmed (1) while a voltage of
less than 4.0V will result in the cell being marked as erased (0). This voltage which demarcates the
different charge state of a flash cell is sometimes referred to as a reference voltage. In a typical SLC, the
reference voltage is ~4 Volts. The voltage range to achieve a specific state with respect to the total
voltage range of the device is called the state’s tolerance voltage. A typical SLC has a total voltage range
of ~8 Volts, therefore the tolerance for each state of an SLC is around 50% (4V / 8V) of the total voltage
range. The precision in which a cell has to control its voltage to accurately store its bits and prevent
charging the wrong state in the cell is called its guard band (also known as the margin of error). These
concepts are shown in the following chart: (Note: in the diagram the four red dots are used to symbolize
a fully charged state)

Figure 9. SLC Charge States

14
EMC CONFIDENTIAL – INTERNAL USE ONLY
SLC Performance
Because SLC has a wide tolerance and guard band, it has exceptional read and write performance as the
voltage control structure does need to be overly precise with its calculations when marking the cell
programmed or erased (> 4.0V for 0, < 4.0V for 1). A wider tolerance always results in faster the write
operations for the cell. Aside from the fast write speed, the wider tolerance also makes SLCs very
durable and resistant to environmental conditions such as shock and temperature extremes. SLCs also
have the lowest power consumption of any flash architecture on the market.

SLC Scalability and Cost / GB

Storing only a single bit per cell is SLC’s main detractor as the single bit architecture makes it difficult to
achieve the storage density required to create larger capacity devices. This makes SLC prohibitively
expensive in terms of cost/GB for larger capacity storage devices. This higher cost factor of SLC has
primarily pushed them into a niche market of high performance, smaller capacity, rugged, Compact
Flash Cards (CFC) used for higher-end digital video and professional DSLR camera applications. Other
areas where SLC storage devices can be found are in industrial and mission critical flight controllers
where shock and vibration resistance is essential.

Multi-Level Cell (MLC) NAND

Multi-Level Cell (MLC) flash technology was primarily developed to address the cost and capacity scaling
limitations of SLC. With MLC flash, a single cell can store two bits of data. To achieve this, MLC flash
must be able to apply charge to the floating gate at four different levels and later be able to detect
which of the four levels is set. The following table shows the four different MLC charge states:

Table 3. MLC Charge States

Floating Gate State Voltage Applied to Cell Referred to as Bit Value

Fully Charged High Voltage Fully Programmed 00
Medium Charged Medium High Voltage Partially Programmed 01
Lightly Charged Medium Low Voltage Partially Erased 10
No Charge Low Voltage Fully Erased 11

The four states and typical voltage range for an MLC flash cell is shown in the following diagram:

15
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 10. MLC Charge States

In the above diagram, the red dots reflect the various charge states, with no dots representing a no
charge state and the six dots representing a fully charged state. The total voltage range for an MLC cell is
often around 8 to 10 Volts (often ~ 20% higher than SLC). The typical voltage tolerance for each of the
four states in MLC is around 25% of the total voltage range with the guard band between the four states
typically around 5% of the total voltage range.

MLC Read and Write Operations

Read and Program/Erase operations with MLC follow the same principles as SLC flash. However, to store
the extra bit, MLC flash needs to tunnel more electrons onto and off of the floating gate. This requires
the use of higher voltages during the P/E cycle than SLC.

MLC Performance
Due to the need for MLC to keep track of four states, the tolerance range and guard band for each of the
four MLC charge states is significantly smaller than SLC. This means that the calculations performed by
the MLC voltage controller to properly control the voltage to achieve the four states are significantly
more precise. This extra precision slows down the writes to the cell making MLC flash have slower write
performance than SLC flash.

MLC Scalability and Cost/GB

By storing 2 bits of data in a single cell, MLC provides double the storage capacity when compared to a
SLC device using the same number of transistors. This doubling of storage capacity in the same transistor
foot print allows for MLC to easily scale to larger capacity storage devices than SLC, making MLC more
affordable from a cost / GB perspective. The affordability of MLC is the primary reason why MLC flash
has been the dominant flash technology on the consumer market today. 2-bit MLC flash drives are
prolific across consumer based thumb drives, PCs, laptops, portable computing devices.

16
EMC CONFIDENTIAL – INTERNAL USE ONLY
The Hypothetical Flash Dead-End
MLC drives can typically scale to capacities reaching 500 GB; however, getting the two bit MLC
architecture to scale up to and beyond 1 TB proved to be technically challenging to manufacture and
unjustifiable in terms of cost / GB. This limitation with two bit MLC technology made many in the
industry begin to believe that flash storage was reaching a “dead-end” as it was becoming evident that
MLC was not going to scale and meet the economics required by the growing demand by the enterprise
for low cost higher capacity flash storage.

Triple Level Cell (TLC or 3-bit MLC) NAND – Avoiding the Dead-End
The development of Triple Level Cell (TLC) flash and 3-D or vertical NAND (3-D NAND) technologies
(described in detail in the next section) has allowed the industry to avoid the flash dead-end. These two
technologies used in combination have yielded the promise of much higher capacity (multi-terabyte)
flash drives at economics which will meet the surging demand for lower cost flash storage in the
enterprise.

As its “3-bit MLC” name suggests, TLC stores three bits on a single flash cell. This yields a total of 8
potential voltage states and bit values for a TLC flash cell. The following table shows the TLC states and
bit values:

Table 4. TLC Charge States

Floating Gate State Voltage Applied to Cell Bit Value

Fully Charged Highest Voltage 000
Highly Charged High Voltage 001
Medium High Charge Medium High Voltage 010
High Medium Charge High Medium Voltage 011
Low Medium Charge Low Medium Voltage 100
Medium Low Charge Medium Low Voltage 101
Lowest Charge Low Voltage 110
No Charge Lowest Voltage 111

These states and the associated cell voltage ranges required to attain them are shown in the diagram
below. The seven red dots represent the fully charged state (000).

17
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 11. TLC Charge States

Comparing the above diagram to the similar diagrams for SLC and MLC, it is evident that TLC requires
higher voltages to achieve the fully charged state, which results in an overall higher total voltage range
(~12 to 15 Volts). Even with the higher voltage range, the tolerance and guard band around the eight
TLC states is much smaller than those of SLC and MLC.

TLC Read and Write Operations

Read and Program/Erase operations with TLC follow the same principles as SLC and MLC flash; however,
in order to store the extra bit, TLC flash needs to tunnel even more electrons onto and off of the floating
gate, which requires the use of even higher voltages during the P/E cycle than MLC.

TLC Performance
Because TLC flash has 8 states, the cell’s voltage controller must work with much tighter tolerances and
guard bands in order to properly charge the cell to the desired state. Like MLC compared to SLC, this
requires more precise calculations which significantly slows the writes to the cell, even more than MLC.
This extra precision makes TLC flash the slowest of the flash technologies for write performance;
however, compared to spinning disk it is significantly faster by orders of magnitude.

TLC Scalability and Cost / GB

Even though has the slowest write performance of the three primary flash technologies, TLCs ability to
scale to terabyte levels results in dramatically reduced economics with respect to cost / GB. This is the
primary reason why the flash manufacturers are rapidly moving to TLC (3-bit MLC) flash technology. This
is shown in the following graphic which shows specific flash technology usage since 2004:

18
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 12. Flash Storage Technology Trends

NAND Basics – Understanding the Technology Behind Your SSD, August 2013, Samsung Electronics

How TLC is Changing the Economics of Flash Storage

In the past, flash drive production costs were cut by shrinking the NAND die. Shrinking the die is
essentially the same strategy that has been done by CPU manufacturers for years by trying to squeeze
more transistors onto the same chip footprint. This is done by shrinking the manufacturing process, e.g.
from 34nm to 25nm. This same principle applies to flash memory where more cells are attempted to be
squeezed on to the same flash die. Increasing the flash cell density per die usually results in smaller die
sizes, which means more dies from a single wafer.

Shrinking die sizes is an effective way to lower cost of flash, but moving to new manufacturing processes
takes time. Yield from the process is likely to be lower at first and real cost savings are not realized until
the new process can mature – which can take a year or more. The cost savings from flash die shrinkage
in the manufacturing process was not able to keep pace with the increasing demand for higher capacity
and lower cost enterprise flash storage devices hence why the industry was beginning to worry that
flash was reaching a dead-end. This is where the 3-bit TLC technology has come in and delivered.

Rather than shrinking the MLC die process to improve density / capacity, TLC increased the number of
bits which could be stored on a single cell from 2 to 3. By doing this, TLC provided a 33% density increase
over MLC flash while not radically modifying the existing manufacturing process. This allows for more
dies per wafer and hence significantly lowers the manufacturing cost of the flash device.

19
EMC CONFIDENTIAL – INTERNAL USE ONLY
Three-Dimensional NAND (3-D NAND)
The three bits per cell of TLC flash increased the storage density of flash devices by 33% over traditional
MLC technology. However, even with this increase, flash devices were still facing technological hurdles
to scale up to storage densities needed by enterprise data centers. Traditional flash manufacturing
processes placed cells together to create NAND blocks in 2-D planes. In order to continue scaling to
higher density levels using cells in 2-D planes, the spacing between cells would need to get smaller.
Using traditional silicon, further shrinkage of cell spacing at levels below 20 nm becomes problematic
due to cell to cell interference. This interference occurs when the cells are so close together that the
electrons can tunnel from one cell to another, making the cells unreliable for storing charge.

Recently, Samsung, Toshiba, and other manufacturers have developed a new vertical flash cell
technology called Three Dimensional NAND (3-D NAND). Instead of layering flash cells into rows and
columns in a 2-D planar manner, 3-D NAND typically uses a new cylindrical Charge Trap Flash (CTF) cell
design which allows the cells to be stacked vertically like building blocks at spacing less than 20 nm
without suffering cell to cell interference. CTF technology uses a non-conductive layer which temporarily
traps electrical charges in the same manner that the floating gate does for traditional NAND. This non-
conductive layer is modified into a three dimensional cylindrical form with Silicon shaft as a core. The
form is then wrapped around by the control gate of the cell. This layering of the cells in a vertical
manner has more than doubled the storage density using the same footprint of traditional 2-D planar
NAND devices.

The major flash manufacturers are all developing their own 3-D NAND technology and manufacturing
process. For example – Samsung uses a technology called Vertical NAND (V-NAND) while Toshiba uses a
technology called BiCS (Bit Cost Scaling). The following diagrams show how a typical 3-D NAND cell is
constructed, and how it is assembled into vertical structures to form a 3-D NAND block. The diagrams
below do not reflect a specific structure or process but represent a generalization of how 3-D NAND
technology is implemented to achieve ultra-high block densities.

20
EMC CONFIDENTIAL – INTERNAL USE ONLY
Figure 13. Typical 3-D NAND Charge Trap Flash Cell Structure

Figure 14. Typical 3-NAND Block Structure

3-D NAND Read and Write Operations

The read, programming, and erase processes for CTF 3-D NAND cells are principally the same as
traditional floating gate flash cells – the primary difference is that CTF technology is able to program all
the bits onto the charge trap at once. Traditional floating gate technology often requires the bits to be

21
EMC CONFIDENTIAL – INTERNAL USE ONLY
programmed serially. Another added benefit of CTF architecture is that the voltage required to fully
charge the cell is significantly less, resulting in less complex calculations for managing the 8 voltage
states of the cell. The ability to charge all of the bits at once and reduced complexity in programming
calculations results in a significant increase (2x observed by manufacturers) of the write performance
over traditional 2-D planar NAND. Also, since they require less voltage to charge the cells, 3-D NAND CTF
cells require less power and run considerably cooler than their 2-D planar counterparts, greatly
enhancing their endurance. As an example, Samsung states that their TLC V-NAND cells can withstand
up to 35000 P/E cycles compared to 3000 P/E cycles for traditional 2-D planar TLC . Flash cell endurance
is discussed in detail in the next section of this document.

NAND Flash Cell Technology Summary

The combination of TLC and 3-D NAND technologies have yielded the promise of much higher capacity
flash drives at economics which will more than meet the surging demand for higher performance lower
cost flash storage in the enterprise. These technologies used in combination will result in enterprise
flash storage devices exceeding 10 TB or more in the near future.

Understanding Flash Cell Endurance

Like a battery that eventually loses its charge after several hundred of cycles, NAND flash components
will also lose their ability to retain charge over time. This time frame is referred to as the Flash Cell
Endurance Cycle.

The Flash Endurance Cycle is most impacted by the tunneling of the electrons onto and off of the
floating gate during the flash cell’s P/E Cycle. Over time the tunneling of the electrons during P/E Cycles
causes a degradation of the of the insulating oxide layer between the silicon and floating gate.
Eventually, the insulating layer degrades to a point where the cell’s floating gate can no longer store
charge. When this occurs, the cell is considered to be in a failed state and is no longer usable for data
storage.

The primary factor which can accelerate the wear on the insulation oxide layer during the P/E cycle is
the voltage needed to reach a fully charged state. The higher the required voltage to reach the fully
charged state, the faster the insulating oxide layer wears. Higher voltage requirements also generate
more heat, which further accelerates the degradation of the insulating layer. The effects of voltage and
the resulting heat is why SLC flash, with its lower voltage requirements, has the highest flash cell
endurance and why TLC flash, with their higher voltage requirements, have the lowest endurance of the
flash technologies.

3-D NAND can be considered a breakthrough technology not only with regards for flash density, but also
for flash endurance. 3-D NAND technology achieves higher 3-bit density at lower voltage levels through
its Cache Trap Architecture. The voltages needed to tunnel charge onto the CTF insulating layer are less
than tunneling electrons onto the floating gate of a traditional NAND cell. This greatly improves the
endurance of CTF based TLC 3-D NAND when compared to traditional TLC 2-D planar floating gate cell
architectures. Endurance ratings for the different flash technologies are shown in the following table:
22
EMC CONFIDENTIAL – INTERNAL USE ONLY
Table 5. Flash Technology Endurance Ratings

Flash Technology Number P/E Voltage Endurance Scalability Write

of Bits Requirement Rating Performance
SLC 1 Lowest Highest Lowest Highest
MLC 2 Medium Low Low Medium
eMLC* 2 Medium Medium Low Low
TLC 3-D NAND (CTF) 3 Medium High Highest Medium
TLC 2-D Planar 3 Highest Lowest Medium Lowest
* Manufacturers improved the flash controller algorithms some of their MLC offerings in order to boost
their endurance to 10000 P/E cycles or more. These improved MLC flash drives are often referred to as
enterprise MLC (eMLC) and carry a pricing premium over traditional MLC (sometimes referred to as
consumer grade MLC or cMLC). Although eMLC flash drives have better endurance over traditional MLC,
their write performance suffers due to the more complex calculations their flash controllers need to
perform.

Flash drive endurance metrics

There are two metrics used by the industry when specifying flash drive endurance. These metrics are
Terabytes Written (TBW) and Drive Writes per Day (DWPD). These metrics are essentially different ways
to express the same thing. Both metrics are dependent on the following factors:

• Maximum number of P/E Cycles - Expresses the manufacturers estimated number of P/E cycles
a flash cell on the drive can endure during the warranty period (typically 3 to 5 years)

• Drive Usable Capacity - Expresses the total usable capacity of the drive in TB

• Drive Over Provisioning - Over Provisioning (OP) is the total physical memory reserved by the
flash controller for usage in background operations, and is not part of the device’s logical
address space. Over Provisioning is typically described as a percentage of the physical capacity
of the device. Manufacturers often use either 28%, 7%, or 0% as OP values. Drive Over
Provisioning can be calculated as follows:

𝐷𝐷𝐷𝐷𝐷 𝑃ℎ𝑦𝑦𝑦𝑦𝑦𝑦 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 − 𝐷𝐷𝐷𝐷𝐷 𝑈𝑈𝑈𝑈𝑈𝑈 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶

𝑂𝑂𝑂𝑂 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 =
𝐷𝐷𝐷𝐷𝐷 𝑈𝑈𝑈𝑈𝑈𝑈 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶

The level of OP affects both write performance and endurance. The higher the OP, the larger the
available pool of flash blocks for background operations and the more efficient the flash
controller can run its wear leveling algorithms, resulting in better performance and endurance;
however, higher levels of OP affect the cost of the device and most manufacturers use a
conservative value in the factory (i.e. 7%). The firmware in the device often allows for
downstream users to alter the OP value for the drive to be higher or lower than the default set
at the factory.

23
EMC CONFIDENTIAL – INTERNAL USE ONLY
• Wear Leveling – the Wear Leveling Factor (WLF) expresses the maximum stress that the wear
leveling method would put onto the most heavily cycled block compared to the average number
of cycles. A WLF of 2x would mean that the manufacturer is assuming that the most heavily
cycled block would have twice the number of cycles compared to the average. Wear leveling
managed by the flash drive controller which uses an algorithm designed by the drive
manufacturer to best determine where to place the data as to minimize the premature wear on
the physical flash blocks. The repeated use of a limited number of blocks can cause the flash
device to prematurely wear out due to excessive P/E cycling. Effective wear leveling algorithms
spread writes over all the flash memory cells, ideally equalizing the cell usage and extending the
device life. However, excessive wear leveling activity on the drive, due to factors such as high
random write workloads, can have a detrimental impact on the endurance.

• Write Amplification - Write amplification is the most important factor which has a detrimental
impact on the endurance of a flash drive. The write amplification factor (WAF) refers to the ratio
of writes committed to the flash device to the writes coming from the host system. A WAF
factor of 2x means that the drive manufacturer is assuming that for every megabyte written by
the host, two megabytes are written to the flash device. WAF on flash drives occurs due to the
nature of flash itself, where a full block of flash memory must be erased before a single host
write can be written to a single page. In other words, the process to perform a single write
operation results in moving or rewriting the entire block. The rewriting of data requires an
already used portion of flash to be read, updated and written to a new location. This means that
much larger portions of flash must be erased and rewritten than are actually required by the
amount of new host data. The moving and rewriting of this extra data also slows down the flash
drive’s write performance. This is one of the primary reasons why flash drives are much faster at
read operations than write operations. The extra erase and write cycles incurred by write
amplification result in more P/E cycles on the device which can significantly reduce the overall
endurance of the flash drive.

Note: Minimizing flash drive WAF is one of the most important features which enterprise flash
drives need to accomplish to help maintain flash drive endurance. Also, enterprise class all flash
storage arrays will also have architecture and embedded features which minimize WAF as well.

The following table lists the various flash drive parameters and features. It also shows how they
have a positive (decreasing) or negative (increasing) impact on a flash drive’s WAF.

24
EMC CONFIDENTIAL – INTERNAL USE ONLY
Table 6. Flash Drive Parameters and Influence on Write Amplification

Flash Drive Factor Description Relationship to WAF

Garbage Collection The efficiency of the algorithm used to pick Positive - As garbage
the next best block to erase and rewrite. Some collection increases,
flash controllers implement background WAF decreases
garbage collection (BGC), where the controller
uses idle time to consolidate blocks of flash
memory before the host needs to write new
data. This enables the performance of the
device to remain high. In 2010, some
manufacturers (notably Samsung) introduced
SSD controllers that extended the concept of
BGC to analyze the file system used on the
SSD, to identify recently deleted files and un-
partitioned space.
Over Provisioning On device Over Provisioning (OP) is the Positive - As Over
difference between the physical capacity of Provisioning increases,
the flash memory and the logical capacity WAF decreases
presented through the operating system (OS)
as available for the user. During the garbage
collection, wear-leveling, and bad block
mapping operations on the flash drive, the
additional space from Over Provisioning helps
lower the write amplification when the
controller writes to the flash memory.

TRIM / SCSI UNMAP These commands must be sent by the Positive - TRIM /
Commands operating system (OS) which tells the storage UNMAP commands
device which sectors contain invalid data. SSDs from the OS result in
consuming these commands can then reclaim decreasing WAF
the pages containing these sectors as free
space when the blocks containing these pages
are erased instead of copying the invalid data
to clean pages.
Secure Erase Erases all user data and related metadata Positive - Secure erase
which resets the SSD to the initial out-of-box frees up blocks for
performance (until garbage collection write operations,
resumes) decreasing WAF
Wear Leveling (WLF) Each time data are relocated without being Negative - Wear
changed by the host system, this increases the leveling being
write amplification and thus reduces the life of performed on the flash
the flash memory. The key is to find an drive increases WAF
optimum algorithm which maximizes them due to the movement
both. of data

25
EMC CONFIDENTIAL – INTERNAL USE ONLY
 High Sequential When an SSD is writing data sequentially, the Positive High sequential
Write Workloads write amplification is equal to one meaning write workloads are the
there is no write amplification best type of write
workload for flash
devices as there is no
WAF.
High Random Write Once the blocks are all written to on a flash Very Negative - High
Workloads drive, garbage collection begins and the random write
performance is limited by the speed and workloads are the
efficiency of that process. Write amplification absolute worst flash
during this phase will increase to the highest drive workloads as the
levels. background garbage
collection occurring on
the flash device greatly
increases WAF.

RAID Protection RAID protection schemes can result in write Negative – RAID 1, 5,
amplification in the following manner: RAID 1 and 6 protections result
and RAID5 result in 1 parity write for every in increased WAF.
data write so the WAF factor is 2. RAID 6
results in 2 parity writes for every data write
so the WAF factor is 3. RAID 0 results in WAF
value of 1.
Data Compression Write amplification goes down and SSD speed Positive - Compression
and Data goes up when data compression and will reduce the amount
Deduplication deduplication eliminates more redundant data of redundant data,
which decreases WAF.

Drive Writes per Day

Drive Writes per day (DPWD) or Writes per Day (WPD) has traditionally been the industry metric used to
specify the endurance limit of a flash drive. This metric is a measure of how many times the drive could
be completely filled and erased over the course of its manufacturer’s warranty period. Each
manufacturer has their own method of calculating DPWD, but it can be generally expressed by the
following formula:

(𝑀𝑀𝑀 P/E 𝐶𝐶𝐶𝐶𝐶) ∗ (1 + 𝑂𝑂)

𝐷𝐷𝐷𝐷 =
(365 𝑑𝑑𝑑𝑑 ∗ 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤 𝑖𝑖 𝑦𝑦𝑦𝑦𝑦) ∗ 𝑊𝑊 ∗ 𝑊𝑊𝑊

As an example, using a 1.5 TB usable capacity drive with 30000 PE Cycle rating, with 7% Over
Provisioning, along with a five year warranty period; and also assuming a WAF of 4, and WLF of 4 yields
the following:

DWPD = (30000 * 1.07) / ((365 * 5 years)*(4*4)) = 1.1 DWPD ~ 1 DWPD

The above DWPD value states that the example drive could sustain a little over 1 full write (a 1.5 TB fill
and erase) per day over the course of its five year warranty period. A drive with this level of endurance
is referred to a 1 drive write per day drive (1 DWPD).
26
EMC CONFIDENTIAL – INTERNAL USE ONLY
To put the matter of flash drive endurance in perspective, consider that most client workloads (8K block,
20% random write, 80% random read) rarely exceed .10 DPWD. This means that a flash drive with 1
write per day endurance rating and a warranty of five years could theoretically last ten times its
warranty period (up to 50 years) with typical client workloads. To put this matter into further
perspective, consider that a traditional 10K RPM hard drive, running a typical client workload, could
wear out simply due to mechanical issues well prior to the 5 year warranty period.

One thing to remember is that flash endurance numbers can change significantly if the workload is
heavy random write. In this case, flash drive WAF values can increase to 10x, meaning that in the
example, the DWPD resulting calculation would yield a value .44 DWPD. This is why enterprise flash
drive and enterprise flash storage array manufacturers put significant effort into minimizing WAF. When
considering an all flash array for the enterprise, the importance selecting flash drives and storage array
architecture which supports WAF minimization cannot be over-emphasized.

Terabytes Written
Terabytes Written (TBW) expresses the total amount of data that can be written into a flash drive before
it is likely to fail. When a manufacturer expresses a drive’s Terabytes Written capability in a specification
sheet, they are most often using the following formula:

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑊𝑊𝑊𝑊𝑊𝑊𝑊 = 𝐷𝐷𝐷𝐷 ∗ 365 ∗ 5 ∗ 𝐷𝐷𝐷𝐷𝐷 𝑈𝑈𝑈𝑈𝑈𝑈 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶

The following example demonstrates a simple TBW calculation for a 1.5 TB flash drive with a 1 DWPD
rating as specified by the manufacturer:

TBW = (1 *365*5*1.5 TB) = 2737.5 TBW

The 2737.5 TBW value means that the manufacturer estimates that the 1.5 TB drive used in the example
could have a total of 2737.5 TB (2.74 Petabytes) written to it before it would reach its end of life.

Comparing Endurance of Flash Drives vs. Traditional HDD

EMC was the first storage array vendor to use flash drives in its products. EMC has performed an in
depth study comparing the mean time between replacement (MTBR) rates of hard drives compared to
flash drives in their storage arrays. The data from this study clearly showed that the replacement rate
for flash drives in EMC arrays was 3x less than the replacement rate for traditional hard drives. This is in
line with other industry studies which examined hard drive failure rates during the typical five year
warrantee period. A 2013 study by cloud storage company BackBlaze calculated that around five
percent of hard drives failed during the sixteen months of the warrantee period and only 74 percent of
drives lasted beyond the fourth year of the warrantee. This produced a mean time to failure rate (MBTF)
of less than 1.5 million hours. Almost all enterprise flash drive manufacturers warranty their products
for a MTBF rate of 2.0 million hours. Based on real industry studies, it can be stated with confidence that
the reliability of flash drives is far exceeding traditional HDD.

27
EMC CONFIDENTIAL – INTERNAL USE ONLY
Flash Endurance Conclusions
Both of the metrics TBW and DWPD are calculations used manufacturers to express flash drive
endurance. Both metrics are valid methods; but when looking at endurance, the TB values expressed in
the TBW calculation tend to put flash drive endurance into a sharper focus. As flash drives grow larger in
capacity, the amount of data which can be put onto a flash drive during its warranty period is truly
staggering. For example, use a 16 TB drive to replace of the 1.5 TB used in the TBW calculation in this
section. When using the same variables for the other parameters in the formula, the TBW calculation for
a 16 TB drive is 29200 TBW (almost 30 Petabytes). To put this into perspective, consider the following:

a) It is estimated that the human brain's ability to store memories is equivalent to about 2.5
petabytes of binary data
b) The experiments in the Large Hadron Collider produce about 15 petabytes of data per year
c) World of Warcraft uses 1.3 petabytes of storage to maintain its game
d) One petabyte of average MP3-encoded songs (for mobile, roughly one megabyte per
minute), would require 2000 years to play
e) The 2009 movie Avatar is reported to have taken over 1 petabyte of local storage at Weta
Digital for the rendering of the 3-D CGI effects
f) As of May 2013, Netflix had 3.14 petabytes of video "master copies," which it compresses
and converts into 100 different formats for streaming

The probability of having to write and erase 30 Petabytes of data onto a single 16 TB flash drive within
its 5 year warranty period is extraordinarily small. It is much more likely that the warranty period for the
drive expires before even a small fraction of the TBW rating is reached.

Conclusion
Modern flash storage technology was introduced in the late 1980s. However, the major breakthroughs
which have made flash storage capable of delivering the performance, density, and the economics
required by the enterprise data center have dramatically progressed in the last two years with the
development of TLC and 3-D NAND. The combination of these two technologies is now allowing flash
storage devices to reach capacities of 16 TB by the end of 2016.

Like every technology, flash storage has its strengths and weaknesses. Its peculiarities such as write
amplification are mitigated by valuable flash drive design and exceptional flash storage array
architecture. When these factors are considered and developed into a solution, flash storage is every bit
as durable and reliable (if not more) as its HDD counterparts for all workloads. The arrival of the larger
capacity flash drives and storage arrays which use them will fundamentally change the storage
landscape in the enterprise data center.

28
EMC CONFIDENTIAL – INTERNAL USE ONLY
References
Larrivee, Steve, “Solid State Drive Primer”, Cactus Technologies, June 8, 2015

“NAND Basics – Understanding the Technology Behind Your SSD”, August 2013, Samsung Electronics

“A Flash Storage Technical and Economic Primer", [Link], March 30, 2015

“Understanding TLC NAND”, [Link], February 23, 2012

Swami,Vijay, “A primer on flash and a look into the challenges of designing an all flash array”, November
11 , 2013

“Samsung V-NAND Technology”, April 2014, Samsung Electronics

29
EMC CONFIDENTIAL – INTERNAL USE ONLY