Technical Report

NetApp Data Compression, Deduplication,

and Data Compaction
Data ONTAP 8.3.1 and Later
Karthik Viswanath, NetApp
February 2018 | TR-4476

Abstract
This technical report focuses on implementing NetApp® data compression, NetApp
deduplication, and NetApp data compaction running on NetApp Data ONTAP® software
version 8.3.1 and later.
TABLE OF CONTENTS

1 Overview ................................................................................................................................................ 5

2 NetApp Storage Efficiency Guarantee................................................................................................ 6

2.1 Storage Efficiency Ratio Visualization .............................................................................................................6

My AutoSupport ...................................................................................................................................................... 6

Starting with ONTAP 9.2, data compaction and aggregate inline deduplication savings are combined and
displayed as a single ratio. ..................................................................................................................................... 7

ONTAP CLI............................................................................................................................................................. 7

OnCommand System Manager ............................................................................................................................ 11

How to Calculate the Data Reduction Ratio ......................................................................................................... 12

3 NetApp Data Compaction .................................................................................................................. 14

Performance ......................................................................................................................................................... 15

Space Savings...................................................................................................................................................... 16

Notes on Data Compaction .................................................................................................................................. 17

4 NetApp Deduplication ........................................................................................................................ 18

4.1 Inline Volume Deduplication..........................................................................................................................19

4.2 Inline Aggregate Deduplication (IAD) or Cross-Volume Deduplication .........................................................19

Performance ......................................................................................................................................................... 20

Space Savings...................................................................................................................................................... 20

Operation .............................................................................................................................................................. 25

Notes on inline aggregate deduplication: .............................................................................................................. 26

4.3 How the Inline Deduplication Engine Identifies Duplicates ...........................................................................26

Deduplicated FlexVol Volumes and Infinite Volume/FlexGroup Data Constituents .............................................. 27

4.4 Background (Postprocess) Volume Deduplication ........................................................................................27

4.5 Background (Postprocess) Aggregate Deduplication ....................................................................................29

5 NetApp Data Compression ................................................................................................................ 30

Difference Between Adaptive Compression and Secondary Compression .......................................................... 30

5.1 How NetApp Data Compression Works ........................................................................................................30

Compression Groups ............................................................................................................................................ 30

Compressed Writes .............................................................................................................................................. 31

Compressed Reads .............................................................................................................................................. 31

5.2 When Data Compression Runs.....................................................................................................................31

Inline Operations .................................................................................................................................................. 31

Postprocess Operations ....................................................................................................................................... 32

5.3 New Features and Behavior of Compression................................................................................................32

2 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 New features introduced in Data ONTAP 8.3.1: ................................................................................................... 32

New features introduced in Data ONTAP 8.3.2: ................................................................................................... 32

Compression Behavior Based on Configuration ................................................................................................... 33

6 General Compression and Deduplication Features ........................................................................ 34

6.1 Compression and Deduplication System Requirements ...............................................................................34

Requirements Overview ....................................................................................................................................... 35

6.2 Maximum Logical Data Size Processing Limits.............................................................................................35

6.3 When to Enable Compression and/or Deduplication .....................................................................................35

General Compression Usage Guidelines ............................................................................................................. 35

7 Space Savings .................................................................................................................................... 36

8 Performance ........................................................................................................................................ 38

9 Feature and Configuration Interoperability ..................................................................................... 40

10 General Workload-Based Storage Efficiency Guidelines ............................................................... 41

11 Best Practices ..................................................................................................................................... 42

Best Practices for Optimal Savings and Minimal Performance Overhead ............................................................ 42

12 Upgrade and Revert............................................................................................................................ 43

12.1 Upgrading to a Later Version of Data ONTAP ..............................................................................................44

12.2 Reverting to an Earlier Version of Clustered Data ONTAP ...........................................................................47

13 Interoperability .................................................................................................................................... 48
13.1 Data Protection .............................................................................................................................................48

13.2 Other NetApp Features .................................................................................................................................52

References ................................................................................................................................................. 73

Version History ......................................................................................................................................... 73

Contact Us ................................................................................................................................................. 74

LIST OF FIGURES
Figure 1) My AutoSupport AFF Efficiency Calculator. ....................................................................................................7
Figure 2) The storage aggregate show-efficiency CLI command. ..................................................................8
Figure 3) ONTAP 9.2 storage aggregate show-efficiency CLI command. ......................................................9
Figure 4) ONTAP 9.2 storage aggregate show-efficiency –details CLI command. ................................10
Figure 5) ONTAP 9.2 storage aggregate show-efficiency –details CLI command. ................................10
Figure 6) ONTAP 9.0 and 9.1: OnCommand System Manager Efficiency and Capacity dashboard. ...........................11

3 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 7) ONTAP 9.2 and later: OnCommand System Manager Efficiency and Capacity dashboard. .........................12
Figure 8) ONTAP 9.0 and 9.1: Data reduction ratio calculation example. ....................................................................13
Figure 9) How data compaction works. ........................................................................................................................14
Figure 10) Data compaction example. ..........................................................................................................................15
Figure 11) Data compaction savings: Oracle Database and ONTAP 9. .......................................................................16
Figure 12) Data compaction savings: SQL Server 2014 and ONTAP 9. ......................................................................17
Figure 13) How NetApp deduplication works at the highest level. ................................................................................18
Figure 14) Inline aggregate deduplication savings: Microsoft SQL Server 2014 and ONTAP 9.2. ...............................21
Figure 15) Inline aggregate deduplication savings: MongoDB and ONTAP 9.2. ..........................................................21
Figure 16) Inline aggregate deduplication savings: VDI and ONTAP 9.2. ....................................................................22
Figure 17) Inline aggregate deduplication savings: VDI and ONTAP 9.2. ....................................................................22
Figure 18) Inline aggregate deduplication savings: VDI and ONTAP 9.2. ....................................................................23
Figure 19) Inline aggregate deduplication savings: VDI and ONTAP 9.2. ....................................................................23

LIST OF TABLES
Table 1) Overview of compression and deduplication requirements for clustered Data ONTAP. .................................35
Table 2) Typical AFF storage efficiency ratios with ONTAP 9 and later. ......................................................................37
Table 3) Typical FAS storage efficiency ratios with ONTAP 9 and later. ......................................................................37
Table 4) Typical AFF storage efficiency ratios with ONTAP 8.3.x. ...............................................................................38
Table 5) Typical FAS storage efficiency ratios with ONTAP 8.3.x. ...............................................................................38
Table 6) Feature configuration interoperability. ............................................................................................................40
Table 7) Workload-based storage efficiency guidelines. ..............................................................................................41
Table 8) Compression behavior after a Data ONTAP 8.3.1 upgrade: AFF configuration. ............................................45
Table 9) Compression behavior after a Data ONTAP 8.3.1 upgrade: FAS configuration. ............................................47
Table 10) Compression behavior after volume move from pre-8.3.1 AFF node to 8.3.1+ AFF node. ..........................56
Table 11) Compression behavior after volume move from pre-8.3.1 FAS node to 8.3.1+ FAS node. ..........................58
Table 12) Compression behavior after volume move from 8.3.1+ FAS node to 8.3.1+ AFF node. ..............................59
Table 13) Compression behavior after volume move from pre-8.3.1 FAS node to 8.3.1+ AFF node. ..........................63
Table 14) Summary of LUN configuration examples. ...................................................................................................70

4 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
1 Overview
One of the biggest challenges for companies today is the cost of storage. Storage is the largest and
fastest growing IT expense. The NetApp portfolio of storage efficiency technologies is aimed at lowering
this cost. NetApp deduplication and data compression are two key components of NetApp storage
efficiency technologies that enable users to store the maximum amount of data for the lowest possible
cost.
This paper focuses mainly on two features: NetApp deduplication and NetApp data compression. These
technologies can work together or independently to achieve optimal savings. Both have the ability to run
either as an inline process as data is written to disk or as a scheduled process. When the two are enabled
on the same volume, the data is first compressed and then deduplicated. Deduplication removes
duplicate compressed or uncompressed blocks in a data volume. Although compression and
deduplication work well together, the savings are not necessarily the sum of the savings when each is run
individually on a dataset.
[July 2016] Updated to include NetApp data compaction, a new storage efficiency technology, and the 4:1
storage efficiency guarantee program, both of which are available with ONTAP® 9.
[January 2017] Updated to include changes to the NetApp Storage Efficiency Guarantee program and
updates on storage efficiency ratio estimates.
[May 2017] Updated to include aggregate inline deduplication, a new storage efficiency technology
available with ONTAP 9.2.
[January 2018] Updated to include background aggregate deduplication and automatic deduplication,
both of which are available with ONTAP 9.3.
As the title indicates, this technical report covers Data ONTAP 8.3.1 and later. For information about
implementation with earlier versions of Data ONTAP, refer to:
• TR-3505: NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide
• TR-3505i: NetApp Deduplication and Data Compression Deployment and Implementation Guide
• TR-3958: NetApp Data Compression and Deduplication Deployment and Implementation Guide, Data
ONTAP 8.1 Operating in 7-Mode
• TR-3966: NetApp Data Compression and Deduplication Deployment and Implementation Guide
The same information applies to FAS, V-Series, and NetApp FlexArray® systems, unless otherwise noted.
Whenever references are made to volumes, the references apply to both NetApp FlexVol® volumes and
Infinite Volumes.
For information about Infinite Volumes, see TR-4178: NetApp Infinite Volume Deployment and
Implementation Guide.

5 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
2 NetApp Storage Efficiency Guarantee
NetApp guarantees that the use of NetApp’s storage efficiency technologies on All Flash FAS (AFF)
systems will reduce the logical capacity needed to store Customer’s data by the ratio shown using the
NetApp Synergy sizing tool for that particular workload or combination of workloads. Data that has
already been compressed, deduplicated, or encrypted is excluded from this Guarantee.
The guarantee is only available for AFF systems running ONTAP 9.0 or later. To qualify for the
guarantee, the AFF systems should be configured with NetApp best practices.

How the Guarantee Works

1. NetApp sales representative or partner sales representative work with the customer to gather
workload and capacity requirements.
2. Sales representative uses NetApp Synergy sizing tool to derive the expected data reduction ratio
based on the information gathered from the customer and generates an efficiency configuration with a
written guaranteed data reduction ratio/effective capacity ratio.
3. Customer fills out a guarantee form along with configuration details such as serial number(s), node
name(s), aggregate name(s), volume name(s), controller location(s), description of the specific
workload(s) (such as home directories, virtual servers, SQL databases, etc.) before or at time of
implementation of NetApp system.
4. The guarantee period starts from the date of installation of the system. Throughout the guarantee
period, customer:
a. Uses all default storage efficiency technologies available with the Flash System (including
compression, thin provisioning, deduplication and compaction)
b. Follows best practices that are published by NetApp. Refer to the KB article about how to
maximize efficiency on ONTAP 9.x systems.
c. Runs version 9.0 or later of the NetApp ONTAP operating system.
d. Continuously enables NetApp AutoSupport (ASUP™).
5. At the end of the guarantee period, customer measures the data reduction ratio on the NetApp
system. If the NetApp system is not delivering the promised data reduction ratio, NetApp provides
free additional disks, as needed, to meet the written guarantee.
To participate and to request additional information, contact your NetApp sales or partner sales
representative.

2.1 Storage Efficiency Ratio Visualization

As part of the storage efficiency guarantee program, to validate the efficiency guarantee, the following
storage efficiency ratio visualizations are available for AFF systems running ONTAP 9 or later.

My AutoSupport
My AutoSupport® has introduced a new tool called Flash Efficiency Calculator that displays the storage
efficiency ratios. It takes serial number of the controller as input and displays the following ratios:
• Cumulative storage efficiency for the cluster
• Cumulative storage efficiency for the nodes in the cluster along with:
 Deduplication and compression node ratios
 Snapshot™ and clone node ratios
 Logical used capacity and physical used capacity for the node
 Cumulative storage efficiency for the aggregates in the node:

6 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
  Cumulative storage efficiency for the volumes in the aggregate

Figure 1) My AutoSupport AFF Efficiency Calculator.

Selecting this
checkbox
displays Data
Reduction
Ratio

Flash Efficiency Calculator can be accessed from the My AutoSupport homepage or using this direct link.

Starting with ONTAP 9.2, data compaction and aggregate inline deduplication savings
are combined and displayed as a single ratio.

ONTAP CLI
[May 2017] With ONTAP 9.0, NetApp introduced a new command, storage aggregate show-
efficiency, to report aggregate-level efficiency ratios along with space-used details. Starting with
ONTAP 9.2, the storage aggregate show-efficiency command has been enhanced to
display the data reduction ratio and has been formatted for better readability.

7 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
ONTAP 9.0 and ONTAP 9.1
The following efficiency ratios are displayed as part of the storage aggregate show-efficiency
command in ONTAP 9.0 and ONTAP 9.1:
• Cumulative storage efficiency ratio
• Aggregate storage efficiency ratio/data compaction storage efficiency ratio
• Volume storage efficiency ratio/data reduction storage efficiency ratio (efficiency from deduplication,
compression, and zero-block deduplication)
• NetApp Snapshot and FlexClone® storage efficiency ratios

Figure 2) The storage aggregate show-efficiency CLI command.

Note: The aggregate show-efficiency command is available only in advanced mode. The
command output is also available through a new NetApp Manageability SDK called aggr-storage-
efficiency-get-iter.

ONTAP 9.2 and Later

Starting with ONTAP 9.2, the aggregate show-efficiency command supports three views:
• Default view (without any options)
• Detailed view (with -details command option)
• Advanced view (with -advanced command option)
Depending on the option passed to the aggregate show-efficiency command, it displays different
ratios and space-used details, as described below.

8 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
1. aggregate show-efficiency (without any options)
When used without any options, the aggregate show-efficiency CLI command displays the
total storage efficiency ratio for all the aggregates in the cluster. This command is available in admin
mode.

Figure 3) ONTAP 9.2 storage aggregate show-efficiency CLI command.

2. aggregate show-efficiency –details

When used with the -details option, the aggregate show-efficiency CLI command displays
the following for each aggregate in the cluster:
 Total storage efficiency ratio (includes Snapshot copies and clones)
 Total data reduction ratio (zero-block deduplication, volume deduplication, aggregate inline
deduplication, compression, and data compaction)
 Aggregate efficiency ratio (data compaction and aggregate inline deduplication combined
savings)
 Volume deduplication ratio
 Compression ratio
 Snapshot ratio
 FlexClone ratio

9 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 4) ONTAP 9.2 storage aggregate show-efficiency –details CLI command.

3. aggregate show-efficiency –advanced

When used with the -advanced option, the aggregate show-efficiency command displays the
ratios as displayed by aggregate show-efficiency -details, along with the space used for
each of the efficiency technologies.
Note: This command option is available only in advanced mode.

Figure 5) ONTAP 9.2 storage aggregate show-efficiency –details CLI command.

10 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
OnCommand System Manager
On AFF systems running ONTAP 9.x, the OnCommand® System Manager dashboard shows a new tab,
called Efficiency, under the Efficiency and Capacity panel. The tab displays the savings ratio at the cluster
and node levels, along with details of logical space used and physical space used.
In ONTAP 9.0 and 9.1 systems, the System Manager dashboard displays:
- Total storage efficiency ratio
- Volume storage efficiency/data reduction storage efficiency ratio (deduplication, compression, and
zero-block deduplication)
- Aggregate storage efficiency/data compaction storage efficiency ratio
- Snapshot and FlexClone storage efficiency ratios
Note: In ONTAP 9.0 and 9.1, OnCommand System Manager does not display the overall data reduction
ratio at the node/cluster level.

Figure 6) ONTAP 9.0 and 9.1: OnCommand System Manager Efficiency and Capacity dashboard.

[May 2017] Starting with ONTAP 9.2, the OnCommand System Manager dashboard is enhanced to
display the following information
- Total storage efficiency ratio
- Total data reduction ratio (zero-block deduplication, volume deduplication, aggregate inline
deduplication, compression, and data compaction)
- FlexClone efficiency ratio
- Snapshot efficiency ratio

11 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 7) ONTAP 9.2 and later: OnCommand System Manager Efficiency and Capacity dashboard.

How to Calculate the Data Reduction Ratio

In ONTAP 9.2 systems, you can use any of the following methods to calculate data reduction ratios:
• My AutoSupport AFF Efficiency Calculator Dashboard to get the aggregate-level, node-level, and
cluster-level data reduction ratios
• OnCommand System Manager to get the node-level and cluster-level data reduction ratio
• The ONTAP CLI command aggregate show-efficiency -details or aggregate show-
efficiency -advanced to get the aggregate-level data reduction ratio
In ONTAP 9.0 and 9.1 systems, you can use the My AutoSupport AFF Efficiency Calculator Dashboard to
get aggregate-level, node-level, and cluster-level data reduction ratios.
On systems that do not have AutoSupport enabled, you can use the following formula, combined with the
aggregate show-efficiency CLI command output, to calculate the data reduction ratio.

12 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 8 shows the calculation for one of the aggregates (n06_ssd_aggr1).

Figure 8) ONTAP 9.0 and 9.1: Data reduction ratio calculation example.

(228.15𝑇𝐵 − 200.16𝑇𝐵 − 0𝑇𝐵) yields (27.99𝑇𝐵) yields

→ → 2.26:1
(15.38𝑇𝐵 − ((3.16𝑇𝐵 + 0𝑇𝐵)/1.06 (12.4𝑇𝐵)

n06_ssd_aggr1
Data Reduction Ratio

13 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
To calculate node-level and cluster-level data reduction ratios:
1. Calculate Data Reduction Logical Used and Data Reduction Physical Used for all the aggregates in
the node/cluster, using the formula just shown.
2. Sum Data Reduction Logical Used and Data Reduction Physical Used for all the aggregates in the
node/cluster to get Node/Cluster Data Reduction Logical Used and ‘Node/Cluster Data Reduction
Physical Used, respectively.
3. Divide Node/Cluster Data Reduction Logical Used by Node/Cluster Data Reduction Physical Used to
get the node/cluster data reduction ratio.

3 NetApp Data Compaction

Starting with ONTAP 9, NetApp has introduced a new patented storage efficiency technology called data
compaction that further reduces the physical used space needed to store data. It is a significant addition
to our storage efficiency portfolio and complements deduplication and compression technologies.
Data compaction takes I/Os that normally consume a 4K block on physical storage and pack multiple
such I/Os into one physical 4K block.

Figure 9) How data compaction works.

Data compaction is an inline operation and occurs after inline compression and inline deduplication. On
an AFF system, the order of execution is as follows:
1. Inline zero-block deduplication. All zero blocks are detected, and no user data is written to physical
storage; only metadata and reference counts are updated.
2. Inline adaptive compression. Compresses 8K logical blocks into 4K physical blocks; very efficient in
determining compressibility of the data and doesn’t waste lot of CPU cycles trying to compress
incompressible data.
3. Inline deduplication. Opportunistically deduplicates incoming blocks to already existing blocks on
physical storage.
4. Inline adaptive data compaction. Combines multiple <4K logical blocks into a single 4K physical
block to maximize savings. It also tries to compress any 4K logical blocks that are skipped by inline
compression to gain additional compression savings.
Data compaction is most beneficial for highly compressible data, very small I/Os and files (<2KB), and
larger I/Os with lots of white space.

14 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Inline data compaction is enabled by default on AFF systems and can optionally be enabled on Flash
Pool™ and HDD systems. On Flash Pool systems, only writes to HDDs are compacted. Data compaction
can be enabled on a per volume basis.
Data compaction can be run independently of deduplication or compression. It is always recommended to
run inline adaptive data compaction along with inline adaptive compression and inline deduplication to
maximize savings. Data compaction is an additive capability to inline compression as it can compress 4K
blocks that are skipped by inline compression. Because inline adaptive compression skips any blocks that
are 4K, the maximum compression savings that can be achieved with inline adaptive compression are
2:1. When inline adaptive data compaction is used along with inline compression, inline adaptive
compression savings can be >2:1. Data compaction helps multiply compression savings.
Figure 10 shows the benefits of using inline data compaction and how inline adaptive compression
savings are multiplied when used along with inline data compaction.

Figure 10) Data compaction example.

1. Figure 10 shows I/Os from three volumes: Volume A with three 8K I/Os (one of which is 50%
compressible, and two are 80% compressible), Volume B with two 4K I/Os (both are 55%
compressible), and Volume C with three 1K I/Os.
2. Without compression or data compaction, the incoming I/Os would have consumed a total of 11 4K
blocks on physical storage. The 1K I/Os from Volume C each require 4K because the minimum block
size in WAFL® is 4K.
3. If inline adaptive compression is used, the 50% compressible 8K I/O from Volume A would be
compressed to 4K, and the two 80% compressible 8K I/Os from Volume A and the three 1K I/Os from
Volume C would also consume 4K each on the physical storage because of the WAFL 4K block size,
resulting in a total of 8 4K blocks on physical storage.
4. With inline adaptive data compaction, the two 80% compressible 8K I/Os from Volume A would be
packed into a single 4K block, the two 55% compressible 4K I/Os from Volume B would be packed
into another 4K block, and the three 1K I/Os from Volume C would be packed into another 4K block,
resulting in a total of 4 4K blocks on physical storage.

Performance
Data compaction by itself does not have any additional performance overhead, but because data
compaction also tries to compress 4K blocks that are skipped by inline compression, there could be an
additional 1% to 2% CPU cycles consumed. As with adaptive compression, data compaction has built-in
heuristics to efficiently detect incompressible blocks.

15 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Space Savings
Space savings from data compaction are highly dependent on the incoming I/O, compressibility, and the
existing data layout on physical storage. Even for the same dataset, it can vary from one system to
another. So far, we have conducted tests on Oracle database and SQL Server 2014 to measure data
compaction savings. Figures 11 and 12 summarize the results. For more information about configuration,
test setup, and detailed results, refer to the following TRs:
 NetApp AFF8080A EX Storage Efficiency and Performance with Oracle Database
 NetApp AFF8080A EX Storage Efficiency and Performance with Microsoft SQL Server 2014

Figure 11) Data compaction savings: Oracle Database and ONTAP 9.

16 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 12) Data compaction savings: SQL Server 2014 and ONTAP 9.

Space Savings Estimation Tool (SSET) has been updated to include data compaction savings estimates.
Starting ONTAP 9.1, NetApp had added support to compact existing data using background compaction
scanner. Use the ‘–compaction true’ option with ‘volume efficiency start …’ command to run
background compaction scanner.

volume efficiency start –vsever <$SVM_NAME> -volume <$VOLUME_NAME> -

compaction true

Notes on Data Compaction

1. Volumes must be thin provisioned (no space guarantee).
2. It works at the aggregate (physical block) level using reference counts. It does not change logical
data.
3. Blocks that are inline deduped are not compacted. Compacted blocks can be deduped by
background dedupe.
4. It can reduce space consumed by Snapshot copies:
 A CLI command to compact individual files in a Snapshot copy is available.
 The command is only available in diagnostic mode and should be used in consultation with
NetApp Support.
5. For accurate space savings reporting, aggregate space utilization should be >10%:
 Data compaction savings are reported at an aggregate level because data compaction works at
the physical block level.
 Data compaction savings for an aggregate can be visualized using the CLI ‘aggregate show-
efficiency’ advanced mode command. System Manager dashboard and My AutoSupport
Flash Efficiency Calculator also display aggregated data compaction savings for a node or
cluster. For more details, see the “Storage Efficiency Ratio Visualization” section, earlier.

17 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
4 NetApp Deduplication
Part of NetApp’s storage efficiency offerings, NetApp deduplication provides block-level deduplication
within a FlexVol volume or data constituent. Essentially, deduplication removes duplicate blocks, storing
only unique blocks in the FlexVol volume or data constituent. Deduplication creates a small amount of
additional metadata in the process.
[May 2017] Starting with ONTAP 9.2, NetApp has expanded existing deduplication technology to allow
block-level sharing across volumes within an aggregate. Inline Aggregate deduplication further reduces
the physical used space needed to store data.
[Jan 2018] Starting with ONTAP 9.3, NetApp has added support for Background Aggregate deduplication.

Figure 13) How NetApp deduplication works at the highest level.

Notable features of deduplication include:

• It works with a high degree of granularity; that is, at the 4KB block level.
• It operates on the active file system of the FlexVol volume or data constituent. Any block referenced
by a NetApp Snapshot copy is not available until the Snapshot copy is deleted.
• It is application transparent, and therefore it can be used to deduplicate data originating from any
application that uses the NetApp system.
• It is enabled and managed by using a simple CLI or GUI such as System Manager or NetApp
OnCommand Unified Manager.
• Volume deduplication can be run as an inline process (starting in Data ONTAP 8.3.2) and/or as a
background process that can be configured to run automatically, be scheduled, or run manually
through the CLI, NetApp System Manager, or NetApp OnCommand Unified Manager.
• Aggregate deduplication is supported as an inline process in ONTAP 9.2.

18 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
4.1 Inline Volume Deduplication
Starting with Data ONTAP 8.3.2, inline deduplication is supported on AFF, HDD, all SSD, and NetApp
Flash Pool systems. Such deduplication is enabled by default on AFF systems. By using inline volume
deduplication in combination with background volume deduplication, users can achieve maximum
deduplication savings and a significant reduction in $/gigabyte and overall TCO, especially on AFF
deployments.
To enable inline deduplication on volumes residing within an HDD/all-SSD (non-AFF) aggregate, the
following option must be set on the controller: sis.idedup_allow_non_aff_hya on. Make sure that
you have done sufficient performance testing before enabling inline deduplication on HDD aggregates,
because spindle speed might become a bottleneck for inline deduplication performance. If you are not
sure whether to enable inline deduplication on your configuration, contact your NetApp representative or
NetApp Support.
On certain AFF systems, there is a way to simulate inline deduplication behavior in Data ONTAP 8.3.1 as
well. It is possible to use background deduplication with a very aggressive deduplication schedule to
maintain storage efficiency all the time. Doing so is essentially like running deduplication inline. This
procedure has proven to be valuable for VDI and server virtualization use cases. For more information
about always-on deduplication and VMware implementation, refer to TR-4428: NetApp All Flash FAS
Solution for VMware Horizon 6 and vSphere Virtual Volumes, TR-4307: NetApp All Flash FAS Solution for
Nonpersistent Desktops with VMware Horizon View, and TR-4335: NetApp All Flash FAS Solution for
Persistent Desktops with VMware Horizon View.
NetApp does not recommend using always-on deduplication on HDD and Flash Pool systems. That is
because running deduplication continuously in the background can adversely affect the performance of
the system, especially when disks are a bottleneck.

4.2 Inline Aggregate Deduplication (IAD) or Cross-Volume Deduplication

Starting with ONTAP 9.2, inline aggregate deduplication is supported on AFF systems, including AFF
MetroCluster™ systems. It is not supported on Flash Pool and HDD systems. Such deduplication is
enabled by default on AFF systems.
IAD is supported on all read/write FlexVol volumes, FlexGroup volumes, Infinite Volumes, and FabricPool.
In the case of FlexGroup and Infinite Volume, cross-volume sharing can happen only among data
constituents. In the case of FabricPool, cross-volume sharing happens only within the flash tier.
All volumes must be thin-provisioned (space guarantee set to None) for IAD to work.
IAD works with existing efficiency technologies such as zero-block deduplication, volume deduplication,
compression, and data compaction. Existing volume savings resulting from volume deduplication and
compression are not affected by IAD.
In the case of compressed volumes, only adaptively compressed volumes can participate in cross-volume
sharing. Secondary compressed volumes do not participate in cross-volume sharing in ONTAP 9.2.
On an AFF system, where all default inline efficiencies are enabled, the workflow is as follows:
1. Inline zero-block deduplication. Zeros are eliminated first. No user data is written to disk. Only
metadata is updated.
2. Inline adaptive compression. The data is scanned and compressed using an 8K compression
group.
3. Inline deduplication. The compressed blocks are scanned to identify duplicates:
a. The deduplication engine tries to find duplicates within the volume first. If duplicates are found,
the block is marked as duplicate. No user data is written to disk. Only metadata is updated.

19 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 b. If no duplicates are found within a volume, the deduplication engine tries to find duplicates across
volumes within the aggregate. If duplicates are found, the block is marked as duplicate. No user
data is written to disk. Only metadata is updated.
4. Inline data compaction. The compaction engine tries to compact blocks that were not inline
deduplicated.

Performance
Inline aggregate deduplication by itself does not have any additional performance overhead because all
the metadata is stored in memory. However, the performance overhead resulting from the entire inline
deduplication operation (inline volume deduplication and inline aggregate deduplication) is between 1%
and 2% in latency increase and throughput reduction. Overall performance overhead of all inline
efficiencies (inline compression, inline deduplication, inline zero-block elimination and inline data
compaction) is less than 10%.
As with inline volume deduplication, IAD is opportunistic and adaptive in nature, giving preference to
client operations over increased savings. It can throttle itself when the latency crosses a certain threshold
or throughput drops below a certain limit.

Space Savings
Space savings from inline aggregate deduplication depend on incoming I/O, similarity to data recently
written to disk, and the presence of fingerprint in the hash store. Primary use cases that benefit from IAD
are creating database copies, patching or updating multiple VMs across volumes, provisioning, patching
or recomposing virtual desktops, and splitting clones.
So far, we have conducted tests on Microsoft SQL Server 2014, MongoDB, and VDI to measure IAD
savings, along with overall savings.

Microsoft SQL Server 2014

A test was conducted to determine the efficiency of an OLTP workload against Microsoft SQL Server
2014 on a 2-node AFF8080CC running ONTAP 9.2.
The database was spread across nine volumes. The results show a total data reduction ratio of 2:1, an
inline aggregate deduplication ratio of 1.31:1, and an inline compression ratio of 1.68:1. There were no
inline volume deduplication savings.

20 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 14) Inline aggregate deduplication savings: Microsoft SQL Server 2014 and ONTAP 9.2.

MongoDB
A test was conducted to determine the efficiency of a public dataset on MongoDB on a 2-node AFF A200
running ONTAP 9.2. The database was spread across five volumes.
The results show a total data reduction ratio of 1.42:1, an inline aggregate deduplication ratio of 1.08:1,
and an inline compression ratio of 1.34:1. There were no inline volume deduplication savings.

Figure 15) Inline aggregate deduplication savings: MongoDB and ONTAP 9.2.

Virtual Desktop Infrastructure

The following VDI-related tests were conducted on a 2-node AFF8040 system to determine storage
efficiency. Each of the tests was run twice, first with inline aggregate deduplication turned off and then
with it turned on.
1. Linked clone provisioning. Provisioned 1,500 linked clone VMs across multiple 12.6TB volumes,
paced in such a way as to maintain a nonsaturated CPU.
Results of this test reflect approximately 33% (1.5:1) storage efficiency improvement with the addition
of inline aggregate deduplication and compaction over ONTAP 9.1 inline efficiencies. The results also

21 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 show a total data reduction ratio of 5.78:1, an inline volume deduplication ratio of 4.28:1, and an inline
compression ratio of 2:1.

Figure 16) Inline aggregate deduplication savings: VDI and ONTAP 9.2.

2. Linked clone recompose. In this test, 1,500 VMs were recomposed that had already been logged
into, the profile had been created, and the desktops had been rebooted.
Results of this test reflect approximately 27% (1.37:1) storage efficiency improvement with the
addition of inline aggregate deduplication and compaction over ONTAP 9.1 inline efficiencies. The
results also show a total data reduction ratio of 3.66:1, an inline volume deduplication ratio of 2.67:1,
and an inline compression ratio of 1.6:1.

Figure 17) Inline aggregate deduplication savings: VDI and ONTAP 9.2.

3. Full clone provisioning. Provisioned 1,500 full clone VMs across multiple 12.6TB volumes, paced in
such a way as to maintain a nonsaturated CPU
Results of this test reflect approximately 28% (1.38:1) storage efficiency improvement with the
addition of inline aggregate deduplication and compaction over ONTAP 9.1 inline efficiencies. The

22 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 results also show a total data reduction ratio of 43.4:1, an inline volume deduplication ratio of 36.6:1,
and an inline compression ratio of 7.81:1.

Figure 18) Inline aggregate deduplication savings: VDI and ONTAP 9.2.

4. Full clone patching. In this test, 1,500 full clone VMs that were already created were applied 1GB
patch, paced in such a way as to maintain a nonsaturated CPU.
Results of this test reflect approximately 27% (1.36:1) storage efficiency improvement with the
addition of inline aggregate deduplication and compaction over ONTAP 9.1 inline efficiencies. The
results also show a total data reduction ratio of 40.4:1, an inline volume deduplication ratio of 33.8:1,
and an inline compression ratio of 7.26:1.

Figure 19) Inline aggregate deduplication savings: VDI and ONTAP 9.2.

23 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
The current implementation of inline deduplication does not eliminate the need for postprocess
deduplication. NetApp recommends running postprocess deduplication in addition to inline deduplication
to maximize storage efficiency.

Space Savings Reporting

Space savings from cross-volume deduplication are reported as deduplication savings at the aggregate
level. Currently, data compaction savings are also reported at the aggregate level. Savings for both
compaction and cross-volume deduplication are reported combined.

Figure 20) ONTAP 9.2 AID space reporting: df –A –S.

Figure 21) ONTAP 9.2 AID space reporting: aggr show-efficiency.

Figure 22) OnCommand System Manager AID space reporting.

24 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 23) My AutoSupport AFF Efficiency Dashboard AID space reporting.

The ONTAP CLI and My AutoSupport report combine aggregate inline deduplication and data compaction
savings. OnCommand System Manager reports aggregate inline deduplication savings as part of the data
reduction ratio at the node and cluster levels. It does not report aggregate-level deduplication savings
separately.

Operation
In ONTAP 9.2, inline aggregate deduplication (cross-volume deduplication) is enabled by default on all
new volumes as well as on all upgraded volumes. In the case of existing volumes, only new writes to the
volumes start participating in cross-volume sharing. Existing data in the volumes is not deduplicated by
cross-volume processes, although new writes may start sharing blocks with existing data.
As noted earlier, inline aggregate deduplication is enabled by default for systems running 9.2. If for some
reason, it must be manually enabled, the preferred method to do so is using the volume efficiency modify
command. This method enables inline aggregate deduplication for the volume, as well as the aggregate
that contains the volume. After being enabled at the aggregate level, this method causes new volumes
created in that aggregate to also have it enabled.
Use the following ONTAP CLI commands to enable, disable, and check the status of cross-volume
deduplication.

Figure 24) AID operation: ONTAP 9.2 CLI: volume commands.

25 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Figure 25) AID operation: ONTAP 9.2 CLI: aggregate commands.

Notes on inline aggregate deduplication:

• Cross-volume shared blocks cannot be relocated.
• For blocks that were skipped by adaptive compression (8K CG compression) or secondary
compression (32K CG compression), data compaction engine tries to compress those blocks using
4K granularity before compacting. Such 4K compressed and compacted blocks cannot participate in
cross-volume sharing.
• IAD is not supported with NetApp Volume Encryption (NVE).
• Block Replication Engine (BRE; SnapMirror®, Volume Move, and Volume Copy) and Logical
Replication with Storage Efficiency (LRSE; SnapMirror and SnapVault®) do not preserve aggregate
deduplication savings. Volume deduplication savings are not affected.

4.3 How the Inline Deduplication Engine Identifies Duplicates

The core enabling technology of deduplication is fingerprints — unique digital signatures for every 4KB
data block. The inline deduplication engine maintains an in-memory data structure (the hash store) to
store the fingerprints of recently written blocks. The hash store is nonpersistent across reboots and
volume offline operations. The contents of the hash store are cycled based on a least recently used
algorithm, which evicts the least recently used block to make room for the new block whenever the hash
store gets full. The size of the hash store depends on the available system memory and varies across
controller models.
NetApp systems also maintain an in-memory cache called a buffer cache that stores recently read/written
4K data blocks.
When data is written to the system, after it goes through inline zero-block deduplication and inline
compression, the inline operation scans the incoming blocks, creates a fingerprint, and stores the
fingerprint in hash store.
After the fingerprint is computed, a lookup in the hash store is performed. Upon a fingerprint match in the
hash store, the data block corresponding to the duplicate fingerprint (donor block) is searched in the
buffer cache:
• If found, a byte-by-byte comparison is done between the current data block (recipient block) and the
donor block as verification to make sure of an exact match. On verification, the recipient block is
shared with the matching donor block without an actual write of the recipient block to disk. Only
metadata is updated to track the sharing details.
• If the donor block is not found in the buffer cache, the donor block is prefetched from disk into the
buffer cache to do a byte-by-byte comparison to make sure of an exact match (ONTAP 9 and later).
On verification, the recipient block is marked as duplicate without an actual write to disk. Metadata is
updated to track sharing details.
 In a Data ONTAP 8.3.2 system, if the donor block is not found in the buffer cache, the current
data block is not marked as duplicate and written to disk.

26 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Deduplicated FlexVol Volumes and Infinite Volume/FlexGroup Data Constituents
A deduplicated volume is a FlexVol volume or data constituent that contains shared data blocks. Data
ONTAP supports shared blocks in order to optimize storage space consumption. Basically, in one FlexVol
volume or data constituent, it is possible to have several references to the same data block. Starting in
ONTAP 9.2, data blocks can be shared across volumes within an aggregate.
In Figure 26, the number of physical blocks used on the disk is 3 (instead of 6), and the number of blocks
saved by deduplication is 3 (6 minus 3). In this document, these blocks are referred to as used blocks and
saved blocks.

Figure 26) Data structure in a deduplicated FlexVol volume or data constituent.

Block Block Block Block Block Block

Pointer Pointer Pointer Pointer Pointer Pointer

Data Data Data

Each data block has a reference count that is kept in the volume or data constituent metadata. In the
process of sharing the existing data and eliminating the duplicate data blocks, block pointers are altered.
The reference count for the block that remains on disk with the block pointer is increased. The reference
count for the block that contained the duplicate data is decremented. When no block pointers have a
reference to a data block, the block is released.
NetApp deduplication technology allows duplicate 4KB blocks anywhere in the FlexVol volume or data
constituent to be deleted, as described in the following sections.
The maximum sharing for a block is 32,767. This fact means, for example, that if there are 64,000
duplicate blocks, deduplication reduces that number to only 2 blocks.

4.4 Background (Postprocess) Volume Deduplication

When deduplication runs for the first time on a volume with existing data, it scans the blocks in the
FlexVol volume or data constituent. Deduplication then creates a fingerprint database that contains a
sorted list of all fingerprints for used blocks in the FlexVol volume or data constituent.
After the fingerprint file is created, fingerprints are checked for duplicates; when duplicates are found, a
byte-by-byte comparison of the blocks is done to make sure that the blocks are identical. If they are found
to be identical, the block’s pointer is updated to the already existing data block, and the new (duplicate)
data block is released.
Releasing a duplicate data block entails updating the block pointer, incrementing the block reference
count for the already existing data block, and freeing the duplicate data block. In real time, as additional
data is written to the deduplicated volume or data constituent, a fingerprint is created for each new block
and written to a change log file. When deduplication is run subsequently, the change log is sorted, its
sorted fingerprints are merged with those in the fingerprint file, and then deduplication processing occurs.
There are two change log files. In that way, as deduplication runs and merges the fingerprints of new data
blocks from one change log file into the fingerprint file, the second change log file logs the fingerprints of
new data written to the FlexVol volume or data constituent during the deduplication process. The roles of
the two files are then reversed the next time that deduplication runs. (For those familiar with Data ONTAP

27 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
usage of NVRAM, this process is analogous to when a switch is made from one half to the other to create
a consistency point.)
Here are some additional details about the deduplication metadata:
• There is a fingerprint record for every 4KB data block, and the fingerprints for all physical data blocks
in the FlexVol volume or data constituent are stored in the fingerprint database file.
• Fingerprints are not deleted automatically from the fingerprint file when data blocks are freed. When a
threshold of the number of new fingerprints is 20% greater than the number of data blocks used in the
volume, the stale fingerprints are deleted. This step can also be taken by a manual operation using
the advanced mode command volume efficiency check.
• The change log file size limit is set to 1% of the volume size except if it is a NetApp SnapVault
destination; then the change log file size limit is set to 4% of the volume size. The change log file size
is relative to the volume size limit. The space assigned for the change log in a volume is not reserved.
• The deduplication metadata for a FlexVol volume or data constituent is inside the aggregate, and it is
also stored in the FlexVol volume or data constituent. The copy inside the aggregate is used as the
working copy for all deduplication operations. Change log entries are appended to the copy in the
FlexVol volume or data constituent.
• During the upgrade of a major Data ONTAP release such as 8.3, the fingerprint and change log files
are automatically upgraded to the new fingerprint and change log structure the first-time volume
efficiency operations start after the upgrade completes. This step is a one-time operation, and it can
take a significant amount of time to complete, during which you can see an increased amount of CPU
on your system. You can see the progress of the upgrade using the volume efficiency show -
instance command.
• The deduplication metadata requires a minimum amount of free space in the aggregate. This amount
is equal to 3% of the total amount of physical data for all deduplicated FlexVol volumes or data
constituents within the aggregate. Each FlexVol volume or data constituent should have 4% of the
total amount of physical data’s worth of free space, for a total of 7%.
• The deduplication fingerprint files are located inside both the volume or data constituent and the
aggregate. This feature allows the deduplication metadata to follow the volume or data constituent
during operations such as a NetApp volume SnapMirror operation. If the volume or data constituent
ownership is changed because of a disaster recovery operation with volume SnapMirror,
deduplication automatically recreates the aggregate copy of the fingerprint database from the volume
or data constituent copy of the metadata the next time it runs. This operation is much faster than
recreating all the fingerprints from scratch.

Deduplication Metadata Overhead Example: Data ONTAP 8.3.x

If you have 100GB of logical data and get 50GB of savings in a single FlexVol volume, then you have
50GB of physical data. Given this fact, there should be 2GB (4% of 50GB) of available space in the
volume and 1.5GB of space available in the aggregate. As a second example, consider a 2TB aggregate
with four volumes, each 400GB in size, in the aggregate. Three volumes are to be deduplicated with the
following:
Vol1: 100GB of logical data, which has 50% savings (50% of 100GB) = 50GB physical data
Vol2: 200GB of logical data, which has 25% savings (75% of 200GB) = 150GB physical data
Vol3: 300GB of logical data, which has 75% savings (25% of 300GB) = 75GB physical data
The required amount of space for deduplication metadata is as follows:
Vol1: 2GB (50GB * 4%)
Vol2: 6GB (150GB * 4%)
Vol3: 3GB (75GB * 4%)

28 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
The aggregate needs a total of 8.25GB ((3% of 50GB) + (3% of 150GB) + (3% of 75GB)) =
1.5+4.5+2.25=8.25GB) of space available in the aggregate.

The primary fingerprint database, otherwise known as the working copy, is outside the volume or data
constituent in the aggregate and is therefore not captured in Snapshot copies. The change log files and a
backup copy of the fingerprint database are located within the volume or data constituent and are
therefore captured in Snapshot copies. Having the primary (working) copy of the fingerprint database
outside the volume or data constituent enables deduplication to achieve higher space savings. However,
the other temporary metadata files created during the deduplication operation are still placed inside the
volume or data constituent. These temporary metadata files are deleted when the deduplication operation
is complete. However, if Snapshot copies are created during a deduplication operation, these temporary
metadata files can get locked in Snapshot copies, and they remain there until the Snapshot copies are
deleted.

4.5 Background (Postprocess) Aggregate Deduplication

Starting with ONTAP 9.3, NetApp has added support for background aggregate deduplication and newly
introduced automatic background deduplication. Both automatic background deduplication and
background aggregate deduplication are supported only on AFF systems.
Automatic background deduplication automatically runs background deduplication jobs for all newly
created/upgraded volumes with a new auto policy that is predefined on all AFF systems. Automatic
background deduplication performs continuous background deduplication on all enabled volumes, without
manual configuration. If the ONTAP version is upgraded to 9.3 or later, all existing volumes have the new
auto policy enabled if they were not previously configured for background deduplication. The automatic
background deduplication engine monitors processing and isolates/removes volumes from automatic
deduplication processing if there are no space savings, focusing on the volumes that provide the most
significant savings.
Background aggregate deduplication extends NetApp ONTAP volume-based deduplication to all volumes
in an aggregate and provides additional incremental deduplication savings. Background aggregate
deduplication is only supported through automatic background deduplication. Only volumes with auto
policy can participate in background cross-volume sharing. On-demand run and user-driven schedules
are not supported with background aggregate deduplication. Volumes with the auto policy are subject to
periodic monitoring. If volumes are not showing at least 15% savings even after scanning more than 20%
of the volume nominal size, then these volumes are deprioritized and no longer participate in the auto
policy. Change logging on deprioritized volumes is disabled, and metadata on these volumes is truncated.
Enabling Background Aggregate Deduplication on New/Upgraded ONTAP 9.3 System
• Background aggregate deduplication, along with automatic background deduplication, is enabled by
default on all new AFF systems.
• All new volumes get the new auto policy.
• Existing volumes with default inline-only policy and where background efficiency is not enabled get
the new auto policy.
• Existing volumes with inline efficiency and background efficiency may be updated to the new auto
policy to enable background aggregate deduplication:
 Background aggregate deduplication should be enabled at the aggregate level. After being
enabled, it is enabled by default for all newly created volumes within the aggregate.
 Volume policy can be changed to auto using the vol efficiency modify… command.

29 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
5 NetApp Data Compression
NetApp data compression is a software-based solution that provides transparent data compression. It can
be run inline or postprocess and includes the capability to compress existing data. No application
changes are required to use NetApp data compression. This process is enabled and managed by using a
simple CLI or GUI such as System Manager or NetApp OnCommand Unified Manager.
Data ONTAP 8.3.1 introduces adaptive compression, an inline compression method that works with I/Os
of varying sizes. The performance impact is minimal, and in some cases enabling adaptive compression
improves overall performance. The compression method available in Data ONTAP 8.3 and earlier is still
available and is now called secondary compression.

Difference Between Adaptive Compression and Secondary Compression

Adaptive Compression
Adaptive compression combines fewer blocks of data into a compression group (8K). The compression
group is then compressed and stored as a single block. When a user requests data from this
compression group, less time is taken to decompress and provide that data to the user, thereby improving
the read performance. In general, adaptive compression is better suited for random workloads. Adaptive
compression provides relatively fewer savings as compared to secondary compression, but with better
performance.

Secondary Compression
Secondary compression combines large groups of data blocks into a compression group (32K). The
compression group is then compressed and stored as fewer blocks, which reduces the size of the data to
a large extent, thereby increasing the free space in the storage system. In general, secondary
compression is better suited for sequential workloads.
Both secondary compression and adaptive compression are supported on all types of disk media (HDD,
AFF, and Flash Pool).

5.1 How NetApp Data Compression Works

NetApp data compression does not compress the entire file as a single contiguous stream of bytes. Doing
so would be prohibitively expensive to service small reads or overwrites from part of a file. That is
because it requires the entire file to be read from disk and uncompressed before the request can be
served. Doing so would be especially difficult on large files.
To avoid this issue, NetApp data compression compresses a small group of consecutive blocks, known
as a compression group. In this way, when a read or an overwrite request comes in, you need to read
only a small group of blocks, not the entire file. This process optimizes read and overwrite performance
and enables greater scalability in the size of the files being compressed.

Compression Groups
The NetApp compression algorithm divides a file into compression groups (CGs). The file must be larger
than 8K or it is skipped for compression and written to disk uncompressed. Compression groups are a
maximum of 8K or 32K, depending on the compression type. For secondary compression, the
compression group size is 32K; for adaptive compression, the compression group size is 8K.
A compression group contains data from one file only. A single file can be contained within multiple
compression groups. If a file were 60K, it would be contained within two secondary compression groups.
The first would be 32K, and the second 28K. Similarly, a 15K file would be contained within two adaptive
compression groups: 8K and 7K.

30 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Compressed Writes
NetApp handles compression write requests at the compression group level. Each compression group is
compressed separately. The compression group is left uncompressed unless a savings of at least 25%
(in the case of 32K CGs) or at least 50% (in the case of 8K CGs) can be achieved on a per–compression
group basis. This policy optimizes savings while minimizing resource overhead.

Figure 27) Compression write request handling.

Because compressed blocks contain fewer blocks to be written to disk, compression reduces the amount
of write I/Os required for each compressed write operation. This reduction not only lowers the data
footprint on disk, but can also decrease the time to complete backups. See the “Feature Interoperability”
section for details about volume SnapMirror.

Compressed Reads
When a read request comes in, we read only the compression group (or groups) that contains the
requested data, not the entire file. This approach optimizes the amount of I/O being used to service the
request. When reading compressed data, only the required compression group data blocks are
transparently decompressed in memory. The data blocks on disk remain compressed. This process has
much less overhead on the system resources and read service times.
In summary, the NetApp compression algorithm is optimized to reduce overhead for both reads and
writes.

5.2 When Data Compression Runs

NetApp data compression, both secondary compression and adaptive compression, can be configured to
run either inline or postprocess, depending on the workload, performance, and savings requirements.

Inline Operations
During inline data compression, as the data is sent to the storage system, it is compressed in memory
before being written to the disk. The advantage of this implementation is that it can reduce the amount of
write I/O. This implementation option can affect your write performance and thus should not be used for
performance-sensitive environments on HDD configurations without proper testing to understand the
impact. The AFF systems introduced with Data ONTAP 8.3.1 and the Flash Pool systems are exceptions.
Inline compression can be used for primary workloads on AFF and Flash Pool systems.
For Flash Pool systems, make sure that the cache size is configured as per Flash Pool best practices. For
more information about Flash Pool, see TR-4070: “Flash Pool Design and Implementation Guide.”
Here are some additional details about inline compression:

31 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
• To provide the fastest throughput, inline compression compresses most new writes but skips more
performance-intensive compression operations. An example of a performance-intensive compression
operation is a small (<4K) partial file overwrite. Postprocess compression, if enabled, tries to
compress any data that was skipped by inline compression.
• Inline adaptive compression compresses data only if the compression savings are >50%.
• Inline secondary compression compresses data only if the compression savings are >25%.
• The incompressible data detection (IDD) option is supported only on secondary compression. The
enhanced compression algorithm used in Data ONTAP 8.3.1 is intelligent enough to identify
incompressible data and return more quickly. There is no need to use the IDD option on 8.3.1. The
option is provided for backward compatibility:
• IDD option is not supported on AFF configurations.
• IDD option is deprecated in ONTAP 9.

Postprocess Operations
NetApp data compression includes the capability to run postprocess compression in addition to inline
compression. Postprocess compression uses the same schedule as deduplication. If compression is
enabled when the volume efficiency schedule initiates a postprocess operation, the operation runs
compression first, followed by deduplication. Postprocess compression also includes the capability to
compress data that existed on disk prior to compression being enabled. Like inline compression,
postprocess compression compresses data only if the compression savings are >50% (adaptive
compression) or >25% (secondary compression).
If both inline and postprocess compression are enabled, then postprocess compression tries to compress
only blocks that are not already compressed. These blocks include those that were bypassed by inline
compression, such as small partial-file overwrites.
Postprocess compression is not supported on AFF configurations.

5.3 New Features and Behavior of Compression

New features introduced in Data ONTAP 8.3.1:

• AFF support: Compression is now officially supported on AFF series controllers (AFF80XX series
controllers).
• There is newly introduced adaptive compression support.
• Read performance improvements: Numerous performance optimizations have gone into Data ONTAP
8.3.1 compression to enhance read performance, especially random read performance. The result of
these optimizations is that Data ONTAP 8.3.1 inline compression performance is comparable to, or
better than, Data ONTAP 8.3.0 performance.
• Complete Flash Pool integration: Starting with Data ONTAP 8.3.1, Flash Pool can cache both read
and write compressed blocks.
• Subfile cloning is supported on adaptive compression–enabled volumes.
• You can size adaptive compression for AFF configurations using the NetApp sizing tool. Note: Sizing
is available only for NetApp representatives.

New features introduced in Data ONTAP 8.3.2:

• Support for background compression scanner to perform in-place conversion of uncompressed data
in a volume to a compressed format on AFF systems.

Volume efficiency start –vserver $SVM_NAME –volume $VOLUME_NAME –compression true –scan-old-
data true

32 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 Note: The preceding command skips any shared blocks and blocks locked in Snapshot copies. To
compress shared blocks and Snapshot blocks, you need to use –shared-blocks true and –
snapshot-blocks true options respectively. Both shared-blocks and snapshot-blocks options
are only available in advanced mode.

Volume efficiency start –vserver $SVM_NAME –volume $VOLUME_NAME –compression true –scan-old-
data true –snapshot-blocks true –shared-blocks true

With shared-blocks option, deduplication savings might be temporarily lost till the next scheduled
dedupe run
With snapshot-blocks option, because blocks trapped in Snapshot copies are locked and cannot be
changed, compressing Snapshot blocks consumes additional space in the file system. Space savings
from the locked blocks are realized after the Snapshot copies are deleted/expired.

• Performance improvement: Compression operations now run in their own domain, which is separate
from the data-serving domain, resulting in a performance improvement of up to 15% in latency and
throughput.

Compression Behavior Based on Configuration

AFF configuration:
• Supports both adaptive and secondary compression.
• Supports only inline compression.
• Adaptive inline compression is enabled by default on all new volumes (only when the effective cluster
version is 8.3.1 and all the nodes in the cluster are on 8.3.1).
• New volumes have an inline-only storage efficiency policy set by default on SAN-optimized AFF
systems.
• Secondary compression can be enabled manually, if required.
• Compression can be turned on and off at the volume level.
• A volume can have either adaptive compression or secondary compression but not both. Active file
systems and Snapshot copies can have different compression types.

Flash Pool configuration:

• Supports both adaptive and secondary compression.
• Supports both inline compression and postprocess compression.
• Compression is not enabled by default on new volumes. Compression must be enabled manually.
• Adaptive compression is enabled by default as long as there are SSDs in the aggregate and the
effective cluster version is 8.3.1.
• Secondary compression can be enabled by specifying the compression type option while turning on
compression.
• Postprocess/background compression is enabled by default. Users can choose background-only,
inline-only, or inline+background compression.
• For inline-only compression, users have to manually assign the inline-only storage efficiency policy to
the volume.

All-HDD configuration:
• Supports both adaptive and secondary compression.

33 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
• Supports both inline compression and postprocess compression.
• Compression is not enabled by default on new volumes. Compression has to be enabled manually.
• Secondary compression is enabled by default.
• Adaptive compression can be enabled by specifying the compression type option while turning on
compression.
• Postprocess/background compression is enabled by default. Users can choose background-only,
inline-only, or inline+background compression.
• For inline-only compression, users have to manually assign the inline-only storage efficiency policy to
the volume.

6 General Compression and Deduplication Features

Both compression and deduplication are enabled on a per-volume basis for FlexVol volumes or Infinite
Volumes. Both processes can be enabled on any number of FlexVol volumes or data constituents in a
storage system. Enabling compression and deduplication on an Infinite Volume automatically enables
them on all data constituents; you can’t enable them on some data constituents and not others.
You can run inline compression and inline deduplication without a scheduled or manually started
background efficiency operation configured on the volume. With a new storage virtual machine, two
predefined efficiency policies, inline-only and default, are automatically created. You can configure a
volume to use the inline-only efficiency policy and enable inline compression to run inline compression on
the volume without any scheduled or manually started background efficiency operations.
Postprocess compression and deduplication share the same scheduler and can be scheduled to run in
one of five different ways:
• Inline
• Scheduled on specific days and at specific times
• Manually, by using the command line
• Automatically, when 20% new data has been written to the volume
• SnapVault software based, when used on a SnapVault destination
Only one postprocess compression or deduplication process can run on a FlexVol volume or data
constituent at a time. Up to eight compression and deduplication processes can run concurrently on eight
different volumes or data constituents within the same NetApp storage system. If there is an attempt to
run postprocess compression or deduplication processes beyond the maximum, the additional operations
are placed in a pending queue and automatically started when there are free processes.
Postprocess compression and deduplication processes periodically create checkpoints so that if the scan
is interrupted, it can continue from the last checkpoint.

6.1 Compression and Deduplication System Requirements

This section discusses the requirements to use deduplication and/or compression and the maximum
amount of data that is compressed and deduplicated. Although the section discusses some basic points,
we assume that the NetApp storage system is already installed and running and that the reader is familiar
with basic NetApp administration.

34 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Requirements Overview
Table 1) Overview of compression and deduplication requirements for clustered Data ONTAP.

Requirement Deduplication Compression

Data ONTAP version Minimum clustered Data ONTAP 8.1

Neither deduplication nor compression requires a license; you enable them on

License
a volume-by-volume basis

All FAS and V-Series and FlexArray systems that are supported with clustered
Hardware
Data ONTAP

Volume type supported • FlexVol or Infinite Volumes

Aggregate type supported 64-bit

Neither compression nor deduplication imposes a limit on the maximum volume

or data constituent size supported; therefore, the maximum volume or data
Maximum volume size
constituent limit is determined by the type of storage system regardless of
whether deduplication or compression is enabled

Supported protocols All

6.2 Maximum Logical Data Size Processing Limits

The maximum logical data size that is processed by postprocess compression and deduplication is
640TB regardless of the size of the volume or data constituent created. After this logical limit is reached,
writes to the volume or data constituent continue to work; however, postprocess compression and
deduplication fail with the error message “maximum logical data limit has reached.”
As an example, in Data ONTAP 8.3.1, if you had a FAS8080 that has a 100TB maximum volume size and
you created a 100TB volume or data constituent, the first 640TB of logical data compress and deduplicate
normally. However, any additional new data written to the volume or data constituent after the first 640TB
is not postprocess compressed or deduplicated until the logical data becomes less than 640TB. If inline
compression is enabled on the volume or data constituent, data continues to be inline compressed until
the volume or data constituent is completely full.

6.3 When to Enable Compression and/or Deduplication

Choosing when to enable compression or deduplication involves balancing the benefits of space savings
against the potential overhead. Your savings and acceptable overhead vary depending on the use case.
You might find some solutions suited for the primary tier and others better suited for the backup/archive
tier only. The amount of system resources the processes consume and the possible savings depend on
the type of data. The performance impact varies in each environment, and NetApp highly recommends
that the performance impact be fully tested and understood before implementing the processes in
production.
Enabling compression or deduplication and deciding which compression type to use depend on the
application characteristics, workload mix, I/O size, and performance requirements. Use the Storage
Space Savings Estimation Tool (SSET) to estimate a dataset’s compression and deduplication savings.

General Compression Usage Guidelines

• Use inline adaptive compression:
When the workload is predominantly random writes or a mix of sequential and random writes

35 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 When read I/Os are the same size or smaller than write I/Os and the workload is mostly reads
• Use inline secondary compression:
When the workload is compressible and mostly large sequential I/Os (32KB or larger)

Inline Versus Postprocess Compression

Inline compression provides immediate space savings; postprocess compression first writes the blocks to
disk as uncompressed and then at a scheduled time compresses the data.
Postprocess compression is useful for environments in which you want compression savings but don’t
want to incur a performance penalty associated with new writes. Common use cases that fall under this
category are primary workloads on all HDD configurations.
Inline compression is useful for customers who aren’t as performance sensitive and can handle some
impact on new write performance as well as on CPU during peak hours. This type of compression is also
useful for customers who want to minimize $/gigabyte and the PE cycle of SSD drives to prolong the life
of flash media. Common use cases that fall under this category are primary and secondary workloads on
AFF and Flash Pool configurations and secondary workloads on all HDD configurations.

Inline Versus Postprocess Deduplication

Inline deduplication provides immediate space savings; postprocess deduplication first writes the blocks
to disk and then at a scheduled time dedupes the data.
Inline deduplication is useful primarily for copy offload and VDI deployments. It is especially useful for VM
patch apply use cases, VM provisioning, and database migrate operations that result in many duplicate
blocks in memory that can be eliminated by inline dedupe.
Inline deduplication might not be a good candidate for entry-level systems in which there is just enough
memory to meet client I/O SLAs. Postprocess deduplication can be considered in such cases. In general,
NetApp does not recommend deduplication for use cases in which data is overwritten at a rapid rate.

7 Space Savings
This section discusses the potential storage savings from data reduction technologies (deduplication,
compression, data compaction). For some virtualized applications that uses cloning technologies, the
savings ratio includes FlexClone savings.
The amount of space savings is primarily a function of data type, and different data types can yield
different savings ratios. The amount also depends on which space-saving technology is being used. In
some cases, deduplication alone or compression alone can yield better savings ratios as compared to
those of the combined savings of deduplication and compression.
In general, secondary compression provides better savings compared to those of adaptive compression.
Adaptive compression provides better performance compared to that of secondary compression. The
maximum savings that one can expect with adaptive compression is 2:1. When used in combination with
data compaction (ONTAP 9 and later) and deduplication, adaptive compression can yield >2:1 savings.
In the NetApp implementation, compression is run before deduplication. Doing so provides the ability to
use inline compression to get immediate space savings from compression followed by additional savings
from deduplication. In our testing of different solutions, we found that better savings were achieved by
running compression before deduplication.
For AFF systems, NetApp recommends using all the default efficiency technologies (inline adaptive
compression, inline deduplication, inline data compaction and postprocess deduplication).

36 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
For FAS systems, because disk is a bottleneck, enabling all the efficiency technologies might consume
significant system resources and interfere with client operations. See Performance section for more
details. NetApp recommends evaluating the dataset using Space Savings Estimation Tool (SSET) to
estimate savings ratios from different efficiency technologies and coupled with performance SLAs,
choosing the efficiency technology combination to maximize efficiency without affecting performance
SLAs.
Tables 2 through 5 list the storage efficiency data reduction ratio ranges for different applications. A
combination of synthetic datasets and real-world datasets has been used to determine the typical savings
ratio range. It is quite possible that the savings that you get might be out of the ratio range listed in the
tables. The savings ratio range mentioned are only indicative.

Table 2) Typical AFF storage efficiency ratios with ONTAP 9 and later.
Workload [with deduplication, data compaction, adaptive compression and FlexClone volumes (where
applicable) technologies] Ratio Range

Home directories 1.5:1–2:1

Software development 2:1–10:1

VDI VMware Horizon full clone desktops (persistent): NetApp Clones 6:1–10:1
VDI VMware Horizon linked clone desktops (nonpersistent) 5:1–7:1
VDI Citrix XenDesktop full clone desktops (persistent): NetApp Clones 6:1–10:1
VDI Citrix XenDesktop MCS desktops (nonpersistent) 5:1–7:1
VDI Citrix Provisioning services desktops (nonpersistent) 3.3:1–5:1
Virtual Servers (OS and Applications) 2:1–4:1
Oracle databases (with no database compression) 2:1–4:1
SQL 2014 databases (with no database compression) 2:1–4:1
Microsoft Exchange 1.6:1
Mongo DB 1.3:1-1.5:1
Precompressed data (such as video and image files, audio files, pdfs, etc.) No Savings

Table 3) Typical FAS storage efficiency ratios with ONTAP 9 and later.
Workload [with deduplication, data compaction, adaptive compression and FlexClone volumes (where
applicable) technologies] Ratio Range

Home directories 1.5:1–2:1

Software development 2:1–10:1

VDI VMware Horizon full clone desktops (persistent): NetApp Clones 6:1–10:1
VDI VMware Horizon linked clone desktops (nonpersistent) 3:1–5:1
VDI Citrix XenDesktop full clone desktops (persistent): NetApp Clones 6:1–10:1
VDI Citrix XenDesktop MCS desktops (nonpersistent) 3:1–5:1
VDI Citrix Provisioning services desktops (nonpersistent) 3.3:1–5:1
Virtual Servers (OS and Applications) 2:1–4:1
Oracle databases (with no database compression) 2:1–4:1
SQL 2014 databases (with no database compression) 2:1–4:1
Microsoft Exchange 1.4:1
Mongo DB 1.3:1-1.5:1

37 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Precompressed data (such as video and image files, audio files, pdfs, etc.) No Savings

Table 4) Typical AFF storage efficiency ratios with ONTAP 8.3.x.

Workload [with deduplication, adaptive compression and FlexClone volumes (where applicable)
technologies] Ratio Range

Home directories 1.5:1–2:1

Software development 2:1–10:1

VDI VMware Horizon full clone desktops (persistent): NetApp Clones 5:1–10:1
VDI VMware Horizon linked clone desktops (nonpersistent) 3:1–5:1
VDI Citrix XenDesktop full clone desktops (persistent): NetApp Clones 5:1–10:1
VDI Citrix XenDesktop MCS desktops (nonpersistent) 3:1–5:1
VDI Citrix Provisioning services desktops (nonpersistent) 3.3:1–5:1
Virtual Servers (OS and Applications) 2:1–4:1
Oracle databases (with no database compression) 1.6:1–2:1
SQL 2014 databases (with no database compression) 1.6:1–2:1
Microsoft Exchange 1.6:1
Precompressed data (such as video and image files, audio files, pdfs, etc.) No Savings

Table 5) Typical FAS storage efficiency ratios with ONTAP 8.3.x.

Workload [with deduplication, adaptive compression and FlexClone volumes (where applicable)
technologies] Ratio Range

Home directories 1.5:1–2:1

Software development 2:1–10:1

VDI VMware Horizon full clone desktops (persistent): NetApp Clones 5:1–10:1
VDI VMware Horizon linked clone desktops (nonpersistent) 3:1–5:1
VDI Citrix XenDesktop full clone desktops (persistent): NetApp Clones 5:1–10:1
VDI Citrix XenDesktop MCS desktops (nonpersistent) 3:1–5:1
VDI Citrix Provisioning services desktops (nonpersistent) 3.3:1–5:1
Virtual Servers (OS and Applications) 2:1–4:1
Oracle databases (with no database compression) 1.6:1–2:1
SQL 2014 databases (with no database compression) 1.6:1–2:1
Microsoft Exchange 1.4:1
Precompressed data (such as video and image files, audio files, pdfs, etc.) No Savings

8 Performance
This section discusses the performance aspects of data compression and deduplication.
Because compression and deduplication are part of Data ONTAP, they are tightly integrated with the
NetApp WAFL file structure. Because of this fact, compression and deduplication are optimized to
perform with high efficiency. The processes are able to leverage the internal characteristics of Data
ONTAP to perform compression and decompression, create and compare digital fingerprints, redirect
data pointers, and free up redundant data areas.

38 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
However, the following factors can affect the performance of compression and deduplication and the I/O
performance of compressed and deduplicated volumes:
• The application and the type of dataset being used
• The data access pattern (for example, sequential versus random access, the size of the I/O)
• The amount of duplicate data
• The compressibility of the data
• The amount of total data
• The average file size
• The nature of the data layout in the volume
• The amount of changed data between compression and deduplication runs
• The number of concurrent compression and deduplication processes running
• The number of volumes or data constituents with compression and deduplication enabled
• The hardware platform: the amount of CPU in the system
• The Data ONTAP release
• The amount of load on the system
• The disk types: SSD, ATA/SAS, and the RPM of the disk
• The number of disk spindles in the aggregate
• The priority set for the volume efficiency operation assigned to the volume
Compression and deduplication can be scheduled to run during nonpeak hours. Doing so allows the bulk
of the overhead on the system during nonpeak hours. When there is a lot of activity on the system,
compression and deduplication run as background processes and limit the system’s resource usage.
When there is not a lot of activity on the system, compression and deduplication speed increases and can
use all available system resources. The potential performance impact should be fully tested before
implementation.
Because compression and deduplication run on a per FlexVol volume or per–data constituent basis, the
more data constituents or FlexVol volumes per node that run in parallel, the greater the impact on system
resources. NetApp recommends that when using compression and deduplication, you stagger the volume
efficiency schedule for volumes to help control the overhead.
Storage QoS gives users the choice of running a volume efficiency operation with a priority of either best
effort or background. This feature enables the administrator to define how volume efficiency operations
compete for resources with user workloads and other system processes not running in the background.
When considering whether to add compression or deduplication, use standard sizing and testing methods
you would use when considering adding applications to the storage system. It is important to understand
how inline compression affects your system and how long postprocess operations take in your
environment. It is also important to know whether you have the bandwidth to run these operations with
acceptable impact on the applications running on your storage system.
We optimized compression performance considerably in Data ONTAP 8.3.1, especially for AFF and Flash
Pool configurations. For AFF use cases that benefit from compression and/or deduplication, we
recommend enabling inline adaptive compression and/or deduplication for primary workloads.
As an example, on Data ONTAP 8.3.1, for a 100% read workload with 8K I/O size, AFF8080 has been
measured to deliver ~30% more IOPS at 1ms with compression than that of Data ONTAP 8.3.0 with no
compression.
However, most workloads are a mix of reads and writes. For a mixed 50/50 R/W workload with
predominantly 8K block size I/O at low to medium load with latency <1ms, AFF8080 delivers 15% to 20%
higher IOPS with inline compression. A smaller platform such as the AFF8020 delivers roughly the same

39 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
performance as that of Data ONTAP 8.3.0. Other platforms’ results are expected to be between these
ranges.
In higher-load scenarios, when latency trends to 2ms or beyond, IOPS may be 10% to 15% lower.
Similarly, workloads with mixed/large block size I/O might see 10% to 15% lower performance compared
with that of Data ONTAP 8.3.0.
Although we optimized compression to minimize impact on your throughput, there might still be an impact
even if you use only postprocess compression. The impact can occur because some data in memory still
must be uncompressed when servicing reads. The impact continues so long as the data is compressed
on disk regardless of whether compression is disabled on the volume at a future point.
Because of these factors, NetApp recommends that performance with compression and deduplication be
measured in a test setup and taken into sizing consideration before deploying the processes in
performance-sensitive solutions, especially for HDD configurations.

9 Feature and Configuration Interoperability

Table 6 shows the support matrix information for feature and configuration combinations.

Table 6) Feature configuration interoperability.

Feature AFF Default Flash Pool Default On? HDD Default On?
Support On? Support Support
Inline zero-block
Yes Yes Yes No Yes No
deduplication

Inline volume
Yes Yes Yes No Yes No
deduplication

Inline aggregate
Yes Yes No No No No
deduplication

Postprocess volume
Yes No Yes No Yes No
deduplication

Postprocess
aggregate Yes Yes No No No No
deduplication

Automatic
background Yes Yes No No No No
deduplication

Inline adaptive
Yes Yes Yes No Yes No
compression

Postprocess adaptive
Not supported Yes No Yes No
compression

Inline secondary
Yes No Yes No Yes No
compression

Postprocess
secondary Not supported Yes No Yes No
compression

Inline data
Yes Yes Yes No Yes No
compaction

40 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
10 General Workload-Based Storage Efficiency Guidelines
Table 7 shows some guidelines for when to use compression, deduplication, data compaction, and/or
inline zero-block deduplication. These guidelines are not rules; your environment may have different
performance requirements for specific use cases. NetApp highly recommends that the performance
impact be fully tested and understood before you decide to implement these processes in production.

Table 7) Workload-based storage efficiency guidelines.

Workload Storage Efficiency Guidelines

AFF Flash Pool (Sized as per Hard Disk Drives

Flash Pool Best Practices)

Database For primary and For primary and For primary workloads, use:
(Oracle,
SQL
secondary workloads, use: secondary workloads, • Inline zero-block
Server) • Adaptive inline use: deduplication
compression • Adaptive inline • For secondary
• Inline zero-block compression workloads, use:
deduplication • Inline zero-block • Adaptive inline
• Inline deduplication compression
deduplication • Inline • Adaptive background
(Data ONTAP deduplication compression
8.3.2 and later) (Data ONTAP
• Inline zero-block
• Inline data 8.3.2 and later)
deduplication
compaction • Inline data
• Inline data compaction
(ONTAP 9 and compaction
(ONTAP 9 and later)
later) (ONTAP 9 and
• Background later)
deduplication • Background
deduplication

VDI and For primary and For primary and For primary workloads, use:
SVI secondary workloads, secondary workloads, • Deduplication
use: use:
• Inline zero-block
• Adaptive inline • Adaptive inline deduplication
compression compression
• Inline zero-block • Inline zero-block For secondary workloads,
deduplication deduplication use:
• Inline • Inline • Adaptive inline
deduplication deduplication compression
(Data ONTAP (Data ONTAP
8.3.2 and later) 8.3.2 and later) • Adaptive background
compression
• Inline data • Inline data
compaction compaction • Deduplication
(ONTAP 9 and (ONTAP 9 and • Inline zero-block
later) later) deduplication
• Background • Background • Inline data compaction
deduplication deduplication (ONTAP 9 and later)

Exchange For primary and For primary and For primary and secondary
secondary workloads, secondary workloads, workloads, use:
use: use: • Inline secondary

41 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 • Adaptive inline • Adaptive inline compression
compression compression • Background secondary
• Inline zero-block • Inline zero-block compression
deduplication deduplication • Deduplication
• Inline data • Inline data • Inline zero-block
compaction compaction deduplication
(ONTAP 9 and (ONTAP 9 and Inline data compaction (ONTAP
later) later) 9 and later)
• Background • Background
deduplication deduplication
• Set schedule to
off-peak hours

File For primary and For primary and For primary and secondary
services secondary workloads, secondary workloads, workloads, use:
use: use: • Adaptive inline
• Adaptive inline • Adaptive inline compression
compression compression • Adaptive background
• Inline zero-block • Inline zero-block compression
deduplication deduplication • Deduplication
• Inline data • Inline data • Inline zero-block
compaction compaction deduplication
(ONTAP 9 and (ONTAP 9 and
• Inline data compaction
later) later)
(ONTAP 9 and later)
• Background • Background
deduplication deduplication

Mixed For primary and For primary and For primary workloads, use:
workloads secondary workloads, secondary workloads, • Deduplication
use: use:
• Inline zero-block
• Adaptive inline • Adaptive inline deduplication
compression compression
• Inline zero-block • Inline zero-block For secondary workloads,
deduplication deduplication use:
• Inline data • Inline data • Adaptive inline
compaction compaction compression
(ONTAP 9 and (ONTAP 9 and
later) later) • Adaptive background
compression
• Background • Background
deduplication deduplication • Deduplication
• Inline zero-block
deduplication
• Inline data compaction
(ONTAP 9 and later)

For additional details refer to workload specific technical reports mentioned in References section.

11 Best Practices

Best Practices for Optimal Savings and Minimal Performance Overhead

This section describes best practices and lessons learned from internal tests and customer deployments:

42 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
• Both deduplication and compression consume system resources and can alter the data layout on
disk. Because of the application’s I/O pattern and the effect of deduplication and compression on the
data layout, the read and write I/O performance can vary. The space savings and the performance
impact depend on the application and the data contents.
• NetApp recommends that the performance impact of deduplication and compression be carefully
considered and measured in a test setup and taken into consideration for sizing before deploying the
processes in performance-sensitive solutions. For information about the impact of deduplication and
compression on other applications, contact the specialists at NetApp for their advice and for test
results for your particular application.
• If there is only a small amount of new data, run deduplication infrequently, because there is no benefit
in running it frequently in such a case, and it consumes system resources. The frequency for running
deduplication depends on the rate of change of the data in the volume.
• The more concurrent compression and deduplication processes you run on a system, the more
system resources are consumed.
• If NetApp Snapshot copies are required, run compression and deduplication processes before
creating the Snapshot copies to minimize the amount of data that gets locked in to the copies. (Make
sure that the compression and deduplication processes complete before creating the Snapshot copy.)
If a Snapshot copy is created on a volume before the deduplication processes complete, the result is
likely to be smaller space savings. If a Snapshot copy is created on a volume before the compression
processes complete, the result is likely to be that more space is used by the Snapshot copies.
• For deduplication to run properly, leave some free space for the deduplication metadata. For
information about how much extra space to leave in the FlexVol volume, data constituent, and
aggregate, see section 4.4.

For additional details on how to maximize storage efficiency on ONTAP 9.x AFF systems, refer this KB
article.

12 Upgrade and Revert

Major and minor nondisruptive upgrades to clustered Data ONTAP are supported with both compression
and deduplication.
NetApp recommends that no active postprocess compression or deduplication operations run during the
nondisruptive upgrade:
• Perform the Data ONTAP upgrade during a time when compression and deduplication operations are
not scheduled to run.
• Determine whether any compression and deduplication operations are active and, if they are, stop
them until the Data ONTAP upgrade is complete.
• You can use the volume efficiency show command to determine whether the status of compression
and deduplication is active or idle. On a system with compression and deduplication enabled, the
output of the volume efficiency show command is similar to the following:

Vserver Volume State Status Progress

Vsone vol1 Enabled Active 122 GB Scanned, 25 GB
Compressed
Vsone vol2 Enabled Active 164 GB Searched
Vsone vol3 Enabled Active 41 GB (45%) Done

43 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 Vsone vol4 Enabled Idle for 00:02:43
• You can use the volume efficiency stop command to abort the active compression and
deduplication operations on the volume and use the volume efficiency start command to
restart it.
For specific details and requirements for performing a nondisruptive upgrade on your system, refer to
Upgrade Advisor in the AutoSupport tool if you have AutoSupport enabled. Otherwise, refer to the release
notes for the version of Data ONTAP to which you are upgrading.

12.1 Upgrading to a Later Version of Data ONTAP

[May 2017] AFF system upgraded to ONTAP 9.2 has all the inline efficiencies enabled by default on all
eligible volumes, regardless of whether it was turned on or not prior to upgrade. This approach includes
inline zero-block deduplication, inline adaptive compression, inline volume deduplication, inline aggregate
deduplication, and inline data compaction.
Upgrading to a later version of Data ONTAP is a nondisruptive operation regardless of whether the
volumes have deduplication enabled or not. However, be aware that part of an upgrade to a later major
release on a deduplication-enabled volume or data constituent includes upgrading the deduplication
metadata. As a result, you might notice that volume efficiency operations take longer to complete during
this operation. This change is a one-time increase on each FlexVol volume or data constituent that had
deduplication enabled before the upgrade and happens only after an upgrade. Subsequent volume
efficiency operations complete as expected.
The process of upgrading the deduplication metadata is done in two stages. The first stage is creating
new change log metafiles in the FlexVol volume or data constituents and is done automatically during the
upgrade, at boot time, while the volume is coming online.
The second stage is sorting the change logs and fingerprints created from the previous version and
recreating with the new format. This work is done automatically the first-time volume efficiency operations
are run on each FlexVol volume or data constituent that had deduplication enabled before the upgrade.
The second step can take a significant amount of time to complete, during which you can see an
increased amount of CPU on your system and see the progress of the upgrade using the volume
efficiency show -instance command. The amount of time for this work to complete might be
different for each FlexVol volume and data constituent. The time depends on the size of the change log
and fingerprint database, the storage efficiency priority used, and the amount of available resources.
During this phase there are no checkpoints, so if you need to stop deduplication, wait for this upgrade
process to complete first.
Even in the case of compression, upgrading to a newer version of Data ONTAP is a nondisruptive
operation.
However, because adaptive compression was newly introduced in Data ONTAP 8.3.1 and the AFF
configuration supports only inline compression, compression behavior on AFF configurations is slightly
different after an 8.3.1 upgrade. This difference is in comparison to compression behavior on HDD and
Flash Pool configurations. Table 8 and Table 9 detail the exact behavior.

44 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 8) Compression behavior after a Data ONTAP 8.3.1 upgrade: AFF configuration.

AFF: Pre-8.3.1 AFF: Post-8.3.1 Upgrade

Volume 1: no compression Volume 1: no compression:
• You can enable inline adaptive and secondary compression
on Volume 1.
• Existing data on the volume remains uncompressed.
• In Data ONTAP 8.3.1, you cannot explicitly convert the
existing data to a compressed format. Existing data is
converted to a compressed format when the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, you can compress existing
uncompressed data using the background scanner. Manually
run the background scanner command to initiate the
process. The background scanner works similarly to
postprocess compression, but scanning is done only once to
convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• If you want to enable inline adaptive compression, first
rehydrate the volume and then enable inline adaptive
compression. Only new writes to the volume are compressed
using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Manually
run the background scanner command to initiate the
process. The background scanner works similarly to
postprocess compression, but scanning is done only once to
convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

45 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
AFF: Pre-8.3.1 AFF: Post-8.3.1 Upgrade
Volume 3: inline secondary Volume 3: inline secondary compression:
compression + background • Background compression is disabled because it is not
secondary compression supported on AFF configurations.
• If you want to enable inline adaptive compression, first
rehydrate the volume and then enable inline adaptive
compression. Only new writes to the volume are compressed
using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Manually
run the background scanner command to initiate the
process. The background scanner works similarly to
postprocess compression, but scanning is done only once to
convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

Volume 4: background secondary Volume 4: no compression:

compression • Background compression is disabled on the volume because
it is not supported on AFF configurations.
• Existing compressed data remains compressed in a
secondary compression format.
• You can enable inline secondary compression.
• If you want to enable inline adaptive compression, you have
to first rehydrate the volume and then enable inline adaptive
compression. Only new writes to the volume are compressed
using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Manually
run the background scanner command to initiate the
process. The background scanner works similarly to
postprocess compression, but scanning is done only once to
convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

46 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 9) Compression behavior after a Data ONTAP 8.3.1 upgrade: FAS configuration.

FAS: Pre-8.3.1 FAS: Post-8.3.1 Upgrade

Volume 1: no compression Volume 1: no compression:
• You can enable inline adaptive or inline secondary
compression on Volume 1.
• Existing data on the volume remains uncompressed.
• Existing data is converted to a compressed format when the
old data is overwritten or background compression is used to
compress the existing data.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• To enable inline/background adaptive compression, first
rehydrate the volume and then enable adaptive compression.

Volume 3: inline secondary Volume 3: inline secondary compression + background secondary

compression + background compression:
secondary compression • To enable inline/background adaptive compression, first
rehydrate the volume and then enable adaptive compression.

Volume 4: background secondary Volume 4: background secondary compression:

compression • You can enable inline secondary compression.
• To enable inline/background adaptive compression, first
rehydrate the volume and then enable adaptive compression.

12.2 Reverting to an Earlier Version of Clustered Data ONTAP

When you are considering reverting a system to an earlier version of clustered Data ONTAP, we typically
recommend contacting NetApp Customer Success Services for assistance.
Scenarios in which customers can revert their NetApp storage systems without guidance from NetApp
Customer Success Services include:
• If the NetApp storage system is new and has not yet been used, then the volume efficiency
revert_to command can be used without guidance from NetApp Customer Success Services.
• If the NetApp storage system is being used as a test system and no production data is present, then
the volume efficiency revert_to command can be used without guidance from NetApp
Customer Success Services.
When reverting a system from clustered Data ONTAP 8.3.1 or later to Data ONTAP 8.3 or earlier, you
must undo all adaptive compression savings from all nodes of the cluster before reverting because
adaptive compression is not supported on releases prior to Data ONTAP 8.3.1.
You should not attempt reverting in any other scenario without assistance from NetApp Customer
Success Services: [Link]
The volume efficiency revert_to command provides sufficient guidance to successfully complete
the revert process, including the process for reverting deduplicated volumes, in these scenarios. If you
encounter problems during or after the revert process, contact NetApp Customer Success Services.

47 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
13 Interoperability
NetApp has several products that work with compression, deduplication, and compaction. This section
discusses how compression, deduplication, and compaction interact with other NetApp products and
features.

13.1 Data Protection

Snapshot Copies
Snapshot copies lock blocks on disk that cannot be freed until the Snapshot copy expires or is deleted.
After a Snapshot copy of data is made on a volume, any subsequent changes to data contained in that
Snapshot copy require additional disk space until the Snapshot copy is deleted or expires. The same is
true with deduplication- and/or compression-enabled volumes.
Active File System (AFS) and any Snapshot copy can contain only one kind of compression: either
adaptive or secondary. It is possible to have different types of compression between AFS and Snapshot
copies:
• For example, AFS with adaptive compression and Snapshot1 with secondary compression
Some best practices to achieve the best space savings from deduplication- and/or compression-enabled
volumes that contain Snapshot copies are:
• Run compression and deduplication before creating new Snapshot copies.
• Wait for postprocess compression and deduplication to complete before creating new Snapshot
copies.
• Change the Snapshot policy to none until the compression of existing data completes.
• If running deduplication without compression, schedule deduplication only after significant new data
has been written to the volume.
• Limit the number of Snapshot copies you maintain.
• If possible, reduce the retention duration of the Snapshot copies.
• Configure appropriate reserve space for the Snapshot copies.
• Delete as many Snapshot copies as possible before running compression against existing data.

Data compaction:
• Data present in Snapshot copies can be compacted by running the ‘wafl pack’ command.
Currently we only support packing of individual files in a Snapshot copy using the pack command.
• The command is only available in diagnostic mode and should only be used in consultation with
NetApp Support.

Replication: Volume SnapMirror

Both compression and deduplication are supported with volume SnapMirror. Volume SnapMirror operates
at the physical block level. Thus, when deduplication is enabled on the source, the data sent over the wire
for replication remains deduplicated, and the savings are inherited at the destination. For compression,
storage efficiency is preserved or lost during replication, depending on the following configuration:
• Source and destination volumes use the same compression type (either adaptive or secondary):
 Logical replication with storage efficiency is used.
 Storage efficiency (compressed and deduplicated blocks) is preserved during data transfer.
• Source and destination volumes use different compression types (adaptive to secondary or vice
versa):

48 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 - Logical replication without storage efficiency is used.
- Blocks are uncompressed at the source and transferred to the destination.
- Data is recompressed using the destination volume’s compression type.
Note: NetApp always recommends using the same compression type on both source and destination volumes
to prevent multiple decompression and recompression operations from occurring.
Additional notes:
• Data can be replicated from AFF systems to HDDs on FAS systems and vice versa.
• If you use compression on both the source and the destination volumes, both must be contained
within a 64-bit aggregate.
• The volume SnapMirror update schedule is not tied to the compression and deduplication schedule.
• When configuring volume SnapMirror and compression and deduplication, consider the compression
and deduplication schedule and the time you want to start a volume SnapMirror initialization. As a
best practice, start volume SnapMirror initialization of a compressed and deduplicated volume after
compression and deduplication are complete. Doing so prevents sending uncompressed and
undeduplicated data and additional temporary metadata files over the network. If the temporary
metadata files in the source volume are locked in Snapshot copies, they also consume extra space in
the source and destination volumes.
• It is important when preparing to initialize a volume SnapMirror relationship on a compressed or
deduplicated volume to decide if you want to perform the optional operation of compressing or
deduplicating existing data on the primary. This fact is especially true if you use the –shared-
blocks true or –snapshot-blocks true options because running compression against existing
data with these options can result in a large number of physical-level changes on disk. If you are
deduplicating existing data, doing so can result in a large number of logical changes on disk. For both
compression and deduplication of existing data, SnapMirror recognizes these as changed blocks and
includes them in the next data transfer to the destination volume. As a result, volume SnapMirror
updates that occur after deduplication or compression of existing data with the –shared-blocks
true or –snapshot-blocks true options are likely much larger than normal. For preexisting
volume SnapMirror relationships, take into consideration the big surge of data involved in the transfer
after compression or deduplication of existing data and plan accordingly.

Data compaction:
• SnapMirror operates at a logical level and does not have any effect on compacted data. During
SnapMirror transfer, the source reads the packed 4K block containing compressed sub-4K chunks
and sends the compressed sub-4K chunks to destination as is. The destination packs the
precompressed sub-4K chunks and writes them to storage. The compaction savings on the
destination might not be exactly the same as those of the source, but they are comparable to those of
the source. The difference in savings is because the sub-4K chunks are repacked at the destination,
and the same sub-4K chunks that were part of a single 4K WAFL block on the source might not end
up being together in a single 4K WAFL block on the destination.
• Regardless of whether the source system supports data compaction, the data on the destination is
compacted on a data compaction supported destination system.
• If the destination system does not support data compaction, the sub-4K chunks are written unpacked
to storage.

Aggregate inline deduplication:

Cross-volume deduplication savings are lost during SnapMirror transfers.

49 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Backup: SnapVault
Both compression and deduplication are supported with SnapVault. The behavior of compression and
deduplication with SnapVault is similar to that of SnapMirror and depends on the efficiency settings on the
source and destination volumes.
If compression is enabled on the SnapVault destination, then the savings from the source are not retained
over the network transfer, but they can be regained.
If the source and destination volumes have a different compression type (source volume: adaptive
compression; destination volume: secondary compression; or vice versa), then the savings from the
source are not retained over the network transfer. Depending on whether the destination has inline or
postprocess compression, the savings are regained.
All other configurations retain the savings over the network and on the destination.
As a best practice, if deduplication is enabled on the source, enable only deduplication on the SnapVault
destination if you expect SnapVault updates to occur from Snapshot copies created before deduplication
completes on the source.
As a best practice, enable only compression on the SnapVault destination if you cannot run compression
on the source.
If you have compression or deduplication enabled on the destination, the process starts automatically
after the transfer completes. You can’t change when this process runs; however, you can change the
volume efficiency priority that is assigned to the volume.
Here are some recommendations for SnapVault destinations when the source has compression enabled:
• If you require compression savings on the destination and your source has compression enabled,
then do not enable compression on the SnapVault destination. The savings are already inherited on
the destination. Second, if you enable compression on the SnapVault destination, then the savings
are lost during the transfer, and you have to redo the savings on the destination. Further, if you ever
enable compression on the destination, even if you later disable it, you never retain the savings from
the source.
• Postprocess compression of existing data results in physical-level changes to the data. This result
means that SnapVault recognizes the changes as changed blocks and includes them in its data
transfers to the destination volume. As a result, SnapVault transfers that follow a volume
efficiency start –scan-old-data true command are likely to be much larger than normal. If
you can do so, NetApp recommends compressing existing data on the source before running
baseline transfers for SnapVault. For preexisting SnapVault relationships, take into consideration the
big surge of data involved in the transfer and plan accordingly.
• As a best practice, have the same compression type on the SnapVault source and destination to
retain savings over the network.
Here are some recommendations for SnapVault destinations when the source does not have
deduplication or compression enabled but the destination does:
• If you require compression savings on the destination and your source does not have compression
enabled, then NetApp recommends using inline compression if you have sufficient CPU resources
during the backup. Inline compression provides the maximum amount of savings without having a
temporary impact on the Snapshot copy size. If you use postprocess compression without inline
compression, then the size of the archival Snapshot copy is temporarily larger because it contains the
uncompressed version of the blocks just transferred. This temporary increase in Snapshot space is
removed after the deduplication process completes and a newer archival Snapshot copy is created.
After you enable compression on the destination, even if you later disable it and enable it on the
source, you never retain the savings from the source.

50 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
• After a SnapVault transfer completes, it automatically creates an archival Snapshot copy. If you run
deduplication or compression on the destination, the archival Snapshot copy is replaced with a new
one after compression and deduplication run on the destination. (The name of this new Snapshot
copy is the same as that of the archival copy, but the creation time of this copy is changed.)
• The compression and deduplication schedule on the destination is tied to the SnapVault schedule
and cannot be modified. The compression and deduplication schedule on the source is not tied to the
SnapVault update schedule, and it can be configured independently on a volume.
• By default, every SnapVault update (baseline or incremental) kicks off the compression and
deduplication process on the destination after the archival Snapshot copy is created if compression
and deduplication are enabled. This feature can be modified by setting the postprocess compression
and deduplication schedule to manual. When this feature is modified, deduplication metadata is not
created, and postprocess compression and deduplication processes are stopped from running. This
option is beneficial for SnapVault destinations that have data with good compression savings but
minimal deduplication savings.
• For maximum savings, set the incompressible data detection option to false on a SnapVault
destination if the volume efficiency schedule is set to manual.
• You cannot modify the compression and deduplication schedule on a SnapVault destination to run at
a specific time or after a specific amount of data has been written to the volume. The volume
efficiency start command is not allowed; however, the volume efficiency start –
scan-old-data true command can be run manually on the destination.
• The SnapVault update is not tied to the compression and deduplication operation. A subsequent
incremental update can run even if the compression and deduplication process on the destination
volume from the previous backup is not complete. In this case, the compression and deduplication
process creates a checkpoint and stops running during the SnapVault update. The process resumes
after the SnapVault update is complete, although the archival Snapshot copy is not replaced. This
operation can result in the active file system temporarily having fewer savings and the Snapshot copy
taking more space until the Snapshot copy expires.
• You can manage deduplication and compression on SnapVault destinations using either the CLI or
OnCommand products. For additional SnapVault information, refer to TR-3487: SnapVault Design
and Implementation Guide.

Data compaction:
• Similar to SnapMirror, SnapVault does not have any effect on compacted data. The network transfer
is precompressed sub-4K chunks to save CPU and network bandwidth. As with SnapMirror, the
destination compaction savings might be different from those of the source due to repacking of sub-
4K chunks at the destination.
• Regardless of whether the source system supports data compaction, the data on the destination is
compacted on the data compaction supported destination system.
• If the destination system does not support data compaction, the sub-4K chunks are written unpacked
to storage.

Aggregate inline deduplication:

Cross-volume deduplication savings are lost during SnapVault transfers.

51 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
SnapRestore
NetApp SnapRestore® functionality is supported with both compression and deduplication, and it works in
the same way with either feature as it does without. When you initiate a SnapRestore operation on a
volume, the restored data retains the original space savings.
If Snapshot copy and the destination volumes being restored have a different compression type, then the
restored data does not retain the original compression space savings.
Deduplication metadata is restored to metadata corresponding to the data maintained in the Snapshot
copy. This process requires additional processing during the first deduplication operation completed after
using volume SnapRestore.
SnapRestore operates at logical level and does not affect data compaction.

SnapLock
NetApp SnapLock® technology is supported with compression, deduplication and data compaction.
However, inline compression is not supported for WORM append files in SnapLock volumes that are
present on 64-bit storage systems with 4 or more processors.

13.2 Other NetApp Features

Deduplication, Compression, Data Compaction, and Zero-Block Deduplication

Interoperability
Deduplication, compression, data compaction, and inline zero-block deduplication work seamlessly with
one another. If a system has all three features enabled, the order of execution is as follows:
1. Inline zero-block deduplication
2. Inline compression
3. Inline volume deduplication (Data ONTAP 8.3.2 and later)
4. Inline aggregate deduplication (ONTAP 9.2 and later; AFF systems only)
5. Inline data compaction (ONTAP 9 and later; AFF systems only)
6. Postprocess compression
7. Postprocess volume deduplication
NetApp internal tests have shown that the order of execution shows results in optimal savings with the
lowest performance and metadata processing overhead.

7-Mode Transition Tool (7MTT)

The 7-Mode Transition Tool preserves dedupe and compression settings when copying volumes from
Data ONTAP operating in 7-Mode to clustered Data ONTAP.
To use compression, the source and the destination volumes must belong to a 64-bit aggregate. All
compression and deduplication savings on the source volume are retained over the network during
transition. After transition, the destination volume inherits all of the compression and deduplication
attributes and storage savings from the source volume.
Transitioning deduplicated and compressed data helps to reduce the network bandwidth during transition
because of these reasons:
• Shared blocks are transferred only once.
• Compression is maintained throughout the transfer.
• Compressed and deduplicated data has smaller transfer sizes as a result of compression and
deduplication space savings; therefore, the transfers are completed more quickly.

52 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
You should not start compression or deduplication of existing data on the source volume during transition.
If deduplication or compression is in progress, start the transition only after the deduplication or
compression operation completes. Doing so avoids sending undeduplicated or uncompressed data and
additional temporary metadata files over the network to the destination volume.
For deduplication and compression to take effect on any new data written on the clustered Data ONTAP
volume, you must enable deduplication and compression schedules after the transition.
Deduplication maintains a partially ordered fingerprint database in the volume along with the aggregate
copy. As a result, the destination system has the space savings from the source volume as well as a copy
of the ordered fingerprint database. After migration, when volume efficiency is run on the new volume for
the first time, the aggregate fingerprint database is automatically constructed from the copy in the
destination volume. This process can result in a one-time increase in the time it takes for volume
efficiency operations to complete.
The same compression type (secondary compression) is maintained for the volume after transition to
clustered Data ONTAP.
Following are compression considerations when moving from 7-Mode to clustered Data ONTAP 8.3.1 or
later using 7MTT.
Scenario 1: Transition from 7-Mode System to Clustered Data ONTAP FAS System
• Case 1: Compression is not enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, compression is not enabled on the volume.
- Users can enable either inline or postprocess adaptive or secondary compression on the volume.
• Case 2: Inline secondary compression is enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, inline secondary compression is enabled on the
volume.
- Users can enable postprocess secondary compression on the volume.
- If users want to enable adaptive compression (inline or postprocess), they must rehydrate the
volume before changing the compression type.
• Case 3: Postprocess secondary compression is enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, postprocess secondary compression is enabled on the
volume.
- Users can enable inline secondary compression on the volume.
- If users want to enable adaptive compression (inline or postprocess), they must rehydrate the
volume before changing the compression type.
• Case 4: Both inline and postprocess secondary compression is enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, both inline and postprocess secondary compression is
enabled on the volume.
- If users want to enable adaptive compression (inline or postprocess), they must rehydrate the
volume before changing the compression type.
Scenario 2: Transition from 7-Mode System to Clustered Data ONTAP AFF System
Note: Postprocess compression is not supported on AFF systems.
• Case 1: Compression is not enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, compression is not enabled on the volume.

53 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 - Users can enable either inline adaptive or inline secondary compression on the volume. Only new
writes to the volume are compressed using the selected compression type.
- In Data ONTAP 8.3.1, the existing uncompressed data in the volume stays uncompressed until all
of the old data is overwritten.
- Starting in Data ONTAP 8.3.2, existing uncompressed data can be compressed using the
background scanner. Users have to manually run the background scanner command to
initiate the process. The background scanner works similarly to postprocess compression, but
scanning is done only once to convert the existing uncompressed data to a compressed format.
The background scanner compresses the existing data using the compression type assigned to
the volume.
• Case 2: Inline secondary compression is enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, inline secondary compression is enabled on the
volume.
- If users want to enable inline adaptive compression, they must rehydrate the data before
changing the compression type. Only new writes to the volume are compressed using the
selected compression type.
- In Data ONTAP 8.3.1, the existing uncompressed data in the volume stays uncompressed until all
the old data is overwritten.
- Starting in Data ONTAP 8.3.2, existing uncompressed data can be compressed using the
background scanner. Users have to manually run the background scanner command to
initiate the process. The background scanner works similarly to postprocess compression, but
scanning is done only once to convert the existing uncompressed data to a compressed format.
The background scanner compresses the existing data using the compression type assigned to
the volume.
• Case 3: Postprocess secondary compression is enabled on the 7-Mode volume:
- After transition to clustered Data ONTAP, postprocess secondary compression is disabled, but
the existing compressed data stays compressed in a secondary compression format.
- Users can enable inline secondary compression on the volume.
- If users want to enable inline adaptive compression, they must rehydrate the data before
changing the compression type. Only new writes to the volume are compressed using the
selected compression type.
- In Data ONTAP 8.3.1, the existing uncompressed data in the volume stays uncompressed until all the
old data is overwritten.
- Starting in Data ONTAP 8.3.2, existing uncompressed data can be compressed using the
background scanner. Users have to manually run the background scanner command to
initiate the process. The background scanner works similarly to postprocess compression, but
scanning is done only once to convert the existing uncompressed data to a compressed format.
The background scanner compresses the existing data using the compression type assigned to
the volume.
• Case 4: Both inline and postprocess secondary compression is enabled on the 7-Mode volume:
 After transition to clustered Data ONTAP, only inline secondary compression is enabled on the
volume. Postprocess compression is disabled on the volume, but the existing compressed data
stays compressed in a secondary compression format.
 If users want to enable inline adaptive compression, they must rehydrate the data before
changing the compression type. Only new writes to the volume are compressed using the
selected compression type.

54 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
  In Data ONTAP 8.3.1, the existing uncompressed data in the volume stays uncompressed until all
the old data is overwritten.
 Starting in Data ONTAP 8.3.2, existing uncompressed data can be compressed using the
background scanner. Users have to manually run the background scanner command to
initiate the process. The background scanner works similarly to postprocess compression, but
scanning is done only once to convert the existing uncompressed data to a compressed format.
The background scanner compresses the existing data using the compression type assigned to
the volume.

Data compaction:
When transitioning data to ONTAP 9 using 7MTT, data is compacted during transition.

Aggregate inline deduplication:

Cross-volume deduplication is not supported with 7MTT, because 7MTT uses SnapMirror to transfer
data.

Volume Move
Because NetApp supports mixed AFF-FAS clusters, we list compression points for you to consider when
moving volumes within a mixed cluster.
• Between AFF nodes and HA pairs:
 Both AFF nodes and HA pairs running a Data ONTAP 8.3.1 or later release:
o No change after the move: all volume configurations and savings are retained.
o All of the AFF compression features in Data ONTAP 8.3.1 and later are supported on the
volume.
 Both AFF nodes and HA pairs running a pre-8.3.1 Data ONTAP release:
o No change after the move: all volume configurations and savings are retained.
o All pre-8.3.1 AFF compression features are supported on the volume.
 Source AFF nodes and HA pairs running the Data ONTAP 8.3.1 or later release and destination
AFF node and HA pairs running a pre-8.3.1 release:
o If the source volume does not have compression enabled, there is no change after the
move. All pre-8.3.1 AFF compression features are supported on the volume following the
move.
o If the source volume has inline secondary compression enabled, there is no change after
the move. Inline secondary compression continues to be enabled, and all pre-8.3.1 AFF
compression features are supported on the volume following the move.
o A mixed-version cluster does not support adaptive compression volumes.

 Source node running a pre-8.3.1 Data ONTAP release and destination node running a Data
ONTAP 8.3.1 or later release:
o Refer to Table 10 for the behavior.

55 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 10) Compression behavior after volume move from pre-8.3.1 AFF node to 8.3.1+ AFF node.

Before the Move After the Move

Pre-8.3.1 AFF Node 8.3.1 or Later AFF Node

Volume 1: no compression Volume 1: no compression:
• You can enable inline and secondary compression on
Volume 1.
• A mixed-version cluster does not support adaptive
compression. After all the nodes in the cluster are upgraded
to 8.3.1 or later, you can enable adaptive compression.
• Existing data on the volume remains uncompressed.
• In Data ONTAP 8.3.1, you cannot explicitly convert the
existing data to a compressed format. Existing data is
converted to a compressed format when the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
must manually run the background scanner command to
initiate the process. The background scanner works similarly
to postprocess compression, but scanning is done only once
to convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

56 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Before the Move After the Move
Volume 3: inline secondary Volume 3: inline secondary compression:
compression + background • Background compression is disabled because it is not
secondary compression supported on AFF configurations.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 4: background secondary Volume 4: no compression:

compression • Background compression is disabled on the volume because
it is not supported on AFF configurations.
• Existing compressed data remains compressed in a
secondary compression format.
• If you want, you can enable inline secondary compression.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

• Between FAS nodes:

 Both FAS nodes running a Data ONTAP 8.3.1 or later release:
o No change after the move: all volume configurations and savings are retained.
o All Data ONTAP 8.3.1 and later FAS compression features are supported on the volume.
 Both FAS nodes running a pre-8.3.1 Data ONTAP release:
o No change after the move: all volume configurations and savings are retained.

57 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 o All pre-8.3.1 FAS compression features are supported on the volume.
 Source FAS node running a Data ONTAP 8.3.1 or later release and destination FAS node
running a pre-8.3.1 release:
o If the volume does not have compression enabled, there is no change after the move. All
pre-8.3.1 FAS compression features are supported on the volume.
o If the volume has inline and/or secondary compression enabled, there is no change after
the move. Inline and/or secondary compression continues to be enabled, and all pre-
8.3.1 compression features are supported on the volume.
o A mixed-version cluster does not support adaptive compression volumes.
 Source FAS node running a pre-8.3.1 Data ONTAP release and destination FAS node running a
Data ONTAP 8.3.1 or later release:
o Refer to Table 11 for the behavior.

Table 11) Compression behavior after volume move from pre-8.3.1 FAS node to 8.3.1+ FAS node.

Before the Move After the Move

Pre-8.3.1 FAS Node 8.3.1 or Later FAS Node

Volume 1: no compression Volume 1: no compression:
• You can enable either inline and/or postprocess secondary
compression on Volume 1.
• A mixed-version cluster does not support adaptive
compression. After all the nodes in the cluster are upgraded
to 8.3.1 or later, you can enable adaptive compression.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.

Volume 3: inline secondary Volume 3: inline secondary compression + background secondary

compression + background compression:
secondary compression • If you want to enable inline or postprocess adaptive
compression, first upgrade all the nodes in the cluster to 8.3.1
or later and rehydrate the volume.

Volume 4: background secondary Volume 4: background secondary compression:

compression • You can enable inline secondary compression.
• If you want to enable inline or postprocess adaptive
compression, first upgrade all the nodes in the cluster to 8.3.1
or later and rehydrate the volume.

• Between FAS and AFF nodes:

 Both FAS node and AFF nodes running a Data ONTAP 8.3.1 or later release:
o Refer to Table 12 for the behavior.

58 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 12) Compression behavior after volume move from 8.3.1+ FAS node to 8.3.1+ AFF node.

Before the Move After the Move

8.3.1 or Later FAS Node 8.3.1 or Later AFF Node

Volume 1: no compression Volume 1: no compression:
• You can enable inline adaptive or inline secondary
compression on Volume 1.
• Existing data on the volume remains uncompressed.
• In Data ONTAP 8.3.1, you cannot explicitly convert the
existing data to a compressed format. Existing data is
converted to a compressed format when the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• If you want to enable inline adaptive compression, first
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

59 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Before the Move After the Move
Volume 3: inline secondary Volume 3: inline secondary compression:
compression + background • Background compression is disabled because it is not
secondary compression supported on AFF configurations.
• If you want to enable inline adaptive compression, first
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 4: background secondary Volume 4: no compression:

compression • Background compression is disabled on the volume because
it is not supported on AFF configurations.
• Existing compressed data remains compressed in a
secondary compression format.
• If you want, you can enable inline secondary compression.
• If you want to enable inline adaptive compression, first
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. Background scanner works similarly to
postprocess compression, but scanning is done only once to
convert the existing uncompressed data to a compressed
format. The background scanner compresses the existing
data using the compression type assigned to the volume.

60 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Before the Move After the Move
Volume 5: inline adaptive Volume 5: inline adaptive compression:
compression • Existing compressed data remains compressed in an
adaptive compression format.
• If you want to enable inline secondary compression, first
rehydrate the volume. Only new writes to the volume are
compressed using secondary compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 6: background adaptive Volume 6: no compression:

compression • Background compression is disabled on the volume because
it is not supported on AFF configurations.
• Existing compressed data remains compressed in an
adaptive compression format.
• If you want, you can enable inline adaptive compression.
• If you want to enable inline secondary compression, first
rehydrate the volume. Only new writes to the volume are
compressed using secondary compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

61 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Before the Move After the Move
Volume 7: inline adaptive Volume 7: inline adaptive compression:
compression + background adaptive • Background compression is disabled because it is not
compression supported on AFF configurations.
• If you want to enable inline secondary compression, first
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

 Both FAS node and AFF node running a pre-8.3.1 Data ONTAP release:
o All volume configurations and savings are retained.
o All pre-8.3.1 AFF compression features are supported on the volume.
 Source FAS node running a Data ONTAP 8.3.1 or later release and destination AFF node
running a pre-8.3.1 Data ONTAP release:
o If the volume does not have compression enabled, there is no change after the move. All
pre-8.3.1 AFF compression features are supported on the volume.
o If the volume had inline and/or postprocess secondary compression enabled, these
processes continue to be enabled after the move. All pre-8.3.1 AFF compression features
are supported on the volume.
o A mixed-version cluster does not support adaptive compression volumes.
 Source FAS node running a pre-8.3.1 Data ONTAP release and destination AFF node running a
Data ONTAP 8.3.1 or later release:
o Refer to Table 13 for the behavior.

62 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 13) Compression behavior after volume move from pre-8.3.1 FAS node to 8.3.1+ AFF node.

Before the Move After the Move

Pre-8.3.1 FAS Node 8.3.1 or Later AFF Node

Volume 1: no compression Volume 1: no compression:
• You can enable inline secondary compression on Volume 1.
• A mixed-version cluster does not support adaptive
compression. After all the nodes in the cluster are upgraded
to 8.3.1 or later, you can enable adaptive compression.
• Existing data on the volume remains uncompressed.
• In Data ONTAP 8.3.1, you cannot explicitly convert the
existing data to a compressed format. Existing data is
converted to a compressed format when the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 2: inline secondary Volume 2: inline secondary compression:

compression • Existing compressed data remains compressed in a
secondary compression format.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

63 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Before the Move After the Move
Volume 3: inline secondary Volume 3: inline secondary compression:
compression + background • Background compression is disabled because it is not
secondary compression supported on AFF configurations.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting in Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

Volume 4: background secondary Volume 4: no compression:

compression • Background compression is disabled on the volume because
it is not supported on AFF configurations.
• Existing compressed data remains compressed in a
secondary compression format.
• If you want, you can enable inline secondary compression.
• If you want to enable inline adaptive compression, first
upgrade all the nodes in the cluster to 8.3.1 or later and
rehydrate the volume. Only new writes to the volume are
compressed using adaptive compression.
• In Data ONTAP 8.3.1, the existing uncompressed data in the
volume stays uncompressed until all the old data is
overwritten.
• Starting with Data ONTAP 8.3.2, existing uncompressed data
can be compressed using the background scanner. Users
have to manually run the background scanner command
to initiate the process. The background scanner works
similarly to postprocess compression, but scanning is done
only once to convert the existing uncompressed data to a
compressed format. The background scanner compresses
the existing data using the compression type assigned to the
volume.

• Between AFF and FAS nodes:

 Both AFF nodes and FAS nodes running a Data ONTAP 8.3.1 or later release:
o No change after the move: all volume configurations and savings are retained.
o All Data ONTAP 8.3.1 and later FAS compression features are supported on the volume.
 Both AFF nodes and FAS nodes running a pre-8.3.1 Data ONTAP release:
o No change after the move: all volume configurations and savings are retained.

64 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
 o All pre-8.3.1 Data ONTAP 8.3.1 FAS compression features are supported on the volume.
 Source AFF node running a Data ONTAP 8.3.1 or later release and destination FAS node
running a pre-8.3.1 release:
o If the volume does not have compression enabled, there is no change after the move. All
pre-8.3.1 FAS compression features are supported on the volume.
o If the volume had inline secondary compression enabled, inline secondary compression
continues to be enabled after the move. All pre-8.3.1 AFF compression features are
supported on the volume.
o A mixed-version cluster does not support adaptive compression volumes.
 Source AFF node running a pre-8.3.1 Data ONTAP release and destination FAS node running a
Data ONTAP 8.3.1 or later release:
o No change after the move: all volume configuration and savings are retained.
o All pre-8.3.1 Data ONTAP FAS compression features are supported on the volume.
Data compaction:
• Data compaction and volume move operate at a physical level, and volume move uses block
replication engine (BRE) to transfer data from source and destination.
• Like SnapMirror and SnapVault, during volume move, precompressed sub-4K chunks are sent to the
destination, where they are repacked on a compaction-supported destination volume or written as is
on a compaction-unsupported destination volume.

Aggregate inline deduplication:

• Cross-volume deduplication savings are lost during volume move.

Infinite Volume
Both compression and deduplication are supported on Infinite Volumes. Compression and deduplication
operate the same way on Infinite Volumes as they do on FlexVol volumes. Here are some important
points to consider:
• Compression and deduplication are enabled at the Infinite Volume level. If you want to enable either
deduplication or data compression, the process is enabled on the entire Infinite Volume; it cannot be
enabled on some data constituents and not on others.
• When deduplication is run, the process looks for duplicate blocks within a single data constituent, not
across the entire Infinite Volume. Blocks that exist in different data constituents are not deduplicated
against one another. For example, if you have four data constituents, each with four copies of the
same block, after deduplication completes on the Infinite Volume, each data constituent stores one
physical copy of the duplicate blocks.
• When postprocess volume deduplication or compression operations are started on the Infinite
Volume, separate operations are run on each of the data constituents up to the maximum of eight per
node. If more than the maximum number of allowed compression and deduplication operations are
scheduled to run at any one time, the operations are queued and run as each deduplication operation
completes.
• As with FlexVol volumes, a data constituent requires free space in the volume and in the aggregate
for deduplication metadata. See the Deduplication Metadata section for more details.
• The namespace constituent is not deduplicated or compressed.
• Some commands do not operate as they do on FlexVol volumes. These include the next two bullet
points.

65 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
• You can choose to view aggregated space savings for an Infinite Volume, or you can see savings at
the individual data constituent level. To see savings at the data constituent level, run the volume
show command, substituting the –volume <volname> option with -is-constituent true. For
example, volume show -vserver <vserver_name> -is-constituent true.
• The volume efficiency show command does not currently display information about status or
progress for an Infinite Volume. To see the status or progress of individual data constituents,
substitute the –volume option for the –is-constituent true option. For example, volume
efficiency show –vserver <SVMname> -is-constituent true –fields progress.

Data compaction and aggregate inline deduplication operate the same way on Infinite Volumes as on
FlexVol volumes.

HA Controller Configuration
HA pair controller configurations are supported with both deduplication and compression. NetApp
recommends that both nodes run the same version of Data ONTAP.
Deduplication and compression run on each node independently.
Writes to the FlexVol volume or data constituents have fingerprints written to the change log. During
failover or giveback to the partner node, inline compression continues as normal and deduplication
change logging continues to happen. Postprocess compression and deduplication processes start at the
next scheduled time on the failed node after failover or giveback completes using the updated change
log, or they can be started manually. A maximum of eight concurrent compression and deduplication
processes is allowed on each node on an HA pair configuration. During failover, the surviving node
continues to be limited to a maximum of eight concurrent compression and deduplication operations,
even though deduplication continues for both nodes.
NetApp recommends that no active postprocess compression and deduplication operations take place
during the planned takeover or planned giveback:
• Perform the planned takeover or giveback during a time when compression and deduplication
operations are not scheduled to run.
• Determine whether any compression or deduplication operations are active and stop them until the
planned takeover or giveback completes.
For additional information about active-active controller configurations, refer to TR-3450: Active-Active
Controller Configuration Overview and Best Practice Guidelines.
Data compaction and aggregate inline deduplication are supported on HA controller configurations.

Quotas
Quotas are based on logical space usage; therefore, for compressed and deduplicated files, the logical
(uncompressed and undeduplicated) size is charged against the quotas. There are several advantages to
this scheme as opposed to charging quotas based on the physical (compressed and deduplicated) size of
the file:
• This scheme is in line with the general design principle of making deduplication and compression
transparent to the end user.
• It is easier for system administrators to manage quotas. They can maintain a single quota policy
across all volumes, whether or not compression or deduplication is enabled on them.
• Overwriting parts of the file does not fail because of quota errors when the new data being written is
not as compressible as the data it is replacing.

66 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Quotas cannot be oversubscribed on a volume. For example, a user with a quota limit of 1TB cannot
store more than 1TB of data in a deduplicated and compressed volume. This point is true even if the data
has been deduplicated and compressed and fits into less than 1TB of physical space on the storage
system.
Because quotas operate at logical level, data compaction does not affect how quotas are reported.

Flash Cache
NetApp Flash Cache™ cards can be used to reduce the number of random disk reads of blocks that are
read repeatedly. This reduction can improve the read performance in environments in which a lot of
shared blocks are read repeatedly.
The amount of performance improvement deduplication-enabled volumes have with Flash Cache
depends on the duplication rate, the access rate, the active dataset size, and the data layout. Adding a
Flash Cache card to a system does not increase the deduplication maximum volume size for that system.
Deduplication also enables efficient use of Flash Cache by retaining the deduplication savings on Flash
Cache that exist on disk. In that way, if you have 32K duplicate blocks on disk, after you run
deduplication, only one block is used on disk. If the blocks are randomly accessed, only one block is used
in Flash Cache as well. This feature can significantly increase the amount of data that can be stored in
Flash Cache.
The Flash Cache card can provide significant performance improvements in VMware VDI environments.
The advantages provided by NetApp Flash Cache cards are further enhanced when combined with other
shared block technologies, such as NetApp deduplication or NetApp FlexClone technology. For additional
information about the NetApp and VMware solution, refer to TR-3705: NetApp and VMware Solution
Guide.
A volume on which compression has been run can contain both compressed and uncompressed blocks.
How Flash Cache works with blocks depends on whether the blocks are written on disk in a compressed
or uncompressed format, regardless of whether compression is enabled or disabled. Flash Cache does
not cache compressed blocks. However, the uncompressed blocks on disk and the metadata continue to
benefit from the performance improvements of Flash Cache. For additional information about the Flash
Cache card, refer to TR-3832: Flash Cache Best Practice Guide.
Data compaction:
• Compacted blocks are treated as any other WAFL block and are cached in Flash Cache cards.

Flash Pool
Both deduplication and compression are supported on Flash Pool aggregates.
In environments with large amounts of shared blocks that are read repeatedly or overwritten randomly,
Flash Pool can significantly reduce the number of HDD reads and writes, thus improving performance.
Flash Pool does this reducing by retaining the deduplication savings on SSD that exist on HDD. If you
have 32K duplicate blocks on disk, after you run deduplication, only one block is used on disk (HDD). If
any requests for duplicate blocks are randomly requested, only one block is used in the Flash Pool
aggregate (SSD) as well. If data is write cached, it is still evaluated for deduplication the next time
deduplication is run. If the data is successfully deduplicated, it remains on SSD until it becomes cold and
is ejected as normal. The amount of performance improvement with Flash Pool depends on the number
of shared blocks, the access rate, the active dataset size, and the data layout.
Starting with Data ONTAP 8.3.1, compressed blocks are eligible for both read and write caching. Any
data that is sequentially written continues to be written to HDD regardless of whether compression is
enabled or not.

67 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
For more information about Flash Pool, refer to TR-4070: NetApp Flash Pool Design and Implementation
Guide.
Data compaction:
• Only blocks written to the HDD tier in Flash Pool are compacted. Blocks written to the SSD tier are
not compacted.
• The system supports reading compacted blocks from the SSD tier.

Aggregate inline deduplication:

Cross-volume deduplication is not supported in the Flash Pool configuration.

FlexClone Volumes
FlexClone volumes support both compression and deduplication. When a FlexClone volume (cloned
volume) is created:
If the parent FlexClone volume has compression or duplication enabled, the new volume inherits the
compression and deduplication savings and the configuration of the parent volume, such as the
compression and deduplication schedule.
The deduplication process also continues for any new data written to the clone. It also recreates the
fingerprint database in the aggregate from the volume copy of the deduplication metadata files (the
fingerprint database and the change log files). Doing so allows the deduplication process to continue to
deduplicate between the new data and the old data.
If you compress existing data with the -shared-blocks true or –snapshot-blocks true options
on the clone, then all compressible data on the parent is written as new data on the clone. This operation
can significantly increase the size of the clone and thus reduce the space saving benefits of cloning.
One use case for FlexClone volumes with compression or deduplication is to quickly set up an
environment with compression or deduplication to determine the possible deduplication and compression
savings on a volume. This feature is useful in the following three scenarios when you do not want to make
changes to the production environment:
• When you have a parent volume that does not have deduplication enabled
• When you have a parent volume that does not have deduplication or compression enabled
• When you have a parent volume that has deduplication but you want to test adding compression

Data compaction:
• FlexClone volumes operate at the logical level and are unaffected by data compaction.

FlexClone Files
FlexClone volumes at the file and LUN levels are available and are allowed on compressed and/or
deduplicated volumes. Both compression and deduplication can be used to regain capacity savings after
a FlexClone volume is broken.
Starting with Data ONTAP 8.3.1, subfile cloning is supported with NetApp data compression for adaptive
compression–enabled volumes.
Data compaction:
• FlexClone volumes operate at the logical level and are unaffected by data compaction.

68 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Volume Splitting
When a cloned volume is split from the parent volume, all of the deduplicated data in the clone that was
part of the parent volume (not including the data that was written to the cloned volume after the clone was
created) is undeduplicated after the volume split operation. If aggregate inline deduplication is enabled on
the cloned volume, this data gets deduplicated again, and the savings are restored. If compression is
enabled on the cloned volume, then, during a split, the data from the parent is transferred to the clone as
uncompressed. The data is recompressed on the clone. Inline compression operations are not attempted
during volume splitting.
Data compaction: Volume splitting is unaffected by data compaction.

NDMP
Both deduplication and compression support backup to a tape through NDMP. The data sent from the
source volume and written to the tape is in an uncompressed and undeduplicated format.

Reallocation (realloc)
Data that is shared within a volume can be reallocated using physical reallocation or read_realloc
space_optimized. Although data can be shared by multiple files when deduplicated, reallocate uses an
intelligent algorithm to reallocate only the data the first time a shared block is encountered. However, data
that is shared across volumes cannot be reallocated. Compressed data is not reallocated by reallocate or
read reallocate, and NetApp does not recommend running reallocate on compressed volumes.

LUNs
When using deduplication or compression on a file-based (NFS/CIFS) environment, both deduplication
and compression are straightforward and automatic. As data is written, it is compressed inline or
postprocess. As duplicate blocks are freed from deduplication, they are marked as available. In both
cases, as blocks of free space become available, the NetApp system recognizes these free blocks and
makes them available to the volume.
Deduplication and compression on a block-based (FCP/iSCSI) LUN environment are slightly more
complicated because of the space guarantees and fractional reservations used by LUNs. For instance,
consider a volume that contains a 500GB LUN and the LUN has LUN reserves enabled. The LUN reserve
causes the space for the LUN to be reserved when the LUN is created.
Now consider that 500GB of data is written to the LUN. The consumed space is exactly 500GB of the
physical disk space. If the data in the LUN is reduced through compression or deduplication, the LUN still
reserves the same physical space capacity of 500GB, and the space savings are not apparent to the
user.
LUN space guarantees and fractional reserves can be configured so that the use by the NetApp system
of the freed blocks changes depending on the configuration. By varying the values of certain parameters,
freed blocks can be returned to either the volume free pool or the aggregate free pool.
This section describes four common examples of LUN configurations and the associated compression
and deduplication behavior, summarized in Table 14.

69 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Table 14) Summary of LUN configuration examples.

A (Default) B C D

LUN space guarantee value Yes Yes No No

Volume fractional reserve value 100 0 0 0

LUN (fractional) overwrite reserved Yes Yes No No

Volume space guaranteed Yes Yes Yes No

Compression and deduplication savings Volume free Aggregate

Neither Neither
are returned to which free pool? pool free pool

Definitions
• LUN (fractional) overwrite reserve. The space that Data ONTAP reserves is available for
overwriting blocks in a LUN when the space guarantee = yes. Because this space is reserved, any
savings from compression or deduplication are not available for other use.
• Volume free pool. This term refers to the free blocks in the parent volume of the LUN. These blocks
can be assigned anywhere in the volume as needed.
• Aggregate free pool. This term refers to the free blocks in the parent aggregate of the LUN. These
blocks can be assigned anywhere in the aggregate as needed.

Configuration A: Default LUN Configuration

The default configuration of a NetApp LUN follows. (The best practice for all NetApp LUNs is for you to
turn the controller Snapshot copy off, delete all scheduled Snapshot copies, and set the snap reserve to
0.)
LUN space reservation value = on Default
Volume fractional reserve value = 100 Default
Volume guarantee = volume Default
Snap reserve = 0% (default = 5%) vol modify -percent-snapshot-space 0
Autodelete = off Default
Autosize = off Default
Try_first = volume_grow Default
This configuration is used so that overwrites to the LUN never fail, even if the LUN is overwritten entirely.
When a LUN is created with LUN space reservation on and volume guarantee set to volume, no apparent
compression or deduplication savings are observed by the storage administrator.
Configuration B: LUN Configuration with Volume Free Space Management
If the user wants to remove LUN overwrite reserve space, the user can do so with the following
configuration:
LUN space reservation value = on Default
Volume fractional reserve value = 0 vol modify -fractional-reserve 0
Volume guarantee = volume Default
Snap reserve = 0% vol modify -percent-snapshot-space 0

70 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Autodelete = on snapshot autodelete modify –enabled true
Autosize = on vol modify –max-autosize X –autosize-increment Y,
where X represents the maximum size to which the volume can grow, and Y represents the increment by
which to grow the volume; both are set in values of <size>[k|m|g|t])
Try_first = volume_grow Default
The differences between this configuration and configuration A are that the value of the fractional reserve
is set to zero, and both autodelete and autosize are on. As a result, in this configuration, no blocks are set
aside for LUN overwrites. To mitigate the possibility of overwrite failures caused by a full volume, NetApp
recommends turning on Snapshot autodelete and volume autosize. Snapshot autodelete frees space in a
volume by automatically deleting old Snapshot copies. Volume autosize expands the volume when it
reaches a predefined threshold.
In this configuration, if compression or deduplication was run, there would be no space savings even if a
space reclamation tool was run. The reason is that with LUN space reservation on, 100% of the LUN
space is allocated at the time of LUN creation whether those blocks contain data or are empty.
Configuration C: LUN Configuration for Maximum Volume Space Savings
If the user wants to make freed blocks available to the volume free pool, the user can do so with the
following configuration:
LUN space reservation value = off lun modify -space-reserve disabled lun_path
Volume fractional reserve value = 0 vol modify -fractional-reserve 0
Volume guarantee = volume Default
Snap reserve = 0% (default = 5%) vol modify -percent-snapshot-space 0
Autodelete = on snapshot autodelete modify –enabled true
Autosize = on vol modify –max-autosize X –autosize-increment Y,
where X represents the maximum size to which the volume can grow, and Y represents the increment by
which to grow the volume; both are set in values of <size>[k|m|g|t])
Try_first = volume_grow Default
The difference between this configuration and configuration B is that the LUN is not space reserved. At
the time of creation, the LUN takes up no space from the volume. Only when data is written to the LUN is
space allocated. This process allows volumes to support more LUNs than they physically have space for.
With LUN space guarantees off, the value for the volume fractional reserve is ignored for all LUNs in this
volume. However, because autodelete and autosize are both on, the volume expands rather than runs
out of space if the total allocated blocks approach the total size of the volume.
In this configuration, after compression and deduplication are run, savings are not seen unless a space
reclamation tool such as NetApp SnapDrive® software is run. If space reclamation is run after
compression and deduplication complete, then the freed blocks are available to the volume.
Configuration D: LUN Configuration for Maximum Volume and Aggregate Space Savings
This configuration provides optimal space savings by returning all freed blocks to the aggregate free pool.
This process is accomplished with the following configuration:
LUN space reservation value = off lun modify -space-reserve disabled lun_path
Volume fractional reserve value = 0 vol modify -fractional-reserve 0
Volume guarantee = none vol modify -space-guarantee none
Snap reserve = 0% (default = 5%) vol modify -percent-snapshot-space 0
Autodelete = on snapshot autodelete modify –enabled true

71 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Autosize = on vol modify –max-autosize X –autosize-increment Y,
where X represents the maximum size to which the volume can grow and Y represents the increment by
which to grow the volume; both are set in values of <size>[k|m|g|t])
Try_first = volume_grow Default

The difference between this configuration and configuration C is that the volume is not space reserved.
This configuration automatically allocates the free blocks from the volume into the aggregate free pool,
where the blocks can be reallocated to any other volumes or LUNs in the aggregate.
The advantage of this configuration is that it provides the highest efficiency in aggregate space
provisioning. The configuration also uses Data ONTAP thin provisioning features, volume autosize, and
Snapshot autodelete to help administer the space in the solution.
In this configuration, any blocks in the volume that are freed by compression and deduplication are
automatically allocated to the aggregate free space pool. If space reclamation is performed on the LUN,
then any freed blocks from the LUN are also allocated to the aggregate free pool.

72 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
References
• NetApp Space Savings Estimation Tool
 Field Portal link (NetApp employees and authorized NetApp partners)
 [Link]
 Utility Toolchest link (NetApp customers, NetApp employees and authorized NetApp partners)
• TR-4514: NetApp AFF8080A EX Storage Efficiency and Performance with Oracle Database
NetApp AFF8080A EX Storage Efficiency and Performance with Oracle Database
• TR-4516: NetApp AFF8080A EX Storage Efficiency and Performance with Microsoft SQL Server
2014
NetApp AFF8080A EX Storage Efficiency and Performance with Microsoft SQL Server 2014
• TR-4403: NetApp AFF8080 EX Performance and Server Consolidation with Microsoft SQL Server
2014
[Link]
• TR-4415: NetApp AFF8080 EX Performance and Server Consolidation with Oracle Database
[Link]
• TR-3633: Oracle Databases on Data ONTAP
[Link]
• TR-4428: NetApp All Flash FAS Solution for VMware Horizon 6 and vSphere Virtual Volumes
[Link]
• TR-4307: NetApp All Flash FAS Solution for Nonpersistent Desktops with VMware Horizon View
[Link]
• TR-4335: NetApp All Flash FAS Solution for Persistent Desktops with VMware Horizon View
[Link]

Version History

Version Date Document Version History

Version 1.0 November 2015 Karthik Viswanath: initial version

Version 2.0 July 2016 Karthik Viswanath: updated to include 4:1 storage efficiency
guarantee and data compaction details

Version 2.0.1 January 2017 Karthik Viswanath: Updated to include the new storage
efficiency guarantee details and updated storage efficiency
ratios

Version 3.0 May 2017 Karthik Viswanath: Updated to include inline aggregate
deduplication content

Version 3.1 January 2018 Karthik Viswanath: Updated to include background aggregate
deduplication and automatic deduplication content

73 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Contact Us
Let us know how we can improve this technical report.
Contact us at docfeedback@[Link].
Include TECHNICAL REPORT 4476 in the subject line.

74 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.
Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact
product and feature versions described in this document are supported for your specific environment. The
NetApp IMT defines the product components and versions that can be used to construct configurations
that are supported by NetApp. Specific results depend on each customer’s installation in accordance with
published specifications.
Copyright Information
Copyright © 2015–2018 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document
covered by copyright may be reproduced in any form or by any means—graphic, electronic, or
mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—
without prior written permission of the copyright owner.
Software derived from copyrighted NetApp material is subject to the following license and disclaimer:
THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY
DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
NetApp reserves the right to change any products described herein at any time, and without notice.
NetApp assumes no responsibility or liability arising from the use of products described herein, except as
expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license
under any patent rights, trademark rights, or any other intellectual property rights of NetApp.
The product described in this manual may be protected by one or more U.S. patents, foreign patents, or
pending applications.
RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to
restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software
clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).
Trademark Information
NETAPP, the NETAPP logo, and the marks listed at [Link] are trademarks of
NetApp, Inc. Other company and product names may be trademarks of their respective owners.

75 TR-4476: NetApp Data Compression and Deduplication: Data ONTAP 8.3.1 and Later ® 2018 NetApp, Inc. All Rights Reserved.