0% found this document useful (0 votes)

118 views482 pages

CDOT Student Guide

Uploaded by

Dheeraj Kumar Singh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

118 views482 pages

CDOT Student Guide

Uploaded by

Dheeraj Kumar Singh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Clustered Data ONTAP Administration

NETAPP UNIVERSITY

Clustered Data ONTAP Administration

Student Guide
Course ID: STRSW-ILT-D8CADM-REV02
Catalog Number: STRSW-ILT-D8CADM-REV02-SG
Content Version: 3.0

NetApp University - Do Not Distribute

ATTENTION
The information contained in this course is intended only for training. This course contains information and activities that,
while beneficial for the purposes of training in a closed, non-production environment, can result in downtime or other
severe consequences in a production environment. This course material is not a technical reference and should not,
under any circumstances, be used in production environments. To obtain reference materials, refer to the NetApp product
documentation that is located at [Link]

COPYRIGHT
© 2012 NetApp, Inc. All rights reserved. Printed in the U.S.A. Specifications subject to change without notice.
No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or
mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written
permission of NetApp, Inc.

U.S. GOVERNMENT RIGHTS

Commercial Computer Software. Government users are subject to the NetApp, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

TRADEMARK INFORMATION
NetApp, the NetApp logo, Go further, faster, AdminNODE, Akorri, ApplianceWatch, ASUP, AutoSupport, BalancePoint,
BalancePoint Predictor, Bycast, Campaign Express, ChronoSpan, ComplianceClock, ControlNODE, Cryptainer, Data
ONTAP, DataFabric, DataFort, Decru, Decru DataFort, DenseStak, Engenio, E-Stack, FAServer, FastStak, FilerView,
FlexCache, FlexClone, FlexPod, FlexScale, FlexShare, FlexVol, FPolicy, GatewayNODE, gFiler, Imagine Virtually
Anything, Infinivol, Lifetime Key Management, LockVault, Manage ONTAP, MetroCluster, MultiStore, NearStore, NetApp
Select, NetCache, NetCache, NOW (NetApp on the Web), OnCommand, ONTAPI, PerformanceStak, RAID DP,
SANscreen, SANshare, SANtricity, SecureAdmin, SecureShare, Securitis, Service Builder, Simplicity, Simulate ONTAP,
SnapCopy, SnapDirector, SnapDrive, SnapLock, SnapManager, SnapMirror, SnapMover, SnapProtect, SnapRestore,
Snapshot, SnapValidator, SnapVault, StorageGRID, StorageNODE, StoreVault, SyncMirror, Tech OnTap, VelocityStak,
vFiler, VFM, Virtual File Manager, WAFL, and XBB are trademarks or registered trademarks of NetApp, Inc. in the United
States and/or other countries.
All other brands or products are either trademarks or registered trademarks of their respective holders and should be
treated as such.

2 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

TABLE OF CONTENTS
WELCOME ........................................................................................................................................................ 1
MODULE 1: OVERVIEW ................................................................................................................................ 1-1
MODULE 2: INSTALLATION AND CONFIGURATION ................................................................................ 2-1
MODULE 3: ARCHITECTURE ....................................................................................................................... 3-1
MODULE 4: USER INTERFACE .................................................................................................................... 4-1
MODULE 5: PHYSICAL DATA STORAGE ................................................................................................... 5-1
MODULE 6: VIRTUAL DATA STORAGE ...................................................................................................... 6-1
MODULE 7: PHYSICAL NETWORKING ....................................................................................................... 7-1
MODULE 8: VIRTUAL NETWORKING .......................................................................................................... 8-1
MODULE 9: NAS PROTOCOLS .................................................................................................................... 9-1
MODULE 10:SAN PROTOCOLS ................................................................................................................. 10-1
MODULE 11: STORAGE EFFICIENCY ....................................................................................................... 11-1
MODULE 12: DATA PROTECTION ............................................................................................................. 12-1
MODULE 13: BASIC TROUBLESHOOTING AND PERFORMANCE ........................................................ 13-1
MODULE 14: CLUSTER MANAGEMENT ................................................................................................... 14-1
MODULE 15: RECOMMENDED PRACTICES ............................................................................................ 15-1

3 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Clustered Data
ONTAP Administration
Course ID:
STRSW-ILT-D8CADM-REV02

NetApp Confidential

CLUSTERED DATA ONTAP ADMINISTRATION

4 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Classroom Logistics

Schedule Safety
 Start time  Alarm signal
 Stop time  Evacuation procedure
 Break times  Electrical safety
guidelines
Facilities
 Food and drinks
 Restrooms
 Phones

NetApp Confidential 2

CLASSROOM LOGISTICS

5 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Objectives
1 of 2
By the end of this course, you should be able to:
 List the benefits of the Data ONTAP operating
system and explain its two modes of operation
 Enumerate and describe the steps for setting up
a cluster
 Manage the physical and virtual resources within
a cluster
 Manage features to guarantee nondisruptive
operations
 Identify the networking components and
networking features of a cluster

NetApp Confidential 3

COURSE OBJECTIVES: 1 OF 2

6 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Objectives
2 of 2
 Implement supported cluster and management
network switches
 Diagram the architecture of a cluster
 Configure a Flash Pool
 Configure a cluster that hosts an Infinite Volume
 Set up and configure SAN and NAS protocols
 Administer mirroring technology and data protection
for a cluster
 Explain the notification capabilities of a cluster
 Scale a cluster horizontally
 Describe the characteristics of a cluster that affect the
cluster’s performance
NetApp Confidential 4

COURSE OBJECTIVES: 2 OF 2

7 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Agenda: Day 1

 Morning
– Module 1: Overview
 Afternoon
– Module 2: Installation and Configuration
– Module 3: Architecture

NetApp Confidential 5

COURSE AGENDA: DAY 1

8 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Agenda: Day 2

 Morning
– Module 4: User Interface
– Module 5: Physical Data Storage
 Afternoon
– Module 6: Virtual Data Storage
– Module 7: Physical Networking

NetApp Confidential 6

COURSE AGENDA: DAY 2

9 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Agenda: Day 3

 Morning
– Module 8: Virtual Networking
– Module 9: NAS Protocols
 Afternoon
– Module 10: SAN Protocols

NetApp Confidential 7

COURSE AGENDA: DAY 3

10 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Agenda: Day 4

 Morning
– Module 11: Storage Efficiency
 Afternoon
– Module 12: Data Protection

NetApp Confidential 8

COURSE AGENDA: DAY 4

11 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Course Agenda: Day 5

 Morning
– Module 13: Basic Troubleshooting and
Performance
– Module 14: Cluster Management
 Afternoon
– Module 15: Recommended Practices

NetApp Confidential 9

COURSE AGENDA: DAY 5

12 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

NetApp University Information Sources

 NetApp Support
[Link]

 NetApp University
[Link]

 NetApp University Support

[Link]

NetApp Confidential 10

NETAPP UNIVERSITY INFORMATION SOURCES

13 Clustered Data ONTAP Administration: Welcome

NetApp University - Do Not Distribute

Module 1
Overview

NetApp Confidential 1

MODULE 1: OVERVIEW

1-1 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Module Objectives
After this module, you should be able to:
 Identify the unique features of Data ONTAP
operating in Cluster-Mode relative to Data
ONTAP operating in 7-Mode
 Explain the three primary benefits of Data ONTAP
clustering
 Identify the scope of Data ONTAP concepts such
as node virtual servers (Vservers), administrative
Vservers, and cluster (data) Vservers
 Discuss the features and advantages of the
NetApp storage tier strategy

NetApp Confidential 2

MODULE OBJECTIVES

1-2 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

1-3 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

A Tale of Two Products

The Data ONTAP The Data ONTAP 7G

Operating System Operating System

SpinFS The Data ONTAP GX Data ONTAP 8

Operating System Operating in:
 7-Mode
 Cluster-Mode

NetApp Confidential 4

A TALE OF TWO PRODUCTS

In 1992, NetApp introduced the Data ONTAP operating system and ushered in the NAS industry. Since then,
NetApp has continued to add features and solutions to the NetApp product portfolio to meet the needs of
NetApp customers. In 2004, NetApp acquired Spinnaker Networks to fold Spinnaker’s scalable clustered file
system technology into the Data ONTAP operating system. In 2006, NetApp released the Data ONTAP GX
operating system, the first clustered product from NetApp. NetApp also continued to enhance and sell the
Data ONTAP 7G operating system.
Offering two products provided a way to meet the needs of the NetApp customers who were happy with the
classic Data ONTAP operating system while enabling customers with specific application requirements to use
the Data ONTAP GX operating system to achieve even higher levels of performance with the flexibility and
transparency that is afforded by the Data ONTAP GX scale-out architecture.
Although NetApp always planned to merge the two products, the migration path for Data ONTAP 7G
customers to get to clustered storage had a big gap. The Data ONTAP 8 operating system helps to bridge that
gap by enabling Data ONTAP 7G customers to run the Data ONTAP 8 operating system in Data ONTAP
operating in 7-Mode in the manner in which customers were accustomed while giving customers a first step
in the eventual move to a clustered environment. Data ONTAP 8 operating in Cluster-Mode enabled Data
ONTAP GX customers to upgrade and continue to operate their clusters as customers are accustomed.
The Data ONTAP 8.1.1 operating system is a major step toward merging the two product lines. The Data
ONTAP 8.1.1 operating system enhances the feature sets of both operating modes. These enhancements make
both predecessor product lines (the Data ONTAP 7G and GX operating systems) obsolete by providing two
modes that are complete supersets of the predecessors.

1-4 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Similarities Between 7-Mode and
Cluster-Mode 1 of 2
 The same controllers and disk shelves
 Unified storage: SAN and NAS
 High-availability (HA) pairs for failover
 RAID 4 and RAID-DP technology
 WAFL (Write Anywhere File Layout)
 Aggregates
 Flexible volumes
 In-place conversion of aggregates from 32-bit to 64-bit
 Snapshot copies
 Quotas (for users, groups, and qtrees)
 Asynchronous SnapMirror
NetApp Confidential 5

SIMILARITIES BETWEEN 7-MODE AND CLUSTER-MODE: 1 OF 2

1-5 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Similarities Between 7-Mode and
Cluster-Mode 2 of 2
 Flash Cache
 Nondisruptive upgrade (NDU)
 Multiprotocol support
– NFS, including NFSv4
– CIFS, including SMB 2.0
– Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE)
– iSCSI
 Storage-efficiency technology (FlexClone volumes,
deduplication, compression, and thin provisioning)
 SnapDrive for Windows and UNIX
 Flash Pools
 OnCommand System Manager
 OnCommand Unified Manager

NetApp Confidential 6

SIMILARITIES BETWEEN 7-MODE AND CLUSTER-MODE: 2 OF 2

1-6 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Some 7-Mode Features Not in
Cluster-Mode
 MetroCluster™
 SnapVault software
 SnapLock software
 Synchronous SnapMirror
 DataMotion for vFiler
 SyncMirror software
 Qtree SnapMirror relationships
 NetApp storage encryption disks
 Internet Protocol version 6 (IPv6)

NetApp Confidential 7

SOME 7-MODE FEATURES NOT IN CLUSTER-MODE

Data ONTAP 8.1 Cluster-Mode systems do not support everything that Data ONTAP 8.1 7-Mode systems
support. Cluster-Mode systems lack support for SnapVault software, MetroCluster, and SnapLock software.
Data ONTAP 7-Mode systems support both synchronous and asynchronous SnapMirror functionality, but
although Cluster-Mode systems have load-sharing mirrors in addition to data protection mirrors, they do not
yet support synchronous SnapMirror. Cluster-Mode systems also lack support for SyncMirror software, Qtree
SnapMirror, and IPv6.

1-7 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Some Cluster-Mode Features Not in
7-Mode 1 of 2
 Vservers
 Scale-out
– 24 nodes (NAS)
– 6 nodes (SAN)
 Namespace (used by NAS and not applicable
to SAN)
 Nondisruptive operation
 DataMotion for Volumes in Cluster-Mode (the
volume move command)

NetApp Confidential 8

SOME CLUSTER-MODE FEATURES NOT IN 7-MODE: 1 OF 2

Clustering carries with it the idea of many nodes that work together but are perceived externally as one
system.
The namespaces—one for each cluster (data) Vserver—are the external, client-facing representation of the
distributed storage. Junctions are the glue that holds the global namespaces together. Junctions are analogous
to symbolic links. Junctions connect volumes to create the namespace of a cluster (data) Vserver.
SAN is supported on clusters of up to 4 nodes as of release 8.1. At release 8.1.1 SAN support is expanded to
clusters of up to 6 nodes.
For nodes to work together as one, constant intracluster communication must occur over a dedicated cluster
interconnect. That cluster interconnect must be reliable.
Flexible volumes can be moved among aggregates and nodes. Moving a volume does not cause the volume’s
path in the namespace to change, nor is the process visible to a client. No NFS mountpoints or CIFS shares
must change. The volume is available for reading and writing during the process.

1-8 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Some Cluster-Mode Features Not in
7-Mode 2 of 2
 Logical interface (LIF) migration and failover
(for NAS)
 Cluster management as a single system
 Leasing (SMB 2.1)
 NFSv4.1, including parallel NFS (pNFS)
 Asynchronous volume-level replication for data
protection (intracluster and intercluster)
 Load-sharing SnapMirror functionality
 Infinite Volumes

NetApp Confidential 9

SOME CLUSTER-MODE FEATURES NOT IN 7-MODE: 2 OF 2

In Cluster-Mode systems, NAS data logical interfaces (LIFs) are not permanently tied to particular network
ports and nodes. As such, they can be migrated (manually or automatically) away from problematic hardware
or hardware that is being heavily taxed. SAN LIFS do not fail over or migrate.
Cluster-Mode systems use data protection and load-sharing mirrors; 7-Mode systems use only data protection
mirrors.

1-9 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 10

LESSON 2

1-10 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Primary Reasons to Use
a Cluster-Mode System
 Scalability: performance and capacity
 Flexibility: data management and movement
 Transparency: namespaces, storage failover,
NAS LIF failover and migration, resource use
and balancing, nondisruptive operation

NetApp Confidential 11

PRIMARY REASONS TO USE A CLUSTER-MODE SYSTEM

Cluster-Mode systems can scale to meet the needs of customers. Adding additional disk shelves to a single
system has always been a method of scaling capacity. The scalability of Cluster-Mode systems is greatly
multiplied by the ability to add nodes to a cluster without any disruption or downtime for the cluster. You can
also scale computing power by adding nodes to a cluster.
Cluster-Mode systems also provide incredible flexibility in how nodes and storage are managed.
Finally, the ability to move volumes among aggregates and nodes means that an administrator isn’t locked
into a particular data scheme. Changing a scheme is transparent and simple.

1-11 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Scalability

 A single system image for 2 to 24 nodes

 Throughput that scales linearly to multiple gigabytes per second
 Capacity that scales to petabytes
 The ability to support continuous operation
 Online (transparent) load balancing and scaling
 Robust high-performance computing (HPC) features
 A fully integrated, single-vendor solution

NetApp Confidential 12

SCALABILITY
Cluster-Mode solutions can scale from 2 to 24 nodes, and are mostly managed as one large system. More
importantly, to client systems a Cluster-Mode system looks like a single file system. The performance of the
Cluster-Mode system scales linearly to multiple gigabytes per second of throughput and capacity scales to
petabytes.
Cluster-Mode systems are built for continuous operation; no single failure on a port, disk, card, or
motherboard will cause data to become inaccessible in a system. Cluster-Mode scaling and load balancing are
both transparent.
Cluster-Mode systems provide a robust feature set, including data protection features such as Snapshot copies,
intracluster asynchronous mirroring, and NDMP backups.
Cluster-Mode is a fully integrated solution. This example shows a 20-node cluster that includes 10 FAS
systems with 6 disk shelves each, and 10 FAS systems with 5 disk shelves each. Each rack contains an HA
pair with storage failover (SFO) capabilities.

1-12 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Scalability: Performance (NAS)

R A Six-Node Data ONTAP Cluster

A B C F R’ A R’ B’
B B’ R
C G H E F D

D E G H

 Linearly scale read performance

with load-sharing mirror
relationships.
 Linearly scale aggregate
read/write performance in a
single namespace.

NetApp Confidential 13

SCALABILITY: PERFORMANCE (NAS)

In the example on this slide, volume R is the root volume of a Vserver and its corresponding namespace.
Volumes A, B, C, and F are mounted to R through junctions. This Cluster-Mode solution provides
performance scaling in two ways:
 Volume B and root Volume R each have two read-only mirror relationships, which enables read requests
to be spread across multiple volumes and nodes (the B′ mirrors) rather than taxing one volume (and node)
with all read requests. High-performance computing (HPC) applications often require scaling of read
throughput well beyond the write throughput.
 Volumes A, C, D, E, F, G, and H provide scaled performance when applications are accessing all six
nodes. Because the volumes are distributed across six nodes, processing is spread across those nodes, but
all of the nodes are in a single namespace. Volumes D and E are mounted to C through junctions.
Likewise, volumes G and H are mounted to F.

1-13 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Scalability: Capacity
 The ability to rapidly and
seamlessly deploy new
storage or applications or both
Projects
 No required downtime
 Movement that is transparent
A B C
to clients and does not alter
A1 A2 A3 B1 B2 C1 C2 C3
the namespace

A B C2 B2
A1 A2 C C3
A3
C1 B1

Offload (move) volumes to newly added storage.

Grow the volumes.

NetApp Confidential 14

SCALABILITY: CAPACITY
In the example on this slide, more capacity is needed for project B. Follow these steps to scale the capacity:
1. Add two nodes to make a 10-node cluster with additional disks.
2. Transparently move some volumes to the new storage.
3. Expand volume B in place.
This movement and expansion is transparent to client machines.

1-14 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Flexibility: The Virtual Storage Tier
Data-Driven Real-Time Self-Managing  Flash Cache
– Storage-level RAID-protected
The Virtual cache
Storage Tier
– PCI-e modules
– Capacities of up to 1 TB
 Flash Pool
– A RAID-protected aggregate
– A solid-state disk (SSD) tier
that is used as cache
– A hard disk tier that is used as
storage
Hard Disk Storage

NetApp Confidential 15

FLEXIBILITY: THE VIRTUAL STORAGE TIER

The NetApp Virtual Storage Tier provides fully automated use and optimization of Flash technology,
controller-based and based on Peripheral Component Interconnect Express (PCIe) and solid-state disks
(SSDs).
 NetApp Flash PCIe modules improve performance for workloads that are random-read-intensive, which
reduces latency by a factor of 10 or more compared to hard disks. Flash Cache modules are available in
capacities of up to 1 terabyte and provide controller-based caching.
 NetApp Flash Pool enables the caching of random read and write operations through the automated use of
SSDs, which enables the use of capacity-optimized hard disk technology across the majority of
application workloads. Flash Pool enables the creation of a Data ONTAP RAID-protected aggregate that
consists of a combination of hard disks and SSDs.

1-15 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Transparency: Load Optimization
 Optimized performance
 Maximized disk use
 Transparency to
Projects applications
A B C

A1 A2 A3 B1 B2 C1 C2 C3

A
B1 B A1 C2 C3
A3
C1 C B2 A2

Project A gets dedicated resources.

NetApp Confidential 16

TRANSPARENCY: LOAD OPTIMIZATION

In the example on this slide, project A needs more computer power than other projects need. With a Cluster-
Mode system, you can:
 Transparently move volumes that are not project-A volumes to free up processing power on other nodes
 Give project A dedicated resources
 Move volumes as needed when project A stops being critical or if another project becomes critical
Volume movement is transparent to client machines.

1-16 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Nondisruptive Operation

 DataMotion for Volumes

 NAS logical interface (LIF) migration
 Storage failover (SFO)

NetApp Confidential 17

NONDISRUPTIVE OPERATION
Nondisruptive operation is a key feature of Data ONTAP clustering. Three critical components of
nondisruptive operation include DataMotion for Volumes (volume move), logical interface (LIF) migration,
and storage failover (SFO).
 SFO is covered in Module 6: Virtual Data Storage
 NAS LIF Migration is covered in Module 8: Virtual Networking
 Volume move is covered in Module 13: Cluster Management

1-17 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Types of Vservers

 Node Vservers
– Represent each physical node
– Are associated with cluster LIFs, node
management LIFs, and intercluster LIFS
 Administrative Vserver
– Represents the physical cluster
– Is associated with the cluster management LIF
 Cluster (data) Vserver
– Is a virtual representation of a physical data
server
– Is associated with data LIFs
NetApp Confidential 18

TYPES OF VSERVERS
Unlike the Data ONTAP GX operating system, which had only one type of Vserver, a Data ONTAP system
operating in Cluster-Mode has three types of Vservers. Vservers like those used in GX are called “cluster
(data) Vservers.” The new types of Vservers are node Vservers, which simply represent nodes, and
administrative Vservers, which represent entire clusters.
Cluster (data) Vservers are also referred to as data Vservers because they are used to read and write data to
and from the cluster. Also, unless the documentation refers specifically to admin Vserver or node Vserver, the
term Vserver refers to cluster (data-serving) Vservers.

1-18 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Cluster Resources
Data
Network
cmg
mg1 lif7 lif8 lif15 mg1 lif16
lif4 lif5 lif6 HA lif12 lif13 lif14
lif1 lif2 lif3 lif9 lif10 lif11
Interconnect

cl1 cl2

cl1 cl2
Cluster
aggr1 aggr2 Interconnect aggr3 aggr4

n1aggr0 n2aggr0
n1vol0 n2vol0

Cluster (data) Vservers:

vserverA
vserverB
vserverC

NetApp Confidential 19

CLUSTER RESOURCES
The example on this slide shows many of the key resources in a cluster: three types of Vservers (node, cluster,
and administrative), plus nodes, aggregates, volumes, and data LIFs.

1-19 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Physical and Logical Elements

 Physical  Logical
– Nodes – Clusters
– Disks – Volumes
– Aggregates – Snapshot copies
– Network ports – Mirror relationships
– FC ports – Vservers
– Tape devices – LIFs

NetApp Confidential 20

PHYSICAL AND LOGICAL ELEMENTS

Physical elements of a system can be touched and seen, like nodes, disks, and ports on those nodes.
Logical elements of a system cannot be touched, but do exist and use disk space. Volumes, Snapshot copies,
and mirror relationships are areas of storage that are divided from aggregates. Clusters are groupings of
physical nodes. Vservers are virtual representations of resources or groups of resources. A LIF is an IP
address that is associated with a single network port.

1-20 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

vserver show (Summary View)
netappu::> vserver show
Admin Root Name Name
Vserver Type State Volume Aggregate Service Mapping
----------- ------- --------- ---------- ---------- ------- -------
netappu admin - - - - -
netappu-01 node - - - - -
netappu-02 node - - - - -
vs1 cluster running vs1 aggr1a file file

4 entries were displayed.

NetApp Confidential 21

VSERVER SHOW (SUMMARY VIEW)

Notice the types of Vservers. When a cluster is created, the administration Vserver is automatically created.
When a node is joined to the cluster, a node Vserver is automatically created to represent it. Administrators
create cluster (data) Vservers to build global namespaces.

1-21 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Cluster (Data) Vservers

 Are also referred to as data Vservers

 Are not necessarily associated with any single
node
 Contain most resources within their scope
– Namespace
– Volumes
– Data LIFs (for client access)
– Protocol “servers”: NFS, CIFS, FC, FCoE, and
iSCSI

NetApp Confidential 22

CLUSTER (DATA) VSERVERS

Cluster (data) Vservers are not necessarily associated with any node or group of nodes within the cluster, but
there may be circumstances where an administrator chooses to limit a Vserver’s volumes and data LIFS to
specific nodes.

1-22 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Node Vservers
Data
Network Node
Network Ports Management
mg1 mg1 LIFs
Cluster LIFs

lif1 lif2

lif1 lif2
Cluster
aggr1 aggr2 Interconnect aggr3 aggr4

Network
Ports

n1aggr0 n2aggr0
n1vol0
Aggregates
n2vol0

NetApp Confidential 23

NODE VSERVERS

1-23 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Administrative Vserver
Cluster
Management
LIF cmg

Administrative Vserver (Physical Cluster)

NetApp Confidential 24

ADMINISTRATIVE VSERVER

1-24 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Cluster (Data) Vservers

lif7 lif8 lif15 lif16

Data LIFs lif4 lif5 lif6 lif12 lif13 lif14
lif1 lif2 lif3 lif9 lif10 lif11

Volumes

Cluster (data) Vservers:

vserverA
vserverB
vserverC

NetApp Confidential 25

CLUSTER (DATA) VSERVERS

1-25 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Putting It All Together
Data
Network
cmg
mg1 lif7 lif8 lif15 mg1 lif16
lif4 lif5 lif6 HA lif12 lif13 lif14
lif1 lif2 lif3 lif9 lif10 lif11
Interconnect

Node Vserver Node Vserver

lif1 lif2

lif1 lif2
Cluster
aggr1 aggr2 Interconnect aggr3 aggr4

n1aggr0 n2aggr0
n1vol0 n2vol0

Cluster (data) Vservers:

vserverA
vserverB
vserverC

NetApp Confidential 26

PUTTING IT ALL TOGETHER

1-26 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Module Summary
Now that you have completed this module, you
should be able to:
 Identify the unique features of Data ONTAP
operating in Cluster-Mode relative to Data
ONTAP operating in 7-Mode
 Explain the three primary benefits of Data ONTAP
clustering
 Identify the scope of Data ONTAP concepts such
as node Vservers, administrative Vservers, and
cluster (data) Vservers
 Discuss the features and advantages of the
NetApp storage tier strategy
NetApp Confidential 27

MODULE SUMMARY

1-27 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Exercise
Module 1: Overview
Time Estimate: 10 Minutes

NetApp Confidential 28

EXERCISE
Please refer to your exercise guide.

1-28 Clustered Data ONTAP Administration: Overview

NetApp University - Do Not Distribute

Module 2
Installation and Configuration

NetApp Confidential 1

MODULE 2: INSTALLATION AND CONFIGURATION

2-1 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Configure nodes and install the Data ONTAP
8.1.1 operating system
 Assign and initialize disks
 Create a multinode cluster
 Add feature licenses to the cluster
 Set the dates, times, and time zones of the
nodes in the cluster

NetApp Confidential 2

MODULE OBJECTIVES

2-2 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Basic Steps for Setting Up a Cluster

1. Connect controllers, disks, and cables.

2. Set up and configure nodes.
3. Install software onto nodes.
4. Initialize disks.
5. Create a cluster.
6. Join additional nodes to the cluster.
7. Create aggregates and volumes.
8. Configure virtual servers (Vservers).

NetApp Confidential 3

BASIC STEPS FOR SETTING UP A CLUSTER

Data ONTAP operating in Cluster-Mode supports V-Series systems. As such, the setup is different when you
use V-Series systems.

2-3 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Hardware Setup

Connect:
 Controllers to disk shelves
 High-availability (HA) interconnect
 Controllers to the networks
 Any tape devices
 Controllers and disk shelves to power

NetApp Confidential 4

HARDWARE SETUP
 Connect controllers to disk shelves. Verify that shelf IDs are set properly.
 If required for your controller type, connect nonvolatile RAM (NVRAM) high-availability (HA) cable
between partners. The connections can be 10-GbE or InfiniBand, depending on your storage controllers.
 Connect controllers to the networks.
 If present, connect any tape devices. This task can be performed later.
 Connect controllers and disk shelves to power.

2-4 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Communication Connections

 Console connection (using ANSI-9600-8N1)

 Remote management device connection,
dependent on model
– Service Processor (SP)
– Remote LAN Module (RLM)
– Baseboard Management Controller (BMC)
 Management network connections
 Cluster network connections
 Data network connections

NetApp Confidential 5

COMMUNICATION CONNECTIONS
Each controller should have a console connection, which is required to get to the firmware and to get to the
Boot menu (for the setup, installation, and initialization options, for example). A remote management device
connection, although not required, is helpful in the event that you cannot get to the UI or console. Remote
management enables remote booting, the forcing of core dumps, and other actions.
Each node must have exactly two connections to the dedicated cluster network. Each node should have at
least one data connection, although these data connections are necessary only for client access. Because the
nodes are clustered together, it’s possible to have a node that participates in the cluster with its storage and
other resources but doesn’t field client requests. Typically, however, each node has data connections.
The cluster connections must be on a network that is dedicated to cluster traffic. The data and management
connections must be on a network that is distinct from the cluster network.

2-5 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Disk Cabling
node1 node2 node3 node4

NVRAM* Interconnect
FC or SAS (simplified)

NetApp Confidential 6

DISK CABLING
A large amount of cabling must be done with a Data ONTAP 8.1.1 cluster. Each node has NVRAM
interconnections to its HA partner. Each node has FC or SAS connections to its disk shelves and to those of
its HA partner.
In a multipath high availability (MPHA) cabling strategy, each storage controller has multiple ways to
connect to a disk. An I/O module (IOM) failure does not require a controller failover. This method is the most
resilient and preferred method of shelf cabling.
Ethernet cabling for alternate control path (ACP) requires one connection to each controller, connected in a
series through all shelves. First you connect stack to stack. Then you connect between IOMs from top to
bottom in each stack.

2-6 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Network Cabling
1 of 2
Cluster Interconnect
Cluster Interconnect

Management

Data

Management Data Network

Network

NOTE: NetApp recommends switch redundancy for both data

and management networks.
NetApp Confidential 7

NETWORK CABLING: 1 OF 2
For customers with strict security requirements, management ports can be connected to a network that is
separate from the data network. In that case, management ports must have a role of management, and network
failover cannot occur between data and management interfaces.

2-7 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Network Cabling
2 of 2
Cluster Interconnect
Cluster Interconnect

Management

Data

Data and
Management
Network

NOTE: NetApp recommends switch redundancy for both data

and management networks.
NetApp Confidential 8

NETWORK CABLING: 2 OF 2
When you cable the network connections, consider the following:
 Each node is connected to at least two distinct networks: one for management (the UI) and data access
(clients) and one for intracluster communication. NetApp supports two 10-GbE cluster connections to
each node to create redundancy and improve cluster traffic flow.
 The cluster can be created without data network connections but not without cluster network connections.
 Having more than one data network connection to each node creates redundancy and improves client
traffic flow.

2-8 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Powering On a Node and Cluster

1. Power on network switches.

2. Power on disk shelves.
3. Power on tape devices (if present).
4. Power on storage controllers.

NetApp Confidential 9

POWERING ON A NODE AND CLUSTER

2-9 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Firmware
 Use LOADER firmware.
 Press any key to enter the firmware.
 Use version to show the firmware version.
 Two boot device images exist: flash0a and
flash0b.
 Use printenv to show the firmware
environment variables.
 Use setenv to set the firmware environment
variables, for example, setenv AUTOBOOT true

NetApp Confidential 10

FIRMWARE
1. Use LOADER firmware.
2. From the console, early in the booting process, press any key to enter the firmware.
3. Use version to show the firmware version.
4. Two boot device images exist (depending on platform): flash0a and flash0b.
– CompactFlash
– USB flash
5. Use printenv to show the firmware environment variables.
6. Use setenv to set the firmware environment variables, for example, setenv AUTOBOOT true
To copy flash0a to flash0b, run flash flash0a flash0b. To “flash” (put) a new image onto the primary flash,
you must first configure the management interface. The auto option of ifconfig can be used if the management
network has a Dynamic Host Configuration Protocol (DHCP) or BOOTP server. If it doesn’t, you must run
ifconfig <interface> addr=<ip> mask=<netmask> gw=<gateway>. After the network is configured, ensure
that you can ping the IP address of the TFTP server that contains the new flash image. To then flash the new
image, run flash t[Link] flash0a.
The environment variables for Cluster-Mode can be set as follows:
 set-defaults
 setenv ONTAP_NG true
 setenv [Link] false
 setenv [Link].boot_clustered true

2-10 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Setup Procedure

From the Boot menu:

1. If necessary, run option 7 (“Install new
software first”).*
2. Run option 4 (“Clean configuration and
initialize all disks”).*
3. Run the cluster setup wizard.

* Steps 1 and 2 are usually done by NetApp before

hardware arrives at a customer site.

NetApp Confidential 11

THE SETUP PROCEDURE

2-11 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Boot Menu
Please choose one of the following:

1. Normal Boot.
2. Boot without /etc/rc (no effect in Clustered ONTAP).
3. Change password.
4. Clean configuration and initialize all disks.
5. Maintenance mode boot.
6. Update flash from backup config.
7. Install new software first.
8. Reboot node.
Selection (1-8)?

NetApp Confidential 12

THE BOOT MENU

2-12 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Installing the Data ONTAP 8.1 Operating
System on a Node
 You need:
– Access to an FTP, TFTP, or HTTP server
– The software image file on that server
 From the Boot menu, complete the following:
1. Select option 7.
2. When prompted, enter a URL to a Data
ONTAP 8.1 tgz image.
3. When complete, allow the system to boot.

NetApp Confidential 13

INSTALLING THE DATA ONTAP 8.1 OPERATING SYSTEM ON A NODE

After you boot the system you boot the system, if the node stops at the firmware prompt by itself (which
happens if the firmware environment variable AUTOBOOT is set to false), type boot_primary to enable
the node to continue to the boot menu. If AUTOBOOT is set to true, the node goes straight to the boot menu.
If you use TFTP, beware of older TFTP servers that have limited capabilities and might cause installation
failures.

2-13 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Initializing a Node

From the Boot menu, select option 4:

 Initialization clears the three disks that the
system uses for the first aggregate that it
creates.
NOTE: This action requires time, depending
on disk size.
 Initialization creates one aggregate (for this
node) and a vol0 root volume on the
aggregate.
 Initialization must be run on both nodes of
each HA pair.
NetApp Confidential 14

INITIALIZING A NODE
Because all disks are initialized parallel to each other, the time that is required to initialize the disks is based
on the size of the largest disk that is attached to the node, not on the sum capacity of the disks. After the disks
are initialized, the node’s first aggregate and its vol0 volume are automatically created.

2-14 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Cluster Setup Wizard
1 of 3
 From the Boot menu of an initialized controller:
1. Boot normally.
2. Log in as “admin” with no password.
3. Follow the prompts.
 You can also run cluster setup from the CLI.

NetApp Confidential 15

THE CLUSTER SETUP WIZARD: 1 OF 3

An initialized storage controller automatically boots to the cluster setup wizard. You can type exit to stop
the wizard and stay at the command prompt. The node is not yet part of a cluster. You can restart the wizard
from the CLI by typing cluster setup.

2-15 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Cluster Setup Wizard
2 of 3
 The first node creates the cluster.
 You need the:
– Cluster name
– Cluster network ports and MTU size (usually best to use
default MTU)
– Cluster base license key
– Cluster management interface port, IP address, network
mask, and default gateway
– Node management interface port, IP address, network
mask, and default gateway
– Domain Name System (DNS) domain name
– IP addresses of the DNS server

NetApp Confidential 16

THE CLUSTER SETUP WIZARD: 2 OF 3

To complete the cluster setup wizard, you need the following information for the first node in the cluster:
 The cluster name. The wizard automatically names the node clustername-0x, where x is the order in
which the node joins the cluster. The node that creates the cluster is 01; the next node to join the cluster is
02; and so on.
 Cluster network ports. On a private, nonroutable cluster network, the wizard automatically assigns cluster
IP addresses.
 The cluster base license key. Additional license keys can be added here.
 The cluster management interface port, IP address, network mask, default gateway, and home port
 The node management interface port, IP address, network mask, default gateway, and home port
 The Domain Name System (DNS) domain name
 IP addresses of the DNS server

2-16 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Cluster Setup Wizard
3 of 3
 Subsequent nodes join the cluster.
 You need the:
– Cluster network ports and MTU size
– Node management interface port, IP address,
network mask, and default gateway

NetApp Confidential 17

THE CLUSTER SETUP WIZARD: 3 OF 3

To complete the cluster setup wizard, you need the following information for subsequent nodes:
 Cluster network ports. On a private, nonroutable cluster network, the wizard automatically assigns cluster
IP addresses.
 The node management interface port, IP address, network mask, default gateway, and home port.

2-17 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

The Normal Boot Sequence
1. The firmware loads the kernel from boot device.
2. The kernel mounts the “/” root image from [Link] on
boot device.
3. Init is loaded, and startup scripts run.
4. NVRAM kernel modules are loaded.
5. The /var partition on NVRAM is created and mounted
(restored from boot device if a backup copy exists).
6. The management gateway daemon (mgwd) is started.
7. The D-blade, the N-blade, and other components are
loaded.
8. The vol0 root volume is mounted from the local D-blade.
9. The CLI is ready for use.

NetApp Confidential 18

THE NORMAL BOOT SEQUENCE

2-18 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Licenses
 “Base” enables clustering (must be applied first).
 “NFS” enables NFS (NAS).
 “CIFS” enables CIFS (NAS).
 “iSCSI” enables iSCSI (SAN).
 “FCP” enables FC (SAN).
 “SnapMirror_DP” enables Data Protection mirroring.
 “SnapRestore” enables volume level Snapshot
restore.
 “FlexClone” enables FlexClone copies.
 “SnapManager” enables SnapManager.

NetApp Confidential 19

LICENSES
NOTE: Load-sharing mirroring requires no license. CompactFlash is no longer needed for storage failover.
Contact NetApp for a hash key to enable Infinite Volumes.

2-19 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Adding More Licenses
 In the clustershell: system license add
 In OnCommand System Manager, select
cluster > configuration > system tools >
licenses.

NetApp Confidential 20

ADDING MORE LICENSES

2-20 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Network Time Protocol

 Manually set the date, time, and time zone

with system date modify.
 Kerberos is time-sensitive and typically
requires the Network Time Protocol (NTP).
 NTP is disabled by default.
 NTP enablement and disablement are cluster-
wide
 The commands for verifying and monitoring
NTP are:
– system services ntp config show
– system services ntp server show
NetApp Confidential 21

NETWORK TIME PROTOCOL

2-21 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Configure nodes and install the Data ONTAP
8.1 operating system
 Assign and initialize disks
 Create a multinode cluster
 Add feature licenses to the cluster
 Set the dates, times, and time zones of the
nodes in the cluster

NetApp Confidential 22

MODULE SUMMARY

2-22 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Exercise
Module 2:
Installation and Configuration
Time Estimate: 30 minutes

NetApp Confidential 23

EXERCISE
Please refer to your exercise guide.

2-23 Clustered Data ONTAP Administration: Installation and Configuration

NetApp University - Do Not Distribute

Module 3
Architecture

NetApp Confidential 1

MODULE 3: ARCHITECTURE

3-1 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Show the end-to-end path of a file write
request through a cluster
 Answer questions about replicated database
(RDB) concepts
 Identify the differences between a vol0 root
volume and a cluster virtual server (Vserver)
root volume

NetApp Confidential 2

MODULE OBJECTIVES

3-2 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

3-3 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

A 14-Node Mixed Cluster

 FAS and V-Series 3x00 and 6x00 systems

can exist in the same cluster.
 NFS and CIFS clients see a single file system.
 Client requests are redirected from one node
to another as necessary.

NetApp Confidential 4

A 14-NODE MIXED CLUSTER

SAN and NAS clients do not see a cluster; they see a cluster (data) virtual server (Vserver) that is capable of
using all the equipment in this cluster. Clients have no way of knowing how many nodes exist in this cluster.
Only the administrator can see how many nodes exist in this cluster.
A client request to any controller is redirected to the controllers that own the requested data.

3-4 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Components

 Four major software components on every

node:
– The N-blade (the network “blade”)
– The D-blade (the data “blade”)
– The SCSI-blade (the SAN “blade”)
– Management (the “M-host”)
 Other key software components on every
node:
– The cluster session manager (CSM)
– RDB units, for example, the volume location
database, or (VLDB)
NetApp Confidential 5

COMPONENTS
The term “blade” refers to separate software state machines that are accessed only by well-defined APIs.
Every node contains an N-blade, a D-blade, a SCSI-blade, and Management (M-host). Any N-blade in the
cluster can talk to any D-blade in the cluster.
The N-blade and the SCSI-blade translate client requests into Spin Network Protocol (SpinNP) requests and
vice versa. The D-blade, which contains the WAFL (Write Anywhere File Layout) file system, manages
SpinNP requests. The cluster session manager (CSM) is the SpinNP layer between the N-blade, the SCSI-
blade, and the D-blade. The SpinNP protocol is another form of RPC interface. It is used as the primary
intranode traffic mechanism for file operations among N-blades and D-blades.
The members of each replicated database (RDB) unit on every node in the cluster are in constant
communication with each other to remain synchronized. The RDB communication is like the heartbeat of
each node. If the heartbeat cannot be detected by the other members of the unit, the unit corrects itself in a
manner that is discussed later in this course. The four RDB units on each node are the Blocks Configuration
and Operations Manager (BCOM), the volume location database (VLDB), VifMgr, and Management. More
information about these RDB units is given later in this course.

3-5 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Data ONTAP 8.1 Cluster-Mode Blades
NAS

NAS
(N-blade)
WAFL  Network blade:
RAID
SAN
(SCSI-blade)
(D-blade)
N
a – “N-blade”
– Provides NAS protocols
V
R

Cluster Interconnect
A
NAS
(N-blade)
M
WAFL
root
 SCSI blade:
RAID
SAN
(SCSI-blade)
(D-blade) – “SCSI-blade”
– Provides SAN protocols
 Data blade:
NAS
WAFL
(N-blade)
RAID
SAN
– “D-blade”
(D-blade) b
(SCSI-blade) N
V
R
A
– Provides storage
access to shelves
NAS M c
WAFL
(N-blade)
RAID
SAN
(D-blade)
(SCSI-blade)

NetApp Confidential 6

DATA ONTAP 8.1 CLUSTER-MODE BLADES

3-6 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Single Node Components (Illustrated)
Node

Client Access (Data) N-Blade and

SCSI-Blade

Management M-Host

CSM Cluster Traffic

D-Blade
RDB Units:
 Management
 VLDB
Cluster (data)
 VifMgr
Vserver
 BCOM
Root Volume Vol0
Root
Vol1
Vol2

NetApp Confidential 7

SINGLE NODE COMPONENTS (ILLUSTRATED)

This graphic is simplified but shows that each node contains the following: an N-blade, an SCSI-blade, a
CSM, a D-blade, an M-host, RDB units (four), and the node’s vol0 volume.

3-7 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The N-Blade

 Manages networking, NFS, and CIFS

 Speaks:
– TCP/IP and UDP/IP
– NFS and CIFS
– SpinNP

NetApp Confidential 8

THE N-BLADE

3-8 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The SCSI-Blade

 Manages networking, FC, Fibre Channel over

Ethernet (FCoE), and iSCSI
 Speaks:
– FC
– SCSI
– SpinNP
– TCP/IP

NetApp Confidential 9

THE SCSI-BLADE

3-9 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The CSM

 Provides a communication mechanism

between any N-blade or SCSI-blade and any
D-blade
 Provides a reliable transport for SpinNP traffic
over UDP/IP
 Is used regardless of whether the N-blade or
the SCSI-blade and the D-blade are on the
same node or on different nodes

NetApp Confidential 10

THE CSM

3-10 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The D-Blade

 Manages the WAFL (Write Anywhere File

Layout) file system, RAID, and storage
 Speaks:
– SpinNP
– FC and SAS to disk and tape devices

NetApp Confidential 11

THE D-BLADE

3-11 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The Path of a (Local) Write Request
Node1 Node2

Requests N-Blade and NBlade and

Responses SCSI-Blade SCSI-Blade

NAS and SAN

Clients

CSM CSM

D-Blade D-Blade

Vol0 Vol0
Root
Root Vol3
Vol1
Vol 1 Vol4
Vol2

NetApp Confidential 12

THE PATH OF A (LOCAL) WRITE REQUEST

A NAS or SAN client sends a write request to a data logical interface (LIF). The N-blade (NAS) or SCSI-
blade (SAN) that is currently associated with that LIF translates the NFS or CIFS (NAS), FC, FCoE, or iSCSI
(SAN) request to a SpinNP request. The SpinNP request goes through the CSM to the local D-blade. The D-
blade sends the data to nonvolatile RAM (NVRAM) and to the disks. The response works its way back to the
client.

3-12 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The Path of a (Remote) Write Request
Node1 Node2

Requests N-Blade and NBlade and

Responses SCSI-Blade SCSI-Blade

NAS and SAN

Clients

CSM CSM

D-Blade D-Blade

Vol0 Vol0
Root
Root Vol3
Vol1
Vol 1 Vol4
Vol2

NetApp Confidential 13

THE PATH OF A (REMOTE) WRITE REQUEST

3-13 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Data ONTAP Architecture

CSM
Cluster Traffic
N-Blade and D-Blade
SCSI-Blade

Network Protocols WAFL RAID Storage

Clients
To HA partner
NVRAM
Physical
Memory

Free BSD
M-Host
NetApp Confidential 14

DATA ONTAP ARCHITECTURE

3-14 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Kernel Modules and User-Space
Processes
 Data ONTAP architecture includes a modified
FreeBSD.
 Data ONTAP operating in Cluster-Mode
consists of multiple components, some of
which are kernel modules and some of which
are user-space processes.
 Kernel modules are loaded during the boot
process, and then user-space processes are
started.
 All kernel modules and all user-space
processes run on every node.
NetApp Confidential 15

KERNEL MODULES AND USER-SPACE PROCESSES

Kernel modules are loaded into the FreeBSD kernel. This gives them special privileges that are not available
to user-space processes. Great advantages to being in the kernel exist; downsides exist, too. For one, it’s more
difficult to write kernel code and the penalty for a coding error is great. User-space processes can be swapped
out by the OS but on the plus side, user-space processes can fail without taking the whole system down and
can be easily restarted dynamically.

3-15 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Kernel Modules
 Common: common_kmod.ko
 NVRAM5: [Link]
 The NVRAM manager: nvram_mgr.ko
 The NVRAM character-device driver: [Link]
 The D-blade: [Link]
 The N-blade: [Link]
 The SCSI-blade: scsi_blade.ko
 SpinVFS: [Link]

NetApp Confidential 16

KERNEL MODULES
All nodes in a cluster have these kernel modules:
 common_kmod.ko: This module is the first kernel module to load. This module contains common
services, which are shared by modules that load after it.
 [Link]: This module is a low-level hardware driver for NVRAM5.
 nvram_mgr.ko: This module segments NVRAM for multiple users, provides common access functions
that NVRAM5 doesn’t provide, and provides centralized power management.
 [Link]: This character-device driver interfaces with individual regions of NVRAM, for example, /var.
 [Link]: This module is the D-blade, a modified Data ONTAP 7G operating system, which includes
the WAFL file system, RAID, storage components, and the SpinNP translation layers.
 [Link]: The N-blade contains the network stack, protocols, and SpinNP translation layers.
 scsi_blade: The SCSI-blade contains the SAN network stack, protocols, and SpinNP translation
layers.
 [Link]: SpinVFS enables the user-space components to access volumes in the cluster.

3-16 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

User-Space Processes
 The management gateway daemon: mgwd
 The volume location database: vldb
 The virtual interface manager: vifmgr
 The block data access configuration: bcomd
 The cluster shell: ngsh
 The NDMP daemon: ndmpd
 NFS and CIFS authentication: secd
 The service process manager daemon: spmd

NetApp Confidential 17

USER-SPACE PROCESSES
All nodes in a cluster have these user-space processes:
 mgwd: This process runs the M-host (the Management component) on each node. The mgwd process of
each node talks to the mgwd processes on the other nodes.
 vldb: This process manages the volume location database and provides the mappings from volume and
container identifiers that are presented in file handles to those containers (or replicas of them) on a
specific D-blade.
 vifmgr: This process manages virtual interfaces, including their ability to migrate, fail over, and revert.
 bcomd: This process manages the configuration of SAN (block) data access.
 ngsh: This process manages the cluster shell on each node.
 ndmpd: This process services all NDMP calls from third-party data-management applications.
 secd: This process manages protocol authentication from NFS and CIFS clients.
 spmd: This process controls the starting, stopping, and restarting of the other processes. If a process gets
hung, the process is killed and restarted by the spmd.

3-17 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The Vol0 Volume

 Contains data for managing the node and

cluster:
– Is used for RDB databases and log files
– Doesn’t contain user or client data
 Cannot be accessed by NAS or SAN clients
 Exists (one) on every node
 Must not be confused with the root volume of
a cluster (data) Vserver
 Cannot be mirrored, moved, or backed up
 Can be re-created after a disaster
NetApp Confidential 18

THE VOL0 VOLUME

The vol0 volume of a node is analogous to the root volume of a Data ONTAP 7G operating system. The vol0
volume contains the data that is needed for the node to function.
The vol0 volume does not contain any user data, nor is it part of the namespace of a Vserver. The vol0
volume lives permanently on the initial aggregate that is created when each node is initialized.
The vol0 volume is not protected by mirror relationships or tape backups, which is valid. Although vol0 is an
important volume (a node cannot boot without its vol0 volume), the data that is contained on vol0 is largely
re-creatable. If the data is lost, the log files are indeed gone. But because the RDB data is replicated on every
node in the cluster, that data can be automatically re-created onto this node.

3-18 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Cluster (data) Vservers
1 of 2
 Are also known as Data Vservers
 Are not the same as MultiStore software
(vFiler units)
 Are virtual entities within a cluster
 Can coexist with other cluster (data) Vservers
in the same cluster
 Are independent of nodes
 Are independent of aggregates
 “Contain” all the volumes of their namespaces

NetApp Confidential 19

CLUSTER (DATA) VSERVERS: 1 OF 2

Think of a cluster as a group of hardware elements (nodes, disk shelves, and more). A Vserver is a logical
piece of that cluster, but a Vserver is not a subset or partitioning of the nodes. A Vserver is more flexible and
dynamic. Every Vserver can use all the hardware in the cluster and all at the same time.
Example: A storage provider has one cluster and two customers: ABC Company and XYZ Company. A
Vserver can be created for each company. The attributes that are related to specific Vservers (volumes, LIFs,
mirror relationships, and others) can be managed separately, while the same hardware resources can be used
for both. One company can have its own NFS server, while the other can have its own NFS, CIFS, and iSCSI
servers.

3-19 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Cluster (data) Vservers
2 of 2
 Represent unique namespaces
 Can and should have multiple data logical
interfaces (LIFs), each of which is associated
with one Vserver
 Can and do have multiple volumes, each of
which is associated with one Vserver

NetApp Confidential 20

CLUSTER (DATA) VSERVERS: 2 OF 2

A one-to-many relationship exists between a Vserver and its volumes. The same is true for a Vserver and its
data LIFs. Cluster (data) Vservers can have many volumes and many data LIFs, but those volumes and LIFs
are associated only with this one cluster (data) Vserver.

3-20 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Volumes, Junctions, and Namespaces
R A Data ONTAP Cluster

A B C F

D E G H
A
B R
C G H E F D

 R is the root of a cluster (data) Vserver.

 A, B, C, and F are mounted to R through
junctions.
 D and E are mounted to C through junctions.
 G and H are mounted to F through junctions.
NetApp Confidential 21

VOLUMES, JUNCTIONS, AND NAMESPACES

These nine volumes are mounted together through junctions. All volumes must have a junction path
(mountpoint) to be accessible within the Vserver’s namespace.
Volume R is the root volume of a Vserver. Volumes A, B, C, and F are mounted to R through junctions.
Volumes D and E are mounted to C through junctions. Likewise, volumes G and H are mounted to F.
Every Vserver has its own root volume, and all nonroot volumes are created within a Vserver. All nonroot
volumes are mounted into the namespace, relative to the Vserver root.

3-21 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Vservers, Namespaces, and Volumes

PopCo PetCo RonCo QuekCo

Namespace Namespace Namespace Namespace

Vserver Vserver Vserver Vserver
Root Root Root Root

Volume Volume
Volume
Volume
Volume
Volume

NetApp Confidential 22

VSERVERS, NAMESPACES, AND VOLUMES

NOTE: This slide is a representation of logical concepts and is not meant to show any physical relationships.
For example, all of the objects that are shown as part of a Vserver are not necessarily on the same physical
node of the cluster. In fact, that situation is unlikely.
This slide shows four distinct Vservers and namespaces. Although the hardware is not shown, these four
Vservers might be living within a single cluster. These namespaces are not separate entities of the Vservers
but are shown merely to indicate that each Vserver has a namespace. The volumes, however, are separate
entities. Each volume is associated with exactly one Vserver. Each Vserver has one root volume, and some
Vservers have additional volumes. Although a Vserver might have only one volume (the Vserver’s root
volume), in real life, it is more likely that a Vserver consists of multiple volumes, possibly thousands.
Typically, a new volume is created for every distinct area of storage. For example, every department and
employee might have its own volume in a Vserver.

3-22 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Namespaces

 A namespace is the file system of a cluster

(data) Vserver.
 A namespace consists of many volumes.
 A namespace is independent of the
namespaces of other cluster (data) Vservers.
 The “root” of the namespace is the cluster
(data) Vserver root volume.
 A client mount or mapping can be to the
cluster (data) Vserver root volume or to a
point further into the “tree.”

NetApp Confidential 23

NAMESPACES
A namespace is a file system. A namespace is the external, client-facing representation of a Vserver. A
namespace consists of volumes that are joined together through junctions. Each Vserver has exactly one
namespace, and the volumes in one Vserver cannot be seen by clients that are accessing the namespace of
another Vserver.

3-23 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The Cluster (data) Vserver Root Volume

 Exists (one) on each cluster (data) Vserver

 Is the root of the cluster (data) Vserver
namespace
 Is a normal flexible volume
 Can and does contain junctions
 Can be moved, copied, and backed up
 Can have Snapshot copies
 Is normally mirrored

NetApp Confidential 24

THE CLUSTER (DATA) VSERVER ROOT VOLUME

Each Vserver has one namespace and, therefore, one root volume. This volume is separate from the vol0
volume of each node.

3-24 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 25

LESSON 2

3-25 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The RDB

 The RDB is the key to maintaining high-

performance consistency in a distributed
environment.
 The RDB maintains data that supports the
cluster, not the user data in the namespace.
 Operations are transactional (atomic): entire
transactions are either committed or rolled
back.
 Four RDB units exist: the volume location
database, Management, VifMgr, and BCOM.

NetApp Confidential 26

THE RDB
The RDB units do not contain user data. The RDB units contain data that helps to manage the cluster. These
databases are replicated, that is, each node has its own “copy” of the database, and that database is always
synchronized with the databases on the other nodes in the cluster. RDB database reads are performed locally
on each node, but an RDB write is performed to one “master” RDB database, and then those changes are
replicated to the other databases throughout the cluster. When reads of an RDB database are performed, those
reads can be fulfilled locally without the need to send requests over the cluster interconnects.
The RDB is transactional in that the RDB guarantees that when data is written to a database, either it all gets
written successfully or it all gets rolled back. No partial or inconsistent database writes are committed.
Four RDB units (the volume location database, Management, VifMgr, and BCOM) exist in every cluster,
which means that four RDB unit databases exist on every node in the cluster.

3-26 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Management

 Is also known as the M-host

 Enables management of the cluster from any
node
 Provides the CLI and GUI (Element Manager)
 Runs as mgwd (the management gateway
daemon) on every node
 Stores its data in the management RDB unit
 Provides BCOM functionality for SAN support

NetApp Confidential 27

MANAGEMENT

3-27 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The VLDB

 The VLDB is:

– One of the RDB units
– An index of which D-blade owns a volume
– An index of which D-blade serves an aggregate
 VLDB content is cached in memory on each
node for instant access by each N-blade and
SCSI-blade to speed up the lookup process
during data access by clients.

NetApp Confidential 28

THE VLDB
Although each RDB unit consists of a process and a database on each node in the cluster, an RDB unit is
considered a single entity. One of the RDB units is the VLDB.
The VLDB tracks where the volumes and aggregates are.
Because the VLDB is potentially referenced (read) frequently for client requests, the VLDB content is cached
in memory on each node so that the N-blade and the SCSI-blade can avoid RDB lookups during client
requests.

3-28 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Other RDB Units

 VifMgr: LIF configuration and failover policies

 BCOM: LUN maps and initiator groups
(igroups)

NetApp Confidential 29

OTHER RDB UNITS

The Management RDB unit contains information that is needed by the mgwd process on each node. The kind
of management data that is stored in the RDB is written infrequently and read frequently. The management
process on a given node can query the other nodes at run time to retrieve a great deal of information. But
some information exists for which it makes sense to store the information locally on each node, in the
Management RDB database.
The VifMgr is responsible for creating and monitoring (NFS, CIFS, and iSCSI) LIFs and for handling NAS
LIF failover (automatic) and migration (manual) of the NAS LIFs to other network ports and nodes.
The BCOM RDB unit hosts the SAN ring that contains the replicated configuration information data for block
data access (including LUN maps and initiator groups, or igroups).

3-29 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The RDB: Details
1 of 2
 Each RDB unit has its own replication “ring.”
 For each of the units, one node is the “master”
and the other nodes are “secondaries.”
 The master node for each unit might be
different than the other units.
 Writes for an RDB unit go to its master and
then get propagated to the secondaries
through the cluster interconnect.

NetApp Confidential 30

THE RDB: DETAILS: 1 OF 2

Each RDB unit has its own ring. If n is the number of nodes in the cluster, each unit or ring consists of n
databases and n processes. At any given time, one of those databases is designated as the master, and the
others are designated as secondary databases. Each RDB unit’s ring is independent of the other RDB units. If
node X has the master database for the VLDB unit, node Y might have the master for the VifMgr unit, and
node Z might have the master for the Management unit and the BCOM unit.
The master of a given unit can change. For example, when the node that is the master for the Management
unit gets booted, a new Management master must be elected by the remaining members of the Management
unit. Note that a secondary can become a master and vice versa. Nothing is special about the database itself
but rather the role of the process that manages the database (master versus secondary).
When data must be written to a unit, the data is written to the database on the master, and then the master
immediately replicates the changes to the secondary databases on the other nodes. If a change cannot be
replicated to a specific secondary, the entire change is rolled back everywhere, which is what “no partial
writes” means. Either all databases of an RDB unit get the change, or none gets the change.

3-30 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The RDB: Details
2 of 2
 An RDB unit is considered to be healthy only
when it is “in quorum” (when a master can be
elected).
 “In quorum” means that a simple majority of
nodes are communicating with each other.
 When the quorum is lost or regained, the master
might change.
 If a master has communication issues, new
masters are “elected” by the members of the unit.
 One node has a tie-breaking ability (epsilon) for
all RDB units.

NetApp Confidential 31

THE RDB: DETAILS: 2 OF 2

RDB Terminology and Definitions
A master can be elected only when a quorum of members is available (and healthy) for a particular RDB unit.
Each member votes for the node that it thinks should be the master for this RDB unit. One node in the cluster
has a special tie-breaking ability called “epsilon.” Unlike the master, which might be different for each RDB
unit, epsilon is a single node that applies to all RDB units.
Quorum means that a simple majority of nodes are healthy enough to elect a master for the unit. The epsilon
power is used only in the case of a voting tie. If a simple majority does not exist, the epsilon node (process)
chooses the master for a given RDB unit.
When cluster communication is interrupted, for example, because of a booting or a cluster interconnect hiccup
that lasts for a few seconds, a unit goes out of quorum. When the cluster communication is restored, the unit
comes back into quorum automatically.

3-31 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

RDB Databases

node1

node2 mgwd VLDB VifMgr BCOM node3

mgwd VLDB VifMgr BCOM mgwd VLDB VifMgr BCOM

node4

mgwd VLDB VifMgr BCOM

NetApp Confidential 32

RDB DATABASES
This slide shows a four-node cluster. The four databases that are shown for each node are the four RDB units
(Management, VLDB, VifMgr, and BCOM). Each unit consists of four distributed databases. Each node has
one local database for each RDB unit.
The databases that are shown on this slide with dark borders are the masters. Note that the master of any
particular RDB unit is independent of the master of the other RDB units.
The node that is shown on this slide with a dark border has epsilon (the tie-breaking ability).
On each node, all the RDB databases are stored in the vol0 volume.

3-32 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Quorum
1 of 2
 A quorum is a simple majority of connected, healthy, and
eligible nodes.
 Two RDB quorum concepts exist: a cluster-wide quorum
and an individual RDB unit that is in or out of quorum.
 RDB units never go out of quorum as a whole; only local
units (processes) do.
 When an RDB unit goes out of quorum, reads from the
RDB unit can still occur, but changes to the RDB unit
cannot.
 Example: If the VLDB goes out of quorum, during the brief
time that the database is out, no volumes can be created,
deleted, or moved; however, access to the volumes from
clients is not affected.

NetApp Confidential 33

QUORUM: 1 OF 2
A master can be elected only when a majority of local RDB units are connected and healthy for a particular
RDB unit on an “eligible” node. A master is elected when each local unit agrees on the first reachable healthy
node in the RDB site list. A “healthy” node is one that is connected, can communicate with the other nodes,
has CPU cycles, and has reasonable I/O.
The master of a given unit can change. For example, when the node that is the master for the Management
unit is booted, a new Management master must be elected by the remaining members of the Management unit.
A local unit goes out of quorum when cluster communication is interrupted for a few seconds, for example,
because of a booting or a cluster interconnect hiccup that lasts for a few seconds. Because the RDB units
always work to monitor and maintain a good state, the local unit comes back in quorum automatically. When
a local unit goes out of quorum and then comes back into quorum, the RDB unit is synchronized again. Note
that the VLDB process on a node might go out of quorum although the VifMgr process on that same node has
no problem.
When a unit goes out of quorum, reads from that unit can be performed, but writes to that unit cannot. That
restriction is enforced so that no changes to that unit happen during the time that a master is not agreed upon.
Besides the example above, if the VifMgr goes out of quorum, access to LIFs is not affected, but no LIF
failover can occur.

3-33 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Quorum
2 of 2
 The members of each RDB unit vote to
determine which node will be their master;
each unit elects its own master.
 Each master might change when a local unit
goes out of and into quorum.
 Before taking a node down for an extended
period of time, it should be marked as
ineligible (so the node doesn’t factor into
quorum):
netappu::> system node modify –node
<node> -eligibility false

NetApp Confidential 34

QUORUM: 2 OF 2
Marking a node as ineligible (by way of the cluster modify command) means that the node no longer
affects RDB quorum or voting. If the epsilon node is marked as ineligible, epsilon is automatically given to
another node.

3-34 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

The Epsilon Node

 When half of the nodes in a cluster are

isolated from the other half, no simple majority
exists.
NOTE: This situation is rare.
 One node has a weighted vote (epsilon).
 The epsilon node is epsilon for the entire
cluster, not only for individual RDB units (such
as the masters).

NetApp Confidential 35

THE EPSILON NODE

One node in the cluster has a special voting weight called epsilon. Unlike the masters of each RDB unit,
which might be different for each unit, the epsilon node is the same for all RDB units. This epsilon vote is
used only in the case of an even partitioning of a cluster, where, for example, four nodes of an eight-node
cluster cannot talk to the other four nodes. This situation is rare, but in this situation, a simple majority does
not exist, and the epsilon node sways the vote for the masters of the RDB units.

3-35 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Quorum Rules

4.2 Ɛ 
3 Ɛ 
2.2 Ɛ 
2 Ɛ 

NetApp Confidential 36

QUORUM RULES

3-36 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Two-Node Clusters

Two-node clusters are a special case:

 No majority exists in the event of a cluster
interconnect partition or during a failover
situation.
 The RDB manages this case “under the
covers” but must be “told” that this cluster
contains only two nodes.
netappu::> cluster ha –enabled true

See TR3450 for more information

NetApp Confidential 37

TWO-NODE CLUSTERS
From Ron Kownacki, author of the RDB:
“Basically, quorum majority doesn’t work well when down to two nodes and there’s a failure, so RDB is
essentially locking the fact that quorum is no longer being used and enabling a single replica to be artificially
writable during that outage.
“The reason we require a quorum (a majority) is so that all committed data is durable: if you successfully
write to a majority, you know that any future majority will contain at least one instance that has seen the
change, so the update is durable. If we didn’t always require a majority, we could silently lose committed
data. So in two nodes, the node with epsilon is a majority and the other is a minority—so you would only
have one-directional failover (need the majority). So epsilon gives you a way to get majorities where you
normally wouldn’t have them, but it only gives unidirectional failover because it’s static.
“In two-node (high-availability mode), we try to get bidirectional failover. To do this, we remove the
configuration epsilon and make both nodes equal—and form majorities artificially in the failover cases. So
quorum is two nodes available out of the total of two nodes in the cluster (no epsilon involved), but if there’s
a failover, you artificially designate the survivor as the majority (and lock that fact). However, that means you
can’t fail over the other way until both nodes are available, they sync up, and drop the lock—otherwise you
would be discarding data.”

3-37 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Putting It All Together
Node

N-Blade and
Client Access (Data)
SCSI-Blade

Management M-Host

CSM Cluster Traffic

D-Blade
RDB Units:
 Management
 VLDB
 VifMgr
Cluster (data)  BCOM
Vol0
Vserver Root
Root Volume
Vol1
Vol2

NetApp Confidential 38

PUTTING IT ALL TOGETHER

This diagram shows that each node contains the following: an N-blade, an SCSI-blade, a CSM, a D-blade, an
M-host, RDB units (four), and vol0.

3-38 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Show the end-to-end path of a file write
request through a cluster
 Answer questions about RDB concepts
 Identify the differences between a vol0 root
volume and a cluster (data) Vserver root
volume

NetApp Confidential 39

MODULE SUMMARY

3-39 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Exercise
Module 3: Architecture
Time Estimate: 15 Minutes

NetApp Confidential 40

EXERCISE
Please refer to your exercise guide.

3-40 Clustered Data ONTAP Administration: Architecture

NetApp University - Do Not Distribute

Module 4
User Interface

NetApp Confidential 1

MODULE 4: USER INTERFACE

4-1 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Compare the command-line interface (CLI)
with the GUIs
 Determine which commands are available for
a command directory
 Determine whether parameters are required
or optional for a command
 Switch between privilege levels

NetApp Confidential 2

MODULE OBJECTIVES

4-2 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

4-3 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Overview
 You can manage resources within a cluster by using the
CLI or the GUIs (NetApp OnCommand System Manager or
Element Manager).
 The CLI and the Element Manager GUI access the
hierarchical command structure.
 The CLI and the Element Manager GUI are generated
from the same source, so they are always synchronized.
 You can access an entire cluster from a “cluster
management” logical interface (LIF).
 You can access nodes by using “node management” LIFs,
for example, to examine an unhealthy node.
 The three administrative privilege levels are admin,
advanced, and diagnostic.

NetApp Confidential 4

OVERVIEW
The command-line interface (CLI) and the GUI provide access to the same information, and you can use both
to manage the same resources within a cluster.
The hierarchical command structure consists of command directories and commands. A command directory
might contain commands, more command directories, or both. In this way, command directories resemble file
system directories and file structures.
Command directories provide groupings of similar commands. For example, all commands for storage-related
actions fall somewhere within the storage command directory. Within that directory are directories for disk
commands and aggregate commands. The command directories provide the context that enables you to
use similar commands for different objects. For example, you use create commands to create all objects
and resources and delete commands to remove objects and resources, but the commands are unique because
of the context (command directory) in which the commands are used. Therefore, storage aggregate
create is different from network interface create.
The cluster login is accessible from a cluster management logical interface (LIF). You can also log in to each
node by using the node management LIF for the node.
NOTE: Element Manager (including ClusterView) is being discontinued in a future release. Element
Manager does not support all of the latest Data ONTAP functionality. You can use the Data ONTAP CLI or
OnCommand System Manager, a web-based graphical management interface, to manage the cluster.

4-4 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Shells

 The cluster shell

– Former name: ngsh (the “ng shell”)
– Scope: the entire cluster
 The node shell
– A subset of the Data ONTAP 7G and Data
ONTAP 7-Mode commands
– Scope: a single node at any one time
 The system shell
– A means of access to the BSD shell
– Scope: a single node at any one time

NetApp Confidential 5

SHELLS

4-5 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The Cluster Shell

 Secure Shell (SSH) is the default method for

nonconsole logins.
 Remote Shell (RSH) is not supported.
 The “root” user is not permitted.
 The “admin” user is predefined, with a
password that is chosen during cluster setup.
 You can access the cluster shell through the
cluster management LIF or the node
management LIFs.

NetApp Confidential 6

THE CLUSTER SHELL

The best way to manage a cluster is to use Secure Shell (SSH) to log in to the cluster shell with the cluster
management LIF IP address. If a node cannot communicate with the rest of the cluster, you can use the node
management LIF for that node. If you cannot use the node management LIF, you can use a remote
management device.

4-6 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Management LIFs

 The cluster management LIF:

– Is a persistent LIF to use for SSH access
– Is unique within the cluster
– Is assigned to a “data” port
– Can fail over and migrate among nodes
 The node management LIF:
– Is unique for a node
– Is assigned to a “data” or “node-mgmt” port
– Can only fail over or migrate to a port on the
same node
– Can access the entire cluster
NetApp Confidential 7

MANAGEMENT LIFS
Data ONTAP operating in Cluster-Mode has one management virtual interface on each node that is called a
“node management” LIF. Node management LIFs do not fail over to other nodes.
Data ONTAP operating in Cluster-Mode also includes a management LIF, the “cluster management” LIF,
that has failover and migration capabilities. Therefore, regardless of the state of each individual node (for
example, if a node is rebooting after an upgrade or is halted for hardware maintenance), a LIF address can
always be used to manage the cluster, and the current node location of that LIF is transparent.

4-7 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Management LIFs
The Output of net int show
netappu::> net int show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
netappu
cluster_mgmt up/up [Link]/24 netappu-01 e0c true
netappu-01
clus1 up/up [Link]/16 netappu-01 e0a true
clus2 up/up [Link]/16 netappu-01 e0b true
mgmt1 up/up [Link]/24 netappu-01 e0c true
netappu-02
clus1 up/up [Link]/16 netappu-02 e0a true
clus2 up/up [Link]/16 netappu-02 e0b true
mgmt1 up/up [Link]/24 netappu-02 e0c true
vs1
vs1_lif1 up/up [Link]/24 netappu-01 e0d true
vs1_lif2 up/up [Link]/24 netappu-01 e0d false
9 entries were displayed.

The Node The Cluster This LIF has failed

Management LIF Management LIF over or migrated.

NetApp Confidential 8

MANAGEMENT LIFS: THE OUTPUT OF NET INT SHOW

The two “mgmt1” LIFs that are shown on this slide are the node management LIFs. Each is associated with
the virtual servers (Vservers) of a node.
The cluster management LIF, “cluster_mgmt” in the example on this slide, is not associated with any one
node Vserver but with the administration Vserver, “netappu,” which represents the entire physical cluster.

4-8 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The Node Shell

 You access the node shell from within the

cluster shell.
 The scope of access is limited to one node at
a time, but you can access any node in the
cluster.
 You can enter the node shell for an interactive
session or execute individual commands
directly from the cluster shell.
 The command for accessing the node shell is
netappu::> system node run.

NetApp Confidential 9

THE NODE SHELL

4-9 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

system node run

 A single command directly from the cluster

shell:
netappu::> system node run –node netappu-02 hostname
netappu-02

 An interactive session:
netappu::> system node run –node netappu-02
Type 'exit' or 'Ctrl-D' to return to the CLI
netappu-02> hostname
netappu-02

NetApp Confidential 10

SYSTEM NODE RUN

In the examples on this slide, the hostname command was invoked from the CLI of one node and executed
on the other node. In the first example, the command was invoked from the cluster shell. In the second
example, the administrator entered the node shell of the other node and then ran the command interactively.

4-10 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The System Shell
 You access the system shell from within the cluster
shell.
 From any node, you can access the system shell on
any other node.
 To access the system shell, do the following:
1. Unlock the “diag” user and set password:
netappu::> security login unlock –username
diag
netappu::> sec log pass –user diag
2. From the cluster shell, use the advanced command:
netappu::*>system node systemshell
3. Can only be accessed by the "diag" user.

NetApp Confidential 11

THE SYSTEM SHELL

You can use the system shell to access the BSD environment that the Data ONTAP operating system runs in.
You should access the system shell only under NetApp Support or Engineering supervision.
You can access the system shell only as the “diag” user and only from within the cluster shell. Root access to
the system shell is not available from Data ONTAP Cluster-Mode systems.
Use the security login unlock -username diag cluster shell command to unlock the
“diag” user. Then use the security login password -username diag command to set the “diag”
user password.
NOTE: You should access the system shell only under the supervision of NetApp technical support.

4-11 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

OnCommand System Manager

 Data ONTAP 8.1.1 operating in Cluster-Mode

works with OnCommand System Manager.
 Start OnCommand System Manager:
1. Enter the IP address (or host name) of the
cluster management LIF.
2. Enter a user name and password.
 The “admin” user is predefined.

NetApp Confidential 12

ONCOMMAND SYSTEM MANAGER

Data ONTAP 8.1.1 operating in Cluster-Mode retains the browser-based Element Manager GUI. However,
NetApp System Manager 2.0R1 and later supports Data ONTAP operating in Cluster-Mode, and
OnCommand System Manager is a more powerful management interface than Element Manager is.
This course covers both of the browser-based GUIs but focuses on the more powerful OnCommand System
Manager interface.

4-12 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

OnCommand System Manager Login Page
1of 3

NetApp Confidential 13

ONCOMMAND SYSTEM MANAGER LOGIN PAGE: 1 OF 3

1. From the Windows Start menu, start OnCommand System Manager 2.1.

4-13 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

OnCommand System Manager Login Page
2 of 3

NetApp Confidential 14

ONCOMMAND SYSTEM MANAGER LOGIN PAGE: 2 OF 3

2. Click Add.
3. Enter the IP address for the cluster management LIF.
4. Click the More button.
5. Select Credentials.
6. Enter the user name admin and the password.
7. Click Add.

4-14 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

OnCommand System Manager Login Page
3 of 3

NetApp Confidential 15

ONCOMMAND SYSTEM MANAGER LOGIN PAGE: 3 OF 3

You should see your cluster appear on the Discover page.

4-15 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 16

LESSON 2

4-16 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Cluster Shell Features
1 of 2
 Has a history buffer
– Enables you to easily reissue commands
– Enables you to retrieve commands and then
easily modify and reissue the commands
 Provides context-sensitive help when you
press the question mark (?) key
 Enables you to reduce the required amount of
typing and get context-sensitive assistance
when you press the Tab key

NetApp Confidential 17

CLUSTER SHELL FEATURES: 1 OF 2

The cluster shell has features that are similar to the popular tcsh shell for UNIX machines, such as the ability
to pull previous commands out of a command history buffer and then edit and reissue those commands.
Editing of commands is similar to editing through tcsh or Emacs, with key combinations such as Ctrl-A and
Ctrl-E that move the cursor to the beginning and end of a command, respectively. The up and down arrows
enable you to cycle through the command history.
Simple online help is also available. You can press the question mark (?) key almost anywhere to get
contextual help. In many places, you can press the Tab key to complete a command or parameter, which
reduces the required amount of typing.

4-17 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Cluster Shell Features
2 of 2
 The cluster shell uses named parameters.
 You can abbreviate command directories,
commands, and parameters to their shortest
unambiguous sequences of characters.
 The search path enables you to run
commands “out of context.”
 You can run queries with patterns and
wildcards.

NetApp Confidential 18

CLUSTER SHELL FEATURES: 2 OF 2

The cluster shell uses named parameters for every command.
Every command directory, command, and parameter can be abbreviated to the shortest string of characters
that is unambiguous within the context. For example, from the top level, the storage aggregate show
command can be abbreviated to sto a s and the network interface show command to n i s.
Commands can be run “out of context.” If you are at the top level of the command hierarchy and enter disk
show, the shell runs the storage disk show command, because the shell can resolve the disk
command as being unique within the whole command hierarchy. Likewise, if you type disk and press Enter,
you are put into the storage disk command directory. This method works even if you’re in an unrelated
command directory, such as the network interface directory.
The cluster shell supports queries and UNIX-style patterns and wildcards to enable you to match multiple
values of specific parameters. For example, assume that you use a naming convention for volumes such that
every volume that is owned by the Accounting department has the prefix “acct_.” You can show only those
volumes by using the volume show –vserver * –volume acct_* command (which shows all
volumes that begin with “acct_” on all Vservers). To further limit your query to volumes that have more than
500 GB of data, you can run this command:
netappu::> volume show –vserver * -volume acct_* -used >500gb.

4-18 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?) Key at the
Top Level 1 of 2
netappu::> ?
up Go up one directory
cluster> Manage clusters
dashboard> Display dashboards
event> Manage system events
exit Quit the CLI session
history Show the history of commands for this CLI session
job> Manage jobs and job schedules
lun> List LUN (logical unit of block storage) commands
man Display the on-line manual pages
network> Manage physical and virtual network connections
qos> QoS settings
redo Execute a previous command
rows Show/Set the rows for this CLI session
run Run interactive or non-interactive commands in
the node shell
security> The security directory
set Display/Set CLI session settings
sis Manage volume efficiency
snapmirror> Manage SnapMirror
statistics> Display operational statistics

NetApp Confidential 19

USING THE QUESTION MARK (?) KEY AT THE TOP LEVEL: 1 OF 2

Press the question mark (?) key at the top level of the command hierarchy to show the command directories
and commands that are available at that top level.

4-19 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?) Key at the
Top Level 2 of 2
storage> Manage physical storage, including disks,
aggregates, and failover
system> The system directory
top Go to the top-level directory
volume> Manage virtual storage, including volumes,
snapshots, and mirrors
vserver> Manage Vservers

NetApp Confidential 20

USING THE QUESTION MARK (?) KEY AT THE TOP LEVEL: 2 OF 2

4-20 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?) Key for
Commands and Directories 1 of 2
netappu::> cluster

netappu::cluster> ?
contact-info Manage contact information for the
cluster.
create Create a cluster
ha Manage high-availability configuration
identity Manage the cluster's attributes,
including name and serial number
join Join an existing cluster using the
specified member's IP address
modify Modify cluster node membership attributes
peer Manage cluster peer relationships
setup Setup wizard
show Display cluster node members
statistics Display cluster statistics

NetApp Confidential 21

USING THE QUESTION MARK (?) KEY FOR COMMANDS AND DIRECTORIES: 1 OF 2
You can also press the question mark key to show the available commands and command directories at any
other level of the hierarchy.

4-21 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?) Key for
Commands and Directories 2 of 2
netappu::cluster> statistics

netappu::cluster statistics> ?
show Display cluster-wide statistics

NetApp Confidential 22

USING THE QUESTION MARK (?) KEY FOR COMMANDS AND DIRECTORIES: 2 OF 2

4-22 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?)
and Tab Keys to Show Parameters 1 of 2
netappu::> storage aggregate

netappu::storage aggregate> modify ?

[ -aggregate] <aggregate name> Aggregate
[ -raidtype|-t {raid_dp|raid4} ] RAID Type
[ -maxraidsize|-s <integer> ] Max RAID Size
[ -hybrid-enabled {true|false} ] Hybrid Enabled
[ -snapshot-autodelete {on|off} ] Automatic Snapshot Deletion
[ -ha-policy {sfo|cfo} ] HA Policy
[ -free-space-realloc {on|off} ] Free Space Reallocation
[ -percent-snapshot-space <percent> ] Space Reserved for Snapshot
Copies

NetApp Confidential 23

USING THE QUESTION MARK (?) AND TAB KEYS TO SHOW PARAMETERS: 1 OF 2
You can press the question mark (?) key to show required and optional parameters and valid keyword values
for parameters that accept keywords.

4-23 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Using the Question Mark (?)
and Tab Keys to Show Parameters 2 of 2
netappu::storage aggregate> modify -aggr aggr1a -state ?
offline
online
restricted

netappu::storage aggregate> modify -aggr aggr1a -state rest<TAB>

netappu::storage aggregate> modify -aggr aggr1a -state restricted

NetApp Confidential 24

USING THE QUESTION MARK (?) AND TAB KEYS TO SHOW PARAMETERS: 2 OF 2
Press the Tab key to show directories, commands, and parameters that are available or to automatically
complete a command (or a portion of a command).

4-24 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

OnCommand System Manager Initial
Page

NetApp Confidential 25

ONCOMMAND SYSTEM MANAGER INITIAL PAGE

This is the initial page that appears when you log in to OnCommand System Manager. It’s a dashboard view
of the system health of the entire cluster. The left pane contains the command directories and commands.
Note the three tabs on the left:
Cluster: Cluster-wide central admin tasks (Administrative Vserver)
Controllers: Node and hardware focused tasks (Node Vservers)
Vservers: Vserver or Virtual Array management (Data Vservers)

4-25 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Storage

NetApp Confidential 26

STORAGE
Notice the expanded Storage directory in the left pane.

4-26 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Storage Aggregate

NetApp Confidential 27

STORAGE AGGREGATE
Notice the Aggregates pane on the right.

4-27 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Editing a Storage Aggregate

NetApp Confidential 28

EDITING A STORAGE AGGREGATE

If you right-click an aggregate and select Edit, the Edit Aggregate dialog box appears.

4-28 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The Edit Aggregate Dialog Box

NetApp Confidential 29

THE EDIT AGGREGATE DIALOG BOX

From the Edit Aggregate dialog box, you can change the RAID type and rename the aggregate.

4-29 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Changing Privilege Levels in the CLI
1 of 2
netappu::> storage disk ?
assign Assign ownership of a disk to a system
fail Fail the file system disk
modify Modify disk attributes
option> Manage disk options
remove Remove a spare disk
removeowner Remove disk ownership
replace Initiate or stop replacing a file-system disk
set-led Turn on a disk's red LED for a number of minutes
show Display a list of disk drives and array LUNs
updatefirmware Update disk firmware
zerospares Zero non-zeroed spare disks

NetApp Confidential 30

CHANGING PRIVILEGE LEVELS IN THE CLI: 1 OF 2

This slide shows the set adv command (an abbreviation of set -privilege advanced) in the
cluster shell. Note the options that are available for the security directory before you change the privilege
level (using admin privileges) and after you change the privilege level (to advanced privileges). With
advanced privileges, the certificate-related commands are available.
If the command prompt contains an asterisk, you are not currently using the admin privilege level.

4-30 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Changing Privilege Levels in the CLI
2 of 2
netappu::> set advanced

Warning: These advanced commands are potentially dangerous; use them

only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y

netappu::*> storage disk ?

assign Assign ownership of a disk to a system
fail Fail the file system disk
modify Modify disk attributes
option> Manage disk options
reassign *Change the default owner of all disks from one
node to another
remove Remove a spare disk
removeowner Remove disk ownership
replace Initiate or stop replacing a file-system disk
set-led Turn on a disk's red LED for a number of
minutes
show Display a list of disk drives and array LUNs
unfail *Unfail a broken disk
updatefirmware Update disk firmware
zerospares Zero non-zeroed spare disks

NetApp Confidential 31

CHANGING PRIVILEGE LEVELS IN THE CLI: 2 OF 2

This slide shows the set adv command (an abbreviation of set -privilege advanced) in the cluster shell. Note
the options that are available for the security directory before you change the privilege level (using admin
privileges) and after you change the privilege level (to advanced privileges). With advanced privileges, the
certificate-related commands are available.
If the command prompt contains an asterisk, you are not currently using the admin privilege level.

4-31 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Element Manager

NetApp Confidential 32

ELEMENT MANAGER
Element Manager provides an additional UI environment that is available with the Data ONTAP Cluster-
Mode operating system without the need to install additional software. Element Manager shares the same
command architecture as the cluster shell CLI and provides an alternative UI environment that maintains all
of the same management capabilities that have been available throughout the clustered Data ONTAP product
line.
To start Element Manager, enter the cluster-management LIF IP address in a Web browser and then the
administrator user ID and password at the prompt.

4-32 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Element Manager from System Manager

NetApp Confidential 33

ELEMENT MANAGER FROM SYSTEM MANAGER

System Manager also provides an alternate way to bring up Element Manager. Click on the Advanced button
in the left pane window.

4-33 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Changing Privilege Levels

NetApp Confidential 34

CHANGING PRIVILEGE LEVELS

This page, selected by clicking PREFERENCES on the left pane, is how you would change the privilege level
from within the GUI.
The privilege level is changed only for the user and interface in which this change is made, that is, if another
admin user is using the cluster shell, that admin user’s privilege level is independent of the level in use here,
even if both interfaces are accessing the same node.

4-34 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

ClusterView
 Is a Flash-based dashboard for viewing system
health, performance, and resource utilization
 Is kicked off from inside Element Manager
 Generates graphs and tables that display
statistics about:
– Performance
– Storage space utilization for aggregates and
volumes
– Alarms for exceeded thresholds
– The system health dashboard
– Jobs that are executing and queued

NetApp Confidential 35

CLUSTERVIEW
ClusterView is a Web-based tool that graphically displays performance, usage, and health information from a
Data ONTAP cluster. ClusterView is implemented as a set of Adobe Flash Web pages that are served up from
any node in the cluster. The user points a Web browser to one particular node, which is referred to as the
“serving node.” Dynamic content is constructed by using performance, health, and resource utilization data
that ClusterView periodically fetches from the serving node. The serving node constructs this data by
querying other nodes in the cluster as appropriate.

4-35 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The ClusterView Dashboard

NetApp Confidential 36

THE CLUSTERVIEW DASHBOARD

This is a “dashboard” view.

4-36 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

The ClusterView Dashboard

NetApp Confidential 37

THE CLUSTERVIEW DASHBOARD

This is a graphical representation of the space utilization: aggregates on the left and volumes on the right.

4-37 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Compare the CLI with the GUIs
 Determine which commands are available for
a command directory
 Determine whether parameters are required
or optional for a command
 Switch between privilege levels

NetApp Confidential 38

MODULE SUMMARY

4-38 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Exercise
Module 4: User Interface
Time Estimate: 20 minutes

NetApp Confidential 39

EXERCISE
Please refer to your exercise guide.

4-39 Clustered Data ONTAP Administration: User Interface

NetApp University - Do Not Distribute

Module 5
Physical Data Storage

NetApp Confidential 1

MODULE 5: PHYSICAL DATA STORAGE

5-1 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Module Objectives
After this module, you should be able to:
 Draw the connections from a high-availability (HA) pair of
controllers to the disk shelves
 Create aggregates
 List the steps that are required to enable storage failover
(SFO)
 Explain and enable “two-node HA” mode for two-node
clusters
 Take over and give back storage between the nodes of an
HA pair
 Enumerate the steps for expanding an aggregate from 32-
bit to 64-bit
 Create a Flash Pool
NetApp Confidential 2

MODULE OBJECTIVES

5-2 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

The FAS3270 System
Hardware Diagram
HA c0b HA c0a FC 0c Management e0M Data Cluster X1107A Dual-
e1b e1a Port 10-GbE Card
Data e0a e2b

X1107A

LINK/ACT

LINK/ACT
Chelsio
Communications

c0a 0c e0a

X1107A

LINK/ACT

LINK/ACT
0a 0b Chelsio
Communications

LNK LNK

c0b 0d e0b

X1139A

10GbEE CNA
3 5

SAN LAN

SAN LAN
PORT 1

PORT 2
4 6

SAS 0a SAS 0b Data e3a and e3b FC 0d Data e0b ACP* Console Cluster
e2a
X1139A Dual-
Port UTA Card
*Alternate Control Path

NetApp Confidential 3

THE FAS3270 SYSTEM: HARDWARE DIAGRAM

The example on this slide shows a FAS3270 system controller. Use this example as a reference, but
remember that as new cards are supported, some of this information might change.

5-3 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

The FAS62x0 System
Hardware Diagram
3

0 e0a e0b
e0c e0d e0e e0f 0a 0b 0c 0d

LINK LINK LINK LINK

7 13

8 14

9 15

10 16

Management: e0M 1Gb Data: e0a and e0b 10-Gb Cluster: e0c, e0e 10-Gb Data: e0d, e0f FC: 0a, 0b, 0c, and 0d Console

NetApp Confidential 4

THE FAS62X0 SYSTEM: HARDWARE DIAGRAM

The example on this slide shows a FAS62x0 series controller. Use this example as a reference, but remember
that as new cards are supported, some of this information might change.

5-4 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

A Typical Disk Shelf with FC Connection
FC A (in)

FC A (out)

OP EN OP EN OPEN OPEN

Fault

2 Gb
Out In

ESH2

SHELF MODULE A
1Gb 2Gb PSU1 PSU2
ID MODULE B

ESH2

Fault
2 Gb
In Out
CAUT ION

CAUT ION
FC B (in)

FC B (out)

NetApp Confidential 5

A TYPICAL DISK SHELF WITH FC CONNECTION

This slide shows the back of a typical disk shelf that has an FC connection to the host storage controller. The
in and out ports of loop A (top) and loop B (bottom) are highlighted.

5-5 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

A Typical Disk Shelf with SAS Connection
ACP ACP SAS SAS ACP ACP SAS SAS
Circle Square Circle Square Circle Square Circle Square

IOM6 IOM6
LNK LNK
A LNK LNK

IOM6 B IOM6

DC AC DC AC
1

x2
2 2 x2
2

IOM6 A IOM6 B

NetApp Confidential 6

A TYPICAL DISK SHELF WITH SAS CONNECTION

This slide shows the back of a typical disk shelf that has a SAS connection to the host storage controller. Each
of the IOM6 modules uses the same orientation. The SAS and alternate control path (ACP) connections use a
circle and square symbol to make cabling connections within a series easier. Each SAS port with a circle
symbol connects to the next shelf’s SAS port with a square symbol. ACP connections work exactly the same
way.

5-6 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

An HA Pair
SAS Storage Configuration

X1107A

X1107A
LINK/ACT

LINK/ACT
1 Chelsio

LINK/ACT

LINK/ACT
Communications 1 Chelsio
Communications

c0a 0c e0a c0a 0c e0a

X1107A

X1107A
2

LINK/ACT

LINK/ACT
Chelsio 2

LINK/ACT

LINK/ACT
0a 0b Communications 0a 0b Chelsio
Communications
LNK LNK LNK LNK

Controller 1 Controller 2
c0b 0d e0b c0b 0d e0b

D
A

B
X2065A
X1139A

X2065A
3

10GbEE CNA
5 3 5

SAN LAN

SAN LAN
PORT 1

PORT 2
4 6 4 6

IOM6 IOM6 IOM6 IOM6

LNK LNK
A LNK LNK LNK LNK
A LNK LNK

Shelf 1 Shelf 1
IOM6 B IOM6 IOM6 B IOM6

DC AC DC AC DC AC DC AC
1 1

x2
2 2 x2
2
x2
2 2 x2
2

IOM6 IOM6 IOM6 IOM6

LNK LNK
A LNK LNK LNK LNK
A LNK LNK

Shelf 2 Shelf 2
IOM6 B IOM6 IOM6 B IOM6

DC AC DC AC DC AC DC AC
1 1

x2
2 2 x2
2
x2
2 2 x2
2

Stack 1 Stack 2
Starting with shelf ID 10 Starting with shelf ID 20

SAS
ACP

VTIC

NetApp Confidential 7

AN HA PAIR: SAS STORAGE CONFIGURATION

The diagram on this slide combines SAS and ACP shelf cabling and the high-availability (HA) cabling for
controller 1 and controller 2 of an HA pair. For a complete course in Data ONTAP Cluster-Mode cabling, see
the Data ONTAP 8 Cabling course (STRHW-WBT-DOTCABL).

5-7 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

HA Interconnect Links
 InfiniBand links connect the two nodes of each HA
pair.
– InfiniBand for FAS6000 and V6000 series
– Dedicated 10-Gb links for FAS3200 and V3200 series
 The HA links are used to mirror nonvolatile RAM
(NVRAM).
 The HA links provide a channel for certain types of
communication traffic between the nodes in a pair:
– Failover
– Disk firmware
– Heartbeats
– Version information

NetApp Confidential 8

HA INTERCONNECT LINKS
InfiniBand links connect the two nodes of each HA pair for all models except the FAS and V-Series 32x0
series model controllers. FAS and V-Series 32x0 model controllers use a dedicated 10-GbE link, internal or
external, depending on the model and enclosure. Visit the NetApp Support site to see the appropriate
hardware configuration guide for your model storage controller.
The types of traffic that flow over the HA interconnect links are:
 Failover: The directives are related to performing storage failover (SFO) between the two nodes,
regardless of whether the failover is:
– Negotiated (planned and in response to an administrator request)
– Not negotiated (unplanned and in response to an improper system shutdown or booting)
 Disk firmware: Nodes in an HA pair coordinate the update of disk firmware. While one node is updating
the firmware, the other node must not perform any I/O to that disk.
 Heartbeats: Regular messages demonstrate availability.
 Version information: The two nodes in an HA pair must be kept at the same major and minor revision
levels for all software components.

5-8 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Disks
 Every disk, volume, and aggregate in an HA pair is
assigned a “home” and is owned by that node.
 A subset of disks in the HA pair serve as “mailbox disks”:
– A mailbox disk provides persistent storage for information
about the SFO state, including some specific replicated
database (RDB) data when in “two-node HA” mode.
– Each node of an HA pair designates two disks in the first
RAID group in the root aggregate.
– Attempts are made to write SFO state information to all
mailbox disks for configuration and status changes.
– Quorum techniques are used to guarantee that at least three
of the four mailbox disks must be available for SFO.

NetApp Confidential 9

DISKS
Each node of an HA pair designates two disks in the first RAID group in the root aggregate as the mailbox
disks. The first mailbox disk is always the first data disk in RAID group RG0. The second mailbox disk is
always the first parity disk in RG0. The “mroot” disks are generally the mailbox disks.
Each disk, and therefore each aggregate and volume that is built upon the disk, can be owned by exactly one
of the two nodes in the HA pair at any given time. This form of software ownership is made persistent by
writing the information onto the disk itself. The ability to write disk ownership information is protected by the
use of persistent reservations. Persistent reservations can be removed from disks by power-cycling the shelves
or by selecting maintenance mode while in boot mode and then issuing manual commands. If the node that
owns the disks is running in normal mode, the node reasserts its persistent reservations every 30 seconds.
Changes in disk ownership are managed automatically by normal SFO operations, although commands exist
to manipulate disk ownership manually if necessary.
Each node in an HA pair can perform reads from any disk to which the node is connected, even if the disk
isn’t that disk’s owner; however, only the node that is marked as that disk’s current owner is allowed to write
to that disk.

5-9 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Disk Ownership
 Software disk ownership is made persistent by writing
the ownership information onto the disk.
 The ability to write disk ownership information is
protected by the use of persistent reservations.
 Changes in disk ownership are managed
automatically by normal SFO operations, although
commands exist to manipulate disk ownership
manually if necessary.
 It is possible for disks to be “unowned.”

NetApp Confidential 10

DISK OWNERSHIP
A disk’s data contents are not destroyed when the disk is marked as “unowned”; only the disk’s ownership
information is erased. Unowned disks that reside on an FC-AL, where the owned disks exist, have ownership
information automatically applied to guarantee that all disks on the same loop have the same owner.

5-10 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

SFO
1 of 2
 Two nodes are connected as an HA pair.
 Each node is a fully functioning node in the larger
cluster.
 Clusters can consist of heterogeneous hardware,
but both nodes of an HA pair must be the same
controller model.
 SFO can be enabled from either node in the pair.
 SFO takeover can be initiated from any node in
the cluster.
 A manual storage takeover forces a booting of
the node that is taken over.

NetApp Confidential 11

SFO: 1 OF 2
Enabling SFO is done within pairs regardless of how many nodes are in the cluster. For SFO, the HA pairs
must be of the same model, for example, two FAS32x0 systems or two FAS62x0 systems. The cluster itself
can contain a mixture of models, but each HA pair must be homogenous. The version of the Data ONTAP
operating system must be the same on both nodes of the HA pair except for the short period of time during
which the pair is upgraded. During that time, one of the nodes is booted with a later version than its partner’s
version, with the partner to follow shortly. The nonvolatile RAM (NVRAM) cards must be installed in the
nodes. Two interconnect cables are required to connect the NVRAM cards (except for FAS and V-Series 32x0
models with single-enclosure high availability).
Remember that this cluster is not simply the pairing of machines for failover; this cluster is the Data ONTAP
cluster.

5-11 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

SFO
2 of 2
 Automatic giveback is disabled by default.
 Both nodes of an HA pair must be booted before
SFO can be enabled for the pair.
 Because the N-blade and D-blade of a write
request can be on separate nodes, the internal
logistics of a failover are different than with the
Data ONTAP 7G operating system.

NetApp Confidential 12

SFO: 2 OF 2

5-12 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

SFO Compared to CFO
1 of 2
 CFO:
– This term now means “controller failover” and is used with the Data
ONTAP 7G operating system and Data ONTAP operating in 7-Mode.
– “Giveback” returns all aggregates to the partner.
– Logical interface (LIF) failover is included in CFO and goes only to its
HA partner.
– The failed system’s identity continues within the surviving partner.
– Some suggest that during a CFO event, it is as if two instances of
Data ONTAP are running on the same controller.
 SFO:
– This term is used with the Data ONTAP GX operating system and
Data ONTAP operating in Cluster-Mode.
– “Giveback” returns the aggregate that contains the vol0 root and
then the other aggregates.
– LIF failover is separate from SFO and can go to any node in the
cluster.
NetApp Confidential 13

SFO COMPARED TO CFO: 1 OF 2

CFO now means “controller failover.”
CFO used to mean “cluster failover,” but the term “cluster” is no longer used in relation to the Data ONTAP
7G operating system or Data ONTAP 8.x operating in 7-Mode.

5-13 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

SFO Compared to CFO
2 of 2
 With CFO, the “down” system is entirely hosted (virtually)
by the HA partner; therefore, the “down” system appears to
be “up.”
 With SFO, the “down” node is gone until the “down” node
comes back:
– The aggregates and volumes are taken over by the HA
partner.
– The NAS data LIFs are migrated to another node (not
necessarily the HA partner).
 With SFO, during a failover event, the surviving HA partner
modifies disk ownership and assimilates the aggregates
and volumes.

NetApp Confidential 14

SFO COMPARED TO CFO: 2 OF 2

5-14 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

The storage failover show Command
netappu::> storage failover show
Takeover InterConn
Node Partner Enabled Possible Up State
-------------- -------------- ------- -------- --------- --------------
netappu-01 netappu-02 true true true connected
netappu-02 netappu-01 true true true connected
2 entries were displayed.

NetApp Confidential 15

THE STORAGE FAILOVER SHOW COMMAND

The example on this slide shows a two-node cluster, which is also an HA pair. Note that SFO is enabled on
both nodes.

5-15 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

What Happens When a Node Fails?
The surviving HA partner immediately initiates the
following process in this order:
1. Releases the failed partner’s reservations
2. Assigns ownership to the failed node’s disks
3. Replays the NV_LOG from its mirrored NVRAM to
complete any in-flight writes and modifications
4. Brings the aggregates online
5. Brings the volumes online and mounts them
6. Updates the volume location database for the new
aggregate location
7. Starts serving requests for the volumes and
aggregates that were served by the failed node
NetApp Confidential 16

WHAT HAPPENS WHEN A NODE FAILS?

5-16 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Two-Node High Availability
 Is an additional configuration step for two-node
clusters only
 Must be configured to enable the cluster to
operate properly when one of the two nodes is
down
 Is needed because of the way that the RDB units
maintain quorum. (RDB units operate differently
when only two nodes exist compared to when
more than two nodes exist.)
 Must be enabled for SFO to work properly in a
two-node cluster

NetApp Confidential 17

TWO-NODE HIGH AVAILABILITY

For clusters of only two nodes, the replicated database (RDB) units rely on the disks to maintain quorum
within the cluster in the case that a node is booted or goes down. This process is enabled by configuring the
two-node HA mechanism. Because of the reliance on the disks, SFO enablement and automatic giveback are
also required by two-node high availability and are configured automatically when two-node high availability
is enabled. For clusters that are larger than two nodes, quorum can be maintained without using the disks. Do
not enable two-node high availability for clusters that are larger than two nodes. When expanding a cluster
beyond two nodes, the HA state must be changed manually. Nodes cannot be added while HA is enabled.

5-17 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Aggregates
 Aggregates:
– Are the same as with the Data ONTAP 7G
operating system
– Have storage containers that consist of disks
– Can be RAID 4 or RAID-DP technology
– Contain volumes
– Can be taken over by their node’s HA partner
– Can be grown by adding disks
 32-bit and 64-bit aggregates are supported.
 Nondisruptive, in-place aggregate expansions are
available from 32-bit aggregates to 64-bit
aggregates.
NetApp Confidential 18

AGGREGATES
In the Data ONTAP 8.1 operating system, nondisruptive, in-place aggregate expansions are available from
32-bit aggregates to 64-bit aggregates. During the conversion, the volumes on the aggregate remain online
and continue to serve data.
For Data ONTAP operating in Cluster-Mode, storage administrators can initiate expansion through the
clustershell by enabling the diagnostic mode and then running the storage aggregate 64bit-
upgrade start command. The expansion runs in the background but can affect overall cluster
performance.
After an aggregate is converted to 64-bit, you can grow the aggregate beyond 16 TB by adding disks through
the storage aggregate add-disks command.

5-18 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

The storage aggregate show Command
netappu::> storage aggregate show
Aggregate Size Available Used% State #Vols Nodes RAID Status
--------- -------- --------- ----- ------- ------ ---------- -----------
netappu-01_aggr0
56.76GB 2.59GB 95% online 1 netappu-01 raid_dp
netappu-01_aggr2
113.5GB 113.2GB 0% online 1 netappu-01 raid4
netappu-01_aggr3
56.76GB 56.70GB 0% online 3 netappu-01 raid_dp
netappu-02_aggr0
56.76GB 2.59GB 95% online 1 netappu-02 raid_dp
netappu-02_aggr1
113.5GB 113.4GB 0% online 4 netappu-02 raid_dp
netappu-02_aggr2
113.5GB 113.5GB 0% online 0 netappu-02 raid4
6 entries were displayed.

NetApp Confidential 19

THE STORAGE AGGREGATE SHOW COMMAND

Here you can see that each node contains three aggregates.

5-19 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

In-Place 32-Bit-to-64-Bit
Aggregate Expansion
 Features:
– You can expand 32-bit aggregates to 64-bit
aggregates.
– You can expand while an aggregate is online
and serving data.
 Considerations:
– 64-bit aggregates consume more space than
32-bit aggregates do.
– The process works in the background but
affects performance.

NetApp Confidential 20

IN-PLACE 32-BIT-TO-64-BIT AGGREGATE EXPANSION

In the Data ONTAP 8.1 operating system, nondisruptive, in-place aggregate expansions are available from
32-bit aggregates to 64-bit aggregates. During the conversion, the volumes on the aggregate remain online
and continue to serve data.
The expansion runs in the background but can affect overall cluster performance.

5-20 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

In-Place 32-Bit-to-64-Bit
Aggregate Expansion: Steps
1. Identify when the 16-TB boundary is crossed.
2. Evaluate how much space is consumed by
the expansion.
3. Add new disks and trigger the expansion.

See TR3978 for more information

NetApp Confidential 21

IN-PLACE 32-BIT-TO-64-BIT AGGREGATE EXPANSION: STEPS

After an aggregate is converted to 64-bit, you can grow the aggregate beyond 16 TB by adding disks through
the storage aggregate add-disks command.

5-21 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

NetApp Virtual Storage Tier

Flash Cache Flash Pool

What is it? What is it?
 A controller-based PCIe card  Storage-level, RAID-protected cache
 A plug and play device (specific to aggregates)

What does it do? What does it do?

 Per controller cache  Caches random reads and
 Caches random reads overwrites
 Provides cached data persistence
through failovers
Where does it fit? Where does it fit?
 With random-read workloads, for  With random-overwrite-heavy
example, File Services workloads, for example, OLTP
 With volumes that span multiple  With consistent performance
aggregates (required)

NetApp Confidential 22

NETAPP VIRTUAL STORAGE TIER

5-22 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Cache
 Is a 256-GB, 512-GB, or 1-TB PCIe module
 Is a plug and play device (no required configuration)
 Is for Data ONTAP 8.0.2 clusters or later
 Supports all protocols
 Acts as an extention to the WAFL buffer cache, and saves evicted
buffers
 Shared by all volumes on a node

For supported platforms and compatibility, see

See TR3832 for more information

NetApp Confidential 23

FLASH CACHE

5-23 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pool
 Flash Pool is an aggregate-level
read and write cache.
 Like Flash Cache, Flash Pool
uses 4-KB block granularity and
Performance Capacity real-time caching.
+  Flash Pool is not a replacement
HDD for Flash Cache
Flash Pool
 Cache remains populated and
available during SFO events.
 Random overwrite data is cached.

NetApp Confidential 24

FLASH POOL
WAFL (Write Anywhere File Layout) aggregates are built with disks of the same type: SATA, hard disks,
and FC and SAS hard disks or solid-state disks (SSD). Flash Pools allow the mixing of SSDs and hard disks
within the same aggregate. The SSD tier aggregate is used as a cache and doesn’t contribute to usable space.
When an aggregate is converted to hybrid, the usable space in the aggregate does not change. The disks that a
hybrid aggregate consists of are treated like any disks in a NetApp storage array, and any class of disk can be
added on demand, subject to best practices around data such as parity ratios and RAID types.
Flash Pools provide:
 Improved cost per performance with fewer spindles, less rack space, and lower power and cooling
requirements
 Highly available storage with a simple administrative model
 Improved cost-to-performance and cost-to-capacity ratios compared to those of an SSD and SATA
combination with pure FC SAS
 Predictable and better degraded mode operation across controller failures and with takeover and giveback
 Automatic, dynamic, policy-based placement of data on appropriate tiers of storage (hard disks or SSDs)
at WAFL block granularity for either data or system metadata

5-24 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pool Components
 A Flash Pool is an aggregate with:
– One or more HDD RAID groups
– An SSD RAID group
 Only one type of HDD can be used
per Flash Pool
– High capacity (SATA)
– Performance (SAS)
 SSDs cache random data
– Previously written data (overwrites)
– Read data expired from main memory
 Existing aggregates can be non
disruptively converted to Flash Pools
NetApp Confidential 25

FLASH POOL COMPONENTS

Flash Pools consist of one or more HDD RAID groups and one or more SSD RAID groups. Each RAID
group within the aggregate must be homogeneous (all SATA HDD, all SAS HDD, or all SSD). Additionally,
on the Data ONTAP 8.1.1 operating system, the RAID Group type of the aggregate must be the same across
both the HDD and SSD RAID groups (RAID4 or RAID-DP). You can nondisruptively convert existing
aggregates to Flash Pools but, after conversion, SSDs cannot be removed.

5-25 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pool Software
and Hardware Support
 Data ONTAP 8.1.1 operating in 7-Mode and Data
ONTAP 8.1.1 operating in Cluster-Mode
 Platform support
– The FAS6200 and FAS6000 series
– The FAS3160, FAS3170, FAS3220, FAS3240, FAS3250,
and FAS3270 systems
– The FAS2220 and FAS2240 series
– Equivalent V-Series systems are also supported
 Shelf configurations
– Mixed SSD and SATA shelves
– V-Series systems with only NetApp SSD and hard disks
 See TR4070: Flash Pools

NetApp Confidential 26

FLASH POOL SOFTWARE AND HARDWARE SUPPORT

5-26 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Blocks in the SSD Tier

 Flash Pool metadata

 Read-cached blocks:
– Are a cached copy of the blocks from the hard
disk tier
– Still exist on the hard disk tier
 Write-cached blocks:
– Are written directly to the SSD tier
– Are not yet written to the hard disk tier

NetApp Confidential 27

BLOCKS IN THE SSD TIER

The following blocks are stored in the SSD tier of the Flash Pool:
 Flash Pool metadata: All metadata that is associated with the Flash Pool is stored in the SSD tier of the
aggregate.
 Read-cached blocks: These blocks are cached copies of a block in the hard disk tier of the aggregate and
are stored in the SSD tier. Almost all data from the active file system in a read/write volume is eligible to
be read-cached into the SSD tier.
 Write-cached blocks: These blocks are associated with a FlexVol volume that is written directly to the
SSD tier of the aggregate. Only one copy of the block exists. The copy is in the SSD tier. A hard disk
block is reserved for write-cached blocks for an eventual move into the HDD tier after access to the block
ceases.

5-27 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pool Read-Caching

 Similar approach to Flash Cache

 Type of operation identified upon first read
‒ Matches insertion policy: data inserted
‒ Doesn’t match policy: data ejected
 Data only cached in SSD, permanent copy
in HDD

NetApp Confidential 28

FLASH POOL READ-CACHING

5-28 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash-Based Read-Caching

A Wait for Hard Disk

Sequential

Most Random
Reads A Short Wait for Flash Read-
Ahead

Recent Random Buffer Cache RAM

Reads

NetApp Confidential 29

FLASH-BASED READ-CACHING
With flash-based read-caching, Flash Pools reduce the wait for data that comes from disk by placing random-
read data onto SSD. You also free up bandwidth to the hard disk, which should be handling mainly sequential
data. The combination of flash-based random reads and the read-ahead mechanism for sequential reads
provides low operational latency for most read operations.

5-29 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Enabling Flash Pools
1. Turn on the hybrid-enabled option.
2. Add a new RAID group by using SSDs.

netappu::> storage aggregate modify

–aggr aggr3 –hybrid-enabled true

netappu::> storage aggregate

-add-disks –aggr aggr3 –disktype
SSD –diskcount 12

NetApp Confidential 30

ENABLING FLASH POOLS

Enable Flash Pools on aggregates by completing the following steps:
1. Turn on the hybrid_enabled option on the aggregate.
2. Add a new RAID group with SSD disks.
This action converts the aggregate to a Flash Pool.

5-30 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pool Compatibility

Flash pools are not compatible with:

 32-bit aggregates
 Infinite volumes
 Aggregates created by using operating
systems earlier than Data ONTAP 7.2

NetApp Confidential 31

FLASH POOL COMPATIBILITY

Flash pools are incompatible with 32-bit aggregates and infinite volumes. Aggregates created by using
operating systems earlier than Data ONTAP 7.2 are not supported; however, this does not include any
aggregate that was created on any Data ONTAP GX or Data ONTAP 8 operating in Cluster-Mode.

5-31 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Flash Pools Additional Considerations

 Available space
 Flash Cache
 HA takeover and giveback
 Volume move
 Volume SnapMirror relationships
 Aggregate Snapshot copies
 Data compression
 V-Series
See TR4070 for more information

NetApp Confidential 32

FLASH POOLS ADDITIONAL CONSIDERATIONS

The capacity of the SSD tier is not reflected in the total aggregate size. For example, if the original aggregate
has a 10-TB capacity and you add an SSD raid group with a 1-TB capacity, the amount of capacity in the
aggregate that can be provisioned is still 10 TB.
Flash pools can coexist in the same cluster or on the same storage controller as Flash Cache, but blocks from
flash pools will not be included in Flash Cache caching. Flash Cache will continue to serve all non-flash-pool
aggregates on the controller.
Not only are flash pools compatible with takeover and giveback, but they also provide performance
acceleration during such events.
Regardless of whether the source aggregate is a flash pool, a volume can be moved to a flash pool aggregate.
The volume will not be immediately cached after the move, and performance may degrade slightly until the
cache is repopulated.
SnapMirror destination volumes can reside in flash pools, but the SnapMirror destination will not be cached.
When the volume is promoted to read/write, then the data from the new active file system will be cached.
Because SSD blocks can become trapped in Snapshot copies, the aggregate Snapshot feature should either be
disabled or configured with automatic deletion so that SSD blocks continue to be recycled.
Flash pools support read caching of uncompressed blocks in a compression-enabled volume, but do not yet
support caching of blocks that are compressed in the HDD tier. Compressed blocks are never write-cached.
Flash pools are supported on V-Series systems with NetApp storage but not with non-NetApp storage.

5-32 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Module Summary
Now that you have completed this module, you should be
able to:
 Draw the connections from an HA pair of controllers to the
disk shelves
 Create aggregates
 List the steps that are required to enable SFO
 Explain and enable “two-node HA” mode for two-node
clusters
 Take over and give back storage between the nodes of an
HA pair
 Enumerate the steps for expanding an aggregate from 32-
bit to 64-bit
 Create a Flash Pool
NetApp Confidential 33

MODULE SUMMARY

5-33 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Exercise
Module 5: Physical Data Storage
Time Estimate: 30 minutes

NetApp Confidential 34

EXERCISE
Please refer to your exercise guide.

5-34 Clustered Data ONTAP Administration: Physical Data Storage

NetApp University - Do Not Distribute

Module 6
Virtual Data Storage

NetApp Confidential 1

MODULE 6: VIRTUAL DATA STORAGE

6-1 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Create a volume
 Build a namespace by using multiple volumes
 Describe user and group quotas
 Configure an Infinite Volume

NetApp Confidential 2

MODULE OBJECTIVES

6-2 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Virtual Servers

 The three types of virtual servers (Vservers)

are:
– Node: represents a node
– Admin: represents a physical cluster
– Cluster: is the context in which volumes are
created
 Creating a cluster (data) Vserver causes a
root volume for that Vserver to be created.
 The root volume is the “root” of the cluster
(data) Vserver’s namespace.

NetApp Confidential 3

VIRTUAL SERVERS
A cluster (data) virtual server (Vserver) connects volumes, logical interfaces, and other elements for a
namespace. No volumes can be created until a cluster (data) Vserver exists with which to associate the
volumes.

6-3 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

The vserver show Command
Summary View
netappu::> vserver show
Admin Root Name Name
Vserver Type State Volume Aggregate Service Mapping
----------- ------- --------- ---------- ---------- ------- -------
netappu admin - - - - -
netappu-01 node - - - - -
netappu-02 node - - - - -
vs7 cluster running vs7 aggr1b file file
4 entries were displayed.

NetApp Confidential 4

THE VSERVER SHOW COMMAND: SUMMARY VIEW

Note the types of Vservers that are shown in this slide. For each node in a cluster, Data ONTAP operating in
Cluster-Mode automatically creates a node Vserver to represent the node. The administrative Vserver is
automatically created when the cluster is created. The cluster (data) Vservers are created by an administrator
to build global namespaces.

6-4 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Volumes
 Flexible volumes in Cluster-Mode are the same as
Data ONTAP 7G or 7-Mode flexible volumes.
 Any single volume can exist within only a single
cluster (data) Vserver.
 Volumes are joined together through junctions to
create the namespace of a Vserver.
 Volumes are the unit of data management: volumes
can be moved, copied, mirrored, backed up, or copied
by using Snapshot copies.
 Data ONTAP 7-Mode volumes cannot be used in
Data ONTAP Cluster-Mode systems, and Data
ONTAP Cluster-Mode volumes cannot be used in
Data ONTAP 7-Mode systems.
NetApp Confidential 5

VOLUMES
Data ONTAP Cluster-Mode flexible volumes are functionally equivalent to flexible volumes in Data ONTAP
operating in 7-Mode and the Data ONTAP 7G operating system. However, Data ONTAP Cluster-Mode
systems use flexible volumes differently than Data ONTAP 7-Mode and Data ONTAP 7G systems do.
Because Data ONTAP clusters are inherently flexible (particularly because of the volume move capability),
volumes are deployed as freely as UNIX directories and Windows folders are deployed to separate logical
groups of data.
Volumes can be created and deleted, mounted and unmounted, moved around, and backed up as needed. To
take advantage of this flexibility, Data ONTAP Cluster-Mode deployments typically use many more volumes
than traditional Data ONTAP 7G deployments use. In an HA pair, aggregate and volume limits apply to each
node individually, so the overall limit for the pair is effectively doubled.

6-5 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Volumes
Sizes and Limits
 Maximum number of volumes per node: 500
 Maximum number of volumes per 24-node
cluster: 12,000
 Minimum flexible volume size: 20 MB
 Maximum flexible volume size with 32-bit
aggregates: 16 TB
 Maximum flexible volume size with 64-bit
aggregates: 50 TB to 100 TB

NetApp Confidential 6

VOLUMES: SIZES AND LIMITS

The maximum flexible volume size with 64-bit aggregates changes depending on the storage controller model
that is used. Current models range from a 50-TB maximum to a 100-TB maximum.

6-6 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

The volume show Command
Summary View
netappu::> volume show
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
netappu-01
vol0 aggr0 online RW 851.5MB 514.8MB 39%
netappu-02
vol0 aggr0_netappu_02_0
online RW 851.5MB 587.1MB 31%
vs7 vs7 aggr1b online RW 20MB 18.88MB 5%
vs7 vs7_vol1 aggr1b online RW 400MB 379.8MB 5%
4 entries were displayed.

NetApp Confidential 7

THE VOLUME SHOW COMMAND: SUMMARY VIEW

The example on this slide shows some volumes. You can see that the Type values are all “RW,” which shows
that these volumes are read/write volumes as opposed to load-sharing mirror copies or data-protection mirror
copies.
The difference between the Size and Available values is the amount of the volume that is used, but the
difference also reflects administrative space that is used by the WAFL (Write Anywhere File Layout) file
system and the space that is reserved for Snapshot copies.

6-7 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Junctions

 Junctions connect volumes to form the

namespace of a cluster (data) Vserver.
 Junctions appear as directories in the
junctions’ parent volumes.
 When a client traverses a junction, the client
has entered a different volume.
 Snapshot copy access does not traverse
junctions.
 A volume can be mounted to the namespace
of its Vserver in exactly one place.

NetApp Confidential 8

JUNCTIONS
Junctions are conceptually similar to UNIX mountpoints. In UNIX, a hard disk can be divided into partitions,
and then those partitions can be mounted at multiple places relative to the root of the local file system,
including in a hierarchical manner. Likewise, the flexible volumes in a Data ONTAP cluster can be mounted
at junction points within other volumes to form a single namespace that is distributed throughout the cluster.
Although junctions appear as directories, junctions have the basic functionality of symbolic links.
A volume is not visible in the namespace of its Vserver until the volume is mounted within the namespace.

6-8 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

The volume show Command
Instance View 1 of 3
netappu::> volume show -vserver vs7 -volume vs7_vol1

Vserver Name: vs7

Volume Name: vs7_vol1
Aggregate Name: aggr1b
Volume Size: 400MB
Volume Data Set ID: 1026
Volume Master Data Set ID: 2147484674
Volume State: online
Volume Type: RW
Volume Style: flex
Is Cluster-Mode Volume: true
Export Policy: default
User ID: root
Group ID: daemon
Security Style: unix
UNIX Permissions: ---rwxrwxrwx

NetApp Confidential 9

THE VOLUME SHOW COMMAND: INSTANCE VIEW 1 OF 3

This is a detailed volume show command. Typing this will show a summary view of all volumes. If you
show a specific Vserver and volume, you will see the instance (detailed) view of the volume rather than the
summary list of volumes.

6-9 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

The volume show Command
Instance View 2 of 3
Junction Path: /vol1
Junction Path Source: RW_volume
Junction Active: true
Parent Volume: vs7root
Comment:
Available Size: 18.88GB
Total User-Visible Size: 19GB
Used Size: 120MB
Used Percentage: 5%
Autosize Enabled (for flexvols only): false
Maximum Autosize (for flexvols only): 23.91GB
Autosize Increment (for flexvols only): 1020KB
Total Files (for user-visible data): 566
Files Used (for user-visible data): 96
Space Guarantee Style: volume

NetApp Confidential 10

THE VOLUME SHOW COMMAND: INSTANCE VIEW 2 OF 3

6-10 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

The volume show Command
Instance View 3 of 3
Space Guarantee in Effect: true
Space Reserved for Snapshots: 5%
Snapshot Reserve Used: 63%
Snapshot Policy: default
Creation Time: Tue Oct 11 14:34:35 2011
Clone Volume: false
NVFAIL Option: off
Is File System Size Fixed: false
Extent Option: off
Reserved Space for Overwrites: 0B
Fractional Reserve: 100%
Snapshot Cloning Dependency: off
Primary Space Management Strategy: volume_grow
Read Reallocation Option: off
Block Type: 64-bit

NetApp Confidential 11

THE VOLUME SHOW COMMAND: INSTANCE VIEW 3 OF 3

6-11 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Mounting a Volume

 You should specify a junction path when you

create a volume. This action immediately puts
the volume into the namespace.
 If you create a volume without mounting the
volume to the namespace of the Vserver, the
volume is inaccessible to clients until the
volume is mounted.
 You use the volume mount and volume
unmount commands to change the junction
path of a volume in its namespace.

NetApp Confidential 12

MOUNTING A VOLUME
When volumes are created by using the volume create command, a junction path is usually specified.
The junction path is optional; a volume can be created and not mounted into the namespace. To put a volume
without a junction path into use, you must use the volume mount command to assign a junction path to the
volume.
When you unmount a volume, you take the volume out of the namespace. An unmounted volume is
inaccessible to NFS and CIFS clients but is still online and can be mirrored, backed up, moved, and so on.
You can then mount the volume again to the same location or a different location in the namespace and in
relation to other volumes. For example, you can unmount a volume from one parent volume and then mount
the volume to another parent volume.

6-12 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Volumes, Junctions, and Namespaces
1 of 3
 Volume: “root”:
– Junction path (relative to the root): /
– NFS mount command:
mount <data_ip>:/ /mnt/vserver1
– NFS path: /mnt/vserver1
 Volume: “smith”:
– Junction path: /user/smith
– “User”: a directory in the root volume in this example,
not a junction
– NFS path: /mnt/vserver1/user/smith

NOTE: The NFS mount command is different from a

cluster shell volume mount command.
NetApp Confidential 13

VOLUMES, JUNCTIONS, AND NAMESPACES: 1 OF 3

You must be able to differentiate how the volume hierarchy looks to an administrator (internally) from how
the namespace looks to an NFS or CIFS client (externally).
An administrator can choose the name of the root volume of a Vserver (and therefore, the root of the
namespace). However, the junction path of the root volume is always /. Note that the junction path for (the
mountpoint of) a volume is not tied to the name of the volume. In this example, the prefix “smith_” is used
for the volume smith_mp3 to associate it with the volume smith, but this approach is only a convention to
make the relationship between the smith volume and its mp3 volume more obvious to a cluster administrator.
Ultimately the volume smith_mp3 is mounted at a directory (media) that was created in the volume named
smith.

6-13 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Volumes, Junctions, and Namespaces
2 of 3
netappu::>
netappu::> mount
mount
netappu::>mount
netappu::> mount –vserver
–vserver
–vserver
–vserver vs1 vs1
vs1
vs1–vol
–vol –vol
–vol acct
smith
smith_mp3
smith_jpg
client% mkdir
client% /user/smith/media
mkdir /user
root
/ –junction-path /acct
/user/smith
/user/smith/media/music
–junction-path
–junction-path /user/smith/media/photos
–junction-path
user

acct /acct smith /user/smith

media

smith_mp3 smith_jpg

/user/smith/media/music /user/smith/media/photos

NetApp Confidential 14

VOLUMES, JUNCTIONS, AND NAMESPACES: 2 OF 3

This slide shows a representation of the volumes of this namespace. The volume names are shown inside the
circles, and the junction paths are listed outside of them. Note that no volume is called “user.” The “user”
entity is a directory within the root volume, and the junction for the smith volume is located in that directory.
The acct volume is mounted directly at the /acct junction path in the root volume.

6-14 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Volumes, Junctions, and Namespaces
3 of 3
Volume: “smith_mp3”:
 Junction path: /user/smith/music
 NFS path:
/mnt/vserver1/user/smith/media/music
 CIFS path (with a share that is called “root_share”):
\\<data_ip>\root_share\user\smith\media\
music

NetApp Confidential 15

VOLUMES, JUNCTIONS, AND NAMESPACES: 3 OF 3

6-15 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

User and Group Quotas

 Limit resource use

vol1
 Track resource use
 Notify users of high
resource use

qtree3
qtree2
qtree1

NetApp Confidential 16

USER AND GROUP QUOTAS

Quotas provide a way for you to restrict and track the disk space and number of files that are used by users,
groups, and qtrees. You apply quotas to specific volumes and qtrees. Data ONTAP 8.0 operating in Cluster-
Mode introduced the ability to create qtrees on Data ONTAP Cluster-Mode volumes but lacked full quota
support. Data ONTAP 8.1.1 operating in Cluster-Mode enables you to apply user and group quota rules to
qtrees. You can use quotas to:
 Limit the amount of disk space or the number of files that can be used by a user or group
 Limit the amount of disk space or the number of files that can be contained by a qtree
 Track the amount of disk space or the number of files that are used by a user, group, or qtree without
imposing a hard limit
 Warn users when their disk use or file use reaches a predefined threshold

6-16 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Quota Policies

vserver1
quota policy - unassigned
quota policy - assigned quota policy
quota rule-–-unassigned
vol1
quota policy
quota rule –unassigned
vol1
quota rule – vol1 quota
quota rule-–unassigned
policy vol1
vol2
quota rule
quota rule –– vol1
vol2
quota rule – vol2 quota rule
quota rule –– vol3
vol2
quota
quota rule –– vol3
quota rule – vol3 quotarule
quota rule –vol2
vol3
quota
quota
quota quota
quota
quota
quota
quota quota
quota
quota
quota quota

 A set of quota rules for all of the volumes of a Vserver

 No sharing among Vservers
 Up to five quota policies per Vserver:
– The Vserver enables backup copies of quota policies.
– One quota policy can be active on a Vserver at a time.

NetApp Confidential 17

QUOTA POLICIES
Quotas are defined by quota rules. Quota rules are collected in the quota policy of a Vserver and are specific
to a volume. A quota rule has no effect on the volume until the quota rule is activated.
A quota policy is a collection of quota rules for all of the volumes of a Vserver. Quota policies are not shared
among Vservers. A Vserver can have up to five quota policies, which enables you to have backup copies of
quota policies. One quota policy is assigned to a Vserver at any given time.
A quota is the actual restriction that the Data ONTAP operating system enforces, the actual tracking that the
system performs, or the actual threshold that triggers the system to send a warning message. A quota rule
always results in at least one quota and might result in many additional derived quotas.
Activation is the process of triggering the Data ONTAP operating system to create enforced quotas from the
current set of quota rules in the assigned quota policy. Activation occurs on a volume-by-volume basis. The
first time that quotas are activated on a volume is called initialization. Subsequent activation of quotas on the
same volume is called either re-initialization or resizing, depending on the scope of the changes.

6-17 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Definition
Applications Applications OnCommand System Manager 2.1 or Cluster Shell

NFSv3

The Data ONTAP 8.1.1 Operating System

Single Namespace, Single Storage Pool

Up to 2 Billion Files (16-TB Maximum File Size)

NAS Container (More than 20 PB of Raw Capacity)

Two to Ten Nodes in

a Cluster
.......

Data constituent
Namespace constituent See TR4037 for more information

NetApp Confidential 18

INFINITE VOLUMES: DEFINITION

Infinite Volumes are boundless, easily administered, scalable containers that exceed the current Data ONTAP
limits for FlexVol capacity. Infinite Volumes are ideal for customers who need a single storage pool in a
single namespace across an entire cluster. Infinite Volumes consist of many FlexVol volumes but are
presented to the storage administrator as a single manageable volume. The constituents of an Infinite
Volume are standard FlexVol volumes. Individual constituents are not managed by the storage administrator.
Although the constituents are not visible in the default displays, the administrator can choose options to view
the constituents.
Data ONTAP clustering currently supports infinite volumes on clusters of up to 10 nodes and are accessed
only by NFS version 3 (NFSv3) clients.

6-18 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Constituent View
NFSv3 access to a single external mountpoint

No NFS client access

directly to the data
constituents
Infinite Volume

Namespace Data Data Data

Constituent Constituent-1 Constituent-2 Constituent-3

Aggregate-1 Aggregate-2 Aggregate-3 Aggregate-4

Node-1 Node-2

NetApp Confidential 19

INFINITE VOLUMES: CONSTITUENT VIEW

An Infinite Volume is composed of one namespace constituent and several data constituents. The namespace
constituent contains the namespace of the Infinite Volume. The namespace constituent maps path names to
data that is stored on data constituents. A data constituent contains the data of one or more regular files and is
one of the containers that contribute to the storage capacity of an Infinite Volume. Each node can host up to
20 data constituents. Access from an NFSv3 client is made to a redirector file within the namespace
constituent. The I/O requests are routed to one of the many data constituents that are referenced by the
redirector file. Because this redirection happens transparently from the client’s perspective, NFS client
administrators are not aware of this redirection. By introducing this redirection technology, an Infinite
Volume can present a single junction namespace that can grow to over 20 petabytes.

6-19 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Redirector and Data Files

Data Con stituents

/NS/
/NS/bak/ /NS/tmp/
Data File Data File
/NS/bak/img /NS/tmp/file

Redirector
Redirector

Namespace Constituent

 Because of storage failover (SFO), access to data continues if only one

node fails.
 Directory operations are unaffected.
 File operations to other data constituents are unaffected.
 The Infinite Volume continues to be available until all constituents fail.

NetApp Confidential 20

REDIRECTOR AND DATA FILES

Infinite Volumes have two types of constituents:
 Data:
– Has multiple data constituents (usually at least one per storage node in a Data ONTAP Cluster-Mode cluster)
– Contains the data of each file
 Namespace:
– The namespace is built on a single FlexVol volume.
– The FlexVol volume contains a pointer (per file) to a file in the data constituent.
As new files are written, Infinite Volumes balance capacity across each data constituent and distribute files
across the cluster. Individual files are not striped; they are fully contained within a data constituent.

6-20 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Resiliency
The Loss of a Data Constituent

Data Constituents

/NS/
/NS/bak/ /NS/tmp/ data file
Data File
/NS/bak/img /NS/tmp/file

Redirector
Redirector

Namespace Constituent

 Because of SFO, access to data continues if just one node fails.

 Directory operations are unaffected.
 File operations to other data constituents are unaffected.
 The Infinite Volume continues to be available until all constituents fail.

NetApp Confidential 21

RESILIENCY: THE LOSS OF A DATA CONSTITUENT

When a data constituent comes offline:
 Directory operations such as mkdir are unaffected.
 File operations (lookup, read, write, getattr, and setattr) to the remaining data constituents are unaffected.
 Operations that are distributed over multiple constituents (create and readdirplus) might require more time
and yield partial results.

6-21 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Resiliency
The Loss of a Namespace Constituent

Data Constituents

Data File Data File

Namespace Constituent

 If the host of the namespace constituent and its SFO partner are both
down:
– Namespace and directory operations do not work.
– File operations on recently accessed files continue to work.
 If only the host fails and the SFO partner takes over, access to the
namespace, and the Infinite Volume functions normally.
NetApp Confidential 22

RESILIENCY: THE LOSS OF A NAMESPACE CONSTITUENT

When a namespace constituent comes offline:
 Namespace and directory operations such as mkdir do not work.
 File operations on recently accessed files (read, write, getattr, and setattr) continue to work because of the
caching of file-handle mapping in memory.

6-22 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Feature Summary
 Manageability  NFSv3 only (no CIFS)
– A single namespace  Configuration
– A simplified setup through – A single Vserver per cluster
OnCommand System Manager 2.1
– A single container of more than 20
– Management through cluster shell, PB (raw)
similar to that of a FlexVol volume
– Support for up to 2 billion files
– Constituent management with
– Up to 10 FAS6280 or V6280 system
diagnostic privilege
nodes
– Automatic capacity balancing at
file-creation time  Reliability: SFO
 Data protection  Flash Cache
– Snapshot copies  Efficiency
– SnapMirror (intercluster) software – Deduplication
– NFS-mounted tape backups – Compression

See TR4078 for Infinite Volume

recommended practices

NetApp Confidential 23

INFINITE VOLUMES: FEATURE SUMMARY

6-23 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Unsupported Features
 FlexClone technology  Quotas
 SIS clones  The FPolicy feature
 SnapLock software  CIFS
 Load-sharing mirror copies  NFSv2 and NFSv4
 SnapMover software  SAN
 Single File SnapRestore  NDMP
 SnapVault software  Hybrid aggregates
 Volume copy  Reverting and downgrades
 Shrinking volume size  FlexCache software
 Volume move*

* Infinite Volumes cannot be moved, but constituents can be moved disruptively to another constituent aggregate.

NetApp Confidential 24

INFINITE VOLUMES: UNSUPPORTED FEATURES

6-24 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Limitations
 FAS6280/V6280 and FAS3270/V3270 systems only
 Clusters of up to 10 nodes in 5 high-availability (HA) pairs
 One cluster (data) Vserver per cluster
 One Infinite Volume per Vserver
 One Infinite Volume constituent per aggregate
 Total raw capacity of approximately 21 PB in a 10-node cluster:
‒ Each aggregate has 2 RAID-DP groups; each group has 18 data and
2 parity disks; each disk is 3-TB SATA.
‒ One constituent exists per aggregate or 175 total constituents.
‒ The maximum useable capacity is approximately 13.15 PB.
 Support for up to 2 billion data files:
‒ The namespace constituent can have up to 2 billion redirector files.
‒ Each data constituent has up to 100 million data files.
 A maximum file size of 16 TB
NetApp Confidential 25

INFINITE VOLUMES: LIMITATIONS

Infinite Volumes have some limitations, depending on the size of the cluster. Infinite Volumes can be used
only on the FAS6280 and V6280 system controllers and on clusters of 2 to 10 nodes. Each aggregate can
contain only one namespace or data constituent, but up to 20 constituents can reside on each node. A cluster
(data) Vserver can contain only one Infinite Volume, and a Vserver that contains an Infinite Volume can be
the only cluster (data) Vserver in the cluster. Any attempt to create a Vserver with an Infinite Volume on a
cluster that has traditional Vservers fails.

6-25 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Infinite Volumes
Examples
netappu::> aggr create -aggregate aggr1 -diskcount 70
netappu::> aggr create -aggregate aggr2 -diskcount 70

netappu::> vserver create -vserver vs0 -rootvolume vs0_root -is-repository true ...
netappu::> volume create -vserver vs0 -volume repo_vol -junction-path /NS -size 768GB

netappu::> volume show -volume repo_vol

Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
vs0 repo_vol - online RW 768.0GB 729.3GB 5%

netappu::> volume show –is-constituent true

Vserver Volume Aggregate State Role Type Size Available Used%
-------- ------------ --------- ------- --------- ---- -------- --------- -----
vs0 repo_vol_default_data0001
aggr2 online data RW 109.7GB 104.2GB 5%
repo_vol_default_data0002
aggr3 online data RW 109.7GB 104.2GB 5%
repo_vol_default_data0003
aggr4 online data RW 109.7GB 104.2GB 5%
repo_vol_default_data0004
aggr5 online data RW 109.7GB 104.2GB 5%
repo_vol_default_data0005
aggr6 online data RW 109.7GB 104.2GB 5%
repo_vol_default_data0006
aggr7 online data RW 109.7GB 104.2GB 5%
repo_vol_ns aggr1 online namespace RW 110.0GB 104.4GB 5%
7 entries were displayed.

NetApp Confidential 26

INFINITE VOLUMES: EXAMPLES

Creating Infinite Volumes and the aggregates that host Infinite Volumes is similar to creating FlexVol
volumes. You create aggregates throughout the cluster to host constituent volumes. You create a Vserver that
can serve Infinite Volumes by using the -is-repository switch. Then you create the Infinite Volume to
fit the capacity of the constituent aggregates. The volume show command shows you the Infinite Volume.
The volume show command with -is-constituent true displays the list of constituent volumes.

6-26 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Create a volume
 Build a namespace by using multiple volumes
 Describe user and group quotas
 Configure an Infinite Volume

NetApp Confidential 27

MODULE SUMMARY

6-27 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Exercise
Module 6: Virtual Data Storage
Time Estimate: 30 minutes

NetApp Confidential 28

EXERCISE
Please refer to your exercise guide.

6-28 Clustered Data ONTAP Administration: Virtual Data Storage

NetApp University - Do Not Distribute

Module 7
Physical Networking

NetApp Confidential 1

MODULE 7: PHYSICAL NETWORKING

7-1 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Draw the connections of the network cables
from the three networks to a controller
 Explain port roles
 Create an interface group and a VLAN
 Discuss switch requirements for the cluster
and management networks

NetApp Confidential 2

MODULE OBJECTIVES

7-2 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The Path of a (Remote) Write Request
Node Node
Requests
N-Blade and N-Blade and
SCSI-Blade SCSI-Blade
Responses
NAS and SAN
Clients
Cluster
Session
Manager
CSM
(CSM)

D-Blade D-Blade

NetApp Confidential 3

THE PATH OF A (REMOTE) WRITE REQUEST

This slide shows the path of a write request. This module focuses on the section of the path between the
clients (NAS and SAN) and the corresponding networking protocol blade (the N-blade for NAS protocols and
the SCSI-blade for SAN protocols).

7-3 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Network Ports
 Physical network ports exist on a controller, with
corresponding network port definitions in the Data
ONTAP operating system:
– Node-management ports (by default, one for each
node)
– Cluster ports (by default, two for each node)
– Data ports (by default, two for each node)
– Intercluster ports (by default, none)
 The defaults might not be the optimal configuration for
your particular installation.
 FC SAN environments use host bus adapter (HBA)
ports as data ports.
NetApp Confidential 4

NETWORK PORTS
Data ONTAP operating in Cluster-Mode distinguishes between physical network ports and logical interfaces
(LIFs). Each port has a role that is associated with the port by default, although that situation can be changed
through the UI. The role of each network port should align with the network to which the port is connected.
Node-management ports are for administrators to connect to the node or cluster, for example, through Secure
Shell (SSH) or a web browser.
Cluster ports are strictly for intracluster traffic.
Data ports are for NAS and SAN client access and for the cluster management LIF.
Intercluster ports are used to communicate with another cluster.

7-4 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The Network Ports of a Node

FC 0c Management e0M Data Cluster X1107A Dual-

e1b and e1a Port 10-GbE Card
Data e0a e2b

X1107A

LINK/ACT

LINK/ACT
Chelsio
Communications

c0a 0c e0a

X1107A

LINK/ACT

LINK/ACT
0a 0b Chelsio
Communications

LNK LNK

c0b 0d e0b

X1139A

10GbEE CNA
3 5

SAN LAN

SAN LAN
PORT 1

PORT 2
4 6

Data e3a and e3b FC 0d Data e0b Console Cluster

e2a
X1139A Dual-
Port UTA Card

NetApp Confidential 5

THE NETWORK PORTS OF A NODE

Using the FAS3270 system as an example, the e1a and e2a ports are defined as having the role of a cluster.
The e0a, e0b, e1b, e2b, e3a, and e3b ports are defined for data. The e1a and e1b ports are on a network
interface card (NIC) in slot 1 at the top right of the controller. The e2 and e3 ports are also found on NICs.
The e0M port is, by default, defined with the role of management. In an FC SAN environment, the FC ports
can also assume the role of serving data.
New to Data ONTAP operating in Cluster-Mode is the ability to create intercluster data-protection mirror
relationships. Intercluster (or cross-cluster) mirror relationships are created over ports that have the role of
intercluster or data. Intercluster port roles are similar to data port roles, except that intercluster port roles are
reserved for intercluster traffic. Intercluster ports can fail over to data ports.

7-5 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The Roles of Network Ports

 “Cluster” is used for communication to the

private cluster network.
 “Data” is used to communicate with client
hosts.
 “Node-Management” is used by
administrators to configure the node.
 “Intercluster” is used for communication to a
peer cluster.

NetApp Confidential 6

THE ROLES OF NETWORK PORTS

Default port roles vary by platform but can be reassigned by using the network port modify command.
Exactly two cluster ports are required per node.

7-6 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The network port show Command
netappu::> net port show
(network port show)
Auto-Negot Duplex Speed (Mbps)
Node Port Role Link MTU Admin/Oper Admin/Oper Admin/Oper
------ ------ ------------ ---- ----- ----------- ---------- ------------
netappu-01
e0a cluster up 9000 true/true full/full auto/1000
e0b cluster up 9000 true/true full/full auto/1000
e0c data up 1500 true/true full/full auto/1000
e0d data up 1500 true/true full/full auto/1000
e1a node-mgmt up 1500 true/true full/full auto/1000
e1b data down 1500 true/true full/half auto/10
netappu-02
e0a cluster up 9000 true/true full/full auto/1000
e0b cluster up 9000 true/true full/full auto/1000
e0c data up 1500 true/true full/full auto/1000
e0d data up 1500 true/true full/full auto/1000
e1a node-mgmt up 1500 true/true full/full auto/1000
e1b data down 1500 true/true full/half auto/10
12 entries were displayed.

NetApp Confidential 7

THE NETWORK PORT SHOW COMMAND

The network port show command shows the summary view of the ports of this two-node cluster. All
the ports are grouped by node. You can see the roles that are assigned to the ports and the ports’ statuses and
maximum transmission unit (MTU) sizes. Note that the e1b data ports are on the nodes but are not connected.

7-7 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The network fcp adapter show
Command
netappu::> network fcp adapter show
Connection Host
Node Adapter Established Port Address
------------ ------- ----------- ------------
netappu-01 0c ptp 4b0038
netappu-01 3a ptp 4b0036
netappu-01 3b loop 0
netappu-01 4a ptp 4b0037
netappu-01 4b loop 0
netappu-02 0c ptp 4b0061
netappu-02 3a ptp 4b0060
netappu-02 3b loop 0
netappu-02 4a ptp 4b005f
netappu-02 4b loop 0
12 entries were displayed.

NetApp Confidential 8

THE NETWORK FCP ADAPTER SHOW COMMAND

The network fcp adapter show command shows the summary view of the FC ports of this two-node
cluster. All the ports are grouped by node. You can see the port addresses that are assigned to the ports and
the ports’ connection types that are established. In an FC SAN environment, these ports can assume a data
port role.

7-8 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The Interface Group (ifgrp)

 Is the same as a virtual interface (vif) in the

Data ONTAP 7G operating system
 Is the same as a port trunk in the Data
ONTAP GX operating system
 Combines two or more physical ports and
presents them as a single logical port
 Provides redundancy and more bandwidth

NetApp Confidential 9

THE INTERFACE GROUP (IFGRP)

An interface group (ifgrp) combines multiple physical ports into a single logical port.
An interface group (ifgrp) is also known as an Ethernet trunk, 802.3ad link aggregation, or port bonding. This
term is the Data ONTAP 8.1.1 name for what was called a virtual interface (vif) in Data ONTAP 7G
architecture and a port trunk in Data ONTAP GX architecture. The N-blade groups multiple physical ports
and makes them available as a single port. The grouping is configured as either an active port and a standby
port (active-passive or single-mode) or with all of the ports in the ifgrp sharing the load (active-active or
multimode). When using multimode, you can activate the Link Aggregation Control Protocol (LACP) to
negotiate port groupings with the switch. The switch must be configured to support link aggregation.

7-9 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Virtual LANs (VLANs)

NetApp Confidential 10

VIRTUAL LANS (VLANS)

A port may be subdivided into multiple virtual LANs, or VLANs. Each VLAN has a unique tag that is
communicated in the header of every packet. The switch must be configured to support VLANs and the tags
that are in use. A VLAN's ID is configured in cluster mode into the name. So vlan "e0a-25" is a VLAN with
tag 25 configured on physical port e0a. VLANs sharing a base port may belong to the same or different IP
spaces, and it follows that the base port could be in a different IP space than its VLANs.

7-10 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

VLANs and Interface Groups

vlan vlan ifgrp

port port port

vlan vlan

ifgrp

port port

NetApp Confidential 11

VLANS AND INTERFACE GROUPS

Ports are the physical network adapters that are cabled to a switch or a virtualization that groups or subdivides
the physical ports. Ports can be arranged in a hierarchy, with VLANs subdividing ifgrps. In Data ONTAP
operating in Cluster-Mode, ifgrps cannot be created from other ifgrps, and VLANs cannot be created from
other VLANs. Additionally, ifgrps cannot be created from VLANs.
NOTE: You cannot create VLANs or interface groups on cluster ports.

7-11 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Cluster Network Standardization
 Approach
– This configuration is standard for cluster interconnect
switches in Data ONTAP Cluster-Mode configurations.
– New clusters require the standard switch configurations for
the cluster and management network.
 Benefits
– This solution is engineered by NetApp.
– Using this solution guarantees that best practices for
networking design are followed:
 Dual cluster interconnect switches for redundancy
 Sufficient ISL bandwidth
 Standard hardware, software, and configurations
 Faster problem resolution (using “known” configurations)

NetApp Confidential 12

CLUSTER NETWORK STANDARDIZATION

To provide a complete storage solution, the Data ONTAP 8 operating system introduced a cluster
interconnect and management switch-configuration standard to which all Data ONTAP Cluster-Mode
deployments are required to adhere. Data ONTAP 8.1.1 operating in Cluster-Mode continues this policy to
provide the best solution in terms of reliability and serviceability.

7-12 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

The NetApp Cluster Interconnect and
Optional Cluster Management Switch
What’s New
 A lower-cost solution for eight-node or smaller cluster sizes
 Support that starts in the Data ONTAP 8.1.1 operating system
 Sixteen ports of 10-GbE cluster or GbE management

Cluster Network: CN1610 Management Network: CN1601

 Low-latency, nonblocking  Sixteen ports GbE RJ-45
sixteen 10-GbE enhanced small  Dual fans; a single power
form-factor pluggable ( SFP+) supply
Ethernet ports
 Dual fans and power supplies

NetApp Confidential 13

THE NETAPP CLUSTER INTERCONNECT AND OPTIONAL CLUSTER

MANAGEMENT SWITCH

7-13 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Cluster Interconnect Requirements
 Cluster interconnect
– NetApp CN1610
– Cisco Nexus 5010 and Cisco Nexus 5020
– Cisco Nexus 5596 (New with Data ONTAP 8.1.2)
– Wire-rate 10-GbE connectivity between storage controllers
 A 1 x 10-GbE connection from each node to each switch
(2 ports per node total)
 Interswitch bandwidth: four ports per switch with CN1610; eight
ports per switch on Cisco Nexus 5010 and 5020
 Cluster management switch for:
– Management connections for storage controllers and shelves
– NetApp CN1601
– Cisco Catalyst 2960

NetApp Confidential 14

CLUSTER INTERCONNECT REQUIREMENTS

The FAS2040 system connects into a cluster by using onboard 1-GbE ports. The first 8 ports of the Cisco
Nexus 5010 and the first 16 ports of the Cisco Nexus 5020 can be 1-GbE or 10-GbE, depending on the small
form-factor pluggable (SFP) that is used. NetApp has released a new 1-GbE SFP to enable the FAS2040
system to participate in clusters. All other controllers remain at 10-GbE. The best practice is to not mix 1G
and 10G nodes.
A Data ONTAP 8.1 Cluster-Mode cluster that uses Nexus 5010 switches for the cluster network can have a
maximum of eight FAS2040 system nodes in the cluster.
A Data ONTAP 8.1 Cluster-Mode cluster that uses Nexus 5020 switches for the cluster network can have a
maximum of 16 FAS2040 system nodes in the cluster.
The NetApp CN1610 and Cisco Nexus 5596 will replace the 5010 and 5020 respectively.

7-14 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Cluster Configuration Overview

2 to 8 Nodes 2 to 18 Nodes 2 to 24 Nodes

 2 NetApp CN1610  2 Cisco Nexus 5010  2 Cisco Nexus 5020
– 16 x 10-Gbps – 20 x 10-GbE ports: – 40 x 10-GbE ports:
Ethernet ports Eight ports are used Eight ports are used
enhanced (SFP+): for ISLs. for ISLs.
Four ports are used – 1 rack unit each – 2 rack units each
for Inter-Switch Links
– An expansion
(ISLs).
module (required for
– 1 rack unit each 12–18 nodes): 1
module (8 x 10-GbE)
 2 NetApp CN1601  2 Cisco Catalyst  2 Cisco Catalyst
– 16 ports of 1-Gbps 2960 2960
Ethernet RJ-45 – 24 ports of 10/100 – 24 ports of 10/100
– 1 rack unit each Ethernet RJ-45 Ethernet RJ-45
– 1 rack unit each – 1 rack unit each

NetApp Confidential 15

CLUSTER CONFIGURATION OVERVIEW

Customers who have their own management networks can use their preferred switches for the management
network. For details, refer to the Compatibility Matrix:
[Link]

7-15 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Configuration Overview
Ability to Be Supported
Maximum
Function Switch Configured in a Network Interface
Nodes
NetApp Cabinet Cards (NICs)
Cluster X1117A-R6 X1107A-
NetApp CN1610 8 Yes R6 X1008A-R6
interconnect
Cluster X1117A-R6 X1107A-
Cisco Nexus 5010 12 No R6 X1008A-R6
interconnect
Cisco Nexus 5010 X1117A-R6
Cluster X1107A-R6
with an expansion 18 No
interconnect X1008A-R6
module
X1117A-R6
Cluster
Cisco Nexus 5020 24 No X1107A-R6
interconnect X1008A-R6
Management NetApp CN1601 16 Yes
Cisco Catalyst
Management 2–24 No
2960-24TT

NetApp Confidential 16

CONFIGURATION OVERVIEW

7-16 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Network Cabling
Cluster Interconect
Cisco Nexus 5010

SLOT2
STAT

PS1
PS2
CONSOLE
L1 L2 MGMTO MGMT1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 10 19 20

Cisco Nexus 5010

SLOT2
STAT

PS1
PS2
CONSOLE
L1 L2 MGMTO MGMT1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 10 19 20

node1 node2 node3 node4

1 1 1
X1107A

X1107A

X1107A
LINK/ACT

LINK/ACT

LINK/ACT
Chelsio Chelsio Chelsio
1

X1107A

LINK/ACT

LINK/ACT
Communications Communications Communications Chelsio
Communications
c0a 0c e0a c0a 0c e0a c0a 0c e0a c0a 0c e0a

2 2 2
X1107A

X1107A

X1107A
0a 0b
2
LINK/ACT

LINK/ACT

0a 0b

LINK/ACT

LINK/ACT
0a 0b

LINK/ACT

LINK/ACT
Chelsio Chelsio Chelsio

X1107A
0a 0b

LINK/ACT

LINK/ACT
Communications Communications Communications Chelsio
Communications
LNK LNK LNK LNK LNK LNK
LNK LNK

c0b 0d e0b c0b 0d e0b c0b 0d e0b c0b 0d e0b

3 5 3 5 3 5 3 5

4 6 4 6 4 6 4 6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Catalyst 2960-S Series PoE+ 10G

Cisco Nexus 5010

CONSOLE M
G

SLOT2
M
T
SYST 1X 11X 13X 23X B
A
RPS
STAT
S

PS1
E
MSTR T

STAT
DPLX
SPED
STCK
PoE

PS2
2X 12X 14X 24X
MODE CONSOLE
1 SFP+ 2 L1 L2 MGMTO MGMT1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 10 19 20

Management Data Network

Network

NOTE: NetApp recommends switch redundancy for both data and

management networks.
NetApp Confidential 17

NETWORK CABLING
This slide shows a four-node cluster. Typically, two distinct networks exist for a cluster. The cluster traffic
must always be on its own network, but the management and data traffic can coexist on a network.
Two cluster connections to each node are required for redundancy and improved cluster traffic flow.
For proper configuration of the NetApp CN1601 and CN1610 switches, refer to the CN1601 and CN1610
Switch Setup and Configuration Guide.

7-17 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Draw the connections of the network cables
from the three networks to a controller
 Explain port roles
 Create an interface group and a VLAN
 Discuss switch requirements for the cluster
and management networks

NetApp Confidential 18

MODULE SUMMARY

7-18 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Exercise
Module 7: Physical Networking
Time Estimate: 20 minutes

NetApp Confidential 19

EXERCISE
Please refer to your exercise guide.

7-19 Clustered Data ONTAP Administration: Physical Networking

NetApp University - Do Not Distribute

Module 8
Virtual Networking

NetApp Confidential 1

MODULE 8: VIRTUAL NETWORKING

8-1 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Create data logical interfaces (LIFs) for each
cluster virtual server (Vserver)
 Create static routes for the data LIFs
 Migrate and revert a NAS data LIF
 Describe LIF failover groups

NetApp Confidential 2

MODULE OBJECTIVES

8-2 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

LIF Characteristics
 An IP address or World Wide Port Name (WWPN) is
associated with a LIF.
 One node-management LIF exists per node (can fail over
to other data or node-management ports on the same
node).
 One cluster-management LIF exists per cluster (can fail
over or migrate throughout the cluster).
 Two cluster LIFs exist per node (can fail over or migrate
only within their node).
 Multiple data LIFs are allowed per data port.
– Are client-facing (NFS, CIFS, iSCSI, and Fibre Channel
access)
– NAS data LIFs can migrate or fail over throughout the cluster

NetApp Confidential 3

LIF CHARACTERISTICS
Each logical interface (LIF) has an associated role and must be assigned to the correct type of network port.
Data LIFs can have a many-to-one relationship with network ports: many data IP addresses can be assigned to
a single network port. If the port becomes overburdened, NAS data LIFs can be transparently migrated to
different ports or different nodes. Clients know the data LIF IP address but do not know which node or port is
hosting the LIF. If a NAS data LIF is migrated, the client might unknowingly be contacting a different node.
The NFS mountpoint or CIFS share is unchanged.
A node can have a maximum of 128 LIFs, regardless of the type of LIF.

8-3 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network interface show Command
1 of 2
netappu::> net int show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
netappu
cluster_mgmt up/up [Link]/24 netappu-01 e0M true
netappu-01
clus1 up/up [Link]/16 netappu-01 e1a true
clus2 up/up [Link]/16 netappu-01 e2a true
mgmt1 up/up [Link]/24 netappu-01 e0a true
netappu-02
clus1 up/up [Link]/16 netappu-02 e1a true
clus2 up/up [Link]/16 netappu-02 e2a true
mgmt1 up/up [Link]/24 netappu-02 e0a true
vs7
vs7_lif1 up/up [Link]/24 netappu-01 e3a true
vs7_lif2 up/up [Link]/24 netappu-01 e3b false

NetApp Confidential 4

THE NETWORK INTERFACE SHOW COMMAND: 1 OF 2

LIF names must be unique within their scope. For data LIFs, the scope is a cluster virtual server (Vserver).
For the cluster and management LIFs, the scopes are limited to their nodes. Thus, the same name (for
example, mgmt1) can be used for all the nodes.

8-4 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network interface show Command
2 of 2
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
vs7
vs7_lif1 up/up [Link]/24 netappu-01 e3a true
vs7_lif2 up/up [Link]/24 netappu-01 e3b false
vs7_fclif1 up/up 20:0f:00:a0:98:13:d5:d4
netappu-01 0c true
vs7_fclif2 up/up 20:10:00:a0:98:13:d5:d4
netappu-01 0d true
vs7_fclif3 up/up 20:14:00:a0:98:13:d5:d4
netappu-02 0c true
vs7_fclif4 up/up 20:12:00:a0:98:13:d5:d4
netappu-02 0d true
13 entries were displayed.

NetApp Confidential 5

THE NETWORK INTERFACE SHOW COMMAND: 2 OF 2

Instead of IP addresses, FC LIFs use worldwide port names (WWPNs).

8-5 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Network Ports and Data LIFs
[Link] (vs1_d2)
[Link] (vs1_d1)
[Link] (vs2_d2)
[Link] (vs2_d1)
[Link] (vs2_d3)

21:00:00:2b:34:26:a6:54 (vs1_d4)
[Link] (vs1_d3)
[Link] (vs3_d1)

e0c e0d e0c 0f

node1 node2

NetApp Confidential 6

NETWORK PORTS AND DATA LIFS

In the environment that is shown on this slide, each of the nodes contains two data network ports. Network
port node2 has three data LIFs that are assigned to one port and two on the other port. This slide shows the
many-to-one relationship between LIFs and network ports. The data LIF name is in parentheses after each
network address.
In a NAS environment, the name is not the actual host name that is associated with the IP address. The name
is an internal name that can be used as the host name for the IP address in the Domain Name System (DNS).
In a NAS environment, all these IP addresses can share one host name such that a DNS round robin is used
and picks an IP address every time that the host name is used, for example, for an NFS mount command.
This slide shows how an environment can randomly distribute client connections across a cluster while the
cluster looks to every user and every client as if there is only one storage host.

8-6 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

LIF Roles and Compatible Ports

LIF Roles Compatible Port Roles

Cluster Cluster
Data Data
Node-management Node-management or
Data
Cluster-management Data
Intercluster Intercluster or
Data

NetApp Confidential 7

LIF ROLES AND COMPATIBLE PORTS

8-7 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

LIF Static Routes

 Are defined paths between LIFs and specific

destination IP addresses through gateways
 Can improve the efficiency of network traffic
that travels through complicated networks
 Have preferences that are associated with
them: When multiple routes are available, the
“metric” specifies the preference order of the
route to use
 Are defined within “routing groups”
 Will be created or chosen automatically when
a LIF is created.
NetApp Confidential 8

LIF STATIC ROUTES

You can control how LIFs in a Vserver use your network for outbound traffic by configuring routing groups
and static routes. A set of common routes are grouped in a routing group that simplifies the administration of
routes.
 A routing group is a routing table in which each LIF is associated with one routing group and uses only
the routes of that group. Multiple LIFs can share a routing group.
NOTE: If, for purposes of backward compatibility, you want one route per LIF, you can create a separate
routing group for each LIF.
 A static route is a defined route between a LIF and a specific destination IP address. The route can use a
gateway IP address.
Routing groups are created automatically as new LIFs are created, unless a routing group covers that port and
role or network combination.
The node-management LIFs on each node have static routes that are automatically set up for them through the
same default gateway.
If more than one static route is defined for a LIF, each static route has a “metric” value. The administrator
uses the metric values to configure routes so that one is preferred over another. The lower the metric value,
the more preferred the route is. The metric value for a node management LIF is 10. When routes are created
for data LIFs, if no metric is defined, the default value is 20.

8-8 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network routing-groups show
Command 1 of 2
netappu::> network routing-groups show
Routing
Vserver Group Subnet Role Metric
--------- --------- --------------- ------------ -------
netappu
c192.168.81.0/24
[Link]/24 cluster-mgmt 20
netappu-01
c169.254.0.0/16
[Link]/16 cluster 30
i192.168.81.0/24
[Link]/24 intercluster 40
n192.168.81.0/24
[Link]/24 node-mgmt 10

NetApp Confidential 9

THE NETWORK ROUTING-GROUPS SHOW COMMAND: 1 OF 2

8-9 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network routing-groups show
Command 2 of 2
Routing
Vserver Group Subnet Role Metric
--------- --------- --------------- ------------ -------
netappu-02
c169.254.0.0/16
[Link]/16 cluster 30
i192.168.81.0/24
[Link]/24 intercluster 40
n192.168.81.0/24
[Link]/24 node-mgmt 10
vs1
d192.168.81.0/24
[Link]/24 data 20
vs2
d192.168.81.0/24
[Link]/24 data 20
9 entries were displayed.

NetApp Confidential 10

THE NETWORK ROUTING-GROUPS SHOW COMMAND: 2 OF 2

8-10 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network routing-groups
route show Command
netappu::> network routing-groups route show
Routing
Vserver Group Destination Gateway Metric
--------- --------- --------------- --------------- ------
netappu
c192.168.81.0/24
[Link]/0 [Link] 20
netappu-01
n192.168.81.0/24
[Link]/0 [Link] 10
netappu-02
n192.168.81.0/24
[Link]/0 [Link] 10
vs1
d192.168.81.0/24
[Link]/0 [Link] 20
vs2
...
5 entries were displayed.
NetApp Confidential 11

THE NETWORK ROUTING-GROUPS ROUTE SHOW COMMAND

As with the network interface show output command, node-management LIFs have a server that
is the node itself. The data LIFs are associated with a cluster (data) Vserver that the data LIFs are grouped
under.

8-11 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

NAS Data LIF Failover and Migration

 “LIF failover” means automatic failover that

occurs because of an outage or reboot.
 “LIF migrate” means manual migration: The
node is not rebooted during manual migration.
 “LIF revert” means manually or automatically
sending a LIF back to its home
(node and port).
 LIF failover is controlled by the failover group.

NetApp Confidential 12

NAS DATA LIF FAILOVER AND MIGRATION

Why migrate a NAS data LIF? It might be needed for troubleshooting a faulty port or to offload a node whose
data network ports are being saturated with other traffic. The LIF fails over if its current node is rebooted.
Unlike storage failover (SFO), LIF failover or migration does not cause a reboot of the node from which the
LIF is migrating. Also unlike SFO, LIFs can migrate to any node in the cluster, not just within the high-
availability (HA) pair. After a LIF is migrated, the LIF can remain on the new node for as long as the
administrator wants.

8-12 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

NAS Data LIF Failover and Migration
Limits
 Node-management LIFs cannot fail over or
migrate to a port on a different node.
 Cluster-management LIFs and NAS data LIFs
can fail over and migrate across ports and
nodes.
 Cluster LIFs can fail over and migrate only
across ports on the same node.
 Data LIFs are bound to a Vserver and do not
fail over or migrate between Vservers.
 SAN data LIFs never fail over or migrate.

NetApp Confidential 13

NAS DATA LIF FAILOVER AND MIGRATION LIMITS

Data LIFs aren’t permanently tied to their home ports. However, the port to which a LIF is migrating is tied to
a node. This example shows the line between the physical and the logical. Also, a port has a node Vserver
scope; a data LIF has a cluster (data) Vserver scope.

8-13 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

LIF Failover Groups

data1
e0c e0d e0c e0d e0c e0d e0c e0d

NetApp Confidential 14

LIF FAILOVER GROUPS

“LIF failover” means the automatic migration of a NAS data LIF in response to a link failure on the LIF’s
current network port. When such a port failure is detected, the LIF is migrated to a working port. A failover
group contains a set of network ports (physical, VLANs, and interface groups) on one or more nodes. A LIF
can subscribe to a failover group. The network ports that are present in the failover group define the failover
targets for the LIF. You can manage failover groups by adding ports to them, removing ports from them,
renaming them, and displaying information about them.

8-14 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Types of Failover Groups

 System-defined
 User-defined
 Clusterwide

netappu::> net int failover-groups create –failover-group

customfailover1 –node netappu-02 –port e0d

netappu::> net int modify –vserver_name vs2 –lif data1

–use-failover-group enabled –failover-group customfailover1

NetApp Confidential 15

TYPES OF FAILOVER GROUPS

Failover groups for LIFs can be system-defined or user-defined. Additionally, one failover group that is called
“clusterwide” is maintained automatically.
Failover groups are of the following types:
 System-defined failover groups: These failover groups automatically manage LIF failover targets on a
per-LIF basis and contain data ports from a maximum of two nodes. The data ports include all the data
ports on the home node and all the data ports on another node in the cluster for redundancy.
 User-defined failover groups: These customized failover groups can be created when the system-defined
failover groups do not meet your requirements. For example, you can create a failover group that consists
of 10-GbE ports and enables LIFs to fail over only to the high-bandwidth ports.
 The clusterwide failover group: This failover group consists of all the data ports in the cluster and
defines the default failover group for the cluster management LIF.

8-15 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

The network interface show Command
netappu::> net int show -vserver vs2 -lif vs2_lif1
Vserver Name: vs2
Logical Interface Name: vs2_lif1
Role: data
Data Protocol: nfs, cifs
Home Node: netappu-02
Home Port: e0d
Current Node: netappu-02
Current Port: e0d
Operational Status: up
Extended Status: -
Is Home: true
Network Address: [Link]
Netmask: [Link]
IPv4 Link Local: -
Bits in the Netmask: 24
Routing Group Name: d192.168.81.0/24
Administrative Status: up
Failover Policy: nextavail
Firewall Policy: data
Auto Revert: false
Use Failover Group: -
Fully Qualified DNS Zone Name: none
Failover Group Name: customfailover1

NetApp Confidential 16

NetApp University - Do Not Distribute

The network interface failover-group
show Command
netappu::> net int failover-groups show
(network interface failover-groups show)
Failover
Group Node Port
------------------- ----------------- ----------
clusterwide
netappu-02 e0c
netappu-02 e0d
netappu-02 e0e
netappu-01 a0a
netappu-01 e0c
customfailover1
netappu-02 e0c
netappu-01 e0c

NetApp Confidential 17

THE NETWORK INTERFACE FAILOVER-GROUP SHOW COMMAND

8-17 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

DNS Load-Balancing Characteristics
 Is built into the N-blade and is optional
 Uses internal “Domain Name System (DNS) zones”
that contain multiple data IP addresses (data LIFs):
– The actual data LIF that is used for an NFS mount is
chosen at NFS mount time.
– NAS data LIFs can be automatically migrated among
nodes to maintain a balanced load.
 Is based on LIF weights:
Weight can be manually or automatically set (based
on the current load in the cluster).
 Provides balanced cluster-wide LIF and N-blade use

NetApp Confidential 18

DNS LOAD-BALANCING CHARACTERISTICS

With DNS load balancing enabled, a storage administrator can choose to enable the new built-in load balancer
to balance the client LIF network access on the basis of the load of the cluster. This DNS server resolves
names to LIFs based on the weight of a LIF. A Vserver can be associated with a DNS load-balancing zone,
and LIFs can be created or modified to be associated with a specific DNS zone. A fully-qualified domain
name can be added to a LIF to create a DNS load-balancing zone by specifying a “dns-zone” parameter in the
network interface create command.
Two methods can be used to specify the weight of a LIF: The storage administrator can specify a LIF weight,
or the LIF weight can be generated based on the load of the cluster. Ultimately, this feature helps to balance
the overall use of the cluster. This feature does not increase the performance of any one individual node;
rather, this feature guarantees that each node is used more evenly. The result is better performance use from
the entire cluster.
DNS load balancing also improves the simplicity of maintaining the cluster. Instead of manually determining
which LIFs are used when mounting a specific global namespace, the administrator can let the system
dynamically decide which LIF is the most appropriate. And after a LIF is chosen, that LIF can be migrated to
a different node automatically to guarantee that the network load remains balanced throughout the cluster.

8-18 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

DNS Load Balancing

Data

Data Network

See KB1012400 for more information

NetApp Confidential 19

DNS LOAD BALANCING

8-19 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Create data LIFs for each cluster (data)
Vserver
 Create static routes for the data LIFs
 Migrate and revert a NAS data LIF
 Describe LIF failover groups

NetApp Confidential 20

MODULE SUMMARY

8-20 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Exercise
Module 8: Virtual Networking
Time Estimate: 30 minutes

NetApp Confidential 21

EXERCISE
Please refer to your exercise guide.

8-21 Clustered Data ONTAP Administration: Virtual Networking

NetApp University - Do Not Distribute

Module 9
NAS Protocols

NetApp Confidential 1

MODULE 9: NAS PROTOCOLS

9-1 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Create an export policy for a cluster (data)
virtual server (Vserver)
 Create an NFS configuration for a cluster
(data) Vserver
 Create a CIFS domain, server, and share
 Mount exports and shares to NFS and CIFS
clients
 Create name mapping rules between NFS
and CIFS

NetApp Confidential 2

MODULE OBJECTIVES

9-2 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

9-3 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFS and CIFS
 NFS
– NFS was developed by Sun Microsystems.
– NFS is the standard for UNIX clients.
– Servers “export”; clients “mount.”
 CIFS
– CIFS was developed by Microsoft.
– CIFS is the standard for Windows clients.
– Servers “share”; clients “use” or “map.”
 NFS and CIFS are configured independently of each
other.
 NFS and CIFS are configured on a per-Vserver basis.

NetApp Confidential 4

NFS AND CIFS

NFS is the standard network file system protocol for UNIX clients while CIFS is the standard network file
system for Windows clients. Macintosh clients can use either NFS or CIFS.
The terminology is slightly different between the two protocols:
 NFS servers “export” their data and the NFS clients “mount” the exports.
 CIFS servers “share” their data and the CIFS clients “use” or “map” the shares.

9-4 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Review the Roles of the Components

 The N-blade on each node does the

translation between the NFS and CIFS
protocols and the Spin network protocol
(SpinNP).
 The cluster session manager (CSM) on each
node understands SpinNP.
 The D-blade on each node understands
SpinNP and reads from and writes to its disks.
 The M-host on each node communicates with
and controls the cluster and nodes.

NetApp Confidential 5

REVIEW THE ROLES OF THE COMPONENTS

The component of the Data ONTAP Cluster-Mode operating system that understands the NFS and CIFS
protocols is the N-blade. The cluster session manager (CSM) understands Spin network protocol (SpinNP), so
the N-blade is a translator between NFS and CIFS and SpinNP.

9-5 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Cluster (data) Vservers

 Cluster (data) Vservers are configured with

name service switches (nis, file, and
ldap).
 If you use Network Information Service (NIS),
a NIS domain must be associated with this
Vserver.
 If you use Lightweight Directory Access
Protocol (LDAP), an LDAP domain must be
associated with this Vserver.

NetApp Confidential 6

CLUSTER (DATA) VSERVERS

The name service switch is assigned at the level of the cluster virtual server (Vserver). Network Information
Service (NIS) and Lightweight Directory Access Protocol (LDAP) domain configurations are likewise
associated at the cluster (data) Vserver level.

9-6 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NIS Domains

 NIS domains must already exist outside of

Data ONTAP architecture.
 Multiple NIS domain configurations can exist
for a Vserver, but only one can be active at
a time.
 Use the vserver services nis-domain
command to view and modify NIS settings in
a Vserver.

NetApp Confidential 7

NIS DOMAINS
The NIS domain is not created within a Data ONTAP cluster. The NIS domain must already exist, and then
configurations can be created to associate the domain with cluster (data) Vservers within Data ONTAP
operating in Cluster-Mode.
Multiple configurations can be created within a Vserver and for multiple Vservers. Any or all of those
configurations can use the same NIS domain or different ones. Only one NIS domain configuration can be
active for a Vserver at one time.
Multiple NIS servers can be specified for an NIS domain configuration when the configuration is created.
Additional servers can be added to the configuration later.

9-7 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

LDAP Domains

 LDAP domains must already exist outside of

Data ONTAP architecture.
 Multiple LDAP domain configurations can
exist for a Vserver, but only one can be active
at a time.
 Use the vserver services ldap
command to view and modify LDAP settings
in a Vserver.

NetApp Confidential 8

LDAP DOMAINS
The LDAP domain is not created within a Data ONTAP cluster. The LDAP domain must already exist, and
then configurations can be created to associate the domain with cluster (data) Vservers within Data ONTAP
operating in Cluster-Mode.
Multiple configurations can be created within a Vserver and for multiple Vservers. Any or all of those
configurations can use the same LDAP domain or different ones. Only one LDAP domain configuration can
be active for a Vserver at one time.

9-8 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Kerberos

 Kerberos can be used with NFS and CIFS.

 Microsoft Active Directory Kerberos and MIT
Kerberos are supported.

NetApp Confidential 9

KERBEROS

9-9 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Kerberos Realms

 A Kerberos realm must already exist outside

of Data ONTAP architecture.
 Kerberos realms can be used by any NFS
configurations.
 CIFS does not require a separate Kerberos
realm definition.
 Use the vserver services kerberos-
realm command to view and modify the
Kerberos settings in a Vserver.

NetApp Confidential 10

KERBEROS REALMS
The Kerberos realm is not created within a Data ONTAP cluster. The Kerberos realm must already exist, and
then configurations can be created to associate the realm for use within the cluster.
Multiple configurations can be created. Each of those configurations must use a unique Kerberos realm.

9-10 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Export Policies and Rules

 Export policies and rules enable the

administrator to restrict access to volumes
based on client IP address and authentication
types.
 Each volume has an export policy. (default
is the default).
 Export policies only apply at the volume
(not qtree) level.

NetApp Confidential 11

EXPORT POLICIES AND RULES

Each volume has an export policy that is associated with it. Each policy can have rules that govern the access
to the volume based on criteria such as a client’s IP address or network, the protocol that is used (NFS,
NFSv2, NFSv3, NFSv4, CIFS, or any), and more. By default, a “default” export policy exists, which contains
no rules.
Each export policy is associated with one cluster (data) Vserver. An export policy name need only be unique
within a Vserver. When a Vserver is created, the default export policy is created for it.
Changing the export rules within an export policy changes the access for every volume that uses that export
policy.

9-11 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Export Policies and Rules Commands
 Use the vserver export-policy
command to view and modify export policy
settings in a Vserver.
 Use the vserver export-policy rule
command to view and modify export policy
rule settings in a Vserver.

NetApp Confidential 12

EXPORT POLICIES AND RULES COMMANDS

9-12 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Export Rule Configuration

1. Set everything wide open to verify that it’s set

up properly.
2. Tighten down security.
3. Add only one export policy per volume.

NOTE: Rules can be added to control specific

access.

NetApp Confidential 13

EXPORT RULE CONFIGURATION

9-13 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFS Configuration

 The scope of an NFS server is a cluster (data)

Vserver.
 Each Vserver can have an NFS server.
 NFS is initially not configured.
 Kerberos, if used, must be configured for an
NFS server.

NetApp Confidential 14

NFS CONFIGURATION
An NFS configuration is limited in scope to a cluster (data) Vserver. A Vserver does not have to have an NFS
configuration. As such, a Vserver must exist before an NFS configuration can be created.
If Kerberos is to be used for an NFS configuration, Kerberos must be configured at the logical interface (LIF)
level within the Vserver. Therefore, if two data LIFs are associated with a Vserver that is running NFS,
Kerberos can be configured for one of the LIFs but not for the other one. Each LIF can use a different
Kerberos configuration and therefore a different Kerberos realm.

9-14 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFS Configuration Options

The following options can be enabled or

disabled for each NFS configuration:
 NFSv2 (not recommended)
 NFSv3
 NFSv4
 NFSv4.1
 NFSv4.1 parallel NFS (pNFS)
 TCP
 UDP

NetApp Confidential 15

NFS CONFIGURATION OPTIONS

The NFS configuration for each Vserver can be configured with the options that are listed on this slide. Each
Vserver can use different NFS options.
netappu::> Vserver nfs modify -
-Vserver -access -v3
-v4.0 -udp -tcp
-spinauth -default-win-user -v4.0-acl
-v4.0-read-delegation -v4.0-write-delegation -v4-id-domain
-v4.1 -rquota -v4.1-pnfs
-v4.1-acl -vstorage

9-15 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFS Differences in Data ONTAP
Operating in Cluster-Mode
 The Data ONTAP 7G operating system:
– An /etc/exports file is used.
– Exports can be persistent or temporary.
– Fencing support enables you to quickly disable write access
to volumes or from users.
– Qtrees and directories can be exported.
 Data ONTAP 8.1.1 operating in Cluster-Mode:
– Export policies and rules are stored in the replicated database
(RDB).
– Exports are persistent.
– An export policy rule can provide the functionality that fencing
provides.
– Export policies can not be applied to qtrees or directories.

NetApp Confidential 16

NFS DIFFERENCES IN DATA ONTAP OPERATING IN CLUSTER-MODE

If you’re familiar with NFS in Data ONTAP 7G architecture (or on UNIX NFS servers), you might wonder
how volumes are tagged to be exported. In Data ONTAP clusters, all volumes are exported as long as they’re
mounted (through junctions) into the namespace of their cluster (data) Vservers. The volume and export
information is kept in the Management replicated database (RDB) unit, so no /etc/exports file exists. This data
in the RDB is persistent across bootings, and, as such, no temporary exports exist.
The Vserver root volume is exported. Because all the other volumes for that Vserver are mounted within the
namespace of the Vserver, you do not need to export anything else. After the NFS client does a mount of the
namespace, the client has NFS access to every volume in this namespace. NFS mounts can also be done for
specific volumes other than the root volume, but the client is limited to being able to see only this volume and
its “descendant” volumes in the namespace hierarchy.
In 7-Mode you can export qtrees and directories. In cluster-Mode, export policies can be applied to volumes,
but not to qtrees or directories.
If a volume is created without being mounted into the namespace, or if a volume gets unmounted, the volume
is not visible within the namespace.

9-16 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFSv4 and NFSv4.1

 NFSv4
– Parity with Data ONTAP 7.3.x software
– NFS referrals
 NFSv4.1
– A minor revision of NFSv4
– An extension, not a modification, of NFSv4
– Delegations on directories and symbolic links
– A new NFS session model
– pNFS
– Fixes to NFSv4

NetApp Confidential 17

NFSV4 AND NFSV4.1

Data ONTAP 8.1 operating in Cluster-Mode introduces support for the NFSv4 protocol specification and for
elements of NFSv4.1. Data ONTAP operating in Cluster-Mode continues to support NFSv2 and NFSv3 fully,
although you should avoid using NFSv2. NFSv4 support brings the Data ONTAP 8.1 operating system in
parity with the Data ONTAP 7.3 operating system.
The key feature for NFSv4 is referrals.
NFSv4.1 is a minor revision and extension, not a modification, of NFS version 4.0, so NFSv4.1 is fully
compliant with the NFSv4 specification. NFSv4.1 extends delegations beyond files to directories and
symbolic links (symlinks), introduces NFS sessions for enhanced efficiency and reliability, and provides
parallel NFS (pNFS). NFSv4.1 also fixes some problems with NFSv4.

9-17 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

NFSv4 Referrals
 The initial request for T is
LIF1 e1a T through LIF4
N
V
 Access to T is over the
R cluster interconnect

Cluster Interconnect
A
M  LIF4 issues a referral
LIF2 e1a
back to the client to use
LIF1
 The client begins to use
LIF3 e1a LIF1 to access T
Referral N
V  This is supported in Data
to LIF1 R
A
ONTAP 8.1 and later
M operating in Cluster-Mode
LIF4 e1a
 This is supported in Red
Hat 5.4 and later
Data Virtual Server (Vserver)

NetApp Confidential 18

NFSV4 REFERRALS
Data ONTAP 8.1 operating in Cluster-Mode introduces NFSv4 referrals. When referrals are enabled in a
virtual server (Vserver), Data ONTAP 8.1 operating in Cluster-Mode provides referrals within that Vserver to
NFSv4 clients. An intraVserver referral occurs when a cluster node that is receiving the NFSv4 request refers
the NFSv4 client to another LIF in the Vserver. The NFSv4 client uses this referral to direct its access over
the referred path at the target LIF from that point onward. The original cluster node issues a referral when it
determines that a LIF exists in the Vserver and is a resident on the cluster node on which the data volume
resides. In other words, if a cluster node receives an NFSv4 request for a nonlocal volume, it can refer the
client to the local path for that volume through the LIF. This therefore allows clients faster access to the data
and avoids extra traffic on the cluster interconnect.
If a volume moves to another aggregate on another node, the volume movement is done nondisruptively.
However, the referral is not updated until the NFSv4 client unmounts and remounts the file system to
understand the new location of the volume and direct the client to the LIF on the node on which the volume
now resides. By default, NFSv4 referrals are enabled on Linux clients like Red Hat Enterprise Linux 5.4 and
later releases.

9-18 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

pNFS: Part 1 (the NFSv3 Process)
NFSv3 Client

Data Logical
Interface (LIF)

Cluster Network

NetApp Confidential 19

PNFS: PART 1 (THE NFSV3 PROCESS)

Remote file access is defined as file access in which a client that is connected through a LIF and uses a
physical port on one controller accesses a file that is hosted on a different controller in the same cluster.
Remote file access has traditionally been a performance concern for clients that use the Data ONTAP Cluster-
Mode operating system. In this example, a client is mounted to a data LIF that is hosted on node 2 and has a
file operation with a destination in a volume on node 5. The request is serviced by the node 2 protocol stack.
That protocol stack looks up the location of the volume and directs the operation to node 5, which hosts the
volume. The request traverses the cluster network. The result is returned to the client along the same path.

9-19 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

pNFS: Part 2 (the Original Data Path)
NFSv4.1 pNFS Client

Data LIF Data LIF

(Metadata Path) (Data Path)

Cluster Network

NetApp Confidential 20

PNFS: PART 2 (THE ORIGINAL DATA PATH)

With pNFS, when a file is opened by an NFS client, the mounted data LIF on node 2 serves as the metadata
path, because this path is used to carry out discovery of the target volume’s location. If the data is hosted by
node 2, the operation is managed locally. In this case, the local node discovers that the data is on node 5.
Based on the pNFS protocol, the client is redirected to a LIF that is hosted on node 5. This request and
subsequent requests to the volume are serviced locally and bypass the cluster network.

9-20 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

pNFS: Part 3 (the New Data Path)
NFSv4.1 pNFS Client

Data LIF Data LIF

(Metadata Path) (New Data Path)

Cluster Network

NetApp Confidential 21

PNFS: PART 3 (THE NEW DATA PATH)

When a volume is moved to an aggregate on a different node, the pNFS client data path is redirected to a data
LIF that is hosted on the destination node.

9-21 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

pNFS

 Can be run with this command:

netappu::> vserver nfs modify
-v4.1-pnfs enabled
 Works with Red Hat Enterprise Linux 6.2,
Fedora 14 or any kernel version 2.6.39
and higher
 Cannot coexist with referrals
 Reduces strain on cluster networks
 Does not need a remount (unlike referrals)

NetApp Confidential 22

PNFS

9-22 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 23

LESSON 2

9-23 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

CIFS Differences in Data ONTAP
Operating in Cluster-Mode
 The Data ONTAP 7G operating system:
– Workgroup or domain authentication
– An explicit choice of “NT4” or “Windows 2000”
authentication
 Data ONTAP 8.1.1 operating in Cluster-Mode:
– Domain authentication only
– Kerberos (enabled or disabled)

NetApp Confidential 24

CIFS DIFFERENCES IN DATA ONTAP OPERATING IN CLUSTER-MODE

Creating a CIFS configuration is the process of enabling a CIFS server for a given cluster (data) Vserver. In
effect, this process creates a CIFS server. A CIFS server is specific to a Vserver. To enable CIFS for another
Vserver, you must create a Vserver-specific CIFS configuration.

9-24 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Server Message Block (SMB) 2.0

 More efficient network use

 Compounding of requests
 Larger reads and writes
 File-property and directory-property caching
 Durable file handles
 Hash-based message authentication code
(HMAC) SHA-256 signing

NetApp Confidential 25

SERVER MESSAGE BLOCK (SMB) 2.0

In addition to the SMB 1.0 protocol, Data ONTAP 8.1 operating in Cluster-Mode supports SMB 2.0 and
SMB 2.1.
SMB 2.0 is a major revision of the SMB 1.0 protocol and includes a complete reworking of the packet format.
SMB 2.0 introduces several performance improvements over earlier versions:
 More efficient network use
 Compounding of requests, which stacks multiple SMBs into a single network packet
 Larger read/write sizes to exploit faster networks
 File and directory property caching
 Durable file handles to enable an SMB 2.0 connection to reconnect to the server transparently if a
temporary disconnection, such as over a wireless connection, occurs
 Improved message signing with improved configuration and interoperability with hash-based message
authentication code (HMAC) SHA-256, which replaces MD5 as a hashing algorithm

9-25 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

SMB 2.1

 The client opportunistic lock (oplock) leasing

model
 Support for large maximum transmission unit
(MTU)
 Improved energy efficiency for client
computers
 Support for earlier versions of SMB

NetApp Confidential 26

SMB 2.1
SMB 2.1 provides important performance enhancements that are listed on this slide.

9-26 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

SMB 2.x

 The following SMB features are not supported

in Data ONTAP 8.1.1 operating in
Cluster-Mode:
– Large MTU
– Resilient handles
– Branch cache
 To enable SMB 2.0 and SMB 2.1, use the
following command:
netappu::> vserver cifs options modify -
vserver vs1 -smb2-enabled true

NetApp Confidential 27

SMB 2.X
The SMB 2.1 protocol provides several minor enhancements to the SMB 2.0 specification. Data ONTAP
8.1.1 operating in Cluster-Mode supports most, but not all, of the SMB 2.1 features. The SMB 2.1 features
that are listed on this slide are not supported.
Support for SMB 2.1 is automatically enabled when you enable the SMB 2.0 protocol on a Vserver. You can
use the command that is shown on this slide to enable SMB 2.x for a Vserver.

9-27 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

The SMB 2.1 Oplock Leasing Model

 File and metadata caching

 Reduced bandwidth consumption
 Retention of cached data after a file is closed
 Full caching with multiple handles

NetApp Confidential 28

THE SMB 2.1 OPLOCK LEASING MODEL

One of the most important features in the SMB 2.1 protocol is the opportunistic lock (oplock) leasing model.
Leasing enables a client to hold oplocks over a wider range of scenarios. The feature offers enhanced file
caching and metadata caching opportunities for the SMB client and provides major performance benefits by
limiting the amount of data that must be transferred between the client computer and the server. This
enhancement particularly benefits networks with high latency. Additionally, because the number of operations
that must be directed toward an SMB file server is reduced, the SMB file server scalability is increased.
The new leasing model in SMB 2.1 enables greater file- and handle-caching opportunities for an SMB 2.1
client computer while preserving data integrity and requiring no current application changes to take advantage
of this capability. New caching levels enable more flexibility for clients to request caching mechanism.
Multiple applications can retain the cached data even after the file handle is closed. Full data caching is
supported even with multiple handles as long as those handles are opened on the same client.

9-28 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

CIFS Notes

 The scope of a CIFS service is a cluster

(data) Vserver.
 Each cluster (data) Vserver can have a single
CIFS service.
 Consult your account team for cluster-level
limits.

NetApp Confidential 29

CIFS NOTES
A CIFS configuration is limited in scope to a cluster (data) Vserver. A Vserver does not require a CIFS
configuration. As such, a Vserver must exist before a CIFS configuration can be created.

9-29 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Domain Controller Configuration
 Two types of CIFS domains exist:
– NT LAN Manager (NTLM), prior to Windows 2000
– Active Directory for Windows 2000 and later
 Domain configuration grants clients single-sign-on
access to the file server.
 To configure the CIFS service, you need:
– The domain name
– The NetBIOS name for the Data ONTAP CIFS
“server”
– An administrative domain controller password so
that a “computer” entity can be automatically
created
NetApp Confidential 30

DOMAIN CONTROLLER CONFIGURATION

Active Directory uses Kerberos authentication. NT LAN Manager (NTLM) is provided for backward
compatibility with Windows clients prior to Windows 2000 Server. Prior to Active Directory, Windows
domains had primary and secondary domain controllers (DCs). With Active Directory, one or multiple
Windows servers that work in cooperation with each other might exist for the Windows domain. A domain
controller is now a role that is played by an Active Directory machine.
When configuring CIFS, the domain controller information is automatically discovered, and the account on
the domain controller is created for you.

9-30 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

CIFS Configuration

 If you use NTLM on the primary domain

controller (PDC), create (on the PDC) a
machine account for the Data ONTAP 8.1.1
CIFS “server.”
 Configure the CIFS service for a Vserver.
 Create CIFS shares for the Vserver.
 Configure name mappings for UNIX-to-
Windows and Windows-to-UNIX identities.

NetApp Confidential 31

CIFS CONFIGURATION

9-31 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

CIFS Configuration Example
netappu::> vserver cifs create -vserver vs1
-domain [Link] –cifs-server
MYCIFS

netappu::> vserver cifs share create -vserver

vs1 -share-name root -path /
-share-properties browsable

netappu::> vserver cifs share create -vserver

vs1 -share-name root_rw -path /.admin
-share-properties browsable

netappu::> vserver cifs share create -vserver

vs1 -share-name %w -path /user/%w
-share-properties browsable,homedirectory

NetApp Confidential 32

CIFS CONFIGURATION EXAMPLE

Some typical steps that are required for configuring CIFS for a cluster (data) Vserver are shown on this slide.
The first step is to create a CIFS configuration. This configuration is the CIFS server for the vs1 Vserver.
Three CIFS shares are created. The first one, “root,” represents the normal path to the root of the namespace.
If the root volume (or any other volume) has load-sharing mirror relationships, this normal path uses the read-
only volumes. Therefore, a read/write share, “root_rw,” must be created. If a client maps to the read/write
share, the client always uses the read/write volumes throughout the namespace of that Vserver. (No load-
sharing mirror relationships are used). More information about mirror relationships, read/write, and read-only
paths is provided later in this course.
The third share uses home directory (dynamic) shares, based on the user name. For example, if the “bill” user
in the nau01 domain connects to the CIFS server, and a path exists in the namespace of /user/bill, the “% w”
is translated dynamically into “bill” such that a share that is called “bill” exists and maps to the junction path
/user/bill. While user “bill” is on his PC in the nau01 domain, he can go to \\mycifs\bill and be put into the
volume that has a junction path of /user/bill.
Clustered ONTAP currently supports three share pattern substitutions:
%d - user's domain
%w - user's Windows login name
%% - to represent a literal "%"

9-32 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Common Kerberos Issues

 Kerberos is sensitive to time skew.

 The default allowable time skew is five
minutes for the Data ONTAP 8.1.1 operating
system.
 Using the Network Time Protocol (NTP) is the
best way to avoid problems.

NetApp Confidential 33

COMMON KERBEROS ISSUES

Kerberos is sensitive to time skew among nodes and between the cluster and a Kerberos server. When
multiple machines work together, as is the case with a Data ONTAP cluster and a Kerberos server, the times
on those machines should be within a few minutes of each other. By default, a five-minute time skew is
allowed. A greater time skew causes problems. Time zone settings manage the issue of machines that are in
different time zones. Network Time Protocol (NTP) is a good way to keep multiple machine times
synchronized with each other. You can also widen the allowable time skew, but it’s best to keep the machine
times synchronized.

9-33 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Connecting to the Domain Controller

NetApp Confidential 34

CONNECTING TO THE DOMAIN CONTROLLER

This slide shows the UI of the Windows Active Directory or domain controller where the machine account is
created for the CIFS configuration of a Vserver.

9-34 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Name Mapping
1 of 2
 Default mapping applies to protocol-specific
Vserver configuration (NFS or CIFS).
 Default mapping provides consistency during
simultaneous access from CIFS and NFS.
 You specify Windows-to-UNIX rules or UNIX-
to-Windows rules.
 Rules use regular expressions with
enhancements.
 You can make direct translations.
 You can map to local users and groups.

NetApp Confidential 35

NAME MAPPING: 1 OF 2
Data ONTAP operating in Cluster-Mode enables concurrent access to files through NFS and CIFS and with
the use of Kerberos. All of these protocols include the concept of a principal (user or group), but they’re
incompatible with each other. Therefore, name mappings provide a level of compatibility.
CIFS principals explicitly contain the CIFS domain as part of the principal. Likewise, Kerberos principals
contain an instance and a realm. NFS uses NIS to store its principal information and is simply a name. (The
NIS domain is implied and so is not needed in the principal). Because of these differences, the administrator
must set up rules (specific or regular expressions) to enable these protocols to resolve the differences and
correlate these principals with each other.

9-35 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Name Mapping
2 of 2
 Name mapping is required to resolve “principals”
(users and groups):
– From Kerberos to UNIX
– From Windows to UNIX
– From UNIX to Windows
 Generally, name mappings are defined by using
regular expressions.
 These name mappings are stored in the
Management RDB unit.

NetApp Confidential 36

NAME MAPPING: 2 OF 2

9-36 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Name Mapping Examples
1 of 2
Map the CIFS “Administrator” user into the UNIX
“root” user and vice versa:
netappu::> vserver name-mapping create
-vserver vs1 -direction win-unix
-position 1
-pattern “learn\\Administrator”
-replacement “root”
netappu::> vserver name-mapping create
-vserver vs1 -direction unix-win
-position 1
-pattern “root”
-replacement “learn\\Administrator”

NetApp Confidential 37

NAME MAPPING EXAMPLES: 1 OF 2

Note the two backslashes between the CIFS domains and the CIFS user. Because these parameters require
regular expressions, and the backslash is a special character in regular expressions, the backslash must be
“escaped” with another backslash. Thus, in a regular expression, two backslashes are required to represent
one backslash.

9-37 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Name Mapping Examples
2 of 2
Map all CIFS users in the “learn” domain into their
like-named UNIX users and vice versa:
netappu::> vserver name-mapping create
-vserver vs1 -direction win-unix
-position 1
-pattern “learn\\(.+)”
-replacement “\1”
netappu::> vserver name-mapping create
-vserver vs1 -direction unix-win
-position 1
-pattern “(.+)”
-replacement “learn\\\1”
See KB1013831 for more information
NetApp Confidential 38

NAME MAPPING EXAMPLES: 2 OF 2

In the example on this slide, a more dynamic mapping is set up. Here, any user in the “learn” domain is
mapped with a like-named UNIX user. Thus, the “yoda” user in the learn domain (learn\yoda) is mapped with
the “yoda” UNIX user.

9-38 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Create an export policy for a cluster (data)
Vserver
 Create an NFS configuration for a cluster
(data) Vserver
 Create a CIFS domain, server, and share
 Mount exports and shares to NFS and CIFS
clients
 Create name mapping rules between NFS
and CIFS
NetApp Confidential 39

MODULE SUMMARY

9-39 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Exercise
Module 9: NAS Protocols
Time Estimate: 120 minutes

NetApp Confidential 40

EXERCISE
Please refer to your exercise guide.

9-40 Clustered Data ONTAP Administration: NAS Protocols

NetApp University - Do Not Distribute

Module 10
SAN Protocols

NetApp Confidential 1

MODULE 10: SAN PROTOCOLS

10-1 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Explain the differences between the
supported SAN protocols
 Identify the components that implement
scalable SAN on a cluster in a Data ONTAP
environment operating in Cluster-Mode
 Configure iSCSI on a cluster and create
a LUN
 Configure a Windows iSCSI initiator

NetApp Confidential 2

MODULE OBJECTIVES

10-2 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Unified Storage

NFS iSCSI
Corporate CIFS
LAN FCoE
FC

NAS SAN

Data ONTAP Operating

in Cluster-Mode
NetApp Confidential 3

UNIFIED STORAGE
A SAN is a block-based storage system that uses FC, Fibre Channel over Ethernet (FCoE), and iSCSI
protocols to make data available over the network. Starting in the Data ONTAP 8.1 operating system, Data
ONTAP systems operating in Cluster-Mode began supporting SANs on clusters of up to four nodes. In the
Data ONTAP 8.1.1 operating system, SAN is supported in clusters of up to six nodes.

10-3 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

SAN Protocol Support
Either FC or IP can be used to implement a SAN:
 FC:
– Uses FC protocol to communicate FC SAN is
covered in
Physical Data FC Frame SCSI-3 SAN Scaling
and
– Uses FCoE to communicate Architecting.

Physical Ethernet FCoE FC Frame SCSI-3

 IP uses iSCSI to communicate.

Physical Ethernet IP TCP iSCSI SCSI-3

NetApp Confidential 4

SAN PROTOCOL SUPPORT

10-4 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Scalable SAN Enhancements

Data ONTAP 8.1 Data ONTAP 8.1.1

Windows Red Hat VMware ESX Windows Red Hat VMware ESX HPUX Solaris

FC, iSCSI, or FCoE FC, iSCSI, or FCoE

LUNs LUNs LUNs LUNs LUNs

NetApp Confidential 5

SCALABLE SAN ENHANCEMENTS

The Data ONTAP 8.1 operating system introduced scalable SAN on clusters of up to four nodes, supporting
FC, iSCSI, and FCoE protocols. With Data ONTAP 8.1.1, scalable SAN support has been expanded to up to
six nodes, increasing capacity for storage, CPU cycles, and network bandwidth for clustered SAN solutions,
with no need to increase management and administrative resources.
Data ONTAP clustering continues to support Windows, VMware ESX, and Red Hat Linux clients, but release
8.1.1 introduces three additional SAN targets. Solaris and HPUX are both supported on FC, iSCSI, and FCoE.
NetApp Host Utilities 6.0 is required on Windows, Red Hat, Solaris, and HPUX. To function with scalable
SAN, all SAN client stacks must have support for Asymmetric Logical Unit Access (ALUA) installed.
Consult the NetApp Supportability Matrix for details on supported versions of SAN hosts.

10-5 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 6

LESSON 1

10-6 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Ports
Application
Initiator File System
TCP/IP Driver iSCSI Driver SCSI Driver FC Driver

Ethernet Port FC Port

SAN Services
Target TCP/IP Driver iSCSI Driver WAFL File System FC Driver

IP LUN FC
SAN SAN
NetApp Confidential 7

PORTS
Data is communicated over ports. In an Ethernet SAN, the data is communicated over Ethernet ports. In an
FC SAN, the data is communicated over FC ports. For FCoE, the initiator has a converged network adapter
(CNA), and the target has a unified target adapter (UTA).

10-7 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Nodes and Portals in iSCSI
Application
Initiator File System
SCSI Driver

Local Network Connection [Link].a:system

Portals Node Name

Target Portal Group (TPG) [Link]:sn.x:vs.1

Cluster (data) Vserver
Target SAN Services
WAFL File System

IP LUN
SAN

NetApp Confidential 8

NODES AND PORTALS IN ISCSI

In IP SAN, the node name describes a cluster (data) Vserver, and the portal describes a LIF. Each iSCSI node
must have a node name. Two node-name formats are possible.
IQN-Type Designator
The format of this node name is conventionally:
[Link]-mm.backward_naming_authority: unique_device_name
This format is the most popular node-name format and the default that is used by a NetApp storage system.
The following are the components of the logical name:
 Type designator, IQN, followed by a period (.)
 The date when the naming authority acquired the domain name, followed by a period
 The name of the naming authority, optionally followed by a colon (:)
 A unique device name

Eui-Type Designator
The format of this node name is:
[Link]
The following are the components of the logical name:
 The type designator itself, “eui,” followed by a period (.)
 Sixteen hexadecimal digits
Example: eui.123456789ABCDEF0

10-8 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Connectivity Between the Initiator and
the Target
Application
Initiator File System
SCSI Driver

Switch

SAN Services
Target WAFL File System

IP LUN
SAN
NetApp Confidential 9

CONNECTIVITY BETWEEN THE INITIATOR AND THE TARGET

10-9 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Direct and Indirect Paths
MPIO
ALUA

Direct

LUN

NetApp Confidential 10

DIRECT AND INDIRECT PATHS

You must configure SAN clients to use:
 Multipath I/O (MPIO) to access LUNs
 ALUA to determine the state of a given data path to the LUNs
The “active-optimized” path to a LUN means the path for which the LIF and LUN are hosted by the same
node. The “active-nonoptimized path” represents the path for which the LIF and LUN are hosted on separate
nodes.
Unlike NAS LIFs, SAN LIFs do not migrate between interfaces or nodes. Therefore, the client host uses
ALUA to determine the most efficient path (or paths) to communicate to the LUN. The active-optimized
paths become the primary paths for data transfer between the host and the LUN.
When a volume that is hosting a LUN is moved to an aggregate on a different node, the virtual server
(Vserver) updates the path status, and the client polls the Vserver for the change. In this way, the new active-
optimized and active-nonoptimized paths are chosen, and the client selects the best possible paths.
When a node goes down and storage fails over to the partner node, the node’s paths also come offline. If an
appropriately zoned SAN LIF is available on the partner node, the path to the takeover node becomes the
active-optimized path until the aggregate is returned to its home node. If the paths to a node become
unavailable so that only non-optimized paths remain, but the storage doesn’t failover, the client chooses a
nonoptimized path, and the data traverses the cluster network until an optimized path is restored.

10-10 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Direct and Indirect Paths
MPIO
ALUA

Indirect

LUN

NetApp Confidential 11

DIRECT AND INDIRECT PATHS

10-11 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Optimized and Nonoptimized Paths
MPIO
ALUA

Indirect Indirect Indirect

LUN

NetApp Confidential 12

OPTIMIZED AND NONOPTIMIZED PATHS

10-12 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Path Priority Selection

ALUA (also called Target Port Group Support or

TPGS) has these features:
 One target port group per state per controller
 Possible states:
– Active-optimized (Direct)
– Active-nonoptimized (Indirect)
– Standby (not implemented in the Data ONTAP
operating system)
– Unavailable

NetApp Confidential 13

PATH PRIORITY SELECTION

ALUA is the Asymmetric Logical Unit Access command set. ALUA is also known as Target Port Group
Support (TPGS) in the SCSI Primary Commands (SPC) standard.
The basic design that is described is for one target port group (TPG) per state per controller. Each of the
controller heads in a high-availability (HA) pair has its own group for each of the possible states:
 Active-optimized
 Active-nonoptimized
 Standby
 Unavailable (For example, a partner port can become unavailable when the interconnect is down.)
With this model, when ports change to a different state, the ports are moved to the new group that represents
that new state.

10-13 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 14

LESSON 2

10-14 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Data ONTAP Cluster-Mode Support
 Data ONTAP 8.1 and later versions operating in
Cluster-Mode support iSCSI.
 To configure iSCSI by using NetApp System Manager or
the CLI:
1. Add the iSCSI license for the cluster.
2. Create or designate an aggregate for the root volume of a virtual
server (Vserver).
3. Create or designate a Vserver for iSCSI.
4. Enable iSCSI traffic for the Vserver.
5. Create iSCSI logical interfaces (LIFs).
6. Create an initiator group (igroup)
7. Create and bind port sets.
8. Create or designate an aggregate and volume for a LUN.
9. Create a LUN.
[Link] the LUN to the appropriate igroup.
NetApp Confidential 15

DATA ONTAP CLUSTER-MODE SUPPORT

10-15 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Licensing iSCSI
 The cluster must have the iSCSI license installed.
 Install the license by using:
– The Cluster Setup Wizard (as discussed previously)
– NetApp System Manager
– The CLI:
netappu::> system license add <xxxxx>
netappu::> system license show
Feature Cluster SN Limit Description
--------------- ----------- ------- -----------
Base 1-80-123456 6 Base License
w/cluster size limit
(nodes)
iSCSI 1-80-123456 6 iSCSI License
2 entries were displayed.

NetApp Confidential 16

LICENSING ISCSI

10-16 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Identifying an Aggregate for Vserver Use
 If needed, create an aggregate:
netappu::> storage aggregate create -aggregate
aggr_iscsi_2 -node netappu-02 -diskcount 7
 Verify the aggregate:
netappu::> aggr show Abbreviated Command

Aggregate Size Available Used% State #Vols Nodes RAID Status

--------- -------- --------- ----- ------- ------ ------ ----- ------
aggr0 900MB 43.54MB 95% online 1 netappu-01 raid_dp, normal
aggr0_scaling_02_0
900MB 43.55MB 95% online 1 netappu-02 raid_dp, normal
aggr_iscsi_1 4.39GB 4.25GB 3% online 2 netappu-01 raid_dp, normal
aggr_iscsi_2 4.39GB 4.39GB 0% online 0 netappu-02 raid_dp, normal
4 entries were displayed.

NetApp Confidential 17

IDENTIFYING AN AGGREGATE FOR VSERVER USE

10-17 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating an iSCSI-Enabled Vserver
 Create a Vserver:
netappu::> vserver create -vserver vsISCSI2 -rootvolume
vsISCSI2_root -aggregate aggr_iscsi_2 -ns-switch file
-nm-switch file -rootvolume-security-style ntfs

 Enable iSCSI protocol:

netappu::> vserver iscsi create -vserver vsISCSI2
-target-alias vsISCSI2 -status up

 Verify the iSCSI-enabled Vserver:

netappu::> vserver iscsi show
Target Target Status
Vserver Name Alias Admin
-------- ---------------- --------------------- ------
vsISCSI2 [Link]:sn.d7d67711cf2:vs.4
vsISCSI2 up

NetApp Confidential 18

CREATING AN ISCSI-ENABLED VSERVER

All volumes within the Data ONTAP operating system have a default security style. In the example on this
slide, New Technology File System (NTFS) was used; however, for a Vserver that is used to expose only
LUNs, it does not matter which security style you use.

10-18 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating iSCSI LIFs
 Create an iSCSI LIF:
netappu::> network interface create -vserver vsISCSI2 -lif
i2LIF1 -role data -data-protocol iscsi -home-node netappu-01
-home-port e0c -address [Link] -netmask [Link]
-status-admin up

 Verify the iSCSI LIFs:

netappu::> net int show -vserver vsISCSI2
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
-------- --------- --------- --------------- ----------- ------- ----
vsISCSI2
i2LIF1 up/up [Link]/24 netappu-01 e0c true
i2LIF2 up/up [Link]/24 netappu-01 e0d true
i2LIF3 up/up [Link]/24 netappu-02 e0c true
i2LIF4 up/up [Link]/24 netappu-02 e0d true
4 entries were displayed.

NOTE: Three additional LIFs were created as shown,

but the creation commands are not shown.

NetApp Confidential 19

CREATING ISCSI LIFS

10-19 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

iSCSI LIFs Considerations

 Failover groups do not apply to SAN LIFs.

 SAN LIFs are single-protocol only, but ports
can be assigned a NAS and a SAN LIF.
 Recommendation: Use at least one LIF per
node but more if you have separate subnets
because of multiple iSCSI initiators per host.

NetApp Confidential 20

ISCSI LIFS CONSIDERATIONS

10-20 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating Port Sets
 Create a port set:
netappu::> lun portset create -vserver vsISCSI2
-portset portset_iscsi2 -protocol iscsi -port-name i2LIF1
i2LIF2 i2LIF3 i2LIF4

 Verify a port set:

netappu::> lun portset show
Vserver Portset Protocol Port Names Igroups
--------- ------------ -------- ----------------------- -------
vsISCSI2 portset_iscsi2
iscsi i2LIF1, i2LIF2,
i2LIF3, i2LIF4
-

Not yet associated with an igroup

NetApp Confidential 21

CREATING PORT SETS

10-21 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Lesson 3

NetApp Confidential 22

LESSON 3

10-22 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Windows Native Multipath I/O
Windows Server 2008 can be configured to support
multipath I/O (MPIO).

Right-click
Features, and
then select Add
Feature. Multipath I/O
added

Booting the system is not required

in Windows Server 2008 R2.

NetApp Confidential 23

WINDOWS NATIVE MULTIPATH I/O

10-23 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Device-Specific Modules

 A device-specific module (DSM) is a driver

that plugs into an MPIO framework.
 Windows MPIO supports:
– A native Windows DSM
– The NetApp Data ONTAP DSM
 If you are using FC and iSCSI paths, the Data
ONTAP DSM is required.
 The Data ONTAP DSM is available at the
NetApp Support site.
 Use the Interoperability Matrix Tool to verify
the recommended version.
NetApp Confidential 24

DEVICE-SPECIFIC MODULES

10-24 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Host Utilities

 NetApp Host Utilities provides Perl scripts to

diagnose and troubleshoot Windows.
 For example, windows_info provides
diagnostic information.
 Use the Interoperability Matrix Tool to verify
compatible host utilities.
 Download and install compatible host utilities.

NetApp Confidential 25

HOST UTILITIES

10-25 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

The iSCSI Software Initiator: Discovery

Click here to
configure.

Add the Vserver’s

LIF address.

Windows Server 2008 R2

Example

NetApp Confidential 26

THE ISCSI SOFTWARE INITIATOR: DISCOVERY

To configure the Microsoft iSCSI Software Initiator:
 Install the Microsoft iSCSI Software Initiator.
 On the Discovery tab, specify the Vserver’s LIF IP address as a target portal.

10-26 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

The iSCSI Software Initiator: Connection

Click here to
The Vserver is enable
discovered. multipath.

Click here to
connect.

Click here to
accept the
connection
method.

NetApp Confidential 27

THE ISCSI SOFTWARE INITIATOR: CONNECTION

The Vserver node name is displayed in the target table.
Select the node name, and then click Connect.

10-27 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

The iSCSI Software Initiator:
Favorite Targets

NetApp Confidential 28

THE ISCSI SOFTWARE INITIATOR: FAVORITE TARGETS

In the Connect To Target dialog box, if you select “Add this connection to the list of Favorite Targets,” the
connection appears on the Favorite Targets tab.

10-28 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating an igroup
 Create an igroup:
netappu::> lun igroup create -vserver vsISCSI2 -igroup
ig_myWin2 -protocol iscsi -ostype windows -initiator iqn.1991-
[Link]:win-frtp2qb78mr –portset portset_iscsi2

 Verify an igroup:
netappu::> igroup show Abbreviated Command
Vserver Igroup Protocol OS Type Initiators
--------- -------- -------- -------- -------------------------
vsISCSI2 ig_myWin2 iscsi windows iqn.1991-
[Link]:win-frtp2qb78mr

NetApp Confidential 29

CREATING AN IGROUP

10-29 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Verifying Connectivity
1 of 2
 Verify the target portal groups:
netappu::> vserver iscsi tpgroup show -vserver vsISCSI2
TPGroup TPGroup Logical
Vserver Name Tag Interface
--------- ---------------- ------- ----------
vsISCSI2 i2LIF1 1032 i2LIF1
vsISCSI2 i2LIF2 1033 i2LIF2
vsISCSI2 i2LIF3 1034 i2LIF3
vsISCSI2 i2LIF4 1035 i2LIF4
4 entries were displayed.

 Verify igroup connection:

netappu::> igroup show –instance ig_myWin2
Vserver Name: vsISCSI2
Igroup Name: ig_myWin2
Protocol: iscsi
…
ALUA: true
Initiators: [Link]:win-
frtp2qb78mr (logged in)

NetApp Confidential 30

VERIFYING CONNECTIVITY: 1 OF 2

10-30 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Verifying Connectivity
2 of 2
 Verify sessions:
netappu::> vserver iscsi session show -vserver vsISCSI2
Tpgroup Initiator Initiator
Vserver Name TSIH Name ISID Alias
--------- ------- ---- ------------------------ --------- -------------
vsISCSI2 i2LIF1 5 [Link]:win-frtp2qb78mr
40:00:01:37:00:00

 Verify connections:
netappu::> vserver iscsi connection show -vserver vsISCSI2
Tpgroup Conn Local Remote TCP Recv
Vserver Name TSIH ID Address Address Size
------------ --------- ----- ----- --------------- ----------- --------
vsISCSI2 i2LIF1 5 1 [Link] [Link] 13140

NetApp Confidential 31

VERIFYING CONNECTIVITY: 2 OF 2

10-31 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating a Volume
 Create a volume:
netappu::> vol create -vserver vsISCSI2 -volume vol1
-aggregate aggr_iscsi_2 -size 150MB -state online -type RW
-policy default -security-style ntfs

 Verify a volume:
netappu::> vol show
Vserver Volume Aggregate State Type Size Available Used%
--------- ---------- --------- -------- ---- ------- --------- -----
…
vsISCSI2 vol1 aggr_iscsi_2 online RW 150MB 142.4MB 5%
…

NetApp Confidential 32

CREATING A VOLUME

10-32 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Creating a LUN
 Create a fully provisioned LUN:
netappu::> lun create -vserver vsISCSI2 -volume vol1
-lun lun_vsISCSI2_1 -size 50MB
-ostype windows_2008 -space-reserve enable

 Verify a LUN:
netappu::> lun show -vserver vsISCSI2
Vserver Volume Qtree LUN State Mapped Type Size
--------- ------ ----- ------------ ------ -------- -------- -------
vsISCSI2 vol1 "" lun_vsISCSI2_1 online unmapped windows_2008 54.91MB

For information about creating thin-provisioned LUNs,

see the SAN Scaling and Architecting course.

NetApp Confidential 33

CREATING A LUN

10-33 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Mapping a LUN
 Map a LUN to an igroup:
netappu::> lun map -vserver vsISCSI2 -volume vol1
-lun lun_vsISCSI2_1 -igroup ig_myWin2

 Verify mapping:
netappu::> lun show -instance /vol/vol1/lun_vsISCSI2_1
Vserver Name: vsISCSI2
LUN Path: /vol/vol1/lun_vsISCSI2_1 …
OS Type: windows_2008
Space Reservation: enabled
Serial Number: BGMc1]-hUDrf
Comment:
Space Reservations Honored: true
Space Allocation: disabled
State: online
LUN UUID: 9d426342-cf8d-11e0-90b1-123478563412
Mapped: mapped
Block Size: 512B …
or use:
netappu::> lun show

NetApp Confidential 34

MAPPING A LUN

10-34 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Lesson 4

NetApp Confidential 35

LESSON 4

10-35 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Scanning for a New LUN

Select Disk
Management.

Right-click, and then

select Rescan Disks.

NetApp Confidential 36

SCANNING FOR A NEW LUN

After creating a LUN with the lun setup command or with NetApp System Manager, use Windows Disk
Management on the host to prepare the LUN for use. The new LUN should be visible as a local disk. If the
new LUN is not visible, in the toolbar, click the Action button, and then click Rescan Disks.

10-36 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Initializing a New LUN

The LUN
appears.

The LUN
is offline.

Right-click,
and then select
Initialize.

NetApp Confidential 37

INITIALIZING A NEW LUN

10-37 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Provisioning a New LUN

The wizard
launches.

Right-click, and
then select New
Simple Volume.

NetApp Confidential 38

PROVISIONING A NEW LUN

To open the New Simple Volume Wizard, right-click the bar that represents the unallocated disk space, and
then select New Simple Volume. Or, from the Action menu in the Computer Management window, select All
Tasks > New Simple Volume.

10-38 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

The Volume Size and Mount Options

Specify the volume size.

Specify the method

to mount.

NetApp Confidential 39

THE VOLUME SIZE AND MOUNT OPTIONS

Choose the partition size and drive letter. Accept the default drive assignment, or use the list to select a
different drive.

10-39 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

The Format and Summary Pages

Specify the format.

Verify the configuration,

and then click Finish.

The LUN is now ready to use.

NetApp Confidential 40

THE FORMAT AND SUMMARY PAGES

10-40 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Additional SAN Resources
 The SAN Implementation instructor-led course:
– Implementation details for when you use Windows, vSphere,
and Linux as initiators
– Information about SnapDrive for Windows and SnapDrive for
UNIX
 The SAN Scaling and Architecting instructor-led course:
– Details about FC and FCoE implementation
– Steps for troubleshooting:
 LIF failure
 Storage failover
 Volume move
 Take both courses and prepare for the NCIE-SAN
certification exams.

NetApp Confidential 41

ADDITIONAL SAN RESOURCES

10-41 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Explain the differences between the
supported SAN protocols
 Identify the components that implement
scalable SAN on a cluster in a Data ONTAP
environment operating in Cluster-Mode
 Configure iSCSI on a cluster and create
a LUN
 Configure a Windows iSCSI initiator

NetApp Confidential 42

MODULE SUMMARY

10-42 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Exercise
Module 10: SAN Protocols
Estimated Time: 45 minutes

NetApp Confidential 43

EXERCISE
Please refer to your exercise guide.

10-43 Clustered Data ONTAP Administration: SAN Protocols

NetApp University - Do Not Distribute

Module 11
Storage Efficiency

NetApp Confidential 1

MODULE 11: STORAGE EFFICIENCY

11-1 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Discuss the storage-efficiency features that
are built into Data ONTAP operating in
Cluster-Mode
 Identify the commands that are needed to
manage storage efficiency on a Data ONTAP
cluster

NetApp Confidential 2

MODULE OBJECTIVES

11-2 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Storage Efficiency

Thin Provisioning FlexClone Software

Deduplication Compression

NetApp Confidential 3

STORAGE EFFICIENCY
Data ONTAP 8.1.1 operating in Cluster-Mode introduces several storage-efficiency features that bring feature
parity between Data ONTAP operating in Cluster-Mode and Data ONTAP operating in 7-Mode. The features
include thin provisioning, deduplication, data compression, and cloning. Each of these powerful features
makes Data ONTAP operating in Cluster-Mode even more attractive to new NetApp customers, current Data
ONTAP 7-Mode customers, current Data ONTAP Cluster-Mode customers, and current Data ONTAP GX
customers.

11-3 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Thin Provisioning
Typical: 40% Use NetApp: More than 70% Use
Buy 50% Less Storage
Save 50% in Power, Cooling,

waste
App 3 8 spindles and Space

waste Shared
App 2 6 spindles capacity

App 3
12 spindles
waste

App 1 6 spindles App 2

App 1

Standard Volume Manager NetApp Thin Provisioning

NetApp Confidential 4

THIN PROVISIONING
If you compare the NetApp storage use approach to the competition’s approach, you find one feature that
stands out. Flexible dynamic provisioning with FlexVol technology provides high storage use rates and
enables customers to increase capacity without the need to physically reposition or repurpose storage devices.
NetApp thin provisioning enables users to oversubscribe data volumes, which results in high use models. You
can think of this approach as “just-in-time” storage.
To manage thin provisioning on a Data ONTAP 8.1.1 cluster, use the volume command.

11-4 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Deduplication

netappu::> volume efficiency on -vserver vs1

-volume vol1

NetApp Confidential 5

DEDUPLICATION
Deduplication improves physical storage-space efficiency by eliminating redundant data blocks within a
FlexVol volume. Deduplication works at the block level on an active file system and uses the WAFL (Write
Anywhere File Layout) block-sharing mechanism. Each block of data has a digital signature that is compared
with all of the other blocks within the data volume. If an exact match is identified, the duplicate block is
discarded, and a data pointer is modified so that the storage system references the copy of the data object that
is stored on disk. The deduplication feature works well with datasets that have large quantities of duplicated
data or white space. You can configure deduplication operations to run automatically or according to a
schedule. You can run deduplication on new data or existing data on any FlexVol volume.
The deduplication feature enables you to reduce storage costs by reducing the actual amount of data that is
stored over time. For example, if you create a 100-GB full backup one night and 5 GB of data changes the
next day, the second nightly backup needs to store only the 5 GB of changed data. This approach amounts to a
95% spatial reduction on the second backup. In operational environments, deduplication of full backups can
save more than 90% of the required space, and deduplication of incremental backups saves an average of 30%
of the space. In nonbackup scenarios, such as the creation of virtual machine images, you can save 40% of the
space. To estimate your own savings, visit the NetApp deduplication calculator at [Link]

11-5 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

The Benefits of Deduplication
 Can reduce space consumption by 20 times or
greater for backups (TR3966)
 Is integrated with the Data ONTAP operating system:
– General-purpose volume deduplication
– Identification and removal of redundant data blocks
 Is application-agnostic:
– Primary storage
– Backup data
– Archival data
 Runs as a background process and is transparent
to clients

NetApp Confidential 6

THE BENEFITS OF DEDUPLICATION

11-6 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Data Compression

netappu::> volume efficiency on -vserver vs1 -volume vol1

-compression true

NetApp Confidential 7

DATA COMPRESSION
Data compression enables you to reduce the physical capacity that is required to store data on a Data ONTAP
8.1.1 cluster by compressing data blocks within a FlexVol volume. Data compression is available only on
FlexVol volumes that are created on 64-bit aggregates. Data compression optimizes the storage space and
bandwidth that is required to replicate data during volume operations, such as moving volumes and
performing SnapMirror transfers. You can compress standard data files, virtual disks, and LUNs, but not file
system internal files, NT streams, or metadata.
To manage compression on a Data ONTAP 8.1.1 cluster, use the volume efficiency command.

11-7 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Characteristics of Data Compression

 Inline compression
– Parallelism is increased.
– Path length is decreased.
– Latency is increased.
 Postprocess compression
– Uncompressed data is compressed during idle
time.
– Only previously uncompressed blocks are
compressed.
– Compression is done before deduplication.
See TR3966 for more information

NetApp Confidential 8

CHARACTERISTICS OF DATA COMPRESSION

There are two types of data compression: inline compression and postprocess compression.
With inline compression, all writes to a volume are compressed immediately before being written to the
volume. Inline compression increases parallelism, because all compression and decompression algorithms are
multiprocessor-capable and because writes are compressed outside of the consistency point. Because
operations do not have to be suspended and resumed, inline compression also reduces path length. However,
because processing is required for compression and decompression, there is a latency impact on performance.
Postprocess compression runs as a background task. Uncompressed data that is written after deduplication is
compressed and rewritten to the volume when the controller is not busy. If inline and postprocess
compression are enabled for the same volume, postprocess compression compresses only the blocks on the
volume that were not compressed previously. If compression and deduplication are enabled, compression is
always executed before deduplication.

11-8 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Cloning

Aggregate
Aggregate
Aggregate

vol1
vol1
vol1 vol1clone
vol1clone
vol1 clone

Data Blocks
Data Blocks

netappu::> volume clone create -vserver vs1 -flexclone vol1clone -parent-volume vol1

netappu::> volume clone split start -vserver vs1 -flexclone vol1clone

netappu::> volume clone split show -vserver vs1 -flexclone vol1clone

NetApp Confidential 9

CLONING
A FlexClone volume is a point-in-time, space-efficient, writable copy of the parent volume. The FlexClone
volume is a fully functional stand-alone volume. Changes that are made to the parent volume after the
FlexClone volume is created are not reflected in the FlexClone volume, and changes to the FlexClone volume
are not reflected in the parent volume.
FlexClone volumes are created in the same virtual server (Vserver) and aggregate as the parent volume, and
FlexClone volumes share common blocks with the parent volume. While a FlexClone copy of a volume
exists, the parent volume cannot be deleted or moved to another aggregate. You can sever the connection
between the parent and the FlexClone volume by executing a split operation.
A FlexClone split causes the FlexClone volume to use its own disk space, but the FlexClone split enables you
to delete the parent volume and to move the parent or the FlexClone volume to another aggregate.
To manage cloning on a Data ONTAP 8.1.1 cluster, use the volume clone command.

11-9 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Discuss the storage-efficiency features that
are built into Data ONTAP operating in
Cluster-Mode
 Identify the commands that are needed to
manage storage efficiency on a Data ONTAP
cluster

NetApp Confidential 10

MODULE SUMMARY

11-10 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Exercise
Module 11: Storage Efficiency
Time Estimate: 60 minutes

NetApp Confidential 11

EXERCISE
Please refer to your exercise guide.

11-11 Clustered Data ONTAP Administration: Storage Efficiency

NetApp University - Do Not Distribute

Module 12
Data Protection

NetApp Confidential 1

MODULE 12: DATA PROTECTION

12-1 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Module Objectives
After this module, you should be able to:
 List the benefits of the mirroring engine
 Create load-sharing and data-protection mirror copies
 Manually and automatically replicate mirror copies
 Promote a mirror copy to replace its read/write volume
 Implement intercluster data protection mirrors
 Recover from a disaster using data protection mirrors
 Create a Snapshot copy of a volume and create Snapshot
policies
 Promote a Snapshot copy to be a read/write volume
 Answer questions about NDMP backup strategies

NetApp Confidential 2

MODULE OBJECTIVES

12-2 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

12-3 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Data-Protection Methods

 Snapshot copies
 Mirror copies for data protection and load
sharing
 Tape backups through third-party backup
software and NDMP for backup and
restoration
 Restores:
– volume snapshot restore
– LS Mirrors: snapmirror promote
– DP Mirrors: snapmirror resync
– NDMP restore
NetApp Confidential 4

DATA-PROTECTION METHODS
A customer’s data-protection plan likely uses all of the methods of protecting data that are shown on this
slide.

12-4 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Disaster Recovery

 Data ONTAP operating in Cluster-Mode uses

NDMP with third-party software for disaster
recovery.
 A Data ONTAP Cluster-Mode system can be
an ndmpcopy source or destination.
 Data-protection intracluster and intercluster
mirrors are built-in, disk-to-disk backups.

NetApp Confidential 5

DISASTER RECOVERY
No native backup or restore commands are available in Data ONTAP 8.1.1 operating in Cluster-Mode. All
tape backups and restores are performed through third-party NDMP applications.

12-5 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Snapshot Functionality

 You can create Snapshot copies manually or

schedule them by using Snapshot policies.
 A user can restore files and directories
through a client, or an administrator can
restore an entire volume.
 SnapRestore functionality:
– Restores an entire volume
– Is provided by means of the volume
snapshot restore command
– Requires the SnapRestore license

NetApp Confidential 6

SNAPSHOT FUNCTIONALITY
Snapshot functionality is controlled by management, which provides the UI for manual Snapshot copies and
the Job Manager policies and schedules for automated Snapshot operations. Each volume can have a Snapshot
policy that is associated with it. A policy can include multiple schedules so that Snapshot copies are created
hourly, daily, weekly, and so on. A policy also indicates how many of each type of Snapshot copy to retain
before deleting older copies. For example, you can keep four hourly Snapshot copies, and when the fifth one
is created, the oldest one is removed, such that a continuously updated group of the previous four hourly
Snapshot copies is retained.
Clients can see and use the .snapshot directories, so users can restore their own data without administrator
intervention. When an entire volume must be restored from a Snapshot copy, an administrator uses the
volume snapshot restore command, which is the same as restoring the volume by using
SnapRestore technology. The entire Snapshot copy is promoted, which replaces the entire volume. Individual
files can be restored only through a client.

12-6 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

The volume snapshot show Command
netappu::> volume snap show -vserver vs7 -volume vs7_vol1
---Blocks---
Vserver Volume Snapshot Size Total% Used%
-------- ------- ---------------------------------- ------------ ------ -----
vs7 vs7_vol1
weekly.2011-09-22_0015 88KB 0% 37%
5min.2011-09-23_1120 76KB 0% 34%
5min.2011-09-23_1125 72KB 0% 33%
5min.2011-09-23_1130 92KB 0% 38%
weekly.2011-09-29_0015 56KB 0% 27%
daily.2011-10-02_0010 56KB 0% 27%
daily.2011-10-03_0010 52KB 0% 26%
hourly.2011-10-03_0605 52KB 0% 26%
hourly.2011-10-03_0705 52KB 0% 26%
hourly.2011-10-03_0805 52KB 0% 26%
hourly.2011-10-03_0905 52KB 0% 26%
hourly.2011-10-03_1005 52KB 0% 26%
hourly.2011-10-03_1105 52KB 0% 26%
13 entries were displayed.

NetApp Confidential 7

THE VOLUME SNAPSHOT SHOW COMMAND

The Snapshot copies that are shown on this slide are scheduled Snapshot copies. The copies include two daily
Snapshot copies, six hourly Snapshot copies, and one weekly Snapshot copy.

12-7 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Mirror Differences Between Data ONTAP
7-Mode and Data ONTAP Cluster-Mode
 Data ONTAP operating in 7-Mode:
– Synchronous and asynchronous mirroring
– Volume SnapMirror and qtree SnapMirror replication
– Data-protection mirror copies only; cascading mirror copies
 Data ONTAP operating in Cluster-Mode:
– A new replication engine
– No qtree SnapMirror replication, but replication by using
volumes
– Asynchronous mirroring only
– Data-protection and load-sharing mirror copies, but no
cascading mirror copies. (All destinations are updated from
the source.)
– Support for intercluster SnapMirror replication

NetApp Confidential 8

MIRROR DIFFERENCES BETWEEN DATA ONTAP 7-MODE AND DATA ONTAP

CLUSTER-MODE

12-8 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

The SnapMirror Engine
 Is used for the volume move, volume
copy, and snapmirror commands
 Uses SpinNP as the transport protocol
between the source and destination volumes
(intracluster only)
 Uses a Snapshot copy of the source,
determines the incremental differences, and
transfers only the differences

NetApp Confidential 9

THE SNAPMIRROR ENGINE

Internally, the incremental data transfer algorithm is used for moving, mirroring, and copying volumes. Data
is transferred over the cluster network by using the proprietary SpinNP protocol. On the source side, Snapshot
copies of the volumes are used as the source of the data transfers so that the source volume itself can still be
read from and written to while the data transfer is happening from the Snapshot copy.

12-9 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Load-Sharing and Data-Protection Mirror
Copies 1 of 2
 Mirror copies are read-only copies of a volume.
 Mirror copies are volumes that have SnapMirror
relationships with source volumes.
 Mirror copies are updated from source volumes
manually or are scheduled (automated).
 Load-sharing mirror relationships stay within the
Vserver of the source volume.
 Data-protection mirror relationships can be within a
Vserver, between Vservers within the cluster, and
between Vservers of two different clusters.
 Mirrors cannot be cascaded.

NetApp Confidential 10

LOAD-SHARING AND DATA-PROTECTION MIRROR COPIES: 1 OF 2

Mirror copies are read-only volumes. Each mirror copy has an association read/write volume and is labeled as
a load-sharing or data-protection mirror copy. Load-sharing and data-protection mirror copies are similar
conceptually, but the type dictates how the mirror copy is used and maintained.
Mirror copies are copies of read/write volumes. Mirror copies are synchronized with the read/write source
volumes only to the extent that an administrator keeps them synchronized through manual replication or
scheduled (automated) replication. Generally, data-protection mirror copies do not need to be as up-to-date as
load-sharing mirror copies do, because of their different purposes.
Each mirror copy can have a replication schedule that is associated with it, which determines when (cron) or
how often (interval) replications are performed on the mirror copy. All load-sharing mirror copies of a volume
are treated as a unified group and use the same schedule. The schedule is enforced by the UI, so if you choose
a different schedule for one load-sharing mirror copy, the other load-sharing mirror copies of that volume are
automatically changed to match. Each data-protection mirror copy is independent—it does not have to use the
same schedule as other data-protection mirror copies.
LS mirror relationships stay within the Vserver of the source volume. DP mirrors can be within a Vserver,
between Vservers within the cluster, and between Vservers of two different clusters.

12-10 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Load-Sharing and Data-Protection Mirror
Copies 2 of 2
 A volume must be created before the volume can be
used as a mirror destination.
 A SnapMirror volume must be created as type DP, a
RW volume cannot be changed to a DP mirror.
 Creating a mirror relationship does not cause an initial
update to be performed.
 An LS mirror copy can be promoted to become the
source volume using the snapmirror promote
command.
 A DP mirror copy can be converted to a writable
volume using the snapmirror break command.
 A mirror copy can be “restored” to its source.

NetApp Confidential 11

LOAD-SHARING AND DATA-PROTECTION MIRROR COPIES: 2 OF 2

All replication is performed directly from the read/write volume to the appropriate mirrors. This method is
different from the cascading that occurs within the Data ONTAP 7G operating system.
Creating a mirror copy, associating the mirror copy with a source volume, and replicating to it are separate
steps.
A load-sharing or mirror copy can be promoted (similar to a restoration by using SnapRestore technology) to
replace its read/write volume.

12-11 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

snapmirror promote (LS Only)

 Performs a failover to destination volume.

 Changes destination volume from read-only to
read-write and assumes the identity of the
source volume and its SnapMirror
relationships (becomes new source vol).
 Destroys the original source volume.
 The destination volume must be a load-
sharing volume.
 Client accesses are redirected from the
original source volume to the promoted
destination volume.
NetApp Confidential 12

SNAPMIRROR PROMOTE (LS ONLY)

12-12 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Mirror Creation Steps
1. Create a (mirror) volume: volume create
2. Create a mirror relationship: snapmirror create
3. Perform baseline replication:
– Data protection: snapmirror initialize
– Load sharing: snapmirror initialize-ls-set
4. Perform incremental replication:
– Data protection: snapmirror update
– Load sharing: snapmirror update-ls-set

NOTE: The update commands work for the baseline

synchronization, too, so the initialize step can be
skipped.
NetApp Confidential 13

MIRROR CREATION STEPS

12-13 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

The snapmirror show Command
netappu::> snapmirror show
Source Destination Mirror Relationship Total
Path Type Path State Status Progress Healthy
------------- ---- ------------ ------------- -------------- ---------- -------
netappu://vs2/vs2root
DP netappu://vs2/vs2root_dp1
Snapmirrored Idle - true
netappu://vs2/vs2root_dp2
Snapmirrored Idle - true
LS netappu://vs2/vs2root_ls2
Snapmirrored Idle - true
3 entries were displayed.

NetApp Confidential 14

THE SNAPMIRROR SHOW COMMAND

This slide shows that the volume that is called vs2root has three mirror copies: two data-protection mirror
copies and one load-sharing mirror copy.

12-14 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

The snapmirror show – instance
Command
netappu::> snapmirror show -source-volume vs2root -type ls -instance

Source Path: netappu://vs2/vs2root

Destination Path: netappu://vs2/vs2root_ls2
Relationship Type: LS
Managing Vserver: vs2
SnapMirror Schedule: 5min
Tries Limit: 8
Throttle (KB/sec): unlimited
Mirror State: Snapmirrored
Relationship Status: Idle
Transfer Snapshot: -
Snapshot Progress: -
Total Progress: -
Snapshot Checkpoint: -
Newest Snapshot: snapmirror.79deda29-e8a6-11e0-b411-
123478563412_4_2147484676.2011-10-05_023500
Newest Snapshot Timestamp: 10/05 02:35:00
Exported Snapshot: snapmirror.79deda29-e8a6-11e0-b411-
123478563412_4_2147484676.2011-10-05_023500
Exported Snapshot Timestamp: 10/05 02:35:00
Healthy: true

NetApp Confidential 15

THE SNAPMIRROR SHOW – INSTANCE COMMAND

The instance view of the vs2root_ls2 mirror copy shows when the mirror copy was last replicated and other
information.

12-15 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Load-Sharing Mirror Copies
 Load-sharing mirror copies are primarily used for load sharing
(balancing) when client read access is used.
 Read access requests for a volume are distributed to the
volume’s load-sharing mirror copies, unless the special “.admin”
path is used.
 Load-sharing mirror copies are automatically available in the
namespace.
 Load-sharing mirror copies are implicitly accessed by clients (for
read access).
 Junctions are accessible in load-sharing mirror copies.
 Load-sharing mirror copies are always replicated as a group.
 A source volume can have a maximum of one load-sharing
mirror copy per node.

NetApp Confidential 16

LOAD-SHARING MIRROR COPIES

The purpose of load-sharing mirror copies is to offload volumes (and a single D-blade) of read activity.
Therefore, all mirror copies must be synchronized at the same data-version level. When a volume is replicated
to its load-sharing mirror copies, all load-sharing mirror copies of the volume are synchronized directly from
the volume (without cascading).
The way that NFS is mounted on a client, or which CIFS share is mapped to the client, changes which data is
accessed (the read/write volume or one of its load-sharing mirror copies). NFS is usually mounted at the root
of a virtual server (Vserver) by using a command such as mount <ip_address>://myvserver. This
command causes the load-sharing selection algorithm to be invoked. If, however, the NFS mount command is
executed by using the “.admin” path, such as mount <ip_address>:/.admin /myvserver, this
mount from the client always accesses the read/write volumes when traversing the namespace, even if there
are load-sharing mirror copies for volumes.
For CIFS, the difference is not in how a share is accessed but in which share is accessed. If you create a share
for the “.admin” path and use that share, the client always has read/write access. If you create a share without
using “.admin,” the load-sharing selection algorithm is used.
Clients are transparently directed to a load-sharing mirror copy for read operations rather than to the
read/write volume, unless the special “.admin” path is used.

12-16 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Load-Sharing Mirror Selection
The Data ONTAP operating system:
 If there are no load-sharing mirror copies of a volume, the N-blade uses
the volume itself.
 If a load-sharing mirror copy is on the same node as the N-blade that
fields the request, the N-blade uses that load-sharing mirror copy.
 If no load-sharing mirror copy is on the same node as the N-blade that
fields the request, the N-blade uses an up-to-date load-sharing mirror
copy on another node.
NFS and CIFS:
 NFS: A new load-sharing mirror can be selected even if a file remains
open.
 NFSv4: Clients are routed to the source of the load-sharing mirror for
direct read and write access.
 CIFS: A new load-sharing mirror is not selected while a file remains
open.

NetApp Confidential 17

LOAD-SHARING MIRROR SELECTION

When the / path is used (that is, the “/.admin” path is not used) and a read or write request comes through that
path into the N-blade of a node, the N-blade first determines if there are any load-sharing mirror copies of the
volume that it needs to access. If there aren’t any load-sharing mirror copies of that volume, the read request
is routed to the read/write volume. If there are load-sharing mirror copies of the volume, preference is given
to a load-sharing mirror copy on the same node as the N-blade that fielded the request. If there isn’t a load-
sharing mirror copy on that node, an up-to-date load-sharing mirror copy from another node is chosen.
If a write request goes to a load-sharing mirror copy, it returns an error to the client, which indicates that the
file system is read-only. To write to a volume that has load-sharing mirror copies, you must use the “.admin”
path.
For NFS clients, a load-sharing mirror copy is used for a set period of time (minutes), after which a new load-
sharing mirror copy is chosen. After a file is opened, different load-sharing mirror copies can be used across
different NFS operations. The NFS protocol can manage the switch from one load-sharing mirror copy to
another.
For CIFS clients, the same load-sharing mirror copy continues to be used for as long as a file is open. After
the file is closed, and the period of time expires, a new load-sharing mirror copy is selected before the next
time that a file is opened. Because the CIFS protocol cannot manage the switch from one load-sharing mirror
copy to another, CIFS clients use this process.

12-17 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Typical LS Mirror Issues
1 of 2
Client machines cannot see volumes that have
been created.
 The volume must be mounted (given a
junction path) to the namespace.
 Replicate the parent volume.

NOTE: You should schedule synchronization of

virtual server (Vserver) root volumes to run
hourly.

NetApp Confidential 18

TYPICAL LS MIRROR ISSUES: 1 OF 2

12-18 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Typical LS Mirror Issues
2 of 2
 Client requests always go to the source volume rather than
to the load-sharing mirror copy. This issue occurs when
the client is mounted by using a .admin path or share.
 Because the client is read-only, client write requests fail.
– This issue occurs when the client is not mounted by using the
.admin path or share.
– For read/write NFS access to a volume that has load-sharing
mirror copies, clients must be mounted by using the .admin
path.
– For read/write CIFS access to a volume that has load-sharing
mirror copies, a specific volume .admin CIFS share must be
created and the clients must connect to that share.

NetApp Confidential 19

TYPICAL LS MIRROR ISSUES: 2 OF 2

12-19 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Data-Protection Mirror Copies
1 of 2
 Data-protection mirror copies are not implicitly
accessed by clients.
 Data-protection mirror copies can be mounted
(through a junction) into the namespace by the
administrator.
 In data-protection mirror copies, junctions are not
accessible.
 Each data-protection mirror copy is replicated
independent of the load-sharing mirror copies and of
other data-protection mirror copies of the same
source volume.
 A source volume can have a maximum of four data-
protection mirror copies.
NetApp Confidential 20

DATA-PROTECTION MIRROR COPIES: 1 OF 2

Data-protection mirror copies are not meant for client access, although they can be mounted into the
namespace by an administrator. Junctions cannot be followed in a data-protection mirror copy, so access is
given only to the data that is contained in that data-protection mirror copy, not to any other volumes that are
mounted to the source read/write volume.
Data-protection mirror copies are primarily meant for disk-based online backups. Data-protection mirror
copies are simpler, faster, more reliable, and easier to restore than tape backups are, although data-protection
mirror copies are not portable for storing offsite. A typical use of data-protection mirror copies is to put them
on aggregates of SATA disks that use RAID-DP technology and then mirror data to them daily during the
least active time in the cluster. One data-protection mirror copy per volume is generally sufficient.

12-20 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Data-Protection Mirror Copies
2 of 2
 Consider using inexpensive, high-capacity (and
slower) SATA disks for data-protection mirror copies.
 Data-protection mirror copies can be restored or
resynchronized:
– To restore a mirror copy is to re-create a broken
SnapMirror relationship such that destination changes
overwrite the source data.
– To resynchronize a mirror copy is to re-create a broken
SnapMirror relationship such that source changes
overwrite the destination data.
– You can restore and resynchronize to a new volume.

NetApp Confidential 21

DATA-PROTECTION MIRROR COPIES: 2 OF 2

A feature that is available only for data-protection mirror copies is the ability to perform a SnapMirror restore.
This action can restore a broken mirror relationship between a source and destination and perform an
incremental overwrite of the source volume with the current contents of the mirror destination. If the restore is
performed between a source and destination that didn’t formerly have a SnapMirror relationship, a baseline
copy of the destination contents are performed to the source volume.
Resynchronizing a source and destination is similar to restoring a source and destination, except that the
source content overwrites the destination content.

12-21 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

New Logical Interface and Port Types
 Intercluster LIFs Ports Cluster LIFs
– New in Data
ONTAP 8.1
– Share data ports ifgrp1
with data LIFs or
e0a e0b e0c e0d
use dedicated
intercluster ports
– Node scoped!
failover only to
other intercluster
capable ports on Data LIFs
Intercluster LIFs
same node (IP Addresses)

NetApp Confidential 22

NEW LOGICAL INTERFACE AND PORT TYPES

12-22 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Intercluster SnapMirror Replication

 Replication between clusters for DR

 Data transfers on intercluster network
RW Source volume

Intercluster LIF Intercluster

connection WAN network

DP Destination volume

NetApp Confidential 23

INTERCLUSTER SNAPMIRROR REPLICATION

Intercluster SnapMirror replication, as opposed to traditional intracluster mirroring, gives you the flexibility
to create an asynchronous SnapMirror volume on a cluster other than the source volume’s cluster for data
protection. The replication is carried out across the WAN by using intercluster logical interfaces (LIFs). You
can use intercluster SnapMirror replication to store online copies of your data off-site for disaster recovery.
To use intercluster SnapMirror replication, you must license the feature on both participating clusters.
You need a full mesh intercluster network to support node failover and volume moves of the source or
destination volumes. For the network to be full mesh, every intercluster LIF on every node in the cluster must
be able to connect to every intercluster LIF on every node in the peer cluster.

12-23 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Intercluster Networking for SnapMirror

 Requires the clusters to be peered together

 Requires full mesh connectivity of intercluster
LIFs

Every intercluster LIF on every

node in the cluster
WAN
must be able to
connect to every intercluster LIF on
every node in the other cluster.

NetApp Confidential 24

INTERCLUSTER NETWORKING FOR SNAPMIRROR

12-24 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Snapshot Copies
1 of 2
bnetappu::> vol snap show -vserver vs2 -volume vs2root
(volume snapshot show)
---Blocks---
Vserver Volume Snapshot Size Total% Used%
-------- ------- ---------------------------------- ------------ ------ -----
vs2 vs2root
weekly.2011-10-02_0015 84KB 0% 1%
daily.2011-10-04_0010 80KB 0% 1%
snapmirror.79deda29-e8a6-11e0-b411-
123478563412_4_2147484684.2011-10-04_052359
0% 1%
92KB
hourly.2011-10-04_2105 72KB 0% 1%
hourly.2011-10-04_2205 72KB 0% 1%
hourly.2011-10-04_2305 72KB 0% 1%
hourly.2011-10-05_0005 72KB 0% 1%
daily.2011-10-05_0010 72KB 0% 1%
hourly.2011-10-05_0105 72KB 0% 1%

NetApp Confidential 25

SNAPMIRROR SNAPSHOT COPIES: 1 OF 2

The Snapshot copies that are shown on this slide are (mostly) scheduled Snapshot copies; three of them are
SnapMirror Snapshot copies. A separate reference Snapshot copy of a read/write volume (vs2root in this
example) is needed for every mirror copy that exists and that has a data version other than the read/write
volume. Because the replication process uses the SnapMirror Snapshot copy for a given mirror copy to
determine what has changed since the previous replication was performed, incremental mirroring is achieved
in this way. Because load-sharing mirrors are synchronized, there should be only one SnapMirror Snapshot
copy for all of the load-sharing mirrors. In this example, the other two SnapMirror Snapshot copies are for
data-protection mirror copies. If all load-sharing and data-protection mirror copies are synchronized, there is
exactly one SnapMirror Snapshot copy, and if there are no mirror copies, there are no SnapMirror Snapshot
copies.

12-25 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Snapshot Copies
2 of 2
---Blocks---
Vserver Volume Snapshot Size Total% Used%
-------- ------- ---------------------------------- ------------ ------ -----
snapmirror.79deda29-e8a6-11e0-b411-
123478563412_4_2147484683.2011-10-05_020500
0% 1%
60KB
hourly.2011-10-05_0205 72KB 0% 1%
snapmirror.79deda29-e8a6-11e0-b411-
123478563412_4_2147484676.2011-10-05_023500
0% 1%
72KB
12 entries were displayed.

NetApp Confidential 26

SNAPMIRROR SNAPSHOT COPIES: 2 OF 2

12-26 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 27

LESSON 2

12-27 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Create
Intracluster or intercluster replication (TR-4015)

Primary Disaster Recovery

Data Center Data Center

R WAN
R
A B B
dp

Create Volume for Mirror (on disaster recovery Vserver):

volume create –vserver vserver -volume datavolume_dp -aggr <aggrname>
-size <equal to datavolume_A size> -type dp
Create Mirror (from disaster recovery site):
snapmirror create -destination-path DR://vserver/datavolume_dp
-source-path PRI://vserver/datavolume_A

NOTE: A cluster peer relationship must be created for intercluster replication.

NetApp Confidential 28

SNAPMIRROR DATA PROTECTION: CREATE

The -type dp argument automatically puts the volume in restricted mode and is required. You cannot
modify an RW volume type (default) to a DP type. The slides work with either intracluster or intercluster
relationships. The peer relationship, intercluster LIFs (and intercluster ports, if required) must also be created
for intercluster SnapMirror.
8.1.1 mirrors can be intracluster (within the cluster) or intercluster (to a peer cluster). No replication is
currently available between 7-Mode and Cluster-Mode.
Asynchronous only
In 8.1.1 no support for:
 Cascading
 Vserver level management
 Vserver DR
 Tape or Disk seeding
All DP mirrors require licenses on source and destination clusters.
Intercluster snapmirror requires a peer cluster relations configuration.
Scheduling must also be configured. This is part of the SnapMirror creation wizard when using System
Manager (recommended).
You can find more details in TR-4015 and the Data Protection Guide.
Because intercluster LIFs are node scoped intercluster LIFs cannot failover to other nodes in the cluster. If
using intercluster ports then intercluster LIFs can fail over to other intercluster ports on the same node. If
intercluster LIFs are assigned to data ports then intercluster LIFs can fail over to any data port on the same
node.

12-28 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Initialize and Update

Primary DR
Data Center Data Center

R
WAN
R
A B B
dp

Initialize Mirror – Baseline Transfer (from DR site):

snapmirror initialize -destination-path
DR://vserver/datavolume_dp -source-path
PRI://vserver/datavolume_A
Update Mirror – Incremental Transfers (from DR site):
snapmirror update -destination-path
DR://vserver/datavolume_dp -source-path
PRI://vserver/datavolume_A

NetApp Confidential 29

SNAPMIRROR DATA PROTECTION: INITIALIZE AND UPDATE

Create, initialize and update separately with DP mirrors. Using the initialize command on a DP mirror
before creating the mirror will result in a “mirror has not been created” error.
With LS mirrors, using an update command alone creates the mirror, initializes (baseline) the mirror and
update (incremental) the mirror. Using the initialize command on LS mirrors also will create the mirror
and then initialize it.
The initialize and update commands work differently in LS and DP. The mirror must be created before
initializing or updating the mirror. Both commands must be run from the destination cluster.

12-29 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Failover Considerations
 SnapMirror does not replicate the Vserver namespace
junction path information
 Because the NAS volumes have no junction path,
they will not be accessible after a SnapMirror break
occurs unless they are pre-mounted before failover,
or until they are mounted after failover
 The security style and permissions on the destination
vserver root volume must be set correctly or the
namespace might be inaccessible after failover
 Use the Cluster Config Dump Tool to collect and
replicate system configuration settings to a disaster
recovery site

NetApp Confidential 30

FAILOVER CONSIDERATIONS
Currently this is a manual task. If there are multiple volumes in the namespace, this will have to repeated for
each volume.
The Cluster Config Dump Tool ([Link] is a Java based
Windows/Linux/Mac utility collects configuration information.
Stores information needed in DR scenario:
 Volume junction paths
 NFS export policies, CIFS shares information
 Snapshot & Storage Efficiency policies
 LUN mapping information
Run locally and replicate or run from remote
Does not restore configuration

12-30 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Disaster
Primary DR
Data Center Data Center

R
WAN
R
A B B
dp

Break Mirror (from DR) – Make destination writeable:

snapmirror break -destination-path
DR://vserver/datavolume_dp
-source-path PRI://vserver/datavolume_A

NOTE: Admin must redirect the clients (or host) of the source
volume on the primary site to the new source volume at the DR
site in a disaster situation.

NetApp Confidential 31

SNAPMIRROR DATA PROTECTION: DISASTER

Breaking the mirror does not delete the relationship (as you will see later) only stops the updates to the mirror
and make the mirrored volume writable. You can then redirect the clients to access the DR site mirrored
volume.

12-31 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Disaster Considerations

 LUNs should be mapped after failover

 CIFs shares/NFS exports must be created
(or verified)
 Assign Snapshot schedules
 Assign Storage Efficiency policies

NetApp Confidential 32

DISASTER CONSIDERATIONS
Currently this is a manual task. If there are multiple volumes in the namespace, the steps are repeated for each
volume.

12-32 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Recovery Scenario A: Dev/Test Recovery
Primary DR
Data Center Data Center

R
WAN
R
A B B
dp

Resync Mirror (From DR) – Resume relationship:

snapmirror resync -destination-path
DR://vserver/datavolume_dp
-source-path PRI://vserver/datavolume_A

NOTE: All “new” data written to the destination after the break will
be deleted.

NetApp Confidential 33

SNAPMIRROR DATA PROTECTION: RECOVERY SCENARIO A:

DEV/TEST RECOVERY
Think of Scenario A as going back to the time before the disaster. Anything that happened before the disaster
(or break) is deleted. Although this scenario can be used in DR, it is more commonly used in test and
development environments. Be aware that in test/dev situations, clients (or host) may not be redirected to DR
site.
Resyncing the mirror will cause the source and destination to revert back to the last common snapshot. Since
any data written to the destination volume happened after the common snapshot point-in-time, the new data
will be deleted.

12-33 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Recovery Scenario B: Source Is Recoverable
Primary DR
Data Center Data Center

R WAN
R
A B B
dp

Delete Mirror (From DR) – Remove Relationship:

snapmirror delete -destination-path
DR://vserver/datavolume_dp
-source-path PRI://vserver/datavolume_A

NOTE: You must delete the mirror to change source to

destination for resync.

NetApp Confidential 34

SNAPMIRROR DATA PROTECTION: RECOVERY SCENARIO B:

SOURCE IS RECOVERABLE
Volumes in cluster-mode can only be a source or a destination, but not both. A source can be a source for
many destinations (fan out), but can not cascade. If the data in the source volume is recoverable, and the data
is still intact, you may have considerable amount of new data on the destination that you do not want deleted.
To prevent a complete re-baselining or deleting of new data, you will need to delete the mirror relationship
first.

12-34 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Recovery Scenario B: Change Relationship
Primary DR
Data Center Data Center

R
WAN
R
A B B
dp

Create a new relationship with DR as source:

snapmirror create -destination-path
PRI://vserver/datavolume_A
-source-path DR://vserver/datavolume_dp
Resync the mirror (From DR) – Copy new data to Primary (no
re-baseline):
snapmirror resync -destination-path
PRI://vserver/datavolume_A -source-path
DR://vserver/datavolume_dp

NetApp Confidential 35

SNAPMIRROR DATA PROTECTION: RECOVERY SCENARIO B:

CHANGE RELATIONSHIP
In this scenario no baseline would be required since a common snapshot is available. Only changed blocks
would be sent. To go back to the original relationship, follow steps on the previous slide.
After deleting the original snapmirror relationship, you can now create a new relationship (switching the
roles). The DR will now be the source and the Primary will now be the destination. To prevent a complete
baseline, use the resync command instead of the initialize command. This will find the common
snapshot and copy only the changes made to the DR site to the Primary site.
You can keep this relationship if desired. Otherwise, you will need to follow steps outlined in the Data
Protection Guide.

12-35 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

SnapMirror Data Protection
Recovery Scenario C: Source Is Unrecoverable
Primary DR
Data Center Data Center

R
WAN
R
A B B
dp

 Create New Volume on source

 Create a new relationship with DR as source
 Initialize Mirror (re-baseline required)

NOTE: You can keep this configuration or, after resolving the
problem with the original source volume by following steps in the
next slide.

NetApp Confidential 36

SNAPMIRROR DATA PROTECTION: RECOVERY SCENARIO C:

SOURCE IS UNRECOVERABLE
Re-baselining required a great amount of resources and bandwidth. Since cluster-mode does not allow seeding
at this time, re-baselining should be avoided at all cost. Scenarios A and B should be avoided if at all possible.
If the volume is corrupted or unrecoverable, you will need to create a new volume and relationship. You will
need to reininitialize the mirror which will require a complete re-baseline.
As with Scenario B, you can keep this relationship if desired. Otherwise, you will need to follow steps
outlined in the Data Protection Guide.

12-36 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Steps To Resume Original Mirror
If after scenarios B and C you want to go back to
original relationship, follow these steps:
1. Redirect clients
2. Update mirrored volumes using snapmirror
update
3. Break the mirror using snapmirror break
4. Delete the mirror using snapmirror delete
5. Recreate a original relationship using
snapmirror create (Dest:DR – Source:
Primary)
6. Resync from DR site using snapmirror resync
(no re-baseline required)
NetApp Confidential 37

STEPS TO RESUME ORIGINAL MIRROR

Currently this is a manual task. If there are multiple volumes in the namespace, this will have to repeated for
each volume.

12-37 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Additional SnapMirror Considerations

 The source and destination Vserver language

type must be the same.
 The Vserver language type can be set only at
time of the creation of the Vserver.
 System Manager 2.0 current issues:
– No support for managing SnapMirror throttle
– No support for creation and management of
intercluster ports; however, SM2.0 can peer
clusters if intercluster ports are preconfigured

NetApp Confidential 38

ADDITIONAL SNAPMIRROR CONSIDERATIONS

12-38 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Lesson 3

NetApp Confidential 39

LESSON 3

12-39 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Job Schedules
 Job schedules can be used:
– Globally (by all Vservers)
– For automated Snapshot copy creation
– For automated mirror updates
 Note the following job schedule syntax:
– @:00,:05,:10...:55 means every five minutes on the
five-minute marks
– @2 means daily at 2:00 a.m.
– @0:10 means daily at 12:10 a.m.
– @:05 means hourly at five minutes after the hour

NetApp Confidential 40

JOB SCHEDULES

12-40 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

The job schedule show Command
netappu::> job schedule show
Name Type Description
----------- --------- ------------------------------------------------
5min cron @:00,:05,:10,:15,:20,:25,:30,:35,:40,:45,:50,:55
8hour cron @2:15,10:15,18:15
avUpdateSchedule
cron @2:00
daily cron @0:10
hourly cron @:05
weekly cron Sun@0:15
6 entries were displayed.

NetApp Confidential 41

THE JOB SCHEDULE SHOW COMMAND

Mirror copies use schedules directly. Snapshot copies are controlled by Snapshot policies. Each Snapshot
policy contains one or more schedules. For mirror copies, the schedule is defined as part of the mirror
definition. For Snapshot copies, the Snapshot policy is defined as part of the read/write volume definition.
Schedules are maintained under the job command directory. The schedules that are shown in this slide are
defined by default. If a mirror copy is assigned the “5min” schedule, for example, the mirror is replicated
every five minutes, based on the system clock. If a Snapshot policy assigns the hourly schedule to a mirror
copy, a Snapshot copy is created at five minutes after every hour.

12-41 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Snapshot Policies
netappu::> volume snapshot policy show
Number Of Is
Name Schedules Enabled Comment
----------------- ---------- ------- ----------------------------------------
-
default 3 true Default policy with hourly, daily &
weekly schedules.
Schedule: hourly Count: 6 Prefix: hourly
daily 2 daily
weekly 2 weekly

none 0 false Policy for no automatic snapshots.

Schedule: - Count: - Prefix: -

2 entries were displayed.

NetApp Confidential 42

SNAPSHOT POLICIES
Two Snapshot policies are automatically created: default and none. If a volume uses none as its Snapshot
policy, no Snapshot copies of the volume are created. If a volume uses the default policy, after two weeks,
there is a total of 10 Snapshot copies of the volume (6 hourly copies, 2 daily copies, and 2 weekly copies).

12-42 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Lesson 4

NetApp Confidential 43

LESSON 4

12-43 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Backup and Restoration with NDMP
1 of 2
 You can perform local NDMP, remote NDMP, and
three-way NDMP backups.
– NDMPv3 and NDMPv4
– Direct Access Recovery (DAR)
 A Data ONTAP Cluster-Mode system does not
provide native backup and restoration, only
NDMP through third-party software.
 Backups are not cluster-transparent.
 Backups do not traverse junctions; you must list
every volume to be backed up.
 You should not back up directly through NFS
or CIFS.
NetApp Confidential 44

BACKUP AND RESTORATION WITH NDMP: 1 OF 2

Backups are the only elements in Data ONTAP clusters that are not cluster-aware. Therefore, a backup
administrator must know which volumes are on which nodes (determined by using volume show queries for
each node).
Backups can be performed across a cluster by using three-way NDMP, provided that the third-party backup
application is given access to the cluster network.
A backup of a source volume does not include all of the volumes that are mounted to the source volume.
NDMP backups do not traverse junctions. Therefore, every volume that is to be backed up must be listed
explicitly. However, if the backup vendor software supports automatic discovery of file systems or the use of
wildcards, not every volume must be specified.
Although backing up through an NFS or CIFS client is possible, doing so uses all of the cluster resources that
are meant to serve data and fill the N-blade caches with data that most clients aren’t actually using. You
should send the data through a dedicated FC connection to the tape device or devices by using NDMP, which
doesn’t use the resources of the N-blade, data network, or cluster network.

12-44 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Backup and Restoration with NDMP
2 of 2
 NDMP is not aware of volume movement.
 NDMP understands nodes and volumes, not
clusters or namespaces.
 The NDMP user name and password (set on
each node and used by the data management
application for authentication) have no
correlation to other administrative user names
or passwords for the cluster.

NetApp Confidential 45

BACKUP AND RESTORATION WITH NDMP: 2 OF 2

12-45 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Configuring for NDMP
1. Enable and configure NDMP on the node or
nodes:
netappu::> system services ndmp modify
2. Identify tape and library attachments:
netappu::> system node hardware tape
drive show
netappu::>system node hardware tape
library show
3. Configure the data management application
(such as Symantec NetBackup) for NDMP.

NOTE: The smallest backup level is the volume.

NetApp Confidential 46

CONFIGURING FOR NDMP

Enabling or disabling NDMP on a node is simple. You must enable NDMP on each node in the cluster that
will be used for backups (probably all of them). The NDMP data management application uses the NDMP
user name and password that you choose during this configuration.
You must know the tape drive and library paths to configure the data management application.

12-46 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Common NDMP Backup Mistakes
 Trying to back up a volume that doesn’t reside
on the node where the backup is performed
 Trying to back up a volume that used to be on a
node
 Specifying the volume’s junction path as the
backup path. (The backup path of a volume is
flat, with only the Vserver name preceding it:
/<vserver>/<volume>.)
 Misconfiguring a data management application

NetApp Confidential 47

COMMON NDMP BACKUP MISTAKES

12-47 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Data ONTAP Cluster-Mode Systems
and NDMP
 Data ONTAP Cluster-Mode systems support
the Symantec NetBackup and IBM Tivoli
Storage Manager (TSM) data management
applications, and more are being added.
 Data ONTAP Cluster-Mode systems support
local NDMP, remote NDMP, and three-way
NDMP backup: NDMPv3 and NDMPv4.
 A data management application with DAR can
restore selected files without sequentially
reading entire tapes.

NetApp Confidential 48

DATA ONTAP CLUSTER-MODE SYSTEMS AND NDMP

Forms of NDMP backup:
 Local NDMP: Data is backed up directly from the storage system (a node, in this case, not a cluster) to a
locally attached (or SAN-attached) tape device.
 Remote NDMP: Data is transported from the storage system to the data management application server.
The server is attached to a tape device and backs up the data to the tape device.
 Three-way NDMP: Data is transported from the storage system to another storage system that has a
locally attached (or SAN-attached) tape device.
Direct Access Recovery (DAR) is the ability of a data management application to restore a selected file or
selected files without the need to sequentially read the entire tape or tapes that are involved in a backup. For
example, if a large backup spans four 100-GB tapes, and the one file that you want to restore is on the fourth
tape, a non-DAR restoration reads the first three tapes in their entirety and then all of the fourth tape up to the
point that it finds the target file. With DAR, the data management application reads some information from
the first tape, and then goes directly to the appropriate spot on the fourth tape.

12-48 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Local, Remote, and Three-Way NDMP
LAN
Remote NDMP
NDMP Backup
Hosts
VERITAS
NetBackup
Server
Local Three-Way
Backup Backup
Tape Drive
Automated Automated
Tape Library Tape Library

NetApp Confidential 49

LOCAL, REMOTE, AND THREE-WAY NDMP

12-49 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

About Tape Devices

 Nodes can share tape libraries through

a SAN.
 Although multiple nodes can see a tape drive
on a SAN, only one node can use the tape
drive.
 You should share a large tape library among
nodes and dedicate multiple tape drives to
each node.

NetApp Confidential 50

ABOUT TAPE DEVICES

12-50 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

A Six-Node Cluster with Data-Protection
Mirror Backups
Cluster Network

Compute Farm

SATA Storage
and
Data-Protection
Mirror Copies

Data Network
Tape Library

NetApp Confidential 51

A SIX-NODE CLUSTER WITH DATA-PROTECTION MIRROR BACKUPS

If a customer can afford to dedicate some nodes to data protection (rather than to serving data to clients), the
customer has an excellent way to protect data. Data-protection nodes are members of the cluster, but their
aggregates consist of slower and denser disks. Data can be mirrored from the other nodes to these nodes and
then backed up from these nodes by using local NDMP, which alleviates the need for tape devices to be
connected to the other nodes. Some of the benefits of this technique are listed below :
 Data-protection mirror copies don’t need to move (or need to move only rarely), which alleviates one
source of headaches for backup administrators.
 Fewer NDMP licenses are needed, which can result in considerable cost savings.
 Fewer nodes must be backed up, which helps backup administrators.
 Nonbackup nodes can use their CPUs and resources for client access.
If these “disaster-recovery” nodes are not used for client access, they don’t need data LIFs and can reassign
those data network ports to be used for cluster network ports for a larger cluster “pipe.”

12-51 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

RDB Disaster Recovery

 The replicated database (RDB) cannot be

backed up.
 As long as one node survives, a copy of the
RDB is available and propagates itself to
reconstructed nodes.
 If all nodes are destroyed, the RDB data is
destroyed.

NetApp Confidential 52

RDB DISASTER RECOVERY

12-52 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Module Summary
Now that you have completed this module, you should be
able to:
 List the benefits of the mirroring engine
 Create load-sharing and data-protection mirror copies
 Manually and automatically replicate mirror copies
 Promote a mirror copy to replace its read/write volume
 Implement intercluster data protection mirrors
 Recover from a disaster using data protection mirrors
 Create a Snapshot copy of a volume and create Snapshot
policies
 Promote a Snapshot copy to be a read/write volume
 Answer questions about NDMP backup strategies

NetApp Confidential 53

MODULE SUMMARY

12-53 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Exercise
Module 12: Data Protection
Time Estimate: 120 minutes

NetApp Confidential 54

EXERCISE
Please refer to your exercise guide.

12-54 Clustered Data ONTAP Administration: Data Protection

NetApp University - Do Not Distribute

Module 13
Basic Troubleshooting
and Performance

NetApp Confidential 1

MODULE 13: BASIC TROUBLESHOOTING AND PERFORMANCE

13-1 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Explain how to configure and use the
AutoSupport tool for a cluster
 Demonstrate the use of the cluster Event
Management System (EMS)
 Identify any offline storage
 Review the required steps for forcing a
system core dump
 Describe the characteristics of a cluster that
affect the cluster’s performance

NetApp Confidential 2

MODULE OBJECTIVES

13-2 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Lesson 1

NetApp Confidential 3

LESSON 1

13-3 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

General Troubleshooting

 The cluster network is vital to overall cluster

health; the cluster network must work reliably.
 The log files on each node contain information
for that node only.
 From the clustershell, use the EMS to see
cluster-wide logs.
 From the systemshell, use log files to see
information for this node.
 The AutoSupport tool contains the log files as
attachments.

NetApp Confidential 4

GENERAL TROUBLESHOOTING
The cluster is maintained by constant communication over the cluster network. As such, the cluster network
must be reliable. One of the first elements to check when problems occur is the health of the cluster.
Each node writes to log files locally on that node. Those log files are only local and do not contain log
messages from the other nodes. Log messages are also written to the Event Management System (EMS),
which enables an administrator on one node (by using the UI) to see the event messages from all nodes in the
cluster.
The AutoSupport tool also provides an effective way to get the log files.

13-4 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Rebooting
 Reboot only as a last resort.
 Rebooting causes the loss of state information at
the time of the failure.
 Rebooting causes the loss of forensic data such
as memory buffers, state information, statistics,
and more.
 If you must reboot, consider capturing a system
core dump, too (if you have time or are directed to
do so by NetApp Support services or
Engineering).
 Do not reboot until you collect as much data as
possible while the problem is live.
NetApp Confidential 5

REBOOTING

13-5 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Log Files
 During a booting, all logging goes to /var/log/ with
resides on the compact flash card.
 After vol0 is available, logging goes to
/mroot/etc/log/mlog/.
 Each process has its own log, for example, [Link].
 Log files are rotated; some old ones are kept.
 EMS logs are at /mroot/etc/log/ems*.
 Use the clustershell debug log command:
– Provides a cluster-wide view of the logs
– Provides a customized list of included logs
– Is available in the diag privilege

NetApp Confidential 6

LOG FILES
While a node is booting, and until the vol0 volume is available, all logging goes to /var/log/. After vol0 is
available, the logging goes to /mroot/etc/log/.
Each process has its own log file, for example, [Link], [Link], and [Link].
Log files are rotated every time that the particular process starts and several previous log files are kept for
each process, for example, [Link].1 and [Link].2.
EMS messages are available to be viewed through the UI. The D-blade, N-blade, and Management event log
messages go to the EMS log. The EMS log is rotated once a week at the same time that the AutoSupport
messages are sent out.
The tail UNIX command prints the last few lines of a file to the screen. The f flag causes it to refresh that
output continuously as new data is written to that file. Using tail f for a log file is an effective way to watch
the logging as it happens. For example, if you run a command in the UI and get an error, you can open
another window to that node, run the tail f command on the log file that you think might provide
information for this error, and then go back to the other window or browser and run the UI command again.
This approach helps to establish the cause-and-effect relationship between a UI command and a log message.

13-6 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Log Files in /mroot/etc/log/mlog/
netappu-01% pwd
/mroot/etc/log/mlog
netappu-01% ls -l
total 9556
-rw-r--r-- 2 root wheel 16705 Oct 7 20:34 apache_access.log
-rw-r--r-- 2 root wheel 16705 Oct 7 20:34 apache_access.log.0000000001
-rw-r--r-- 2 root wheel 8912 Oct 7 15:19 apache_error.log
-rw-r--r-- 2 root wheel 8912 Oct 7 15:19 apache_error.log.0000000001
-rw-r--r-- 2 root wheel 21213 Oct 7 00:27 [Link]
-rw-r--r-- 2 root wheel 21213 Oct 7 00:27 [Link].0000000001
-rw-r--r-- 2 root wheel 56462 Oct 7 20:57 [Link]
-rw-r--r-- 2 root wheel 56462 Oct 7 20:57 [Link].0000000001
drwxr-xr-x 2 root wheel 4096 Oct 5 17:31 dead_logs
-rw-r--r-- 2 root wheel 241887 Oct 7 20:56 [Link]
-rw-r--r-- 2 root wheel 241887 Oct 7 20:56 [Link].0000000001
-rw-r--r-- 2 root wheel 1277743 Oct 7 20:57 [Link]
-rw-r--r-- 2 root wheel 1277743 Oct 7 20:57 [Link].0000000001
-rw-r--r-- 2 root wheel 2680581 Oct 7 20:57 [Link]
-rw-r--r-- 2 root wheel 2680581 Oct 7 20:57 [Link].0000000001
....
....

NetApp Confidential 7

LOG FILES IN /MROOT/ETC/LOG/MLOG/

13-7 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Cluster

 Are all the nodes healthy?

netappu::> cluster show
netappu::*> cluster ring show
(advanced privilege)
 For two-node clusters only, is two-node high
availability configured?
netappu::> cluster ha show
 Is the cluster network OK?
netappu::*> cluster ping-cluster
<node> (advanced privilege)

NetApp Confidential 8

CLUSTER
The cluster commands provide a quick check on the health of the cluster. Remember that a two-node cluster
must have two-node high availability enabled. If this step is forgotten, problems arise, especially during
storage failover (SFO).
The cluster ping-cluster command is a great way to ensure that all the cluster ports and cluster
logical interfaces (LIFs) are working properly.

13-8 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The cluster ping-cluster Command
netappu-01::*> cluster ping-cluster –node
netappu-01
Host is netappu-01
Getting addresses from network interface table...
Local = [Link] [Link]
Remote = [Link] [Link]
Ping status:
4 paths up, 0 paths down at 1500 size
4 paths up, 0 paths down at 4500 size
4 paths up, 0 paths down at 9000 size
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)

NetApp Confidential 9

THE CLUSTER PING-CLUSTER COMMAND

13-9 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Data Storage
 Are the aggregates online?
netappu::> storage aggregate show
netappu::> storage aggregate show -state
!online
 Are any disks broken or reconstructing?
netappu::> storage disk show -state
broken
netappu::> storage disk show -state
reconstructing
 Are the volumes online?
netappu::> volume show -state !online

NetApp Confidential 10

DATA STORAGE
For the most part, the commands on this slide are self-explanatory. Most show commands provide a view of
what’s happening in a particular area of the cluster. Also, most show commands have some powerful query
capabilities that, if you take the time to learn, can help you to pinpoint potential problems.
In the volume show -state !online command, the exclamation point means “not” (negation).
Therefore, this command shows all volumes that do not have a state of “online.” Because you’ll want to know
about other states that exist, it’s important to use !online rather than offline.

13-10 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

SFO

 Is storage failover (SFO) happy?

netappu::> storage failover show
netappu::> storage failover show
-instance
 Every node needs its vol0 (and thus, the
aggregate that contains its vol0) in order to
boot.
 After a takeover, the vol0 aggregate must be
given back first.
 A persistent “waiting for giveback” state
indicates an SFO problem.
NetApp Confidential 11

SFO
When the aggregates of one node fail over to the high-availability (HA) partner node, the aggregate that
contains the vol0 volume of that node goes, too. Each node needs its vol0 to boot, so when the rebooted node
begins to boot, the first thing that happens is that it signals the partner to do a giveback of that one aggregate
and then waits for that to happen. If SFO is working properly, giveback happens quickly, the node has its vol0
and can boot, and when it gets far enough in its boot process, the rest of the aggregates are given back. If
problems exist, you probably see the rebooted node go into a “waiting for giveback” state. If this happens, it’s
possible that its aggregates are stuck in a transition state between the two nodes and may not be owned by
either node. If this happens, contact NetApp Global Support.

13-11 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Common SFO Issues
 Incorrect cabling: Both nodes in a pair should see
the same disk with the same loop disk ID, for
example, node1:0a.16 and node2:0a.16.
 Nodes: Nodes must be rebooted after enabling
SFO before SFO can work properly.
 Disk-shelf module hardware or firmware failure
 Disk ownership or reservations that do not
transfer properly on takeover or giveback
 Problems with cluster-network connectivity to the
partner node to complete disk-count calculations
during giveback

NetApp Confidential 12

COMMON SFO ISSUES

13-12 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Additional Common SFO Issues
 If a disk on the loop is failing or has failed, SFO can
fail, wedge, or take a long time.
 SFO times are impacted by the number and size of
volumes that are taken over or given back.
 If a node is in takeover state but cannot perform the
giveback, a booting of the node in question is usually
required.
 If a disk count mismatch is detected, giveback can be
aborted (to prevent giveback to a broken loop).
 If long-running operations or open locks (generally
CIFS) exist, giveback can be aborted. The giveback
can be forced by using the overridevetoes flag.

NetApp Confidential 13

ADDITIONAL COMMON SFO ISSUES

It is a best practice for the vol0 aggregate to contain only the vol0 volume, because during giveback, the vol0
aggregate must be given back to the home node and must complete that giveback before any other aggregates
can be given back. The more volumes that are on that aggregate, the more time that the vol0 giveback
requires, and thus the longer the delay before all the other aggregates can be given back. The exception to this
best practice is during an evaluation or proof of concept, where a configuration might contain only one or two
disk shelves per node.

13-13 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Analyzing SFO Issues
When analyzing SFO issues, examine the:
 HA interconnect cables
 [Link] file
 Shelf logs (for engineering analysis):
– Locate the log in /mroot/etc/log/shelflog.
– Attach your case to firmware BURTs as appropriate.
 [Link], which shows manual and operator-initiated
takeover and giveback
 Clustershell and nodeshell status outputs
 Disk visibility status
 Disk-shelf module status
(failed, amber LEDs, and so on)
NetApp Confidential 14

ANALYZING SFO ISSUES

13-14 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Networking

 Are all the ports OK?

netappu::> network port show
 Are all the logical interfaces (LIFs) OK and
home?
netappu::> net int show
netappu::> net int show –is-home false

NetApp Confidential 15

NETWORKING
You must verify that all the network ports, including the cluster ports, are OK. If those ports are fine, verify
that the LIFs are working properly, and note which ones are home and which ones aren’t home. If the LIFs
are not home, it doesn’t mean that a problem exists, but this condition might give you a sense of what’s
happening.

13-15 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The Service Process Manager (spmd)

 The Service Process Manager:

– Monitors the management gateway daemon
(mgwd) and the other user-space applications
at regular intervals to verify that the processes
are running
– Generates an EMS message when a process
dies, and then restarts the process
 spmctl is the systemshell command to
control spmd.
 The diagnostic-level clustershell command
spm show lists monitored processes.
NetApp Confidential 16

THE SERVICE PROCESS MANAGER (SPMD)

 A process must register with the Service Process Manager (SPM) to be managed.
 If the process is not running, the SPM restarts the process. The SPM generates an EMS message when a
process dies.
 If a process reaches its threshold number of restarts, the SPM shifts to an interval-based restart and
generates an AutoSupport message. Currently the threshold is 10 restarts in an hour.
 Interval-based restarts range from 5 minutes to a maximum of 60 minutes per process. The interval
between restarts is twice that of the last value, for example, 5 minutes, then 10 minutes, then 20 minutes,
up to 60 minutes. Any further retries happen once every hour after the first 15 (10 + 5) retries.
 The SPM never gives up managing a process.

13-16 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

VLDB and D-Blade Inconsistencies
 If the volume location database (VLDB) and the
D-blade become inconsistent with each other,
they must be brought back in synchronization.
 The debug vreport show clustershell
command displays a list of discrepancies of
information in the Write Anywhere File Layout
(WAFL) file system and the VLDB.
 The debug vreport fix clustershell
command fixes discrepancies by updating the
VLDB.
 Currently, the fix command fixes only the VLDB,
not the D-blade.
NetApp Confidential 17

VLDB AND D-BLADE INCONSISTENCIES

13-17 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Statistics Objects
netappu::statistics> show -object ?
node
latency
disk
processor
port
volume
aggregate
protocol-request-size
nfs3
nfs4
cifs
iscsi_conn
fcache
lif
client

NetApp Confidential 18

STATISTICS OBJECTS
The category parameter has a finite set of key words, as shown on this slide.
NOTE: Only a partial list is shown.

13-18 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The statistics show Command
netappu::> statistics show -object nfs* -instance failure* -counter write* -node
netappu-01

Node: netappu-01
[Link] Value Delta
----------------------------------------------- ------------- -------------
[Link] 0 -
[Link]-pct 0% -
[Link] 3 0/s:30s
[Link]-pct 0% -
[Link] 0 -
[Link]-pct 0% -

NetApp Confidential 19

THE STATISTICS SHOW COMMAND

The output on this slide shows statistics that are specifically for all NFS “categories” and only for this node.

13-19 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Core Files
 User-space core files are:
– Located in mroot/etc/crash/cores
– Named as <procname>.core.<pid>
 Kernel core files are:
– Located in /mroot/etc/crash/kcores
– Named as
core.<sysid>.<panic_date>.<panic_time>.nz

NetApp Confidential 20

CORE FILES
User-space core dumps are named according to the process name (for example, mgwd) and use the process ID
(pid) of that instance of the process that generates the core file.
Kernel core dumps include the sysid, which is not the node name but a numerical representation of this node.
The date and time in the core dump name indicate when the panic occurred.
Remote LAN Module (RLM) is an out-of-band connection to a node that allows for some management of a
node even when the node is inaccessible from the console and UI. The RLM connection has a separate IP
address and its own shell. Examples of RLM commands are system power off, system power
on, system reset, and system console.

13-20 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

User-Generated Core Files
A system core dump can be generated manually.
 If the clustershell is responsive, enter:
netappu::> system reboot -node <node> -dump true
 If the node is in bad shape, from the Remote LAN
Module/Storage Partition Management session
(RLM/SPM), enter:
netappu::> system core
 If the RLM/SPM is not configured, from the
systemshell, enter:
netappu-01% sysctl debug.debugger_on_panic=0
netappu-01% sysctl [Link]=1

NetApp Confidential 21

USER-GENERATED CORE FILES

When a node panics, a kernel core dump is generated. Sometimes, however, a node is up and running but has
issues that cannot be debugged live. NetApp Support services might request that a system core dump be
generated for one or multiple nodes to capture the complete picture of what is happening at that time. If a
node is healthy enough to issue UI commands, a system reboot command can be entered with the dump
true parameter. If a node is not healthy enough for that, from the RLM/SPM session to that node, the system
core command can be used to generate a core dump. If the RLM/SPM isn’t configured and the system shell
is responsive, typing a series of two sysctl commands forces a core dump.

13-21 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Managing Core Files

 All kernel cores in the cluster are listed in:

netappu::> system coredump show

 Core files must be saved:

netappu::> system coredump save

 Core files can be uploaded to NetApp:

netappu::> system coredump upload

NetApp Confidential 22

MANAGING CORE FILES

Core files are meant to be examined by NetApp Support services and should be reported and uploaded to
NetApp Support services. The default location to which core dumps should be uploaded (as shown through
system coredump config show) is [Link]

13-22 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Lesson 2

NetApp Confidential 23

LESSON 2

13-23 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Data ONTAP Cluster-Mode Performance

 Running on the same hardware, Data ONTAP

operating in 7-Mode is slightly faster than
Data ONTAP operating in Cluster-Mode.
 The Data ONTAP Cluster-Mode components
have more work to do to support the cluster
functionality, so this comparison is not valid.
 When two or more nodes are combined into a
cluster, the benefits outweigh the “overhead.”

NetApp Confidential 24

DATA ONTAP CLUSTER-MODE PERFORMANCE

13-24 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Performance Tuning

 Not many “knobs” require tuning for

performance within Data ONTAP operating in
Cluster-Mode.
 Some areas must be monitored and possibly
adjusted.
 You can distribute resources, which affects
performance.
 The good news is that Data ONTAP operating
in Cluster-Mode is flexible.

NetApp Confidential 25

PERFORMANCE TUNING

13-25 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The Cluster Network
 The cluster network:
– Must be set up properly and must be stable for Data
ONTAP operating in Cluster-Mode to function optimally
– Must be set to a maximum transmission unit (MTU) size
of 9,000
– Must use jumbo frames
– Must be isolated, dedicated, and not routed
– Requires two cluster links per node for redundancy and
performance
 10-GbE cluster NICs are required.
 Virtual LANs (VLANs) and interface groups (ifgrps)
are not supported on cluster ports.

NetApp Confidential 26

THE CLUSTER NETWORK

13-26 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The Data Network

 The data network is typically shared with the

management network.
 Additional data NICs can be added.
 10-GbE can be added for data NICs.

NetApp Confidential 27

THE DATA NETWORK

13-27 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Data LIF Balancing

 Consider having more than one data LIF per

data port to provide granularity in balancing
the connection load.
 Clients should spread their connections
across all the data LIFs in a Vserver.
 Data ONTAP operating in Cluster-Mode can
use DNS load balancing.
 Ifgrps support the Link Aggregation Control
Protocol (LACP) but are of limited value in
Data ONTAP operating in Cluster-Mode.

NetApp Confidential 28

DATA LIF BALANCING

13-28 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Volume Balancing
 A namespace should be spread across the nodes of a
cluster.
 Flexible volumes should be evenly distributed across
nodes and aggregates.
 Use load-sharing mirror copies to avoid “hot spots” for
reads.
– Load-sharing mirror copies of root volumes require no license.
– Suggest one Vserver root volume load-sharing mirror copy
per node, including the node that contains the volume.
 When a volume becomes a “hot spot” for writes, that
volume can be manually divided into multiple volumes.

NetApp Confidential 29

VOLUME BALANCING
Using load-sharing mirror copies of Vserver root volumes is important for HA access to the other volumes in
the namespace. As such, load-sharing mirror copies do not require a separate license but rather are included in
the Base license. A best practice is to create load-sharing mirror copies of the Vserver root volume and to
situate one on each node, including the node that contains the Vserver root volume itself.
“Splitting” a volume is a manual process. For example, if a volume has two directories to which many writes
are sent and such that the volume is a hot spot, the volume can be divided into two volumes. With a new
volume on another node, the contents of one of those directories can be moved (by using NFS or CIFS
commands) into the new one, and the new volume can be mounted into the namespace at the same location as
the original directory’s location. The clients use the same path to write the data, but the writes go to two
separate volumes rather than to one.

13-29 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Client Performance Tuning
 NFS clients:
– Keep up-to-date with NFS client patches.
– For Linux, use Linux version 2.6.18 (or later) kernels.
– Use NFSv3 or NFSv4. (NFSv2 is not recommended.)
– Use User Datagram Protocol ( UDP) or TCP (TCP for better
predictability).
– The default MTU size is 1,500, but you can use 9,000.
– Use an NFS client cache.
– Maximize the read size (rsize) and the write size (wsize).
 CIFS clients:
– Options exist for each CIFS share to disable change-notify
and opportunistic locks (oplocks) for CIFS clients.
– Use SMB 2.0 or higher whenever possible.

NetApp Confidential 30

CLIENT PERFORMANCE TUNING

For more details on client tuning, refer to TR3580.

13-30 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Scale-Up Versus Scale-Out
 Scaling up is upgrading equipment to achieve
better performance and is bound by the most
powerful controller.
 Scaling out is adding equipment to achieve better
performance:
– This method is bound by the number of controllers
that can be clustered.
– Scale-up can be performed in addition to scale-out.
– More variables are involved in performance
evaluations.
 Customers should work with their account teams
before beginning to scale up or out.
NetApp Confidential 31

SCALE-UP VERSUS SCALE-OUT

13-31 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Dashboard Commands
 dashboard alarm show
 dashboard alarm thresholds show
 dashboard health vserver show
 dashboard health vserver show-aggregate
 dashboard health vserver show-all
 dashboard health vserver show-port
 dashboard health vserver show-protocol
 dashboard health vserver show-lif
 dashboard health vserver show-volume
 dashboard storage show
 dashboard performance show

NetApp Confidential 32

DASHBOARD COMMANDS
The dashboard commands provide quick views of the nodes and the cluster.

13-32 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The dashboard storage show
Command
netappu::> dashboard storage show -week
~1 day ~2 days ~3 days ~7 days
Aggregate Size Used Vols Used Vols Used Vols Used Vols Used Vols
--------- -------- ------- ---- ----------- ------- --- ------- --- ------- ---
netappu-01_aggr0
113.5GB 99.91GB 1 620KB 0 1.18MB 0 1.77MB 0 4.36MB 0
netappu-01_aggr2
908.3GB 50.00GB 1 4KB 0 12KB 0 16KB 0 40KB 0
netappu-02_aggr0
113.5GB 99.91GB 1 612KB 0 1.13MB 0 1.68MB 0 4.02MB 0
netappu-03_aggr0
229.1GB 109.9GB 2 648KB 0 1.23MB 0 1.84MB 0 4.34MB 0
netappu-03_aggr1
687.3GB 110.1GB 7 48KB 0 80KB 0 128KB 0 344KB 0
netappu-04_aggr0
229.1GB 99.92GB 1 624KB 0 1.18MB 0 1.74MB 0 4.06MB 0
netappu-04_aggr1
687.3GB 90.08GB 8 56KB 0 108KB 0 164KB 0 436KB 0
7 entries were displayed.

NetApp Confidential 33

THE DASHBOARD STORAGE SHOW COMMAND

By default, the storage dashboard displays the:
 Aggregate name
 Aggregate size, in GB
 Aggregate available space, in GB
 Aggregate used space, in GB
 Percentage of space that is used
 Number of volumes
 Four-hour change in used size
 Four-hour change in number of volumes
 Eight-hour change in used size
 Eight-hour change in number of volumes
 Operational status

13-33 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The dashboard performance show
Command
hydra::> dashboard performance show
Average ---Data-Network--- -Cluster--Network- ---Storage---
Total Latency CPU Busy Recv Sent Busy Recv Sent Read Write
Ops/s in usec Busy Util MB/s MB/s Util MB/s MB/s MB/s MB/s
------ ------- ---- ---- ------ ------ ---- ------ ------ ------ ------
hydra01
0 0 2% 0% 0 0 0% 0 0 0 0
hydra02
0 2224 2% 0% 0 0 0% 0 0 0 0
hydra03
0 3517 2% 0% 0 0 0% 0 0 0 0
hydra04
0 4260 70% 0% 0 0 30% 71 1 19 88
...
hydra23
27113 10903 62% 11% 140 9 29% 5 71 5 80
hydra24
0 2743 76% 0% 0 0 29% 71 5 6 82
cluster:summary
42729 11090 76% 25% 237 20 30% 183 181 42 285
24 entries were displayed.
In this example, there are 24 nodes in the cluster,
but we’re suppressing the output from 18 of them.

NetApp Confidential 34

THE DASHBOARD PERFORMANCE SHOW COMMAND

By default, the performance dashboard displays the following information about system and cluster
performance:
 Node name or cluster summary
 Average operation latency, in microseconds
 Total number of operations
 Percentage of data network use
 Data that is received on the data network, in MBps
 Data that is sent on the data network, in MBps
 Percentage of cluster network use
 Data that is received on the cluster network, in MBps
 Data that is sent on the cluster network, in MBps
 Data that is read from storage, in MBps
 Data that is written to storage, in MBps
This command can display a wide range of performance information. See the reference page for the command
for further details.
This performance view can be used in conjunction with statistics show –node <node> –object
<object> to get more detailed statistics.

13-34 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Statistics
1 of 2
netappu::statistics> show -object processor

Node: netappu-01
[Link] Value Delta
----------------------------------------------- ------------- -------------
processor.processor0.processor_busy 2% -
processor.processor1.processor_busy 2% -
...
Node: netappu-05
[Link] Value Delta
----------------------------------------------- ------------- -------------
processor.processor0.processor_busy 43% -
processor.processor1.processor_busy 53% -
processor.processor2.processor_busy 56% -
processor.processor3.processor_busy 48% -

Node: netappu-06
[Link] Value Delta
----------------------------------------------- ------------- -------------
processor.processor0.processor_busy 29% -
processor.processor1.processor_busy 31% -
processor.processor2.processor_busy 48% -
processor.processor3.processor_busy 65% -
68 entries were displayed.

NetApp Confidential 35

STATISTICS: 1 OF 2
Under the “admin” privilege, the statistics show –object processor command shows a basic
view of the use of each processor of a node.

13-35 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Statistics
2 of 2
netappu::statistics*> show -node netappu-02 -object processor -instance processor1
Node: netappu-02
[Link] Value Delta
----------------------------------------------- ------------- -------------
processor.processor1.instance_name processor1
processor.processor1.node_name - -
processor.processor1.instance_uuid - -
processor.processor1.processor_busy 3% -1
processor.processor1.sk_switches 32225379 615/s:189s
processor.processor1.hard_switches 22140953 -
processor.processor1.domain_busy - -
idle 95% -
kahuna 0% -
storage 0% -
...
sm_exempt 0% -
cluster 0% -
protocol 0% -
nwk_exclusive 0% -
nwk_exempt 0% -
nwk_legacy 0% -
hostOS 2% -
7 entries were displayed.

NetApp Confidential 36

STATISTICS: 2 OF 2
Under the “diag” privilege, the statistics show –node netappu-05 –object processor –
instance processor1 command shows a detailed view of the use of processor1 of node netappu-05.

13-36 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

The statistics periodic Command
netappu::statistics> periodic
cpu total data data data cluster cluster cluster disk disk
busy ops nfs-ops cifs-ops busy recv sent busy recv sent read write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- --------
88% 50053 50053 0 92% 339MB 23.8MB 45% 261MB 291MB 40.8MB 422MB
82% 66182 66182 0 75% 383MB 16.9MB 38% 309MB 250MB 42.4MB 385MB
68% 50602 50602 0 69% 317MB 34.9MB 49% 263MB 297MB 44.7MB 396MB
73% 44095 44095 0 68% 284MB 19.9MB 38% 230MB 240MB 38.4MB 395MB
75% 48998 48998 0 75% 319MB 27.1MB 49% 263MB 295MB 44.3MB 404MB
85% 56650 56650 0 89% 346MB 16.5MB 43% 271MB 212MB 49.2MB 432MB
87% 62995 62995 0 69% 379MB 41.9MB 52% 306MB 343MB 42.5MB 347MB
72% 43038 43038 0 64% 287MB 11.8MB 47% 220MB 218MB 41.1MB 410MB
cluster: 2/13/2008 11:10:15
cpu total data data data cluster cluster cluster disk disk
busy ops nfs-ops cifs-ops busy recv sent busy recv sent read write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- --------
Minimums:
68% 43038 43038 0 64% 284MB 11.8MB 38% 220MB 212MB 38.4MB 347MB
Averages for 8 samples:
78% 52826 52826 0 75% 332MB 24.1MB 45% 265MB 268MB 42.9MB 399MB
Maximums:
88% 66182 66182 0 92% 383MB 41.9MB 52% 309MB 343MB 49.2MB 432MB

This point is when the administrator pressed Ctrl-C, and the

summary, min, max, and average values were displayed.

NetApp Confidential 37

THE STATISTICS PERIODIC COMMAND

The statistics periodic command runs until Ctrl-C is pressed, with each line of output reporting the
stats since the previous line of output (interval). The default interval is one second. When Ctrl-C is pressed,
some summary data is presented.
This output can tell you much. If the “cluster busy” values are nonzero, it’s a good indication that the user
data isn’t sent over the cluster links. The same is true if “cluster recv” and “cluster sent” values are in the KB
range. Therefore, if operations occur with no data sent over the cluster network, the output shows that data is
served locally, which is similar to when many reads are performed to load-sharing mirror copies that are on
the same nodes as the data LIFs that are accessed by the clients. When cluster traffic occurs, the “cluster recv”
and “cluster sent” values are in the MB range.
Other good options to use with this command are:
 statistics periodic –object latency –node node
 statistics periodic –object volume –node node –interval 1
 statistics show –node node –object volume –counter *latency (This wildcard
shows all the latency counters.)

13-37 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Perfstat
 Collects four types of data:
– On-disk log files (FILE)
– Configuration-related data (CONFIG)
– Performance-related data, single execution (PERF)
– Performance-related data, timed execution (STAT)
 Is distributed as an executable file
 Has a modifiable configuration file that enables
quick and easy tailoring
 Runs on Windows, Linux, and Mac OS

See [Link]

NetApp Confidential 38

PERFSTAT

13-38 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Module Summary
Now that you have completed this module, you should
be able to:
 Explain how to configure and use the AutoSupport
tool for a cluster
 Demonstrate the use of the cluster EMS
 Identify any offline storage
 Review the required steps for forcing a system core
dump
 Describe the characteristics of a cluster that affect the
cluster’s performance

NetApp Confidential 39

MODULE SUMMARY

13-39 Clustered Data ONTAP Administration: Basic Troubleshooting and Performance

NetApp University - Do Not Distribute

Module 14
Cluster Management

NetApp Confidential 1

MODULE 14: CLUSTER MANAGEMENT

14-1 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Module Objectives
After this module, you should be able to:
 Nondisruptively upgrade (NDU) the Data ONTAP
software
 Scale the cluster’s storage capacity by adding
disks to an aggregate
 Move volumes off an aggregate and delete the
aggregate
 Remove a node from a cluster
 Describe the virtual server (Vserver)
administrative roles

NetApp Confidential 2

MODULE OBJECTIVES

14-2 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

The volume move Command

 Volume moves are performed within the cluster.

 Volumes can be moved from one aggregate to
another, anywhere in the cluster.
 Data is transferred over the cluster network.
 Clients continue to have access (reads and
writes) to the volume while the volume is moved.
 Clients do not know that the volume moved.
 Use the command volume move start and
volume move trigger-cutover.
 Use the –validation-only option to check
validity without moving the volume.
NetApp Confidential 3

THE VOLUME MOVE COMMAND

Volume move start [-cutover-action {abort_on_failure|defer_on_failure|force|wait}] - Action for Cutover
Specifies the action to be taken for cutover. Default is "defer_on_failure". If "abort_on_failure" action is
specified, the job will try to cutover until cutover attempts are exhausted. If it fails to cutover, it will cleanup
and end the operation. If "defer_on_failure" action is specified, the job will try to cutover until the cutover
attempts are exhausted. If it fails to cutover, it will move into "cutover deferred" state. This is the default
option. The volume move job waits for the user to issue a volume move trigger-cutover command to restart
the cutover process. If "force" action is specified, the job will try to cutover until the cutover attempts are
exhausted and force the cutover at the expense of disrupting the clients. If "wait" action is specified, when the
job hits the decision point, it will not go into cutover automatically, instead it will wait for the user to issue a
volume move trigger-cutover command as the signal to try the cutover.

14-3 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

The system image show Command
netappu::> system image show
Is Is Install
Node Image Default Current Version Date
-------- ------- ------- ------- ------------ ------------------
netappu-01
image1 false false 8.0.2 11/23/2010 13:41:00
image2 true true 8.1.0 10/10/2011 13:59:20
netappu-02
image1 false false 8.0.2 11/23/2010 13:43:50
image2 true true 8.1.0 10/10/2011 14:01:30
4 entries were displayed.

NetApp Confidential 4

THE SYSTEM IMAGE SHOW COMMAND

From any node in the cluster, you can see the images from all other nodes. Of the two images on each node,
the one that has an Is Current value of true is the one that is currently booted. The other image can be booted
at any time, provided that the version of the Data ONTAP 8 operating system on that image is compatible
with that of its high-availability (HA) partner and the rest of the cluster.

14-4 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Upgrading Nodes

 Two images exist on each node.

 Updates are installed on the inactive image.
 When “Is Default” is “true,” that image is used
after the next boot process.
 Updates can be loaded onto each node’s
secondary image at the same time and the
images made active when a boot process is
convenient:
netappu::> system image update -node * -package
[Link] -setdefault true

NetApp Confidential 5

UPGRADING NODES

14-5 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Nondisruptive Upgrades (NDUs)

The NDU Upgrade Path for Data ONTAP Cluster-Mode

Source Version Destination Version NDU?

8.0 8.0.1, 8.0.2, 8.1, or 8.1.1 Yes*
8.0.1 8.0.2, 8.1 or 8.1.1 Yes*
8.0.2 8.1 or 8.1.1 Yes

*Clients may experience a brief outage during a storage send

home operation.

NetApp Confidential 6

NONDISRUPTIVE UPGRADES (NDUS)

A nondisruptive upgrade (NDU) is an upgrade to the OS within a cluster with no disruption of service to NAS
or SAN clients. Cluster-Mode features such as storage failover, logical interface (LIF) migration, and the
ability to store two boot images on the boot device make an NDU possible.
You can download a new version of Data ONTAP Cluster-Mode to one boot image of an HA pair while the
other image in the pair operates normally.
You can verify the statuses of both images with the system image show command. Set the image that contains
the new version as the default. If storage and LIF failover are enabled, reboot one node of the HA pair.
Storage and logical network interfaces continues to be served by the partner node.
The first node comes up and runs the upgraded software. Storage and data LIFs are sent back home. The
upgrade is halfway complete.
Repeat the process to upgrade the second node in the pair.
You have upgraded the cluster without affecting client access. It should be noted that switch software and
firmware along with disk and disk shelf firmware can be upgraded during the NDU process. Also note that 7-
Mode-to-Cluster-Mode NDU is not supported.
The NDU feature is supported between Data ONTAP 8 versions. All upgrades, except for those from Data
ONTAP 8.0.2 to 8.1, may result in brief outage during the send home of aggregates. The table shows NDU
availability by source and destination Data ONTAP Cluster-Mode versions.

14-6 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

NDU: Rolling Upgrade

UPGRADED UPGRADED UPGRADED UPGRADED UPGRADED UPGRADED

NetApp Confidential 7

NDU: ROLLING UPGRADE

To minimize the time it takes to upgrade large clusters, you can use a rolling upgrade strategy. Using the same
NDU procedure, you can upgrade up to a third of the nodes in a cluster in parallel. [2] Be sure to upgrade
nodes in an HA pair separately, so that one node is always available to serve data for the pair. It is also critical
for the health of the cluster that you make sure that fewer than half of the nodes in the cluster are down at any
one time.
A partially upgraded cluster in which multiple versions of Data ONTAP Cluster-Mode are running is referred
to as a mixed-mode cluster. This condition should only exist while the cluster is being upgraded—you should
not allow it to continue for an extended period of time. [4] When a cluster is running in mixed mode, new
cluster-wide features will not be available to any of the nodes in the cluster. Node-specific features will be
available on the upgraded nodes when both nodes of the HA pair are running the new software version.

14-7 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Rebooting Nodes

 A simple majority of the nodes must be “up” at

all times within the cluster.
 For storage high availability, don’t reboot both
nodes of a high-availability (HA) pair at the
same time.
 Reboot nodes in an orderly fashion, and verify
that all storage (aggregates and volumes) and
all data logical interfaces (LIFs) remain up.

NetApp Confidential 8

REBOOTING NODES

14-8 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Ways to Scale a Cluster

 Create cluster Vservers and namespaces.

 Create volumes.
 Add disk shelves to a node.
 Create additional aggregates.
 Add disks to aggregates.
 Join nodes to the cluster.
 Add more network ports and LIFs.
 Add a NetApp Flash Cache module.

NetApp Confidential 9

WAYS TO SCALE A CLUSTER

In the process of managing a cluster, it is typically necessary to scale out. Clusters provide many ways to
scale out, some of which you’ve seen and performed. This slide lists some of the ways that a cluster can scale.

14-9 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Adding Nodes to the Cluster

 Join nodes to the existing cluster.

 Is storage failover (SFO) automatically
configured for the new nodes?

NetApp Confidential 10

ADDING NODES TO THE CLUSTER

14-10 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Scaling Other Resources

Use the data ports on the new nodes to add

data LIFs to the Vserver. Then consider the
following questions:
 What happens if you don’t perform this
action?
 What about your root volume load-sharing
mirror copies?
– Should you add more?
– What are the implications of adding or not
adding load-sharing mirror copies?
 Do your NDMP plans change?
NetApp Confidential 11

SCALING OTHER RESOURCES

14-11 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Deleting Aggregates

 Reasons for deleting an aggregate:

– To free up disks for other purposes
– To re-create an aggregate in a different way
 Conditions for deleting an aggregate:
– The aggregate must have no volumes on it.
(volume move makes this process easy.)
– You cannot delete the aggregates that contain
vol0.

NetApp Confidential 12

DELETING AGGREGATES
If it’s determined that an aggregate must be re-created, or if the disks are needed for another purpose (for
example, to grow a different aggregate), you might need to delete an aggregate. The volumes must be
removed from the aggregate first, which can be accomplished by volume move if you don’t want to delete the
volumes.

14-12 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

EMS

 The Event Management System (EMS):

– Writes events to the event log
– Sends and routes notifications of events
– Is a cluster-wide collection
– Can view events of all nodes from any node
(event log show)
 Each event contains a:
– Message name
– Severity level
– Description
– Corrective action, if applicable
NetApp Confidential 13

EMS

14-13 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Event Severity Levels

The event severity levels, from most to least

severe, are:
 EMERGENCY
 ALERT
 CRITICAL
 ERROR
 WARNING
 NOTICE
 INFORMATIONAL
 DEBUG
NetApp Confidential 14

EVENT SEVERITY LEVELS

The definitions of the event severity levels are:
 EMERGENCY: The system is unusable.
 ALERT: An action must be performed immediately to prevent system failure.
 CRITICAL: A critical condition occurred.
 ERROR: An error condition occurred.
 WARNING: A warning condition occurred.
 NOTICE: A normal but significant condition occurred.
 INFORMATIONAL: An informational message was sent.
 DEBUG: A debugging message was sent.

14-14 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Event Notifications
 A three-step process to set up:
1. Modify the event configuration to contain the mail
host and other attributes.
2. Create at least one event destination.
3. Modify routes to use a destination.
 Examples:
netappu::> event config modify -mailfrom
admin@[Link] -mailserver [Link]
netappu::> event destination create -name
crits -mail frank@[Link]
netappu::> event route modify -messagename
coredump* -destinations crits

NetApp Confidential 15

EVENT NOTIFICATIONS
Only one event configuration exists. The named event destinations must be created or modified appropriately,
for example, to indicate to which email address certain event notifications should be sent. Event routes are
associations between predefined event messages and event destinations. You can enable the notification to a
destination of a message by modifying the message’s destination value. This action can be performed all at
the same time by using a regular expression when specifying the event name in the event route
modify command.

14-15 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Event Destinations

 An event destination is a named combination

of any or all of the following:
– The email destination
– The SNMP destination
– The syslog destination
 The named destination can be associated
with specific event messages through the use
of an event route (event route modify).

NetApp Confidential 16

EVENT DESTINATIONS

14-16 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Event Routes

 Are associations between event messages

and event destinations
 Allow for frequency thresholds and time
thresholds:
– Prevent floods of event notifications
– Stop notifications for a specific number of
iterations or for a period of time (for example, if
you know that a disk is bad and you want to be
reminded only once a day)

NetApp Confidential 17

EVENT ROUTES
Event routes have nothing to do with network routes but are merely associations between event messages and
receivers of notifications associated with the messages.

14-17 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

The AutoSupport Tool
 Is the NetApp “phone home” proactive support
mechanism, must be configured
 Functions per node, not per cluster
 Sends:
– A weekly email on Sundays at a randomized time
throughout the day to avoid mass mailing from many
systems
– EMS logs and other configuration information
– Nightly log files
 Must be configured and enabled
 Is tested with:
netappu::> system node autosupport invoke
-node netappu-02 –type test
NetApp Confidential 18

THE AUTOSUPPORT TOOL

14-18 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Administrative Security
 Use the security login command to configure role-
based administrative access to the cluster.
 Configure by application: console, HTTP, SNMP, Secure
Shell (SSH), and the ONTAPI interface library
 To enable and disable security audit logging, use:
netappu::> security audit modify -cliset on
-httpset on -cliget on -httpget on
 Audited commands go to [Link] files, with the “console”
or the “ssh” prefix:
From the systemshell: netappu-01% egrep
“console|ssh” /mroot/etc/mlog/[Link]*
 Nodes keep track of local SSH and console commands in
/mroot/etc/mlog/[Link].

NetApp Confidential 19

ADMINISTRATIVE SECURITY

14-19 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles

 admin
 vsadmin
 vsadmin-protocol
 vsadmin-readonly
 vsadmin-volume
 readonly
 none

netappu::> security login role show

NetApp Confidential 20

VSERVER ADMINISTRATIVE ROLES

Data ONTAP 8.1.1 operating in Cluster-Mode includes administrative access-control roles that can be used to
subdivide administration duties for virtual server (Vserver) administration tasks.
The “admin” role is still the most capable role and should be used for administrators who require full
administration privileges. New to Data ONTAP 8.1 operating in Cluster-Mode is the “vsadmin” role. The
“vsadmin” role grants the Vserver administrator full administrative privileges for the Vserver.
Additional new roles include the “vsadmin-protocol” role, the “vsadmin-readonly” role, and the “vsadmin-
volume” role. Each of these roles provides a unique Vserver administration privilege. The “readonly” and
“None” roles are still available and can be used to further restrict administrative capabilities.

14-20 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
admin
 Grants all possible capabilities
 Is a cluster superuser

NetApp Confidential 21

VSERVER ADMINISTRATIVE ROLES: ADMIN

14-21 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
vsadmin
This role is the superuser role for a Vserver. A Vserver
administrator with this role has the following capabilities:
 Manages its own user account, local password, and
public key
 Manages volumes, quotas, qtrees, Snapshot copies,
FlexCache devices, and files
 Manages LUNs
 Configures protocols
 Configures services
 Monitors jobs
 Monitors network connections and network interfaces
 Monitors the health of a Vserver
NetApp Confidential 22

VSERVER ADMINISTRATIVE ROLES: VSADMIN

14-22 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
vsadmin-protocol
A Vserver administrator with this role has the
following capabilities:
 Configures protocols
 Configures services
 Manages LUNs
 Monitors network interfaces
 Monitors the health of a Vserver

NetApp Confidential 23

VSERVER ADMINISTRATIVE ROLES: VSADMIN-PROTOCOL

14-23 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
vsadmin-readonly
A Vserver administrator with this role has the
following capabilities:
 Monitors the health of a Vserver
 Monitors network interfaces
 Views volumes and LUNs
 Views services and protocols

NetApp Confidential 24

VSERVER ADMINISTRATIVE ROLES: VSADMIN-READONLY

14-24 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
vsadmin-volume
A Vserver administrator with this role has the
following capabilities:
 Manages volumes, quotas, qtrees, Snapshot
copies, FlexCache devices, and files
 Manages LUNs
 Configures protocols
 Configures services
 Monitors network interfaces
 Monitors the health of a Vserver

NetApp Confidential 25

VSERVER ADMINISTRATIVE ROLES: VSADMIN-VOLUME

14-25 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Vserver Administrative Roles
readonly and None
 A Vserver administrator with the role of
“readonly” can grant read-only capabilities.
 A Vserver administrator with the role of None
cannot grant capabilities.

NetApp Confidential 26

VSERVER ADMINISTRATIVE ROLES: READONLY AND NONE

14-26 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Module Summary
Now that you have completed this module, you should
be able to:
 Nondisruptively upgrade the Data ONTAP software
 Scale the cluster’s storage capacity by adding disks to
an aggregate
 Move volumes off an aggregate and delete the
aggregate
 Remove a node from a cluster
 Describe the Vserver administrative roles

NetApp Confidential 27

MODULE SUMMARY

14-27 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Exercise
Module 14: Cluster Management
Time Estimate: 60 minutes

NetApp Confidential 28

EXERCISE
Please refer to your exercise guide.

14-28 Clustered Data ONTAP Administration: Cluster Management

NetApp University - Do Not Distribute

Module 15
Recommended Practices

NetApp Confidential 1

MODULE 15: RECOMMENDED PRACTICES

15-1 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Module Objectives

After this module, you should be able to:

 Identify some recommended practices for
Data ONTAP Cluster-Mode high availability
 List some of the ways to balance Data
ONTAP Cluster-Mode resources
 Describe some of the methods for managing a
cluster
 Enumerate some of the ways to protect and
recover a cluster from a disaster

NetApp Confidential 2

MODULE OBJECTIVES

15-2 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Nondisruptive
Operations (NDO) RAID-DP Technology
 Create a dedicated, three-disk RAID-DP root
aggregate on each node.
 Best practices for ONTAP 7G and Data
ONTAP 7-Mode RAID and storage still apply.
 Use RAID-DP technology for all user-data
aggregates.
 Use RAID-DP technology to enable online
disk firmware upgrades.
 Maintain two spare disks per disk type to
allow for disk maintenance center and NDU of
disk firmware.
NetApp Confidential 3

RECOMMENDED PRACTICES FOR NONDISRUPTIVE OPERATIONS (NDO): RAID-DP

TECHNOLOGY
Below are notes about node root aggregates (of which there is one per node):
 A node’s vol0 resides (permanently) on the root aggregate.
 Only vol0 should be placed on root aggregates.
 Root aggregates improve resiliency.
 Root aggregates speed up takeover and giveback.
 If you have few available disks, you can use two-disk RAID 4 instead of three-disk RAID-DP
technology.

15-3 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for NDO
SFO
 Enable storage failover (SFO).
 Reboot the high-availability (HA) pair after
enabling SFO for the first time.
 Enable “two-node high availability” for clusters
that contain only two nodes.
 Consider the advantages and disadvantages
of automatic giveback and discuss them with
the customer.

NetApp Confidential 4

RECOMMENDED PRACTICES FOR NDO: SFO

15-4 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for NDO
Nondisruptive Upgrade (NDU)
 You can perform upgrades in stages.
 Rolling upgrades are becoming the norm.
 You can reboot multiple nodes in parallel, depending
on number of nodes in the cluster.
 You should use an HTTP or FTP server as your
primary means of performing package downloads.
 You can upgrade a local host, if necessary, but must
first use the scp command to send the image file to
/mroot on each node.
 Remember to revert the logical interfaces (LIFs) back
to their home ports after you boot the LIFs (or set the
automatic reversion option).

NetApp Confidential 5

RECOMMENDED PRACTICES FOR NDO: NONDISRUPTIVE UPGRADE (NDU)

You can perform upgrades in stages by leaving the old image as the default, so that booting does not bring up
the upgraded image.
Rolling upgrades of a high-availability (HA) pair are faster than parallel reboots.
You can boot multiple nodes in parallel (though only one per HA pair), but remember that quorum rules
demand that fewer than half of the nodes in a cluster be down or booting at any given time. Also, be aware of
the logical interface (LIF) failover rules to guarantee that the data LIFs are not all failing over to nodes that
are also being rebooted.

15-5 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for NDO
Mixed-Version Clusters and LIF Failover
 Mixed-version clusters are supported with
caveats that are specific to each version.
 You should use the default configuration of
LIF failover and manually assign policies for
any exceptions:
– First-level failover: same node, different
network interface card (NIC)
– Second-level failover: different node (not the
HA partner)

NetApp Confidential 6

RECOMMENDED PRACTICES FOR NDO: MIXED-VERSION CLUSTERS AND LIF

FAILOVER

15-6 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for NDO
Load-Sharing Mirror Copies
Place load-sharing mirror copies of the virtual
server (Vserver) root volume onto all nodes or
at least onto one node of each HA pair:
 This configuration enables continuous access,
even if the node with the Vserver root volume
is down.
 Because default access is to a load-sharing
mirror copy (a read-only volume), this
configuration prevents the root volume from
filling up accidentally.

NetApp Confidential 7

RECOMMENDED PRACTICES FOR NDO: LOAD-SHARING MIRROR COPIES

15-7 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for NDO
Servers and Locality
 Where possible, configure multiple Domain
Name System (DNS), Network Information
Service (NIS), Lightweight Directory Access
Protocol (LDAP), and Network Time Protocol
(NTP) servers.
 Time zone settings should be the same
across all nodes.
 Language settings should be consistent
among Vservers and volumes.

NetApp Confidential 8

RECOMMENDED PRACTICES FOR NDO: SERVERS AND LOCALITY

Certain management operations (like a Network Information Service–NIS–lookup) happen over the
management network port, which can be a single point of failure. The cluster management LIF can use LIF
failover functionality.

15-8 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Resource
Balancing 1 of 2
 Balance resources across the cluster:
– Data and cluster interfaces
– Flexible volumes of a namespace
– Load-sharing mirror copies
 Maintain a junction-only Vserver root volume
with a low change rate, and create multiple
load-sharing mirror copies of the volume.

NetApp Confidential 9

RECOMMENDED PRACTICES FOR RESOURCE BALANCING: 1 OF 2

Note the maximum number of volumes that are allowed per controller.
Balance the number of volumes per controller across the cluster. When you create volumes, distribute the
volumes evenly across the cluster. When the cluster becomes unbalanced (for example, because of volume
deletions or volume size changes), use the volume move capability to redistribute volumes accordingly.

15-9 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Resource
Balancing 2 of 2
 Use built-in DNS load balancing to balance
client connections across network interfaces:
– Create many data LIFs for the cluster.
– Assign LIFs evenly to available network ports.
– Monitor network use levels and migrate LIFs to
different ports as needed to rebalance the load.
– When many clients are attached, the clients are
evenly spread across the system.

NetApp Confidential 10

RECOMMENDED PRACTICES FOR RESOURCE BALANCING: 2 OF 2

15-10 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for
Load-Sharing Mirror Copies
 Use load-sharing mirror copies:
– For read-only or mostly read-only data
– When data is updated only by a few authorized
individuals or applications
– When the data set is relatively small, or the cost of
the mirror copies in disk space is justified
– To netboot many clients at the same time (which is
a read-only operation and a popular use of load-
sharing mirror copies)
 Schedule load-sharing mirror copies to be
automatically replicated every hour.

NetApp Confidential 11

RECOMMENDED PRACTICES FOR LOAD-SHARING MIRROR COPIES

15-11 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for
Intercluster Mirror Copies
 A full mesh intercluster network supports node
failover and volume moves of the source or
destination volumes.
 Intercluster LIFs can be created on ports that
have an intercluster role or a data role
(through the CLI).

NetApp Confidential 12

RECOMMENDED PRACTICES FOR INTERCLUSTER MIRROR COPIES

15-12 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for
Manageability Granularity
 When you are deciding whether to create a
volume, a directory, or a qtree, ask these
questions:
– Will this element benefit from being managed
or protected separately?
– How large will this element get?
 Greater volume granularity is beneficial for
many workflows and enables movement of
volumes and resource distribution.
 Larger volumes tend to yield better
compression and dedup ratios.
NetApp Confidential 13

RECOMMENDED PRACTICES FOR MANAGEABILITY: GRANULARITY

15-13 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for
Manageability Volume Naming Conventions
 Volume names and junction names are
distinct.
 Each volume name must be unique within the
Vserver.
 Volume names should be wildcard-friendly.
 Volumes can be grouped by name (in
alphanumeric order).

NetApp Confidential 14

RECOMMENDED PRACTICES FOR MANAGEABILITY: VOLUME NAMING

CONVENTIONS

15-14 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Networking

 Modify port roles and create all additional

cluster LIFs before you create any data LIFs.
(Cluster LIFs are necessary to prevent cluster
ports from being automatically added to data
LIF failover rules.)
 Enable the PortFast option on data switch
ports for fast LIF failover and migration.

NetApp Confidential 15

RECOMMENDED PRACTICES FOR NETWORKING

15-15 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Disaster
Recovery 1 of 2
 Enable Snapshot copies and data-protection
mirror copies for critical volumes.
 Consider putting data-protection mirror copies
on SATA disks:
– The use of data-protection mirror copies on
SATA disks is a disk-based backup solution.
– Intercluster data-protection mirror copies can
be used for off-site backups.

NetApp Confidential 16

RECOMMENDED PRACTICES FOR DISASTER RECOVERY: 1 OF 2

15-16 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Recommended Practices for Disaster
Recovery 2 of 2
 Plan disaster-recovery implementations
carefully by considering taking quorum and
majority rules. (You can recover an out-of-
quorum site, but doing so is not customer-
friendly.)
 Use NDMP to back up important volumes to
tape.
 Have a policy for rotating backups off-site for
disaster recovery.

NetApp Confidential 17

RECOMMENDED PRACTICES FOR DISASTER RECOVERY: 2 OF 2

15-17 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Module Summary

Now that you have completed this module, you

should be able to:
 Identify some recommended practices for
Data ONTAP Cluster-Mode high availability
 List some of the ways to balance Data
ONTAP Cluster-Mode resources
 Describe some of the methods for managing a
cluster
 Enumerate some of the ways to protect and
recover a cluster from a disaster

NetApp Confidential 18

MODULE SUMMARY

15-18 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Course Summary
1 of 2
Now that you have completed this course, you should
be able to:
 List the benefits of the Data ONTAP operating system
and explain its two modes of operation
 Enumerate and describe the steps for setting up a
cluster
 Manage the physical and virtual resources within a
cluster
 Manage features to guarantee nondisruptive
operations
 Identify the networking components and networking
features of a cluster

NetApp Confidential 19

COURSE SUMMARY: 1 OF 2

15-19 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

Course Summary
2 of 2
 Implement supported cluster and management
network switches.
 Diagram the architecture of a cluster
 Configure a Flash Pool
 Configure a cluster that hosts an Infinite Volume
 Set up and configure SAN and NAS protocols
 Administer mirroring technology and data protection
for a cluster
 Explain the notification capabilities of a cluster
 Scale a cluster horizontally
 Describe the characteristics of a cluster that affect the
cluster’s performance
NetApp Confidential 20

COURSE SUMMARY: 2 OF 2

15-20 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

NetApp Confidential 21

THANK YOU

15-21 Clustered Data ONTAP Administration: Recommended Practices

NetApp University - Do Not Distribute

NETAPP UNIVERSITY
Clustered Data ONTAP Administration

Student Guide

Course ID: STRSW-ILT-D8CADM-REV02
Catal

2
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

3
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

4
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

5
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

6
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

7
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

8
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

9
Clustered Data ONTAP Administration: Welcome
© 2012 NetApp, Inc. This material is intended only for training. Reproductio

STRSW Ilt Dataprot Rev6 Studentguide
No ratings yet
STRSW Ilt Dataprot Rev6 Studentguide
364 pages
STRSW Ilt Perfcdot Exerciseguide
No ratings yet
STRSW Ilt Perfcdot Exerciseguide
109 pages
Data ONTAP 7-Mode Administration. StudentGuide PDF
100% (1)
Data ONTAP 7-Mode Administration. StudentGuide PDF
840 pages
VMware On NetApp Solutions Student Guide and Labs
No ratings yet
VMware On NetApp Solutions Student Guide and Labs
314 pages
D8CADM-REV04 StudentGuide
No ratings yet
D8CADM-REV04 StudentGuide
706 pages
Guia Completa de Netapp
No ratings yet
Guia Completa de Netapp
740 pages
Nondisruptive Operations For Netapp Ontap 9.0 V1.3-Lab Guide
No ratings yet
Nondisruptive Operations For Netapp Ontap 9.0 V1.3-Lab Guide
43 pages
Five Little-Known Tips To Increase Netapp Storage Resiliency
No ratings yet
Five Little-Known Tips To Increase Netapp Storage Resiliency
4 pages
STRSW Ilt Ociarc Studentguide
No ratings yet
STRSW Ilt Ociarc Studentguide
44 pages
SnapLock ONTAP Technical FAQ
No ratings yet
SnapLock ONTAP Technical FAQ
29 pages
Tr-4211 NetApp Storage Performance Primer
No ratings yet
Tr-4211 NetApp Storage Performance Primer
42 pages
S3 For ONTAP 9.10 Technical FAQ 2022.02.15
No ratings yet
S3 For ONTAP 9.10 Technical FAQ 2022.02.15
12 pages
Data OnTap NetApp Tutorial Basic Administration Ariane Dibble 12082014
No ratings yet
Data OnTap NetApp Tutorial Basic Administration Ariane Dibble 12082014
19 pages
NetApp Active/Active Cluster Overview
No ratings yet
NetApp Active/Active Cluster Overview
28 pages
h16463 Isilon Advanced Networking Fundamentals
No ratings yet
h16463 Isilon Advanced Networking Fundamentals
70 pages
Tr-4517 Ontap Select Product Architecture and Best Practices
No ratings yet
Tr-4517 Ontap Select Product Architecture and Best Practices
60 pages
Unified Storage Management Final Lab Guide VNX Lab Guide 2014
100% (1)
Unified Storage Management Final Lab Guide VNX Lab Guide 2014
233 pages
NetApp Basic Deployment Service Guide
No ratings yet
NetApp Basic Deployment Service Guide
6 pages
SL10496 Basic Concepts For NetApp ONTAP 9.4-NAS Services-CLI Edition-V2.0.0 PDF
No ratings yet
SL10496 Basic Concepts For NetApp ONTAP 9.4-NAS Services-CLI Edition-V2.0.0 PDF
45 pages
VNX Unified Storage Management - Lab Guide
No ratings yet
VNX Unified Storage Management - Lab Guide
219 pages
Brocade SAN Design Principles Overview
No ratings yet
Brocade SAN Design Principles Overview
46 pages
TR-4476-0716 Dedupe Compression Compaction
No ratings yet
TR-4476-0716 Dedupe Compression Compaction
63 pages
Cmode Explained Latest
100% (1)
Cmode Explained Latest
547 pages
VMAX3 Business Continuity Management Student Guide
No ratings yet
VMAX3 Business Continuity Management Student Guide
264 pages
STRSW Ilt Sanimp Rev2 Labsetupguide
No ratings yet
STRSW Ilt Sanimp Rev2 Labsetupguide
12 pages
NimbelStorage NTS-2001-I Student Guide R5 Feb2017
100% (1)
NimbelStorage NTS-2001-I Student Guide R5 Feb2017
379 pages
Isilon Architecture for Big Data Storage
No ratings yet
Isilon Architecture for Big Data Storage
1 page
SnapMirror Cascade Configuration Guide
No ratings yet
SnapMirror Cascade Configuration Guide
22 pages
ECS Cloud Storage for Modern Archiving
No ratings yet
ECS Cloud Storage for Modern Archiving
32 pages
ONTAP 9 NDMP Configuration Express Guide
No ratings yet
ONTAP 9 NDMP Configuration Express Guide
22 pages
Netapp Upgrade Coursepartner
No ratings yet
Netapp Upgrade Coursepartner
31 pages
h17133 Non Disruptive Migration Best Practices and Operational Guide
No ratings yet
h17133 Non Disruptive Migration Best Practices and Operational Guide
148 pages
Isilon Design Consideration For SMB Environment PDF
No ratings yet
Isilon Design Consideration For SMB Environment PDF
43 pages
STRSW Ilt Perfcdot Rev01 Exerciseguide
No ratings yet
STRSW Ilt Perfcdot Rev01 Exerciseguide
108 pages
ONTAP Cluster Setup Guide
No ratings yet
ONTAP Cluster Setup Guide
20 pages
Basic Concepts For Netapp Ontap 9 V1.3-Lab Guide
50% (2)
Basic Concepts For Netapp Ontap 9 V1.3-Lab Guide
259 pages
NetApp's Ransomware Defense Strategies
No ratings yet
NetApp's Ransomware Defense Strategies
102 pages
Clustered Data ONTAP - NFS Best Practice and Implementation Guide
No ratings yet
Clustered Data ONTAP - NFS Best Practice and Implementation Guide
161 pages
Module 10-Data Protection - Participant Guide
No ratings yet
Module 10-Data Protection - Participant Guide
89 pages
MetroCluster FC FAQ-9.7 - v1.10 PDF
No ratings yet
MetroCluster FC FAQ-9.7 - v1.10 PDF
30 pages
Module 1 VMAX Performance Workshop
0% (1)
Module 1 VMAX Performance Workshop
38 pages
NetApp User Group AFF Updates
No ratings yet
NetApp User Group AFF Updates
16 pages
STRSW Ilt D8cadm Rev03 Studentguide
No ratings yet
STRSW Ilt D8cadm Rev03 Studentguide
568 pages
NetApp Training OnTap Clustering Student Guide Rev4
69% (13)
NetApp Training OnTap Clustering Student Guide Rev4
699 pages
STRSW Ilt Ontapadm Studentguide
No ratings yet
STRSW Ilt Ontapadm Studentguide
505 pages
STRSW Ilt Ontapadm Studentguide
No ratings yet
STRSW Ilt Ontapadm Studentguide
443 pages
STRSW ILT ONTAPADM - StudentGuide PDF
100% (1)
STRSW ILT ONTAPADM - StudentGuide PDF
551 pages
ONTAP Cluster Admin Student Guide
No ratings yet
ONTAP Cluster Admin Student Guide
534 pages
Netapp Ilt Ontapadm Studentguide Ver5.2019
100% (1)
Netapp Ilt Ontapadm Studentguide Ver5.2019
505 pages
STRSW ILT ONTAPADM - StudentGuide
No ratings yet
STRSW ILT ONTAPADM - StudentGuide
503 pages
ONTAP Cluster Administration Guide
No ratings yet
ONTAP Cluster Administration Guide
503 pages
STRSW ILT ONTAPADM - CourseDescription
No ratings yet
STRSW ILT ONTAPADM - CourseDescription
3 pages
ONTAP Data Protection Administration Student Guide
100% (2)
ONTAP Data Protection Administration Student Guide
249 pages
ONTAP Data Protection Student Guide
No ratings yet
ONTAP Data Protection Student Guide
249 pages
STRSW ILT DATAPROT REV06 - ExerciseGuide PDF
No ratings yet
STRSW ILT DATAPROT REV06 - ExerciseGuide PDF
72 pages
NCDA Study Guide
100% (1)
NCDA Study Guide
38 pages
NetApp Cluster Mode CDOT 8
No ratings yet
NetApp Cluster Mode CDOT 8
10 pages
Learning Path for ONTAP 9 Essentials
No ratings yet
Learning Path for ONTAP 9 Essentials
9 pages
01 Overview SG PDF
No ratings yet
01 Overview SG PDF
22 pages
Student Guide NetApp Accredited Storage Architecture Professional Workshop
No ratings yet
Student Guide NetApp Accredited Storage Architecture Professional Workshop
540 pages
Symantec Netbackup 7.5 For Unix: Install, Configure, and Deploy (Appendix B)
No ratings yet
Symantec Netbackup 7.5 For Unix: Install, Configure, and Deploy (Appendix B)
338 pages
Accounting and Finance Assignment Guide
No ratings yet
Accounting and Finance Assignment Guide
30 pages
MS-03 Economic and Social Environment
No ratings yet
MS-03 Economic and Social Environment
22 pages
MS07 WithQuestionPaper
No ratings yet
MS07 WithQuestionPaper
16 pages
MS05 WithQuestionPaper
No ratings yet
MS05 WithQuestionPaper
19 pages
MS04 WithQuestionPaper
No ratings yet
MS04 WithQuestionPaper
27 pages
CCNA Lab: VLANs and DHCP Setup
No ratings yet
CCNA Lab: VLANs and DHCP Setup
5 pages
Philips EPIQ Ultrasound System Overview
No ratings yet
Philips EPIQ Ultrasound System Overview
1 page
Huawei Router Basic Config Commands
No ratings yet
Huawei Router Basic Config Commands
6 pages
Enensys Asiipguard Datasheet A
No ratings yet
Enensys Asiipguard Datasheet A
4 pages
Understanding Data Communication Basics
No ratings yet
Understanding Data Communication Basics
9 pages
Dell EMC Networking N1100 Series Spec Sheet
No ratings yet
Dell EMC Networking N1100 Series Spec Sheet
3 pages
Understanding USB Types and Speeds
0% (1)
Understanding USB Types and Speeds
11 pages
Advantages of Time Division Multiplexing
No ratings yet
Advantages of Time Division Multiplexing
17 pages
AVIWEST Streamhub Transceiver Overview
No ratings yet
AVIWEST Streamhub Transceiver Overview
2 pages
Zabbix for Smart Campus Network Monitoring
No ratings yet
Zabbix for Smart Campus Network Monitoring
18 pages
Basic Cisco Switch Configuration Guide
100% (1)
Basic Cisco Switch Configuration Guide
5 pages
Telecom Networks Intro
No ratings yet
Telecom Networks Intro
29 pages
Cisco Firepower Security Exam Guide
No ratings yet
Cisco Firepower Security Exam Guide
100 pages
Simulated DDoS Attack Lab Setup - Project Report
No ratings yet
Simulated DDoS Attack Lab Setup - Project Report
19 pages
Introduction to Computer Networks 2020
No ratings yet
Introduction to Computer Networks 2020
7 pages
Session 2
No ratings yet
Session 2
10 pages
Understanding QoS in IP Networks
No ratings yet
Understanding QoS in IP Networks
22 pages
Flow Control: Stop-and-Wait Protocols
No ratings yet
Flow Control: Stop-and-Wait Protocols
55 pages
SIM7000 Series SPEC 20201023
No ratings yet
SIM7000 Series SPEC 20201023
3 pages
01 SD-WAN Solution (Online Reading)
100% (1)
01 SD-WAN Solution (Online Reading)
73 pages
Hierarchical Robotic System for AI
No ratings yet
Hierarchical Robotic System for AI
93 pages
Underwater Sensor Network Attacks
No ratings yet
Underwater Sensor Network Attacks
3 pages
Understanding Network Topologies and Protocols
No ratings yet
Understanding Network Topologies and Protocols
4 pages
Mastering Traceroute for Network Troubleshooting
No ratings yet
Mastering Traceroute for Network Troubleshooting
54 pages
Aruba CX ACL Management Guide
No ratings yet
Aruba CX ACL Management Guide
7 pages
GSM-R Interface Specs for ETCS 2.5
No ratings yet
GSM-R Interface Specs for ETCS 2.5
28 pages
Wireless Network Security Overview
No ratings yet
Wireless Network Security Overview
23 pages
TAS Overview: MMTel & SCC Roles
No ratings yet
TAS Overview: MMTel & SCC Roles
14 pages
IPv6 BRKIPV 2191
No ratings yet
IPv6 BRKIPV 2191
67 pages
Evolving To An Open C-RAN Architecture For 5G
No ratings yet
Evolving To An Open C-RAN Architecture For 5G
10 pages