CSS4E01d - Storage Area Networks Unit III (Part 1)

Unit III
Fiber Channel Protocol Stack - Fiber Channel SAN – IP Storage.

Fibre Channel Basics

The enormous advancement in the performance of computers paved way to the

popularization of bandwidth- and storage- intensive applications. The existing
framework has been slow to keep up with the increasing demands related to
performance and storage, which has resulted as a bottleneck in the performance in
in the areas of speed, distance, and device connectivity. For example, Small
Computer System Interface (SCSI) —one of the popular infrastructures — is
limited to 80 Mbps of speed, up to 25 meters of bus length, and a maximum of 32
devices per bus. These restrictions are major bottlenecks for online trading and
transactions.

Fibre Channel has emerged as an ideal solution for storing, retrieving, and
transferring data between servers, storage devices, other network devices, and
users. Despite the high-speed, high-performance, and reliable solution that Fibre
Channel offers, it remains relatively inexpensive. Because of these advantages,
Fibre Channel has been adopted as the native technology in storage area networks
(SANs).

 Fibre channel is a serial data transfer architecture developed by a consortium

of computer and mass storage device manufacturers and now being
standardized by ANSI. The most prominent Fibre Channel standard is Fibre
Channel Arbitrated Loop (FC-AL).
 The fibre channel standard defines a high-speed data transfer interface that
can be used to connect workstations, mainframes, supercomputers, storage
devices, and displays. The fibre channel standard addresses the need for very
fast transfers (up to 1 gigabit per second) of large amounts of information.
Currently, fibre channel's main use is as an interface to storage.
 It is conceived as a generic, efficient physical transport system that can
support multiple protocols, variety of channels and networks currently in
place – hence it provides one standard for networking, storage, and data
transfer.
 FC has strong industry support. Vendors such as McData, Vixel, Emulex,
Qlogic and StorageTek develop Host Bus Adapters, FC-AL Hubs, Fabrics
and management software. They work together with solution providers such
as Compaq, Del, HP, IBM and Sun to create entire solutions.

Department of Computer Science, Farook College Page 1

CSS4E01d - Storage Area Networks Unit III (Part 1)

 FC need a upper level protocol on top of it - ATM, IP and FICON, with

SCSI being the predominant usage. FC is the low-level transport that ships
data, but hosts are normally communicating via SCSI as far as they're
concerned. The hubs, switches and HBAs in a SAN all speak FC, while the
applications that use SAN storage continue to use familiar protocols, like
SCSI.

In a networking environment, processors can communicate with other processors

and peripheral devices by using two technologies—network and channel.

Network Vs Channel

A channel is a direct or switched point-to-point communication between two

devices. As a result, channel communication is hardware-intensive, fast, and
produces low overhead. The network is a multi-point communication which make
use of protocols. Because of multi-point connections, and high dependency on
protocols, network communication is slower than channel communication.
Network communication also produces high overhead. However, networks can
support extensive communication between various devices, as compared to
channels that operate with a limited number of devices.

Department of Computer Science, Farook College Page 2

CSS4E01d - Storage Area Networks Unit III (Part 1)

 Currently network is connectionless. Data and the addressing information

are bundled as packets and transmitted over a shared medium. These
networks are simple, straight forward to implement, scalable and flexible.
But reliability is a grave concern to ensure correct delivery of packets,
especially when there are a large number of nodes in the network.
 A channel is concerned with IO performance. An IO channels responsibility
is to transmit data from one end to the other with least latency. Thus a
channel is typically hardware intensive with minimal software overhead.
The tradeoff here is minimal error correction.

Fibre Channel (FC) is designed to combine the best features of both networks
and channels. FC maintains the speed and low overhead of a channel while adding
the flexibility (through connectivity) and the longer distances that are characteristic
of a network.

FC incorporates channel type features such as point-to-point dedicated connections

and hardware intensive processing for buffer control, error handling and
encoding/decoding. FC incorporates networking type features such as circuit and
packet switching and use of data structures to pass control in formation in the data
stream. Other features include full-duplex transmission, support for both optical
and copper as a transport medium and the ability to serve as a transport for other
protocols, both channel and network.

Fibre Channel uses Fabric — a high-performance switched interconnection scheme

— to connect network devices. Fabric provides a point-to-point connection
between two communicating devices. The point-to-point connection helps the
communicating devices to exploit the full bandwidth of the Fibre Channel
connection until the transaction is over.

Advantages of Fibre Channel over SCSI

SCSI FC
Maximum data 80Mbps 133 Mbps, 266 Mbps, 530 Mbps, 1 Gbps,
transfer rate 2 Gbps, 4 Gbps and 8 Gbps (As of 2017)
Maximum number of 32 126 to 224 (approximately 16 million).
device of supported
simultaneously

Department of Computer Science, Farook College Page 3

CSS4E01d - Storage Area Networks Unit III (Part 1)

The maximum length 25m 30 meters (for Copper Cables). Ranges

of bus permitted from 500 meters to 10 kms in the case of
between storage optical fiber links. Switching modules are
devices and servers designed that can overcome this distance
limitation. For eg, Catalyst 5000 switching
modules (CISCO) can support Fibre
Channel connectivity up to 70 kms (43.5
miles).
Fault tolerance Low High
Administrative costs High. Due to Low. Because of reduced network
high downtime, high fault tolerance, and
downtime increased data availability.
and reduced
data
availability

The Fibre Channel Architecture

 The generic Fibre Channel network is composed of one or more bi-

directional point-to-point channels. The links support 1Gbps (or 100MBps)
data rates in each direction. The transport media may be fiber optic cable,
copper twisted pair or coax cable.
 The links in the FC network are between communication ports known as
N_ports. N_port stands for Node Port where a node is a device on the FC
network. The links may be point-to-point between N_ports or the may be set
up as a Fabric.
 A Fabric consists of several N_Ports connected to a switch. Ports on the
switch are called F_ports.
 The ports may be “daisy chained” to form a ring (In electrical and electronic
engineering a daisy chain is a wiring scheme in which multiple devices are
wired together in sequence or in a ring. A daisy chain is an interconnection
of computer devices, peripherals, or network nodes in series, one after
another). This is called an Arbitrated Loop (FC-AL). In this configuration
the ports are referred to as L_ports. No switch is necessary for FC-AL.
These basic layouts may be combined in different ways to create more
complex topologies.

Department of Computer Science, Farook College Page 4

CSS4E01d - Storage Area Networks Unit III (Part 1)

Fibre Channel Topologies

FC is typically realized in one of 3 topologies:

o Point-To-Point
o Loop
o Switched Fabric.

Point-To-Point connection

The simplest type of connection. It can exist by itself or as a subset in a Fabric or

Loop topology. An example of a point-to-point topology would be a FC tape unit
connected directly to a server.

Department of Computer Science, Farook College Page 5

CSS4E01d - Storage Area Networks Unit III (Part 1)

Because the connection between two nodes is direct, there is no media sharing.
This allows the two connected nodes to use the complete bandwidth of the link.

In this topology, a network node can also be connected to more than one node at
the same time, maintaining a point-to-point connection with the two (See the above
Figure). For this, the node should have more than one N_Port.

The Arbitrated Loop (FC-AL) Topology

Department of Computer Science, Farook College Page 6

CSS4E01d - Storage Area Networks Unit III (Part 1)

 In this design, all devices are in a loop or ring, similar to token ring
networking. Up to 126 nodes can be connected to form a complete loop.
 Fibre Channel hubs exist to connect multiple devices together and may
bypass failed ports. A loop may also be made by cabling each port to the
next in a ring.
 A minimal loop containing only two ports. Only one pair of ports can
communicate concurrently on a loop.
 To transfer data, all nodes that are part of the loop must try to gain control of
the loop. Once a node gains control of the loop, it establishes a virtual point-
to-point connection with the intended recipient and starts transferring data.
During the transmission, the entire bandwidth is available to the two
communicating ports. When the transmission is complete, the connection is
terminated and the rest of the loop nodes can arbitrate for control of the loop.
 Advantages: Up to 126 nodes can be connected to form a complete loop.
Loops are easy to setup and maintain and are relatively inexpensive. Several
loops may be connected via switch to share the load. The virtual point-to-
point connection enables high-speed transmissions and full use of the
available bandwidth. Addition or deletion of nodes from the loop by the
implementation of hubs and bypass ports is dynamic.
 Disadvantages: The devices are subject to a failure similar to a series
circuit. If one node on the loop goes. The larger the number of nodes
attached to a loop, the lower the performance of the loop.

The Switched Fabric Topology

 can be considered the “ideal” FC topology.

Department of Computer Science, Farook College Page 7

CSS4E01d - Storage Area Networks Unit III (Part 1)

 The configuration of the Fabric can be either cascaded or non-cascaded.

 In the case of switched, non-cascaded Fabric, the switches are not
interconnected. The basic idea behind this topology is that each device is
connected to a central switch - network nodes and storage devices are all
connected to a switch.
 In the switched, cascaded Fabric configuration, all the switches are
interconnected with each other to form a mesh. The configuration resembles
a large logical switch. By being connected to one switch, a node can access
any other node that is connected to one of the switches that belongs to the
Fabric.
 Advantages: (i) It allows for several communications to occur
simultaneously (ii) Availability of high bandwidth for each communication
because media is not shared with other LAN-side network traffic (iii)
Support to a large number of nodes (224) (iii) Increased fault tolerance due to
the implementation of virtual mesh, which prevents other nodes and
communications from being affected when a node or link fails (iv) Improved
scalability because devices (nodes) can be hot-swapped without affecting
other nodes, ports, or ongoing communications

Layers or Levels of FC

Figure: Fibre Channel Protocol Architecture.

Department of Computer Science, Farook College Page 8

CSS4E01d - Storage Area Networks Unit III (Part 1)

 Fibre Channel uses a multi layer protocol architecture.

o There are 5 layers.
 FC-0: Physical layer
 FC-1: Encode/Decode layer
 FC-2: Framing Protocol/Flow Control
 FC-3: Common Services
 FC-4: Upper Level Protocol Support.
 Additionally, there is another layer, which although is not
typically considered part of the basic architecture - FC-AL
(Arbitrated Loop) layer.
 FC–0 through FC–2 are called as The Physical and signaling
level (FC-PH).
 The Upper level (FC-UL) functional level - also called as the
Node level – include FC-3 and FC-4.

FC-0: Physical Layer

 Describes the physical interface. The purpose of the physical interface is to
take a stream of bits in at the transmitter, send them over the media, receive
them and convert them back to bits at the output.
 This layer was designed to accommodate the maximum number of physical
requirements, such as connectors, electrical parameters, and data rates.

Department of Computer Science, Farook College Page 9

CSS4E01d - Storage Area Networks Unit III (Part 1)

 This layer offers high flexibility in the use of physical media. For example,
copper cabling can be used for short-distance communications, and optical
fiber cabling can be used for signals that traverse longer distances. This
flexibility allows network designers and administrators to go for specialized
configurations according to the requirements of the company, which makes
the network cost-effective.
 One of the safety features of the optical implementation requires the
transmitter to stop transmitting if a fiber optic cable is disconnected or
broken. This is to prevent any chance of the laser from causing eye damage.

FC-1: Encode/Decode layer

 This layer is called the transmission protocol layer.

 This layer is responsible for defining the following: (i) Character conversion
and coding rules (ii) Error detection (iii) Byte synchronization and word
alignment (iv) Link maintenance
 Describes the means by which user data is encoded for transmission and
decoded at the other end.
 FC uses an 8/10 bit encode/decode scheme originally developed by IBM.
This code accomplishes several things.
o It attempts to minimize errors by equalizing the number of 1’s and 0’s
transmitted and not allowing more than 5 consecutive bits of the same
type in a row.
o It allows for distinguishing “Special Characters” and also provides for
simplifying byte and word alignment.
o Additionally, the evening out of 1’s and 0’s also has the effect of
minimizing low frequency components in the transmitted signal. This
allows for the design of relatively inexpensive transmitter/receiver
circuitry which can perform at the required bit error rate (1012).

FC-2: Framing Protocol/Flow Control

o The most complex layer of the protocol.

o The major elements of FC-2 are: the encapsulation of data using
frames, flow control and classes of service. It also provides error
control.
o In order to serve as a transport mechanism, FC must be able to
encapsulate user data and deliver it to the intended recipient. Frames
are the basic package used to encapsulate and transport the data. There
are two basic types of frames, the Data Frame and the Link Control

Department of Computer Science, Farook College Page 10

CSS4E01d - Storage Area Networks Unit III (Part 1)

Frame. Both types have the same basic format. A frame is composed
of 6 basic sections:
 Start of Frame (SOF): The SOF is comprised of 4 bytes, the
“comma” and 3 bytes indicating the type of connection service.
 Frame Header (FH): The FH contains information to control
link operations, perform routing, do protocol processing and
detect missing or out of order Frames. One of the major
functions of the FH is to uniquely identify active Frames. A
Frame is uniquely identified by a combination of the following
fields: S_ID, D_ID, OX_ID RX_ID, SEQ_ID and SEQ_CNT.
A Sequence Qualifier identifies active and open Sequences. The
Sequence Qualifier is composed of all the fields listed above
except the SEQ_CNT. The SEQ_CNT is used to identify the
Frame’s location within the Sequence.

Frames and Frame Header Formats

 The Optional Header section can contain up to 4 types of
headers:
 Expiration Security Header (how long until Frame should
be discarded)
 Network Header (contains info pertaining to network
related ULPs e.g. IP)
 Association Header (associates Frames with different
Exchange Id’s)
 Device Header (contains other ULP info e.g. SCSI).

Department of Computer Science, Farook College Page 11

CSS4E01d - Storage Area Networks Unit III (Part 1)

o The Payload: For a Data Frame, the payload contains the user Data.
For a Link Control Frame, the payload is not used. The CRC field is
used to verify the data integrity of the FH and Payload.
o The End of Frame (EOF) delimiter is an ordered set that designates
the end of the Frame content. The EOF contains information about the
validity of the Frame’s content, whether to keep the connection open
or normally close and if necessary, abort the connection.
o One logical level up from the Frame is the Sequence. Sequences
provide the means for ensuring data integrity of blocks of data that are
transmitted and received. A Sequence is composed of a group of
related Frames transmitted in one direction.
o FC-2 controls the flow of Frames between ports so that receiver
buffers are not overrun. A count of the number of buffers available is
maintained by the Sequence Initiator (transmitter) and is used to
throttle the transmission of Frames.
 There are two basic types of flow control. These are End to End
and Buffer to Buffer.
 End to End flow control is used in cases of N_port to N_port
communications. In this type, the receiver responds to all valid
Frames it receives with an ACK Frame. In the case of no
buffers available on the receiver, it responds with a Busy
Frame. In the case of an invalid Frame, it sends a reject frame.
The Sequence Initiator is responsible for maintaining the
current amount of credit (EE_Credit_CNT). It decrements the
credit count when it sends a Frame and increments it only when
an ACK response is received.
 Buffer to Buffer flow control is used in the case of an N_port
talking to a Fabric or in the case of an N_port to N_port
connection in a Point to Point topology. In this case each side is
responsible for maintaining its own BB_Credit_Count. As with
End to End, the BB_Credit_CNT is decremented for every
Frame transmitted, but is incremented only when it receives a
R_RDY Ordered Sequence (Primitive Signal). This indicates
that the receiver has resources available to accept another
Frame.
 FC-4: Common Services
o The FC-3 level is not currently fully defined.
o Its intended purpose is to support “common services” - a service that
would utilize multiple N ports working together on a single node. An
example would be “striping” where data is sent out of a node on
several N_ports such as data acquired by a high speed data acquisition

Department of Computer Science, Farook College Page 12

CSS4E01d - Storage Area Networks Unit III (Part 1)

process being “striped” to several RAIDs. This would increase the

effective bandwidth by sending data out in parallel.
 FC-4: Upper Level Protocol Support
o Supports the mapping of Upper Level Protocols (ULP) onto Fibre
Channel data structures.
o This allows FC to be a lower level transport mechanism for ULPs.
Because of its high speed, low overhead and reliability, FC can be
used to carry channel, peripheral interface and networking protocols.
o There are mappings for several major protocols such as SCSI (Small
Computer Systems Interface), IPI-3 (Intelligent Peripheral Interface-
3), HiPPI (High Performance Parallel Interface), IP (Internet
Protocol) - IEEE 802.2 (TCP/IP) data, ATM/AAL5 (ATM adaptation
layer for computer data) and SBCCS (Single Byte Command Code
Set).
 FC serves as a transport for ULPs is by mapping the ULP messages (known
as Information Units) into FC Sequences and/or Exchanges. The flexibility
of Sequences and Exchanges are what make mapping many existing ULPs to
FC feasible.
 A primary use of FC technology is as a transport for SCSI. It is so
significant that an acronym has been created for it - Fibre Channel Protocol
for SCSI (FCP). The transport is accomplished by wrapping SCSI
command, response, status and data blocks. SCSI operations have bi-
directional communications scheme and thus are suited for operations using
FC Exchanges.

Department of Computer Science, Farook College Page 13

CSS4E01d - Storage Area Networks Unit III (Part 1)

Classes Service of FC

 Fibre Channel offers six communication strategies to ensure successful,

efficient, and fast delivery of different types of traffic. These strategies are
commonly known as classes of service (CoS).
 The six classes of service are Class 1, Class 2, Class 3, Class 4, Class 5, and
Class 6. Each class of service is related to different aspects of
communication, such as connection establishment, allocation of bandwidth
to the connection, delivery of frames between the two communicating ends,
Acknowledgment (negative and positive) of a delivery, if required and Flow
control mechanism.

Class 1

 Class 1 service provides acknowledged, reliable, connection-oriented

transactions. The main features of this service class include the following:
o Dedicated circuit-switched connection through the Fabric between
two communicating entities.
o Maximum-bandwidth and high-throughput transactions.
o Reliable delivery of frames in the same order as they were
transmitted.
 This class of service is ideal for bandwidth-intensive applications that
require the data to be transferred quickly and continuously, such as backup,
restore, and various multimedia applications.
 Class 1 provides a dedicated connection between two ports including Fabric
ports known as the Sequence Initiator and the Sequence Recipient.
 Once established, a Class 1 connection is retained and guaranteed by the
Fabric.
 The flow control management of a Class 1 connection is as shown below.

Department of Computer Science, Farook College Page 14

CSS4E01d - Storage Area Networks Unit III (Part 1)

Class 1 data flow. Note R_RDY on Connect Request only.

Class 2

 Class 2 provides an acknowledged, multiplexed, and connectionless service.

 The main features of Class 2 service are the following:
o Nondedicated connections, due to which the bandwidth is shared with
other transactions
o Multiplexed connections that allow a port to communicate with more
than one port simultaneously
o Guaranteed delivery of frames with acknowledgments
o Out-of-order delivery of frames because a single transmission link is
shared between many simultaneous transactions
 Class 2 service is ideal for traffic, where timeliness and order of delivery are
not critical. Class 2 service is commonly used for transferring IP packets and
FTP data.

Department of Computer Science, Farook College Page 15

CSS4E01d - Storage Area Networks Unit III (Part 1)

Class 2 data flow.

 Both Class 1 and Class 2 send acknowledgment Frames confirming Frame

delivery. If delivery cannot be made due to congestion, a Busy frame is
returned and the sender tries again.

Class 3

Class 3 data flow. Note the lack of ACK. Only R_RDY for link maintenance
 Class 3 provides unacknowledged connectionless service. Its main features
are:
o Non-dedicated connections - the bandwidth is shared with other
transactions
o Non-guaranteed data delivery – no acknowledgement after the
delivery of frames.
o Out-of-order delivery of frame.

 Class 3 is also called as a datagram service.

Department of Computer Science, Farook College Page 16

CSS4E01d - Storage Area Networks Unit III (Part 1)

 Class 3 service is used when timeliness and the order of frames delivery is
not important. Situations related to the loss or corruption of frames during
transmissions are handled by upper-level protocols and applications.
 Class 3 service is generally used to transfer SCSI data and for network
broadcasts and multicasts (sending data to a set of nodes or ports). It is also
used in FC-AL topology also use to transfer data.
 It sacrifices reliability for performance.

Class 4
 Class 4 is similar to Class 2. It provides acknowledged, multicast, and
connection-oriented service.
 The main difference between the two service classes is that in Class 2,
bandwidth is shared between simultaneous transactions. In Class 4,
bandwidth is shared; but each transaction is allocated a fraction of the
available bandwidth, which cannot be shared with the other transactions that
are happening at the same time
 Features of Class 4 include:
o Dedicated connection, which is allocated a part of the total bandwidth.
This Class 4 connection is referred to as a virtual circuit (VC)
o Shared bandwidth - multiple Class 4 connections can be established
simultaneously
o Multicast connections that allow a port to communicate with more
than one port simultaneously.
o Reliable delivery of frames - with the help of acknowledgments
o In-order delivery of frames

Class 5
 Class 5 service is reserved for future operations and is still under
development.

Class 6

o Class 6 acknowledged, connection-oriented, and reliable transactions.

o It is similar to Class 1 service except that it is unidirectional. It also provides
a multicast facility by allowing one port to transfer the same data to more
than one port simultaneously.
o This class is ideal for video broadcasts (which are unidirectional) and other
real-time applications that need to transfer large quantities of data. This class
uses end-to-end flow control.

Department of Computer Science, Farook College Page 17

CSS4E01d - Storage Area Networks Unit III (Part 1)

A summary of features of all classes of service are as shown below:

Department of Computer Science, Farook College Page 18