<Course

High Availability
Title>

LY
N
O
SE
U
AL
N
R
TE
IN
High Availability

LY
N
O
SE
U
AL

High Availability Defined

N

Users want their systems, (for example, computers, telephones, video games, or televisions) to work
properly and at all times. Availability, in general terms, refers to the ability of a user group to access a
R

desired system or resource. When the user group cannot access a given system or resource, the
resource is considered unavailable. Often, the term downtime is used to describe a period when a
TE

system or resource is unavailable. Therefore, high availability is the ability to ensure a high degree of
operational continuity for a given user community to a system or some other resource.
Continued on next page.
IN

2 [Link]
 High Availability
High Availability Defined (contd.)
Note that uptime and availability are not the same thing. A system can be up, but unavailable
because of other issues. Availability is typically measured as a percentage of uptime over a given
duration. The following table provides a mapping for availability percentages and the corresponding
amount of time a system is considered to be unavailable:

Availability and Downtime Mappings

Availability Downtime per Downtime per Downtime per
% year month (30 days) week

90% 36.5 days 72 hours 16.8 hours

LY
99% 3.65 days 7.20 hours 1.68 hours

99.9% 8.76 hours 43.2 minutes 10.1 minutes

99.99% 52.6 minutes 4.32 minutes 1.01 minutes

N
99.999% 5.26 minutes 25.9 seconds 6.05 seconds

99.9999% 31.5 seconds 2.59 seconds 0.605 seconds

O
The obvious objective is to achieve the highest level of availability possible. Note that downtime
calculations can vary between organizations, which can make them somewhat misleading. Because

SE
the manner in which downtime is calculated can vary, you might find that overall user satisfaction is
a better method of evaluating success.
U
AL
N
R
TE
IN

[Link] 3
High Availability

LY
N
O
SE
U
AL

High Availability Networks

N

When designing high availability networks, you should ensure that all network components function
properly and are available to their respective user community. As previously mentioned, there is a
R

difference between uptime and availability, but both are equally important and must coexist for full
operational continuity. All high availability networks include provisions in the form of features and
TE

physical characteristics that allow for maximum uptime and accessibility.

To maximize uptime and accessibility in a network, you should consider the following as you design
and implement your network:
1. Is the base functionality of the network’s hardware, software, protocols, and overall
IN

operations protected?
2. When a failure occurs, how will the overall system or network react, and what level of
recovery can be expected?
To properly protect a network, your network design should include some level of redundancy.
Although redundancy can provide a large amount of protection for the network, it does come with a
cost. Many devices running the Junos operating system include redundant hardware components,
such as Routing Engines (REs), control boards (CBs), power supplies, and cooling fans. Refer to the
technical publications at [Link] for details on a specific Junos device.
In addition to redundant hardware, you can also use various software features that accommodate
redundancy and rapid failure detection. We cover some of these high availability features on the next
slide.

4 [Link]
 High Availability

LY
N
O
SE
U
AL

Supported High Availability Features

N

The Junos OS supports a number of software features that can increase availability in a network.
Some high availability features compliment other features, whereas in other cases, they are mutually
R

exclusive and cannot be enabled together.

The following is a brief summary of the high availability features shown on the slide:
TE

• Graceful restart (GR): This feature allows uninterrupted packet forwarding and
temporary suppression of all routing protocol updates. GR enables a router to pass
through intermediate convergence states that are hidden from the rest of the network.
• Graceful Routing Engine switchover (GRES): This feature enables a routing platform
IN

with redundant REs to continue forwarding packets even if one RE fails. Graceful RE
switchover preserves interface and kernel information and ensures that traffic is not
interrupted. Graceful RE switchover does not, however, preserve the control plane.
Continued on next page.

[Link] 5
High Availability
Supported High Availability Features (contd.)
• Nonstop active routing (NSR): This feature uses the same infrastructure as graceful RE
switchover to preserve interface and kernel information. However, NSR also saves
routing protocol information by running the routing protocol process (rpd) on the backup
RE. By saving this additional information, NSR is self contained and does not rely on
helper routers to assist the routing platform in restoring routing protocol information.
NSR is advantageous in networks where neighbor routers do not support GR. As a result
of this enhanced functionality, NSR is a natural replacement for GR. NSR and GR are
mutually exclusive and cannot be enabled at the same time. Note that graceful RE
switchover must be configured for NSR to function properly.
• Bidirectional Forwarding Detection (BFD): This feature is a simple hello mechanism that
detects failures in a network. BFD sends hello packets at a specified, regular interval. A

LY
neighbor failure is detected when the routing device stops receiving a reply after a
specified interval. BFD works with a wide variety of network environments and
topologies. The failure detection timers for BFD have shorter time limits than default
failure detection mechanisms, providing faster detection.

N
• Virtual Router Redundancy Protocol (VRRP): This feature enables hosts on a LAN to
make use of redundant routing platforms on that LAN without requiring more than the
static configuration of a single default route on the hosts. The VRRP routing platforms

O
share the IP address corresponding to the default route configured on the hosts. At any
time, one of the VRRP routing platforms is the master (active) and the others are
backups. If the master fails, one of the backup routers becomes the new master router,

• SE
providing a virtual default routing platform and enabling traffic on the LAN to be routed
without relying on a single routing platform.
Unified In-Service Software Upgrade (ISSU): This feature allows you to upgrade between
two different Junos OS releases with no disruption on the control plane and with
minimal disruption of traffic. Unified ISSU is supported only on dual Routing Engine
U
platforms. In addition, graceful Routing Engine switchover (GRES) and nonstop active
routing (NSR) must be enabled.
Note that support for these highlighted features can vary between Junos devices. Refer to the
AL

documentation for your specific product for details. We cover the highlighted features in more detail
in the subsequent sections. The Junos OS supports several other HA features and supporting
technologies not shown on the slide. You can refer to the technical publications at http://
[Link]/techpubs/ for more details.
N
R
TE
IN

6 [Link]
 High Availability

LY
N
O
SE
U
AL

What If ...?
N

There are a number of possible events that can cause network disruptions. One such event is
presented on the slide, where R1’s routing process (rpd) restarts. When the routing process restarts,
R

all configured protocols are affected, which means the established network topology and
communication paths are also affected. In the scenario presented on the slide, all of R1’s OSPF
TE

neighbors (and in fact all OSPF routers in this case) must recalculate any path that traverses R1
because of the topology change.
Typically, when rpd restarts, the effect on the network is temporary. In other words, once rpd restarts
and the affected protocols re-establish their respective adjacencies or peering sessions, the topology
and data paths return to their original state, thus causing multiple, networkwide disruptions—one
IN

when the initial topology change occurs because of the rpd restart event, and one when the topology
is restored to its original form. These temporary disruptions can have a significant impact on a user’s
experience, especially considering today’s modern networks, which include voice and video
communications.

[Link] 7
High Availability

LY
N
O
SE
U
AL

Introducing Graceful Restart

N

Graceful restart (GR) addresses the situation described on the previous slide. GR allows a router
undergoing a restart event, including a restart of the routing protocol process (rpd), to inform its
R

adjacent neighbors and peers of its condition. The restarting router requests a grace period from the
neighbor or peer, which can then cooperate with the restarting router. When a restart event occurs
TE

and GR is enabled, the restarting router can still forward traffic during the restart period, and
convergence in the network is not disrupted. The neighbors or peers of the restarting router, also
known as helper routers, hide the restart event from other devices not directly connected to the
restarting router. In other words, the restart is not visible to the rest of the network, and the
restarting router is not removed from the network topology.
IN

The graceful restart request occurs only if the following conditions are met:
• The network topology is stable;
• The neighbor or peer cooperates;
• The restarting router is not already cooperating with another restart already in progress;
and
• The grace period does not expire.

8 [Link]
 High Availability

LY
N
O
SE
U
AL

GR Support
N

As shown on the slide, GR is supported by several standards-based protocols. A number of RFCs and
drafts exist that document the operational details for GR and each of the protocols for which GR is
R

supported. While these different protocols implement GR slightly differently, the basic concepts and
operations are the same from a high availability point of view.
TE

GR Requirements
Routers must have GR enabled to support both GR router modes—the restarting router mode and
helper router mode. By default, Junos devices can operate as helper routers but not as restarting
IN

routers; restarting router mode functionality must be enabled through configuration. We cover GR
configuration on subsequent slides.
In addition to having the GR functionality enabled, the router must support nonstop forwarding
operations, which simply means the router must be able to continue forwarding traffic during times
of control plane instability. Nonstop forwarding is an inherent attribute of Junos devices because of
the architectural design, which cleanly separates the control and forwarding planes.

[Link] 9
High Availability

LY
N
O
SE
U
AL

Configuring GR: Part 1

N

GR helper mode is enabled by default on all Junos devices. You can disable GR helper mode globally
for all supported protocols at the [edit routing-options] hierarchy or on a per-protocol,
R

per-group, or per-neighbor basis, depending on the specific protocol. The slide illustrates the syntax
required to disable GR helper mode globally, enable GR helper mode for the BGP protocol, and
TE

disable GR for a BGP peer. As with many similar configuration scenarios, the most specific definition
is used.
IN

10 [Link]
 High Availability

LY
N
O
SE
U
AL

Configuring GR: Part 2

N

GR’s restarting router mode is not enabled by default. You can enable GR restarting router mode
through configuration at the [edit routing-options] hierarchy. The slide provides a sample
R

configuration used to enable GR’s restarting router mode globally and for all protocols along with a
sample configuration that disables GR for a specific BGP peer.
TE

Continued on next page.

IN

[Link] 11
High Availability
Configuring GR: Part 2 (contd.)
The configuration options used with GR vary between the supported protocols. The following are the
available GR configuration options for OSPF and BGP:
[edit protocols]
user@R1# set ospf graceful-restart ?
Possible completions:
+ apply-groups Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
disable Disable OSPF graceful restart capability
helper-disable Disable graceful restart helper capability
no-strict-lsa-checking Do not abort graceful helper mode upon LSA changes
notify-duration Time to send all max-aged grace LSAs (1..3600 seconds)

LY
restart-duration Time for all neighbors to become full (1..3600 seconds)

[edit protocols]
user@R1# set bgp graceful-restart ?

N
Possible completions:
<[Enter]> Execute this command
+ apply-groups Groups from which to inherit configuration data

O
+ apply-groups-except Don't inherit configuration data from these groups
disable Disable graceful restart
restart-time Restart time used when negotiating with a peer (1..600)
stale-routes-time Maximum time for which stale routes are kept (1..600)
| Pipe through a command

SE
For other protocol-specific GR configuration options, refer to the technical publications.
U
AL
N
R
TE
IN

12 [Link]
 High Availability

LY
N
O
SE
U
AL

Monitoring GR
N

You monitor GR differently depending on the protocol with which it is configured. For BGP, you use
the show bgp neighbor command along with the peer address. If you have GR restarting router
R

mode enabled under the [edit routing-options] hierarchy, the GracefulRestart option
appears in the output generated by the show bgp neighbor command; otherwise this option is
TE

omitted from the options list. The other references of restart shown in the sample output are
indicative that this router supports the GR helper router mode, which is enabled by default.
Continued on next page.
IN

[Link] 13
High Availability
Monitoring GR (contd.)
For OSPF, and other protocols such as IS-IS, you monitor GR operations using traceoptions. You must
first enable traceoptions using the graceful-restart flag option. Then you monitor the
associated log file using the show log log-file-name command. The following output
illustrates a sample GR transaction from the perspective of the restarting router:
user@R1> show log trace-ospf
Jan 15 10:08:53.652803 OSPF Restart: phase now 2
Jan 15 10:08:55.300166 OSPF Restart: sending grace lsas
Jan 15 10:08:55.301066 OSPF Restart: estimated restart duration timer triggered
Jan 15 10:08:55.301139 OSPF Restart: area [Link] triggered restart maxwait timer of
40 seconds
Jan 15 10:08:56.312242 OSPF Restart: sending more grace lsas

LY
Jan 15 10:08:57.312678 OSPF Restart: sending more grace lsas
Jan 15 10:09:01.320373 OSPF Restart: graceful restart OK to send hellos
Jan 15 10:09:01.320486 OSPF Restart: phase now 3
Jan 15 10:09:01.336244 RPD_OSPF_NBRUP: OSPF neighbor [Link] (realm ospf-v2 ge-0/

N
0/1.0 area [Link]) state changed from Init to 2Way due to 2WayRcvd (event reason:
neighbor detected this router)
Jan 15 10:09:35.308470 OSPF Restart: area [Link] restart maxwait timeout

O
Jan 15 10:09:37.703251 OSPF Restart: area [Link] building the router lnk nbr tree
Jan 15 10:09:37.703554 OSPF Restart: lnk_nbr_tree link type 2 id [Link] data
[Link], numberred 1
Jan 15 10:09:37.703800 OSPF Restart: build intf_lnk_nbr tree interface ge-0/0/1.0
area [Link]

[Link], numberred 1 SE
Jan 15 10:09:37.703859 OSPF Restart: lnk_nbr_tree link type 2 id [Link] data

Jan 15 10:09:37.756306 RPD_OSPF_NBRUP: OSPF neighbor [Link] (realm ospf-v2 ge-0/

0/1.0 area [Link]) state changed from Loading to Full due to LoadDone (event reason:
U
OSPF loading completed)
Jan 15 10:09:37.756403 OSPF Restart: remove neighbor id [Link] interface ge-0/0/
1.0 area [Link] from interface lnk_nbr_tree
AL

Jan 15 10:09:37.756465 OSPF Restart: removed neighbor [Link] id [Link]

interface ge-0/0/1.0 area [Link] from area lnk_nbr_tree
Jan 15 10:09:37.756540 OSPF Restart: lnk_nbr_tree is empty for area [Link]
Jan 15 10:09:37.756588 OSPF Restart: phase now 4
Jan 15 10:09:37.756731 OSPF Restart: all neighbors acquired. Notifying all that TED
N

database is populated
Jan 15 10:09:37.756779 OSPF Restart: exit imminent
R

Jan 15 10:09:47.793197 OSPF Restart: the restart estimated duration is up

Jan 15 10:09:47.793330 OSPF Restart: phase now 5
Jan 15 10:09:47.793569 OSPF Restart: purging grace lsas
TE

Jan 15 10:09:47.997263 OSPF Restart: purging old LSAs

Jan 15 10:09:47.997361 OSPF Restart: phase now 0
Jan 15 10:09:47.997791 OSPF Restart: phase 0 restart complete
IN

14 [Link]
 High Availability

LY
N
O
SE
U
AL

Graceful RE Switchover Defined

N

Many Junos devices offer hardware component redundancy, such as redundant REs. Graceful RE
switchover enables a routing platform with redundant REs to continue forwarding packets, even if
R

one RE fails. Graceful RE switchover preserves interface and kernel information and ensures that
traffic forwarding is not interrupted during a mastership change. Graceful RE switchover does not,
TE

however, preserve the control plane, which means the routing protocol process (rpd) must restart
and the information learned through that process must be relearned (unless NSR is also configured).
The slide shows a brief comparison that highlights the benefits of using graceful RE switchover.
Without graceful RE switchover enabled, the PFE restarts and all hardware and interfaces are
discovered by the new master RE when a mastership change occurs. The new master RE restarts
IN

rpd, so all adjacencies are aware of the topological changes. Because all interfaces go down during
this process, the system generates interface alarms. The network also undergoes a topology change
because all protocol adjacencies are affected, albeit temporarily.
With graceful RE switchover enabled, the PFE is not restarted and interface and kernel information is
preserved. By allowing the PFE to remain up during a mastership switchover and preserving
interface and kernel information, graceful RE switchover greatly reduces the time the RE failover
process takes. With no other high availability features enabled, the new master RE must restart rpd,
so all adjacencies are aware of the routing change. To preserve routing during a switchover, graceful
RE switchover must be combined with either GR or NSR. We discuss NSR in the next section of this
material.

[Link] 15
High Availability

LY
N
O
SE
U
AL

Graceful RE Switchover Operations

N

By default, when two REs are installed in a router, the RE installed in slot 0 (known as RE0) assumes
the master RE role, whereas the RE installed in slot 1 (known as RE1) assumes the backup RE role.
R

The following sample output illustrates this concept:

{master}
TE

user@R1-re0> show chassis routing-engine

Routing Engine status:
Slot 0:
Current state Master
IN

Election priority Master (default)

...
Routing Engine status:
Slot 1:
Current state Backup
Election priority Backup (default)
...
Continued on next page.

16 [Link]
 High Availability
Graceful RE Switchover Operations (contd.)
To allow some distinction between both REs and ensure that both REs are accessible regardless of
their current role, you can define and apply RE groups, as shown in the following sample capture:
{master}[edit groups]
user@R1-re0# show
re1 {
system {
host-name R1-re1;
backup-router [Link];
}
interfaces {
fxp0 {

LY
unit 0 {
family inet {
address [Link]/24;
}

N
}
}
}

O
}
re0 {
system {
host-name R1-re0;

}
backup-router [Link];

interfaces {
fxp0 {
unit 0 {
SE
U
family inet {
address [Link]/24;
}
AL

}
}
}
}
N

You apply the re0 and re1 groups at the root hierarchy level:
{master}[edit]
R

user@R1-re0# set apply-groups [re0 re1]

TE

The previous sample configuration ensures each RE can be distinguished by a unique name and
gives each RE its own management IP address for direct out-of-band (OoB) management access. By
default, the backup RE does not have rpd running, which means it does not have any routes
installed. You can install a route entry (in this case a default route) on the backup RE using the
backup-router statement. The specified address represents the gateway address for this route
IN

entry. Note that the backup-router statement can alternatively be included under the [edit
system] hierarchy level as a single entry and its definition is not required but highly recommended.
Continued on next page.

[Link] 17
High Availability
Graceful RE Switchover Operations (contd.)
Once graceful RE switchover is enabled, you should synchronize the configurations using the
commit synchronize command. If you do not synchronize configurations, the configuration
applied on the backup RE will be used when a failover occurs. If graceful RE switchover is enabled
and you do not issue commit synchronize, you will see the following commit warning:
{master}[edit chassis]
user@R1-re0# commit
warning: graceful-switchover is enabled, commit synchronize should be used
commit complete
You can, alternatively, add the synchronize functionality through the configuration. With this
functionality enabled, issue commit to synchronize configurations, as follows:

LY
{master}[edit system]
user@R1-re0# set commit synchronize

{master}[edit system]

N
user@R1-re0# commit
re0:
configuration check succeeds

O
re1:
commit complete
re0:
commit complete

SE
Note that in some virtualization implementations, the commit synchronize operation is automatic.
Check the documentation for your specific platform for details.
Once the REs are synchronized, they exchange keepalives. If the backup RE does not receive a
keepalive from the master RE after a specified timeout (typically 2 seconds), it determines that the
U
master RE has failed and takes mastership. When a mastership change occurs, the PFE seamlessly
disconnects from the old master RE and reconnects to the new master RE. The PFE does not reboot
and continues forwarding traffic based on the existing forwarding table entries. The new master RE
AL

then synchronizes its state with the PFE. If the new master RE detects that the PFE state is not up to
date, it resends state update messages.
Continued on next page.
N
R
TE
IN

18 [Link]
 High Availability
Graceful RE Switchover Operations (contd.)
When graceful RE switchover is enabled, the router alters the information flow from the RE to the
PFE. The router now duplicates changes made on the master RE to the backup RE prior to signalling
those changes to the PFE. When graceful RE switchover is not enabled, the master RE notifies only
the PFE.
If graceful RE switchover is not enabled, we recommend that you implement RE failover protection.
You can enable failover protection, adjust the keepalive interval and alter the default mastership
assignments under the [edit chassis redundancy] hierarchy. When RE failover protection is
enabled, the default keepalive interval is 300 seconds.
[edit chassis redundancy]
user@R1# set ?
Possible completions:

LY
+ apply-groups Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
> failover Failover to other Routing Engine
keepalive-time Time before Routing Engine failover (2..10000 seconds)

N
> routing-engine Redundancy options for Routing Engines
If needed, you can perform a manual RE mastership switchover using the request chassis

O
routing-engine master commands, as follows:
user@R1> request chassis routing-engine master ?
Possible completions:
acquire
release
switch SE
Attempt to become master Routing Engine
Request that other Routing Engine become master
Toggle mastership between Routing Engines
Redundancy configuration options are platform specific. Check the documentation for your specific
product for support details.
U
AL
N
R
TE
IN

[Link] 19
High Availability

LY
N
O
SE
U
AL

Configuring Graceful RE Switchover

N

You enable graceful RE switchover under the [edit chassis] hierarchy using the set
redundancy graceful-switchover command. Once you activate the configuration change,
R

the banner should change, indicating the RE’s current role (either master or backup) as shown on
the slide. In some virtualization implementations, such as the virtual chassis (EX Series) or chassis
TE

cluster (SRX Series), the role indicator (either master or backup) is automatically displayed,
regardless of whether graceful RE switchover is enabled.
IN

20 [Link]
 High Availability

LY
N
O
SE
U
AL

Monitoring Graceful RE Switchover

N

The slide illustrates the show system switchover command, which is used to verify whether
graceful RE switchover is enabled and that the databases are synchronized. Note that you can issue
R

this command on the backup RE only.

For graceful RE switchover to function properly, the master RE replicates its state to the backup RE
TE

and PFE. The following are the three specific states that must be replicated:
• Configuration database: The configuration database, or repository of configuration files,
is replicated through the commit synchronize process. Several system processes
require the configuration database to perform their designated functions; for example,
IN

the device control daemon (dcd) checks this database when it brings interfaces online,
the chassis process (chassisd) uses this database to manage hardware components,
and rpd uses this database to control routing protocols.
• Kernel and related entries: Enabling graceful RE switchover starts a custom Junos
process known as ksyncd. The ksyncd process is responsible for kernel state replication
tasks between various hardware components.
• PFE state: The Junos OS uses chassisd to perform PFE state replication. When a
mastership change occurs, chassisd performs a soft restart to query the system’s
hardware inventory. When the hardware components respond to the query, the system
re-attaches them to the backup RE and brings them online without any disruption.

[Link] 21
High Availability

LY
N
O
SE
U
AL

Nonstop Active Routing

N

Nonstop Active Routing (NSR) enables a routing platform with redundant REs to switch from a
primary RE to a backup RE without alerting peer nodes. NSR uses the same infrastructure as
R

graceful RE switchover to preserve interface and kernel information. In addition to maintaining

interface and kernel information, NSR also saves routing protocol information by running the routing
TE

protocol process (rpd) on the backup RE. By saving this additional information, NSR is self-contained
and does not rely on helper routers to assist the routing platform in restoring routing protocol
information. NSR is advantageous in networks where neighbor routers do not support GR protocol
extensions, as well as failure scenarios where GR cannot negotiate a grace period. As a result of this
enhanced functionality, NSR is a natural replacement for GR. If you enable both NSR and GR, the
IN

commit operation will fail.

NSR requires that the participating REs run the same version of the Junos OS. Although NSR
supports most protocols, it does not support all protocols. For all protocols supported by NSR, the
state information is preserved during a switchover event. If you configure a protocol that is not
supported by NSR, the protocol operates as usual. When a switchover occurs, the state information
for the unsupported protocol is not preserved and must be refreshed using the normal recovery
mechanisms inherent in the protocol. Also note that NSR is not supported on all Junos devices. Refer
to the technical publications for platform and protocol support details.

22 [Link]
 High Availability

LY
N
O
SE
U
AL

Configuring NSR
N

The slide provides a sample configuration used to enable NSR. Note that you must also enable
graceful RE switchover for NSR to function. We covered graceful RE switchover in detail in the
R

previous section of this material. In addition to enabling NSR and graceful RE switchover, you should
also ensure that the commit operation synchronizes the configuration file from the master RE to the
TE

backup RE. You can issue the commit synchronize command each time you perform a commit
operation or enable the synchronize functionality in the configuration, as follows:
{master}[edit]
user@R1-re0# set system commit ?
IN

Possible completions:
+ apply-groups Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
synchronize Synchronize commit on both Routing Engines by default
Once NSR is enabled and the configurations on the master and backup REs are synchronized, the
routing protocol process on the backup RE actively gathers information sent to and from the routing
protocol process on the master RE. This process allows the backup RE to keep its state up-to-date
with the network just as the master RE does.

[Link] 23
High Availability

LY
N
O
SE
U
AL

Monitoring NSR
N

As shown on the slide, you use the show task replication command to verify NSR
synchronization. You should see all configured protocols that support NSR synchronization listed in
R

the output, along with their complete status.

Continued on next page.
TE
IN

24 [Link]
 High Availability
Monitoring NSR (contd.)
Alternatively, you can log in to the backup RE and issue the same operational show commands you
would issue on the master RE to determine protocol and routing information, as follows:
{master}
user@R1-re0> request routing-engine login other-routing-engine
€
--- JUNOS 10.1R1.8 built 2010-02-12 18:31:54 UTC
{backup}
user@R1-re1> show ospf neighbor
Address Interface State ID Pri Dead
[Link] fe-0/0/1.0 Full [Link] 128 0
[Link] fe-0/0/2.0 Full [Link] 128 0

LY
{backup}
user@R1-re1> show bgp summary
Groups: 1 Peers: 2 Down peers: 0

N
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0 10 10 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn

O
State|#Active/Received/Accepted/Damped...
[Link] 64700 55 54 0 0 24:29 5/5/5/
0 0/0/0/0
[Link] 64700 54 52 0 0 23:53 5/5/5/
0

{backup}
0/0/0/0

user@R1-re1> show route protocol ospf

SE
U
inet.0: 21 destinations, 34 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
AL

[Link]/32 *[OSPF/10] 00:39:47, metric 1

> to [Link] via fe-0/0/1.0
[Link]/32 *[OSPF/10] 00:39:47, metric 1
> to [Link] via fe-0/0/2.0
[Link]/32 *[OSPF/10] 00:44:22, metric 1
N

MultiRecv
Note that you can enable traceoptions with the nsr-synchronization flag option and monitor
R

the associated log file to view NSR synchronization details for a given protocol.
{master}[edit protocols ospf]
TE

user@R1-re0# show
traceoptions {
file nsr-trace;
flag nsr-synchronization detail;
IN

}
[Trimmed]

[Link] 25
High Availability

LY
N
O
SE
U
AL

Detecting Network Failures

N

All modern protocols, such as OSPF and BGP, include some mechanism to detect network failures.
One problem with protocol failure detection mechanisms is that they can be slow (especially with the
R

default timers). One example, illustrated on the slide, is OSPF, which, with its default timers, can take
up to 40 seconds before a neighbor is declared dead.
TE

You can adjust the default timers to lower the time it takes a protocol to detect failures and declare a
neighbor or peer dead. In fact, you can adjust the dead timers for some interior gateway protocols
(IGPs), such as OSPF and IS-IS, to detect failures in about 1 second. There is, however, a cost
associated with lowering protocol timers. Lowering a protocol’s timers often equates to an increase
of hellos or keepalives and, consequently, more processing overhead for the related protocol and
IN

routing process. Increasing the load on a protocol or the routing process can potentially cause
undesirable results and adversely affect a router’s overall performance, especially in large networks.

26 [Link]
 High Availability

LY
N
O
SE
U
AL

Bidirectional Forwarding Detection

N

The Bidirectional Forwarding Detection protocol (BFD) is a simple protocol designed to rapidly detect
link failures. Once two devices negotiate and establish a BFD session, BFD continuously sends
R

hellos to monitor the associated link. If BFD stops receiving hellos from its neighbor, it takes down
the session and notifies the system that a communication problem exists. BFD can detect link
TE

failures in less than a second, which means hellos are exchanged and processed frequently and
efficiently.
BFD can provide a number of key benefits within a high availability network. BFD relieves protocols
from being required to provide fast-failure detection; in fact, routing protocol timers for hellos or
keepalives can be left at their default values or even be increased to reduce the associated
IN

processing. BFD also provides a single, common method for managing protocol timers. Rather than
modifying the relevant timers for each protocol running in the network, you can leave the protocol
timers at their default settings and simply implement BFD to use consistent timer values for all
protocols. Another key benefit provided by BFD is that it provides a failure detection mechanism for
static routes, which, unlike modern routing protocols, do not have such a mechanism otherwise.
The slide lists the protocols that supports BFD. When a failure occurs, BFD notifies the protocol it is
configured to support, at which time the protocol can take the needed action; in most cases, it routes
around the detected failure. BFD is configured directly under the protocol or static route you want it
to support. We examine the configuration details on a subsequent slide.

[Link] 27
High Availability

LY
N
O
SE
U
AL

Configuring BFD
N

The slide highlights some BFD configuration examples and a recommendation for the transmit and
receive intervals. You can set the transmit and receive intervals separately or define a
R

minimum-interval for both directions, as shown on the slide. Although you can set the minimum
interval to a value lower than 300 ms, we do not recommend a value below that interval. With the
TE

minimum interval set to 300 ms, BFD still achieves a subsecond failure detection. This subsecond
failure detection assumes the default multiplier value of three, which simply means if three
consecutive hellos are missed, the link is considered failed (3 * 300 ms = 900 ms failure detection).
On some Junos devices, BFD uses periodic packet management (PPM) on the PFE. PPM off loads the
processing overhead typically placed on the RE by delegating some of the processing responsibilities
IN

to the PFE. PPM is enabled by default on the Junos devices on which it is supported. For PPM
support details, check the product documentation for your specific platform.
Depending on the media type, BFD might not provide a significant benefit. For example, some media
types like SONET and ATM already provide rapid detection for link failure, so using BFD to monitor
those link types will not add a significant amount of value. BFD does, however, add significant value
for Ethernet interfaces, especially when Layer 2 switches are positioned between two devices.
Alternatively, you might consider Ethernet OAM, which provides rapid link failure detection for
Ethernet connections. Ethernet OAM, unlike BFD, operates at Layer 2.
Continued on next page.

28 [Link]
 High Availability
Configuring BFD (contd.)
By default, BFD sessions are adaptive, which means you can adjust the intervals for the session. You
define a minimum interval value for the transmit and receive directions on the router. If the neighbor
router’s interval for either direction is higher, that BFD session uses the higher value.
The hierarchy at which you enable BFD and the actual configuration options vary between the
supported protocols. The configuration options for OSPF and BGP include the following:
[edit protocols]
user@R1# set ospf area 0 interface ge-0/0/1.0 bfd-liveness-detection ?
Possible completions:
+ apply-groups Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
> authentication Authentication options

LY
> detection-time Detection-time options
full-neighbors-only Setup BFD sessions only to Full neighbors
minimum-interval Minimum transmit and receive interval (milliseconds)
minimum-receive-interval Minimum receive interval (1..255000 milliseconds)

N
multiplier Detection time multiplier (1..255)
no-adaptation Disable adaptation
> transmit-interval Transmit-interval options

O
version BFD protocol version number

[edit protocols]
user@R1# set bgp bfd-liveness-detection ?
Possible completions:
+ apply-groups

> authentication
SE
Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
Authentication options
U
> detection-time Detection-time options
holddown-interval Time to hold the session-UP notification to the client
minimum-interval Minimum transmit and receive interval (milliseconds)
minimum-receive-interval Minimum receive interval (1..255000 milliseconds)
AL

multiplier Detection time multiplier (1..255)

no-adaptation Disable adaptation
> transmit-interval Transmit-interval options
version BFD protocol version number
N

You can define BFD options for BGP at the protocol, group, or neighbor hierarchy levels. As with
similar configuration scenarios, the most specific definition is used (neighbor then group, group then
R

protocol). For BFD configuration options for other protocols, refer to the technical publications for the
specific protocol.
TE
IN

[Link] 29
High Availability

LY
N
O
SE
U
AL

Monitoring BFD
N

The slide shows the key command and options for monitoring BFD. In the sample output, the default
multiplier value is 3, which means three consecutive BFD hellos must be missed prior to declaring
R

the BFD session dead. You can also see that transmit interval of 400 ms is used for the session with
the [Link] neighbor, which is a higher value than what was configured. (Refer to the previous
TE

slide for details.) The higher transmit interval value is indicative of this BFD session operating in
adaptive mode, which is the default behavior. You can disable BFD adaptive mode using the
no-adaptation configuration option.
Continued on next page.
IN

30 [Link]
 High Availability
Monitoring BFD (contd.)
In addition to the show bfd session commands, you can also issue the show bgp neighbor
command for EBGP neighbors to verify BFD status for a given peer, as follows:
user@R1> show bgp neighbor [Link]
Peer: [Link]+179 AS 65510 Local: [Link]+49363 AS 64700
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ adv-aggregates ]
Options: <Preference AdvertiseInactive GracefulRestart PeerAS Refresh>
Options: <BfdEnabled>
Holdtime: 90 Preference: 170

LY
Number of flaps: 0
Peer ID: [Link] Local ID: [Link] Active Holdtime: 90
Keepalive Interval: 30 Peer index: 0
BFD: enabled, up

N
[Trimmed]
If you do not see the BFD session in the up state, check the configurations on both devices to ensure

O
they are compatible. You might also need to check any configured firewall filters on the Junos
devices configured for BFD to ensure that the defined firewall filter permits BFD communications.

SE
U
AL
N
R
TE
IN

[Link] 31
High Availability

LY
N
O
SE
U
AL

What If ...?
N

The slide presents a sample scenario and tests your knowledge of the overall impact the potential
failures might have in this sample network environment. Because R1 serves as the only gateway
R

beyond the [Link]/24 subnet, if it failed or either of its directly connected links became
unusable, the users on the referenced subnet would not be able to communicate with any remote
TE

subnet. We present a recommended solution for this scenario throughout the subsequent slides in
this section.
IN

32 [Link]
 High Availability

LY
N
O
SE
U
AL

VRRP Defined
N

The Virtual Router Redundancy Protocol (VRRP) is a standards-based election protocol that can
facilitate redundancy in a LAN environment and eliminate the single point of failure highlighted on
R

the previous slide. VRRP elects one of the participating routers (known as VRRP routers) to function
as gateway device, while all other VRRP routers serve in a backup capacity. All communications
TE

between VRRP routers occur through a common switch. We cover more details regarding the election
process and general VRRP communications on subsequent slides.
Note that VRRP is very similar in functionality to Cisco Systems’ Hot Standby Router Protocol (HSRP).
VRRP is most commonly found in Ethernet environments but can also be used in LAN environments
that use Token Ring or Fiber Distributed Data Interface (FDDI). VRRP is an industry standard and is
IN

defined in RFC 2338.

[Link] 33
High Availability

LY
N
O
SE
U
AL

Terms and Concepts

N

Any router participating in VRRP including the master and backup routers, is known as a VRRP router.
The master router is the VRRP router responsible for forwarding packets on a given LAN segment.
R

The master router also performs the Address Resolution Protocol (ARP) functions for the virtual
router that it represents. A backup router is a VRRP router that is available to assume the role of the
TE

master router if a failure occurs. Multiple backup routers can exist for a given VRRP group.
From the user’s perspective, a single gateway device exists when VRRP is deployed. This single
gateway device, in actuality, is a virtual device known as the virtual router. The virtual router is a
logical entity that functions as the default router for a LAN. The virtual router consists of a virtual
router identifier (VRID) and virtual IP (VIP) address. The VRID uniquely identifies one virtual router
IN

from another. The VIP address is managed by the virtual router and is attached to the VRRP router
functioning as the master for that network at any point in time.

34 [Link]
 High Availability

LY
N
O
SE
U
AL

VRRP Communications
N

VRRP version 2 uses a common advertisement packet to facilitate communication between VRRP
routers. VRRP uses this common advertisement packet primarily to relay the priority and state
R

details of the master router for a given virtual router. The VRRP advertisement packet is
encapsulated in an IP packet and sent to the Internet Assigned Numbers Authority (IANA) designated
TE

multicast address for VRRP, which is [Link]. The VRRP advertisement packet uses a time-to-live
(TTL) value of 255 and cannot be forwarded beyond the local subnet on which it is sent. This value
cannot be adjusted, and any packet that has a TTL value other than 255 is discarded. The default
interval for VRRP advertisements is 1 second. You can modify this interval to a value in the range of
1–255 seconds. If subsecond VRRP advertisements are required, configure the fast-interval
IN

option with a range of 100–999 milliseconds. All participating VRRP routers must support this
option.
Certain fields within the VRRP advertisements must match on all VRRP routers for a given group or
virtual router. Some examples are the VRID value and authentication parameters. If the values for
these fields, which require a match, are not the same, the packets are discarded and VRRP will not
work properly.
When a VRRP router sends a VRRP packet, the router uses the virtual MAC address as the source
MAC address. If a host on a LAN segment sends an ARP request for the VIP address, the master
router responds with the virtual MAC address associated with the virtual router or group. The virtual
MAC address is deterministic and uses the VRID or group number as its unique identifier. The virtual
MAC address uses the following format: 00-00-5E-00-01-VRID.

[Link] 35
High Availability

LY
N
O
SE
U
AL

Determining the Master Router

N

VRRP uses priority to elect the master router. The priority value range is 1–255, and higher priority
values are preferred. The Junos OS assigns a default priority value of 100 to each VRRP router,
R

unless the VRRP router owns the VIP address. If a VRRP router owns the VIP address, the priority
value must be set to 255. In this case, this router always assumes the role of master if it is
TE

functioning properly. If this router fails, the VRRP router or backup router with the next highest priority
at that time then assumes the role of master. When the router that owns the VIP address returns to
normal operation, it preempts the existing master and assumes the role of master once again.
In environments where the VRRP routers do not own the VIP address, you can administratively
disable preemption. We cover VRRP configuration details later in this section.
IN

36 [Link]
 High Availability

LY
N
O
SE
U
AL

VRRP States
N

Prior to electing the master and backup routers for a VRRP group, all routers begin in the initialize
state. In the initialize state, a startup event occurs, which essentially announces each VRRP router’s
R

capability and includes each router’s priority setting as well as other required parameters used in the
election process. During the initialize state, no forwarding occurs because there is no master router
TE

to represent the virtual router.

Based on the VRRP exchanges during the initialize state, VRRP elects the master router based on the
priority values; all other routers assume the backup state. The master router assumes forwarding
responsibilities for the LAN and responds to ARP requests sent to the VIP address. The master router
sends periodic announcements to all other VRRP routers within the VRRP group. These
IN

announcements indicate the master router’s state and priority. If the announcements are not
received for a deterministic period of time, the backup router with the next highest priority assumes
mastership. If a backup router that has a higher priority value than the current master router
becomes available, it assumes the role of master unless preemption is administratively disabled.
Routers in the backup state monitor the health or state of the master router and stand ready to
assume the mastership if necessary.
In the event of a mastership change, the backup router might, for a very brief moment, be in the
transition state. This state is simply a transitional step, in which a router changes from the backup
state to the master state. While in this state, no forwarding occurs for the LAN.

[Link] 37
High Availability

LY
N
O
SE
U
AL

Sample VRRP Configuration

N

The slide provides sample VRRP configurations for two routers participating in the same VRRP group.
Based on the configurations shown and the details covered on the previous slides, you know that R1
R

should assume the role of the master VRRP router because of its higher priority value.
To implement load balancing in your network, configure multiple VRRP groups for the same subnet
TE

with a different VIP for each group. You must ensure, using priority, that one router is the master
VRRP router for one group while the second router is the master VRRP router for the other group. This
configuration introduces two gateways for the same subnet, where each gateway device (the master
VRRP router for each group) has a backup gateway device (the backup VRRP router for each group).
Note that you must configure the end hosts to use one of the available gateways—the VIP for one of
IN

the two VRRP groups. Note that the load balancing distribution will vary between environments and
relies on the number of hosts assigned to each gateway and the overall traffic patterns associated
with those hosts.
Note that VRRP is not supported on all Junos devices. Check the product-specific documentation for
your product for support information.

38 [Link]
 High Availability

LY
N
O
SE
U
AL

Additional Configuration Options

N

This slide highlights some additional VRRP configuration options, along with a brief summary for
each option. Refer to the following additional details for these options:
R

• Use the track configuration option to monitor interfaces or routes that are essential
when forwarding traffic received through the VRRP interface. When a tracked interface
TE

or route becomes unavailable, the current priority value for the related VRRP group is
reduced to a new priority value based on a user-defined setting. The reduction of priority
for the VRRP group on the VRRP router involved can then trigger a mastership change.
The track configuration option is an excellent way to maintain external reachability
during a failure scenario.
IN

• By default, the master router representing the virtual router does not respond to
Internet Control Message Protocol (ICMP) requests sent to the VIP address unless the
master router owns that IP address. If the behavior is to have the master router respond
to ICMP requests sent to the VIP address—even if that router does not own that specific
IP address—use the accept-data configuration option. Note that using the
accept-data configuration option to facilitate ICMP responses violates RFC 2338.
The RFC strictly prohibits any ICMP response from a VRRP router unless that router
owns the VIP address. The use of this option can, however, dismiss unnecessary
problem reports indicating that the gateway router is not responding to ICMP requests.
Continued on next page.

[Link] 39
High Availability
Additional Configuration Options (contd.)
• Environments that have potential security risks should incorporate some level of
authentication. Use the authentication-type and authentication-key
configuration options to authenticate VRRP messages. The authentication options for
VRRP are none, simple, and Message Digest 5 (MD5). For LANs with security concerns,
we recommend MD5 authentication because it is the only authentication option that
encrypts the VRRP exchanges. These configuration options must match on all VRRP
routers within the same VRRP group.
• By default, the VRRP router with the highest priority setting becomes the master router
for a given virtual router. If a new router becomes available that has a higher priority, it
preempts mastership from the existing master. In situations where the VIP address is
not owned by any of the participating VRRP routers within a specific VRRP group, you

LY
can administratively disable preemption using the no-preempt configuration option.
Additional VRRP configuration options exist. The following output provides a full list of the VRRP
configuration options:

N
[edit interfaces ge-0/0/4 unit 0 family inet address [Link]/24]
user@R1# set vrrp-group 10 ?
Possible completions:

O
accept-data Accept packets destined for virtual IP address
advertise-interval Advertisement interval (1..255 seconds)
+ apply-groups Groups from which to inherit configuration data
+ apply-groups-except Don't inherit configuration data from these groups
authentication-key

no-accept-data
no-preempt
Authentication key
authentication-type Authentication type
fast-interval SE
Fast advertisement interval (100..999 milliseconds)
Don't accept packets destined for virtual IP address
Don't allow preemption
U
> preempt Allow preemption
priority Virtual router election priority (0..255)
> track Interfaces to track for VRRP group
AL

+ virtual-address One or more virtual IPv4 addresses

virtual-link-local-address Virtual link-local addresses
> vrrp-inherit-from VRRP group to follow for this VRRP group
You can use the vrrp-inheret-from configuration option when multiple VRRP groups are
N

defined on the same physical interface. This option allows one or more inheriting groups to acquire
certain configuration parameters from the active group. The inheriting groups acquire the
advertise-interval, authentication-key, authentication-type,
R

fast-interval, no-preempt, preempt, track interface, and track route

configuration options from the active group.
TE

In addition to simplifying the configuration tasks when multiple VRRP groups are defined on a single
interface, the vrrp-inheret-from configuration option also reduces the amount of VRRP traffic
that passes through that interface. Only the active group exchanges VRRP messages. Inheriting
groups do not exchange VRRP messages, but rather inherit the VRRP state associated with the
IN

active group.

40 [Link]
 High Availability

LY
N
O
SE
U
AL

Monitoring VRRP Operations

N

You can use the show vrrp commands shown on the slide to gather varying levels of detail. In the
example on the slide, we see that R1 is master and R2 is backup for VRRP group 10.
R

If you see that both VRRP routers show the master state, check the configuration and physical
connectivity through the switch. You might also need to check any configured firewall filters on the
TE

VRRP routers to ensure those firewall filter permit VRRP communications.

IN

[Link] 41
High Availability

LY
N
O
SE
U
AL

Unified ISSU
N

A unified in-service software upgrade (ISSU) enables you to upgrade between two different Junos OS
releases with no disruption on the control plane and with minimal disruption of traffic. Unified ISSU
R

is supported only on dual Routing Engine platforms. In addition, graceful Routing Engine switchover
(GRES) and nonstop active routing (NSR) must be enabled. Note that NSR is not supported on all the
TE

Junos devices. Refer to the technical publications for platform and protocol support details.
The master Routing Engine and backup Routing Engine must be running the same software release
before you can perform a unified ISSU. You cannot take any PICs online or offline during a unified
ISSU.
IN

42 [Link]
 High Availability

LY
N
O
SE
U
AL

Perform a Unified ISSU

N

The slide lists the high-level process for performing a unified ISSU.
R

To verify the status of Flexible PIC Concentrators (FPCs) and their corresponding PICs after the most
recent unified ISSU, issue the show chassis in-service-upgrade command on the master
Routing Engine:
TE

user@host> show chassis in-service-upgrade

Item Status Reason
FPC 0 Online
FPC 1 Online
IN

FPC 2 Online
PIC 0 Online
PIC 1 Online
FPC 3 Offline Offlined by CLI command
FPC 4 Online
PIC 1 Online
FPC 5 Online
PIC 0 Online
FPC 6 Online
PIC 3 Online
FPC 7 Online

[Link] 43
High Availability

LY
N
O
SE
U
AL

Troubleshooting Unified ISSU

N

The slide lists a troubleshooting procedure for unified ISSU.

R
TE
IN

44 [Link]