NSF/GR

High-end routers such as the ASR 9000 series and ASR 1000 series have redundant route processors. The route processors run in active/standby mode, and if one route processor fails, the standby route processor can take over. This takeover process is called SSO (stateful switchover).

When this feature was implemented, it required every routing protocol processes to restart when an SSO event occured. This reset neighborships and the routing protocol databases needed to be rebuilt on the standby RP (route processor). The problem with this was that traffic would stop breifly for 1-2 seconds while the neighborship was brought down and then re-established.

The workaround was to implement NSF (nonstop forwarding), also called GR (graceful restart). These two terms are effectively interchangable. NSF refers to the fact that the line cards have the FIB programmed locally, and can continue forwarding with the slightly stale FIB entries while the RIB gets rebuilt and routing protocol neighborships are re-established. When an SSO event happens, the router continues forwarding with the old forwarding entries even though they might be a few seconds stale.

GR (graceful restart) refers to the role the routing protocol neighbor plays during an SSO event. The neighbor must allow for a grace period and not consider the neighbor down for a few seconds. Then once the neighbor is back up, it helps the standby RP rebuild its RIB. The neighbor must not remove routes and cause adjacency flaps during this process.

NSF/GR works different on each protocol, but as you will see, the idea is generally the same. The neighbors advertise their capability to preform GR. When a SSO event occurs, the neighbors have some mechanism to understand that they shouldn’t flap the adjacency or remove routes during the rebuilding process.

NSF/GR only works with SSO. You must have redundant RPs to even use this feature in the first place. However, a router without SSO can still help a neighbor that does do SSO by being a GR “helper.” In GR there are two roles: NSF-capable and NSF-aware. NSF-capable means the device has dual RPs and is capable of preforming SSO. NSF-aware means the device is aware of how NSF works and can be a “helper” by allowing for a grace period while its neighbor preforms a SSO event. A NSF-capable router is generally always NSF-aware itself. However a device can be NSF-aware and not be NSF-capable.

While it is not really possible to lab an SSO event in a viritual setting, we can still examine the configuration and show commands.

Lab

We’ll simply use two routers, XRv and CSR1000v connected back-to-back.

OSPF NSF

OSPF NSF support was added to IOS before the IETF standard was ratified. For this reason, you have two options when configuring nsf under OSPF: cisco mode or ietf mode.

RP/0/0/CPU0:XRv(config)#router ospf 1
RP/0/0/CPU0:XRv(config-ospf)#nsf ?
  cisco             Enable Cisco Non Stop Forwarding
  flush-delay-time  Maximum time allowed for external route learning (seconds)
  ietf              Enable ietf graceful restart
  interval          Minimum interval between NSF restarts (seconds)
  lifetime          Maximum route lifetime following restart (seconds)

csr1000v(config)#router ospf 1
csr1000v(config-router)#nsf ?
  cisco    Cisco Non-stop forwarding
  enforce  Cancel NSF restart when non-NSF-aware neighbors detected
  ietf     IETF graceful restart
  <cr>     <cr>

Also note that by default, NSF helper support (the “NSF-aware” role) is enabled by default on both XE and XR, for both cisco mode and ietf mode.

csr1000v(config-router)#do show ip ospf | in NSF
 IETF NSF helper support enabled
 Cisco NSF helper support enabled

<I cannot find equivalent show output for XR>

From what I can tell, there doesn’t seem to be any advantage to using cisco mode over ietf mode. The benefit of ietf mode is that it conforms to the IETF standard (RFC 3623) and will interoperate with other vendor’s equipment. I believe cisco mode will not interoperate due to its proprietary nature.

No matter which mode you use, the goal is the same. During a RP switchover event, you want to avoid a change in the IGP topology, which would cause the neighborship adjacency flap, LSA flooding, and SPF calculations. While one router undergoes the switchover the event, the other router should wait “gracefully” and then help its neighbor rebuild its RIB, all without ever bringing down the adjacency.

The issue in OSPF is that the OSPF Hellos contain the activate neighbor. If a router undergoes an RP switchover event, the standby RP will not know the active neighbor, and therefore the field will be blank. The router needs a mechanism to tell the neighbor that it has just switched over, so it should ignore the fact that the active neighbor field is missing. The solution was to add an extended TLV to the OSPF Hello. This includes a restart signal and a LSDB resync bit. The restart signal is only set when the switchover has occured. The LSDB resync bit is essentially a capability advertisement, and signals that the router supports NSF.

When a router sees a Hello with the restart signal bit set, it keeps the neighborship in the Full state, even though the active neighbor field is missing. When the router sees DBD packets from the router which has switchovered with the restart signal bit set, it then moves the neighbor from Full to Exstart and starts LSDB sync. From here, DBD sync works as normal.

You cannot enable both cisco and ietf mode at the same time. If you try, it will not overwrite the existing mode with the new mode. Instead it will give an error and reject the command.

csr1000v(config)#router ospf 1
csr1000v(config-router)#nsf cisco
csr1000v(config-router)#nsf ietf
% OSPF: Cannot enable both Cisco NSF and IETF Graceful Restart

The IETF mode uses the Grace LSA to indicate that a router is restarting. This is an Opaque type 9 LSA, which has link-local flooding. The restarting router sends a Grace LSA to its neighbors, indicating that it is restarting and the grace period time (how long to hold the adjacency). Once the router switches over to the standby RP, database sync happens again, and the router flushes its Grace LSAs. Essentially, the Grace LSAs take place of the Restart Signal bit in the Hello, which is used in cisco mode.

ISIS NSF

The goal of NSF in ISIS is identical to OSPF - maintain neighbor adjacencies and assist in rebuilding the RIB when a RP switchover event happens on a neighbor.

Like with OSPF, Cisco created a prior NSF feature before the IETF NSF capability was published. But unlike the cisco mode in OSPF, the cisco mode in ISIS is an independent “internal” mechanism. The router statefully saves information to the standby RP so that during a switchover, the standby RP already has all adjacency and LSDB information. This is actually called NSR (nonstop routing) which we will look at in the next article. For this reason, we are only concerned with the ietf mode for ISIS right now.

ISIS uses a new TLV, called the Restart TLV (#211) and is seen in the Hello PDU. Just like with OSPF, by default, XE and XR will preform the role of “NSF-aware” (NSF helper mode) and support restart signaling.

Just like we saw in OSPF, the ISIS Hello contains the neighbors seen.

Therefore the Restart Signaling TLV is used to circumvent the issue when the standby RP becomes primary and does not have the adjacency state yet. When this happens, the router sets the Restart Request bit. The neighboring router, preforming the helper role, responds with the Restart Acknowledgment bit set in its own Hello. Next, LSPs are exchanged as normal.

BGP Graceful Restart

BGP only allows for the IETF implementation of NSF. This is called “BGP graceful restart” instead of “BGP NSF” because the command to enable the feature is graceful-restart and not nsf like we saw previously for OSPF/ISIS. The feature is the same though. Remember that GR and NSF are essentially interchangable terms.

BGP neighbors advertise their ability to support GR using a BGP capability in the Open message. This capability parameter includes the address families for which GR is supported.

In addition, BGP GR uses a end of RIB marker (EOR) which indicates that there are no more Updates to be sent, and the peer can immediately start running bestpath. The EOR marker is simply an update message with no prefixes. This speeds up convergence by explicity letting a neighbor know when it is finished sending updates.

To enable BGP GR we simply use the following command:

RP/0/0/CPU0:XRv(config)#router bgp 65000
RP/0/0/CPU0:XRv(config-bgp)#bgp graceful-restart

csr1000v(config)#router bgp 65000
csr1000v(config-router)#bgp graceful-restart 
All BGP sessions must be reset to take the new GR config

We need to restart the peering session in order for GR to be active. This is because the capability is advertised in the Open message.

We can now confirm in the CLI that the peering session is using GR:

csr1000v#show bgp neighbors | in neighbor is|Grace
BGP neighbor is 10.0.0.1,  remote AS 65000, internal link
    Graceful Restart Capability: advertised and received
  Graceful-Restart is enabled, restart-time 120 seconds, stalepath-time 360 seconds

RP/0/0/CPU0:XRv#show bgp neighbor | in "neighbor is|Grace"
Sat Oct  8 22:55:55.066 UTC
BGP neighbor is 10.0.0.2
  Graceful restart is enabled
    Graceful Restart (GR Awareness): received
    Graceful Restart capability advertised
    Graceful Restart capability received

The restart time is used as a “grace” period on the helper router. It starts once the TCP session fails. The restart timer stops once the Open message is received. The stalepath timer is how long the helper router will hold onto the original routes until they are flushed. This is longer than the restart timer because if there are a lot of routes to exchange, it may take a few extra minutes between the Open message being seen, and the peering router receiving all updates, preforming bestpath, and re-advertising the routes again.

Although the XR output doesn’t explicitly list the timers, they are the same as XE by default - 120 seconds for restart-time and 360 seconds for stalepath-time.

Now that GR is enabled on our session, we can see the EOR markers. We see an empty Update from each neighbor when the neighborship is cleared and brought up agian.

LDP GR

The concept of GR in LDP is the same as the routing protocols we’ve examined thus far. When a router preforms an SSO event, the LDP neighbor should allow for a grace period of missed Hellos, and then assist with rebuilding the LIB when the standby RP becomes active. The difference between LDP and BGP/ISIS/OSPF is that LDP GR holds stale label bindings, while the other protocols hold stale routes and adjacencies. Remeber that GR is needed on all protocols in order to not drop customer traffic during an SSO event.

Enabling LDP GR on IOS-XR resets the LDP neighborship with no warning message. On IOS-XE it gives you a warning and you must clear neighborships yourself. However, if GR is enabled on both sides, this shouldn’t result in traffic disruption, because the neighbors should treat it as a GR event.

csr1000v(config)#mpls ldp graceful-restart 
% Previously established LDP sessions may not have graceful restart protection.

We can now see a new TLV being exchanged in the LDP Initialization message, which is called FT session. FT stands for Fault Tolerant. The only flag you need to pay attention to for GR is the L flag. It is only set when a GR event happens and the peers need to relearn previous state to refresh stale bindings.

The reconnect timeout is the time the peer should hold onto the LDP session before marking it as Down. This effectively changes the hold time for the session. The field is in msec. The value is what the other neighbor uses for the local router. If a new LDP session is not re-established when this timer hits zero, the router will delete all stale LIB bindings.

The recovery time is the time the router will hold onto stale LDP bindings while the state is being refreshed between the peers. The timer is started after the new TCP session is established, and stopped once all mapping messages are exchanged again.

We can also verify that GR is operational using show commands:

csr1000v#show mpls ldp graceful-restart 

Neighbor Liveness Timer: 120 seconds
Max Recovery Time: 120 seconds
Forwarding State Holding Time: 600 seconds
Down Neighbor Database (0 records):
Graceful Restart-enabled Sessions:
  VRF default:
    Peer LDP Ident: 1.1.1.1:0, State: estab

RP/0/0/CPU0:XRv#show mpls ldp graceful-restart 
Mon Oct 10 15:07:15.685 UTC

  Forwarding State Hold timer : Not Running
  

   Neighbor ID      Up  Connect Count  Liveness Timer      Recovery Timer    
   ---------------  --  -------------  ------------------  ------------------
   2.2.2.2          Y        1                -                   -

The connect count on XR increments every time a session has to re-connect.

Conclusion

The terms NSF and GR are generally interchangable. NSF refers to the fact that a router can continue forwarding traffic in the data plane using stale FIB/LFIB entries while the control plane switches from the active RP to the standby RP. GR refers to the fact that a neighbor can allow for a grace period, keeping adjacencies and routes active while its neighbor preforms a switchover event.

In all protocols, GR/NSF needs to be negotiated and configured on both peers.

As we will see in the next article, NSR (nonstop routing) is a feature which allows a router to preform an SSO event without needing “help” from its neighbor. This allows SSO to work even if a peer does not support GR/NSR.

The (Unofficial) CCNP-SP Study Guide