BFD

BFD (Bidirectional forwarding detection) allows you to quickly detect link failures at layer 2 without lowering routing protocol hello timers.

Imagine you have a customer connected at layer 2, and you’re running eBGP with them. There are potentially multiple layer 1 devices in the path, making it possible for the link to go down somewhere in the path while the link still appears up at either end.

In the image above, if the fiber path connecting PE1 to the NID goes down, CE1 will still see the link as up with the NID. CE1 will wait 180 seconds before failing over to a redundant but less-prefered BGP path.

We could lower the routing protocol hello/keepalive timers but we will never be able to achieve sub-second failover times. Most routing protocols allow a minimum hello interval of one full second. Further, lowering timers increases CPU overhead, because every hello/keepalive is punted to the CPU for processing.

BFD solves this problem. BFD runs as its own standalone application and routing protocols register with it in order to use BFD for failure detection. This allows a single BFD session to be used by multiple routing protocols at the same time on a given interface.

BFD uses control packets and echo packets. The control packets are used to control the session and are sent at a slower interval - anywhere from 1-30 seconds. The echo packets are used for fast detection and can be sent in as little as 50msec on XE or as low as 3msec on XR. The echo packets are looped through the data plane without the CPU needing to process the packet. (You will see how this works in detail in this article). The client routing protocols such as BGP can continue to use normal 60 second keepalives for the session. If the BFD session with the neighbor goes down, BFD alerts BGP and BGP will bring down the neighbor. This effectively allows you tell the routing protocol that the interface is down at layer 2 even though layer 1 is still up.

BFD is an IETF standard and can interoperate on equipment from different vendors. BFD has two modes, asynchronous mode and demand mode. Only asynchronous mode is supported on Cisco equipment. The demand mode does not use periodic Control packets, instead it uses polling only when needed. We won’t pay attention to demand mode in this article.

BFD is a very powerful solution to providing quick failover times for routing protocols such as OSPF, ISIS and BGP in situations where two routers connect via multiple pieces of layer 1 equipment.

Lab

We’ll use the following topology to test BFD between IOS-XE devices and between XE and XR. You will need to use an XRv9K, not an XRv.

Basic Configuration

The basic configuration for BFD is fairly straightforward. You define the echo timers on each interface. BFD neighbors will not form until a client routing protocol registers with BFD, such as OSPF, BGP or ISIS.

#csr1000v1
int Gi2
 bfd interval 500 min_rx 500 multiplier 3

#csr1000v2
int Gi2
 bfd interval 500 min_rx 500 multiplier 3

We’ll use OSPF as our client protocol. There are two ways to activate BFD on an OSPF interface. On csr1000v1, I activate BFD for OSPF on the interface, and on csr1000v2 I activate BFD on all interfaces under the OSPF config.

#csr1000v1
router ospf 1 
 network 10.1.1.0 0.0.0.3 area 0
!
int Gi2
 ip ospf bfd

#csr1000v2
router ospf 1
 network 10.1.1.0 0.0.0.3 area 0
 bfd all-interfaces

ISIS configuration (for reference)

router isis
 bfd all-interfaces
!
OR
int Gi2
 isis bfd

BGP configuration (for reference)

router bgp 65000
 neighbor 1.1.1.1 fall-over bfd

Static route configuration (for reference)

csr1000v1(config)#ip route static bfd gi2 10.1.1.2
csr1000v1(config)#ip route 0.0.0.0 0.0.0.0 gi2 10.1.1.2
*Oct 11 21:25:01.951: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 10.1.1.2 proc:IPv4 Static, idb:GigabitEthernet2 handle:1 act

You must include the outgoing interface in the static route

In all above configs, BFD will only work if the BFD interval is defined under the interface. You must both define the BFD interval on the interface and activate BFD on the routing protocol.

Once a protocol registers with BFD, we can confirm that BFD is up.

csr1000v1#show bfd neighbors 

IPv4 Sessions
NeighAddr                              LD/RD         RH/RS     State     Int
10.1.1.2                             4097/4097       Up        Up        Gi2

The above gives you a basic understanding of BFD and how to use it. Let’s get into the details of how BFD really works.

BFD Echo Timers

The BFD timers in IOS XE are a little hard to understand. The first value is the min_tx value and the second value is the min_rx value. The local min_tx is compared against the min_rx received from the peer, and the slowest value is used for locally sent echos. The other router preforms the same comparison of its own min_tx compared against its peer’s min_rx to determine how quickly to send its own echos.

For example, let’s say we have this configured:

csr1000v1(config-if)#do sho run int gi2 | in min_rx
 bfd interval 450 min_rx 550 multiplier 3

csr1000v2(config)#do sho run int gi2 | in min_rx
 bfd interval 500 min_rx 500 multiplier 3

csr1000v1 will use a value of 500msec to send BFD echos, and csr1000v2 will use a value of 550. This is because csr1000v1 compared 450 against the min_rx value of 500 from it’s peer, and chose the slower value. csr1000v2 compared its own min_tx value of 500 and compared it against csr1000v1’s min_rx of 550, and chose to send Echos at 550msec.

Only the min_rx value is signaled to the peer in the Control message. This allows a router to control the speed at which it will receive echos and prevents another router from being able to overwhlem it.

We can confirm this on the router by looking at the interface configured values, and seeing the actual echo interval that was chosen.

csr1000v1#show bfd int | sec Ethernet2
Interface Name: GigabitEthernet2
 Interface Number: 8
 Configured bfd interval using bfd interval command
 Min Tx Interval: 450000, Min Rx Interval: 550000, Multiplier: 3

csr1000v1#show bfd neigh detail | in Min Echo
             Min Echo interval: 500000

Here we see the min_tx and min_rx configured on interface Gi2, and the neighbor’s min_rx value. The min echo interval is the min_rx as seen on the Control packet from the neighbor. So we can determine that csr1000v1 will choose an echo interval of 500msec, because it compares the received value of 500msec to its local min_tx of 450msec.

What value will we see on csr1000v2?

csr1000v2#show bfd int | sec Ethernet2
Interface Name: GigabitEthernet2
 Interface Number: 8
 Configured bfd interval using bfd interval command
 Min Tx Interval: 500000, Min Rx Interval: 500000, Multiplier: 3
csr1000v2#show bfd neighbors details | in Min Echo
             Min Echo interval: 550000

Here we confirm that Gi2 is configured for min_tx of 500msec and min_rx of 500msec, and that the peer is asking for a min_rx of 550msec. So csr1000v2 must transmit echos at an interval of 550msec.

BFD Control timers

While these echo packets are going on, control packets happen simultaneously at an interval of one second by default.

You can change the control timer interval with the following command in IOS-XE

csr1000v1(config)#bfd slow-timers ?
  <1000-30000>  Value in ms to use for slow timers

This allows for a configurable range of 1-30seconds. This is completely independent of the echo interval.

Just like echo packets, a BFD session will use the slowest interval value configured on each peer. The difference is that you don’t specify two separate min_rx and min_tx values, you only configure one single value. The value you configure is effectively both the min_tx and min_rx for the control packets. This means that the control packets cannot have a different transmit rate on each peer. Since the min_tx and min_rx is always identical on each peer, both peers will end up using whatever the higher value is.

Let’s change csr1000v1 to set a value of 3 seconds.

csr1000v1(config)#bfd slow-timers 3000

csr1000v1#show bfd neighbors detail

IPv4 Sessions
NeighAddr                              LD/RD         RH/RS     State     Int
10.1.1.2                             4097/4097       Up        Up        Gi2
Session state is UP and using echo function with 500 ms interval.
Session Host: Software
OurAddr: 10.1.1.1       
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0


Holddown (hits): 0(0), Hello (hits): 3000(553)


Echo Rx Count: 3055, Echo Rx Interval (ms) min/max/avg: 371/797/440 last: 58 ms ago
Echo Tx Count: 3055, Echo Tx Interval (ms) min/max/avg: 378/798/440 last: 60 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF 
Uptime: 00:22:25
Last packet: Version: 1                  - Diagnostic: 0
             State bit: Up               - Demand bit: 0
             Poll bit: 0                 - Final bit: 0
             C bit: 0                                   
             Multiplier: 3               - Length: 24
             My Discr.: 4097             - Your Discr.: 4097
             
             Min Echo interval: 500000

What is a bit frustrating is that the slow timers and echo timers are not clearly distinguished. Highlighted above are all the references to the slow timers.

The value on the second-to-last line in the show output above is the interval from the peer learned in the Control packet. In fact that entire bottom section of the show output is essentially a pcap of the Control packet received from the peer.

All values in this pcap are seen in the show command above.

csr1000v1 has a min_tx and min_rx of 3000, and csr1000v2 has a min_tx and min_rx of 1000, csr1000v1 compares its own min_tx (3000) to the min_rx of csr1000v2 (1000) and uses 3000msec. Likewise, csr1000v2 compares its own min_tx (1000) to the min_rx of csr1000v1 (3000) and uses 3000msec. This is why the control packet interval will end up being the same on each peer.

It is important to note that if either 3x echo packets or 3x control packets go missing, the BFD sesssion is brought down. This means that if for some reason, echo packets continue working, but control packets do not, the session will not stay up. Also note that the echo packet multipler is configurable from 3-50, but the control packet multipler appears to be fixed at 3.

A closer look at Echo packets

Do you notice anything unique about the Echo packets?

The source and destination IP is the same on each packet! Why is this?

This allows the peer to loop the packet through hardware. If the destination IP was the router itself, the packet would be punted to the CPU. Instead, a router sets the source and destination IP to itself, so that when the interface receives the packet, it will simply do a CEF lookup on the destination IP and forward the frame back to the router.

Why do we see each packet twice?

This is because the first instance on the wire is when one router originates the packet. The second instance is that packet being echoed right back to the original router. The only difference between the two packets it the MAC addresses.

Echo packets use a UDP destination port of 3785, and control packets use a UDP destination port of 3784.

Turning off BFD Echos

You have the option to use only control packets instead of BFD echos. I’m not aware of a good reason to do this, as the echos provide for hardware offload (they are processed only in hardware). Control packets must be processed in the CPU, so typically you don’t want to disable the use of echos.

If we disable echos on one side, the session will no longer use echo packets.

csr1000v1(config)#int gi2
csr1000v1(config-if)#no bfd echo

The packets still flow at the same rate as before, but they are control packets instead of echo packets. Also notice that csr1000v1 sets a min echo interval (min_rx) of 0 ms. This signals that it does not want to use echos. The min_tx and min_rx that we configured earlier on the interface (450ms and 550ms) is now used for the control packet intervals.

BFD on IOS-XR

BFD configuration on IOS-XR is a little more intuitive in my opinion. The configuration follows the theme of IOS-XR in general, in which all commands are present under the routing protocol config stanza instead of the interface.

To lab BFD in CML you will need an XRv9000. You can configure BFD on XRv and commit successfully, but you won’t be able to form BFD neighborships.

OSPF configuration

router ospf 1
 bfd fast-detect
 bfd minimum-interval <50-30000 in msec>
 bfd multiplier <3-50>

The bfd commands can be configured under the process, area, or interface

ISIS configuration

router isis 1
 int Gi0/0/0/0
  bfd fast-detect ipv4|ipv6
  bfd minimum-interval <3-30000 in msec>
  bfd multiplier <2-50>

BFD can only be configured under the interface

BGP configuration

router bgp 65000
 bfd minimum-interval
 bfd multiplier
 !
 neighbor 1.1.1.1
  bfd fast-detect
  bfd minimum-interval <3-30000 in msec>
  bfd multiplier <2-16>

Static route configuration

router static
 address-family ipv4 unicast
  0.0.0.0/0 10.0.0.2 bfd fast-detect minimum-interval 500 multiplier 3

Notice that in IOS-XR there is no min_tx and min_rx for the echo interval. It appears that the minimum-interval is used for min_tx. In my lab, min_rx is set to 1ms on BFD control packets. I’m not sure if this is expected or a bug with the virtualized router. Perhaps the router accepts a value that low because it is capable of handling traffic that quickly. The echos are switched in the data plane after all.

RP/0/RP0/CPU0:XRv9k#show run router ospf
Tue Oct 11 19:12:21.632 UTC
router ospf 1
 bfd fast-detect
 area 0
  interface GigabitEthernet0/0/0/0
   bfd minimum-interval 300
   network point-to-point

Control packets from XRv9k show a min echo of 1ms (last line)

We can confirm the BFD min_tx interval with the following show command:

RP/0/RP0/CPU0:XRv9k#show ospf int gi0/0/0/0 | in BFD
Tue Oct 11 19:13:03.135 UTC
  BFD enabled, BFD interval 300 msec, BFD multiplier 3, Mode: Default

The following command shows us the actual echo transmit interval that is used. csr1000v1 still has a min_rx of 550 configured, so that is used because it is slower than 300msec.

RP/0/RP0/CPU0:XRv9k#show bfd session                
Tue Oct 11 19:13:53.606 UTC
Interface           Dest Addr           Local det time(int*mult)      State     
                                    Echo             Async   H/W   NPU     
------------------- --------------- ---------------- ---------------- ----------
Gi0/0/0/0           10.0.0.2        1650ms(550ms*3)  6s(2s*3)         UP        
                                                             No    n/a

The Async tab is the interval for control packets. By default on IOS-XR it is 2 seconds.

We can also get detailed output about the session. This can be useful if you are troubleshooting BFD flaps with a customer. You can see how long the session has been up and how many times it has flapped.

RP/0/RP0/CPU0:XRv9k#show bfd session interface Gi0/0/0/0 detail
Tue Oct 11 19:15:03.724 UTC
I/f: GigabitEthernet0/0/0/0, Location: 0/0/CPU0
Dest: 10.0.0.2
Src: 10.0.0.1
 
 Session type: PR/V4/SH

 Version: 1, desired tx interval: 1 s, required rx interval: 1 s
 Required echo rx interval: 550 ms, multiplier: 3, diag: None
 My discr: 4097, your discr: 2148532226, state UP, D/F/P/C/A: 0/0/0/0/0

 Version: 1, desired tx interval: 2 s, required rx interval: 2 s
 Required echo rx interval: 1 ms, multiplier: 3, diag: None
 My discr: 2148532226, your discr: 4097, state UP, D/F/P/C/A: 0/0/0/1/0

 
 
 
 Echo detection time: 1650 ms(550 ms*3), async detection time: 6 s(2 s*3)
Local Stats:
 Intervals between async packets:
   Tx: Number of intervals=100, min=636 ms, max=2 s, avg=1817 ms
       Last packet transmitted 1552 ms ago
   Rx: Number of intervals=100, min=3 ms, max=2 s, avg=1732 ms
       Last packet received 637 ms ago
 Intervals between echo packets:
   Tx: Number of intervals=100, min=550 ms, max=552 ms, avg=550 ms
       Last packet transmitted 230 ms ago
   Rx: Number of intervals=100, min=545 ms, max=555 ms, avg=550 ms
       Last packet received 229 ms ago
 Latency of echo packets (time between tx and rx):
   Number of packets: 100, min=1060 us, max=6715 us, avg=1827 us
Session owner information:
                            Desired               Adjusted
  Client               Interval   Multiplier Interval   Multiplier
  -------------------- --------------------- ---------------------
  ospf-1               300 ms     3          2 s        3

The detailed output is a little easier to read than XE detailed neighbor output. We can clearly see the received and transmitted echo parameters, and see the negotiated echo tx interval of 550.

What is also nice about IOS-XR is that interfaces appear to have default BFD intervals. For example, the following configuration is valid:

no router ospf 1
!
router bgp 65000
 bgp router-id 10.0.0.1
 address-family ipv4 unicast
 neighbor 10.0.0.2
  remote-as 65000
  bfd fast-detect
  address-family ipv4 unicast

The only BFD related command here was bfd fast-detect. Once the neighborship comes up, we can see that the interface has a default BFD echo interval of 333msec for BGP:

RP/0/RP0/CPU0:XRv9k#show bfd session detail | in echo
Tue Oct 11 19:20:15.885 UTC
 Required echo rx interval: 550 ms, multiplier: 3, diag: None
 Required echo rx interval: 1 ms, multiplier: 3, diag: None
 Desired echo tx interval: 333 ms, local negotiated echo tx interval: 550 ms
 Intervals between echo packets:
 Latency of echo packets (time between tx and rx):

If we remove BGP and do the same for ISIS, we see a default echo tx interval of 150msec:

RP/0/RP0/CPU0:XRv9k#show run router isis
Tue Oct 11 19:22:27.136 UTC
router isis 1
 net 49.0001.0000.0000.0001.00
 interface GigabitEthernet0/0/0/0
  bfd fast-detect ipv4
  point-to-point
  address-family ipv4 unicast

RP/0/RP0/CPU0:XRv9k#show bfd session detail | in echo
Tue Oct 11 19:21:59.554 UTC
 Required echo rx interval: 550 ms, multiplier: 3, diag: None
 Required echo rx interval: 1 ms, multiplier: 3, diag: None
 Desired echo tx interval: 150 ms, local negotiated echo tx interval: 550 ms
 Intervals between echo packets:
 Latency of echo packets (time between tx and rx):

150msec is also used for OSPF (not demonstrated).

BFD on Bundles in IOS-XR

To test BFD on a bundle interface, I connected two XRv9Ks on Gi0/0/0/0 and Gi0/0/0/1.

Here is the configuration:

hostname XR1
!
bfd
 multipath include location 0/0/CPU0
!
interface Bundle-Ether1
 ipv4 address 10.0.0.1 255.255.255.252
!
interface GigabitEthernet0/0/0/0
 bundle id 1 mode on
!
interface GigabitEthernet0/0/0/1
 bundle id 1 mode on
!
router ospf 1
 area 0
  interface Bundle-Ether1
   bfd fast-detect
   network point-to-point

hostname XR2
!
bfd
 multipath include location 0/0/CPU0
!
interface Bundle-Ether1
 ipv4 address 10.0.0.2 255.255.255.252
!
interface GigabitEthernet0/0/0/0
 bundle id 1 mode on
!
interface GigabitEthernet0/0/0/1
 bundle id 1 mode on
!
router ospf 1
 area 0
  interface Bundle-Ether1
   bfd fast-detect
   network point-to-point

Configuring BFD on bundle interfaces is very similar to physical interfaces. The interface is activated for BFD under the routing protocol configuration.

The only difference is that you must include all line cards of the members of the bundle under the bfd config stanza. The XRv9Ks only have a single virtual card on 0/0, so this is included.

bfd
 multipath include location 0/0/CPU0

Interestingly, echos are not used on the bundle interface. I believe that echo mode might not be supported for a bundle interface.

RP/0/RP0/CPU0:XR2#show bfd session
Tue Oct 11 20:49:34.759 UTC
Interface           Dest Addr           Local det time(int*mult)      State     
                                    Echo             Async   H/W   NPU     
------------------- --------------- ---------------- ---------------- ----------
BE1                 10.0.0.1        0s(0s*0)         450ms(150ms*3)   UP        
                                                             No    n/a

The async interval is the familiar 150msec with a multiple of 3, which is the default we saw in OSPF on a physical interface.

A pcap confirms that only BFD Control packets are used:

Conclusion

BFD allows for link failure detection at the layer 2 level, which is extremely important when there are one or more layer 1 devices in the path between two routers. Routing protocols register themselves with BFD in order for a BFD neighborship session to form.

BFD uses control packets, and optionally echo packets. Echo packets are looped in hardware and do not hit the CPU.

BFD configuration on IOS-XE ,consists of two parts:

Set the BFD interval with bfd interval <min_tx> min_rx <min_rx> multipler <value>
Activate BFD for the routing protocol

BFD configuration on IOS-XR requires no bfd config on the physical interface. Instead you activate BFD under the routing protocol configuration stanza.

The BFD echo transit interval is a negotiated value. A router will use the highest value between its local min_tx value and the received min_rx value. It is also important to understand that echos are unidirectional. A router will send its own echos at the locally negotiated value (the lesser of its own tx interval and the min rx interval from the peer). Likewise the other peer does the same, and locally negotiates its own transmitted echos which can be a completely different rate.

Finally, if a router wishes to only use control packets, it sets the min_rx value to zero.