BFD
Last updated
Last updated
BFD (Bidirectional forwarding detection) allows you to quickly detect link failures at layer 2 without lowering routing protocol hello timers.
Imagine you have a customer connected at layer 2, and you’re running eBGP with them. There are potentially multiple layer 1 devices in the path, making it possible for the link to go down somewhere in the path while the link still appears up at either end.
In the image above, if the fiber path connecting PE1 to the NID goes down, CE1 will still see the link as up with the NID. CE1 will wait 180 seconds before failing over to a redundant but less-prefered BGP path.
We could lower the routing protocol hello/keepalive timers but we will never be able to achieve sub-second failover times. Most routing protocols allow a minimum hello interval of one full second. Further, lowering timers increases CPU overhead, because every hello/keepalive is punted to the CPU for processing.
BFD solves this problem. BFD runs as its own standalone application and routing protocols register with it in order to use BFD for failure detection. This allows a single BFD session to be used by multiple routing protocols at the same time on a given interface.
BFD uses control packets and echo packets. The control packets are used to control the session and are sent at a slower interval - anywhere from 1-30 seconds. The echo packets are used for fast detection and can be sent in as little as 50msec on XE or as low as 3msec on XR. The echo packets are looped through the data plane without the CPU needing to process the packet. (You will see how this works in detail in this article). The client routing protocols such as BGP can continue to use normal 60 second keepalives for the session. If the BFD session with the neighbor goes down, BFD alerts BGP and BGP will bring down the neighbor. This effectively allows you tell the routing protocol that the interface is down at layer 2 even though layer 1 is still up.
BFD is an IETF standard and can interoperate on equipment from different vendors. BFD has two modes, asynchronous mode and demand mode. Only asynchronous mode is supported on Cisco equipment. The demand mode does not use periodic Control packets, instead it uses polling only when needed. We won’t pay attention to demand mode in this article.
BFD is a very powerful solution to providing quick failover times for routing protocols such as OSPF, ISIS and BGP in situations where two routers connect via multiple pieces of layer 1 equipment.
We’ll use the following topology to test BFD between IOS-XE devices and between XE and XR. You will need to use an XRv9K, not an XRv.
The basic configuration for BFD is fairly straightforward. You define the echo timers on each interface. BFD neighbors will not form until a client routing protocol registers with BFD, such as OSPF, BGP or ISIS.
We’ll use OSPF as our client protocol. There are two ways to activate BFD on an OSPF interface. On csr1000v1, I activate BFD for OSPF on the interface, and on csr1000v2 I activate BFD on all interfaces under the OSPF config.
You must include the outgoing interface in the static route
In all above configs, BFD will only work if the BFD interval is defined under the interface. You must both define the BFD interval on the interface and activate BFD on the routing protocol.
Once a protocol registers with BFD, we can confirm that BFD is up.
The above gives you a basic understanding of BFD and how to use it. Let’s get into the details of how BFD really works.
The BFD timers in IOS XE are a little hard to understand. The first value is the min_tx value and the second value is the min_rx value. The local min_tx is compared against the min_rx received from the peer, and the slowest value is used for locally sent echos. The other router preforms the same comparison of its own min_tx compared against its peer’s min_rx to determine how quickly to send its own echos.
For example, let’s say we have this configured:
csr1000v1 will use a value of 500msec to send BFD echos, and csr1000v2 will use a value of 550. This is because csr1000v1 compared 450 against the min_rx value of 500 from it’s peer, and chose the slower value. csr1000v2 compared its own min_tx value of 500 and compared it against csr1000v1’s min_rx of 550, and chose to send Echos at 550msec.
Only the min_rx value is signaled to the peer in the Control message. This allows a router to control the speed at which it will receive echos and prevents another router from being able to overwhlem it.
We can confirm this on the router by looking at the interface configured values, and seeing the actual echo interval that was chosen.
Here we see the min_tx and min_rx configured on interface Gi2, and the neighbor’s min_rx value. The min echo interval is the min_rx as seen on the Control packet from the neighbor. So we can determine that csr1000v1 will choose an echo interval of 500msec, because it compares the received value of 500msec to its local min_tx of 450msec.
What value will we see on csr1000v2?
Here we confirm that Gi2 is configured for min_tx of 500msec and min_rx of 500msec, and that the peer is asking for a min_rx of 550msec. So csr1000v2 must transmit echos at an interval of 550msec.
While these echo packets are going on, control packets happen simultaneously at an interval of one second by default.
You can change the control timer interval with the following command in IOS-XE
This allows for a configurable range of 1-30seconds. This is completely independent of the echo interval.
Just like echo packets, a BFD session will use the slowest interval value configured on each peer. The difference is that you don’t specify two separate min_rx and min_tx values, you only configure one single value. The value you configure is effectively both the min_tx and min_rx for the control packets. This means that the control packets cannot have a different transmit rate on each peer. Since the min_tx and min_rx is always identical on each peer, both peers will end up using whatever the higher value is.
Let’s change csr1000v1 to set a value of 3 seconds.
What is a bit frustrating is that the slow timers and echo timers are not clearly distinguished. Highlighted above are all the references to the slow timers.
The value on the second-to-last line in the show output above is the interval from the peer learned in the Control packet. In fact that entire bottom section of the show output is essentially a pcap of the Control packet received from the peer.
All values in this pcap are seen in the show command above.
csr1000v1 has a min_tx and min_rx of 3000, and csr1000v2 has a min_tx and min_rx of 1000, csr1000v1 compares its own min_tx (3000) to the min_rx of csr1000v2 (1000) and uses 3000msec. Likewise, csr1000v2 compares its own min_tx (1000) to the min_rx of csr1000v1 (3000) and uses 3000msec. This is why the control packet interval will end up being the same on each peer.
It is important to note that if either 3x echo packets or 3x control packets go missing, the BFD sesssion is brought down. This means that if for some reason, echo packets continue working, but control packets do not, the session will not stay up. Also note that the echo packet multipler is configurable from 3-50, but the control packet multipler appears to be fixed at 3.
Do you notice anything unique about the Echo packets?
The source and destination IP is the same on each packet! Why is this?
This allows the peer to loop the packet through hardware. If the destination IP was the router itself, the packet would be punted to the CPU. Instead, a router sets the source and destination IP to itself, so that when the interface receives the packet, it will simply do a CEF lookup on the destination IP and forward the frame back to the router.
Why do we see each packet twice?
This is because the first instance on the wire is when one router originates the packet. The second instance is that packet being echoed right back to the original router. The only difference between the two packets it the MAC addresses.
Echo packets use a UDP destination port of 3785, and control packets use a UDP destination port of 3784.
You have the option to use only control packets instead of BFD echos. I’m not aware of a good reason to do this, as the echos provide for hardware offload (they are processed only in hardware). Control packets must be processed in the CPU, so typically you don’t want to disable the use of echos.
If we disable echos on one side, the session will no longer use echo packets.
The packets still flow at the same rate as before, but they are control packets instead of echo packets. Also notice that csr1000v1 sets a min echo interval (min_rx) of 0 ms. This signals that it does not want to use echos. The min_tx and min_rx that we configured earlier on the interface (450ms and 550ms) is now used for the control packet intervals.
BFD configuration on IOS-XR is a little more intuitive in my opinion. The configuration follows the theme of IOS-XR in general, in which all commands are present under the routing protocol config stanza instead of the interface.
To lab BFD in CML you will need an XRv9000. You can configure BFD on XRv and commit successfully, but you won’t be able to form BFD neighborships.
The bfd commands can be configured under the process, area, or interface
BFD can only be configured under the interface
Notice that in IOS-XR there is no min_tx and min_rx for the echo interval. It appears that the minimum-interval is used for min_tx. In my lab, min_rx is set to 1ms on BFD control packets. I’m not sure if this is expected or a bug with the virtualized router. Perhaps the router accepts a value that low because it is capable of handling traffic that quickly. The echos are switched in the data plane after all.
Control packets from XRv9k show a min echo of 1ms (last line)
We can confirm the BFD min_tx interval with the following show command:
The following command shows us the actual echo transmit interval that is used. csr1000v1 still has a min_rx of 550 configured, so that is used because it is slower than 300msec.
The Async tab is the interval for control packets. By default on IOS-XR it is 2 seconds.
We can also get detailed output about the session. This can be useful if you are troubleshooting BFD flaps with a customer. You can see how long the session has been up and how many times it has flapped.
The detailed output is a little easier to read than XE detailed neighbor output. We can clearly see the received and transmitted echo parameters, and see the negotiated echo tx interval of 550.
What is also nice about IOS-XR is that interfaces appear to have default BFD intervals. For example, the following configuration is valid:
The only BFD related command here was bfd fast-detect. Once the neighborship comes up, we can see that the interface has a default BFD echo interval of 333msec for BGP:
If we remove BGP and do the same for ISIS, we see a default echo tx interval of 150msec:
150msec is also used for OSPF (not demonstrated).
To test BFD on a bundle interface, I connected two XRv9Ks on Gi0/0/0/0 and Gi0/0/0/1.
Here is the configuration:
Configuring BFD on bundle interfaces is very similar to physical interfaces. The interface is activated for BFD under the routing protocol configuration.
The only difference is that you must include all line cards of the members of the bundle under the bfd config stanza. The XRv9Ks only have a single virtual card on 0/0, so this is included.
Interestingly, echos are not used on the bundle interface. I believe that echo mode might not be supported for a bundle interface.
The async interval is the familiar 150msec with a multiple of 3, which is the default we saw in OSPF on a physical interface.
A pcap confirms that only BFD Control packets are used:
BFD allows for link failure detection at the layer 2 level, which is extremely important when there are one or more layer 1 devices in the path between two routers. Routing protocols register themselves with BFD in order for a BFD neighborship session to form.
BFD uses control packets, and optionally echo packets. Echo packets are looped in hardware and do not hit the CPU.
BFD configuration on IOS-XE ,consists of two parts:
Set the BFD interval with bfd interval <min_tx> min_rx <min_rx> multipler <value>
Activate BFD for the routing protocol
BFD configuration on IOS-XR requires no bfd config on the physical interface. Instead you activate BFD under the routing protocol configuration stanza.
The BFD echo transit interval is a negotiated value. A router will use the highest value between its local min_tx value and the received min_rx value. It is also important to understand that echos are unidirectional. A router will send its own echos at the locally negotiated value (the lesser of its own tx interval and the min rx interval from the peer). Likewise the other peer does the same, and locally negotiates its own transmitted echos which can be a completely different rate.
Finally, if a router wishes to only use control packets, it sets the min_rx value to zero.
Short BFD blog: https://journey2theccie.wordpress.com/2020/05/21/bidirectional-forwarding-detection/
BFD on IOS-XE: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bfd/configuration/xe-16-6/irb-xe-16-6-book/irb-bi-fwd-det.html
Detailed BFD support article for IOS-XR: https://community.cisco.com/t5/service-providers-knowledge-base/bfd-support-on-cisco-asr9000/ta-p/3153191
