802.3ah (Ethernet OAM)

Ethernet OAM is a protocol used for monitoring and troubleshooting Ethernet on a last-mile link. Ethernet OAM reminds me of BFD but for layer 2 Ethernet. Both protocols are used to detect failures at the layer 2 or layer 3 level. However, Ethernet OAM has many other features as well which we will see in this article. Ethernet OAM is sometimes called “Link OAM” or “Ethernet for the First Mile OAM.”

Ethernet OAM was invented in order to use the same OAM tools that traditional WAN technologies like ATM and SONET had, for Ethernet. Ethernet was invented as a LAN protocol and did not originally have the same requirements that we impose on Ethernet today as a WAN technology.

Ethernet OAM is only supported on full duplex, point-to-point links. Ethernet OAM PDUs use the slow protocol destination MAC of 0180.c200.0002. You may recognize this because it is the same MAC that LACP uses! Slow protocols cannot exceed a maximum tramission of 10 frames per second. The idea behind “slow protocols” is that they have very low impact on overall bandwidth of the link. The ethertype for slow protocols is 8809. LACP is subtype 1 and Ethernet OAM is suptype 3.

You typically run Ethernet OAM on a point-to-point link between a PE and the demarcation device at the customer premises. The main features when using Ethernet OAM are: link monitoring (detecting errors and acting on them based on a threshold), remote link fault detection (notifying the peer of faults), and the ability to turn the remote partner into a loopback. We can easily lab Ethernet-OAM using CSR1000v. Ethernet OAM is not supported on XRv or XR9kv in my testing.

Lab

Throughout this lab we will simply use two CSR1000v routers called CE1 and PE1 connected back-to-back on Gi1. We will also have CE2, which is the Z end of an E-Line service. We will only use CE2 at the very end of the lab.

To configure Ethernet OAM you simply use the following command under the interface:

#CE1
int Gi1
 ethernet oam

#PE1
int Gi1
 ethernet oam

I will also put CE1 in passive mode. By default it is in active mode. Active/passive mode works just like in LACP. The active side tries to initiate a session by actively sending OAMPDUs, and the passive side only sends PDUs in response. Two devices in passive mode will not become OAM peers.

CE1(config-if)#ethernet oam mode ?
  active   Active mode
  passive  Passive mode

CE1(config-if)#ethernet oam mode passive

Discovery

  1. PE1 was configured first, and sends OAM PDUs describing its capabilities. We also see here that Remote Evaluating and Remote Stable are both False because a remote partner has not been seen yet.

  2. In the next frame, CE1 has seen PE1 so it sets Local Stable to True and Remove Evaluating to False. This is because it does not know whether PE1 has seen itself yet. It waits to receive a PDU with Local Stable from PE1. It also includes the Remote Information TLV, which is the same information in PE1’s Local Information TLV.

  3. PE1 now sets Local Stable and Remote Stable, and also includes the Remote Information TLV:

  4. CE1 can now set Remote Stable:

In the CLI we see the following syslog message on each router:

*Oct 31 13:46:21.719: %ETHERNET_OAM-6-ENTER_SESSION: The client on interface Gi1 has entered the OAM session.

Using the following show command we can see some of the capabilities that we see in the pcaps above:

CE1#show ethernet oam discovery 
GigabitEthernet1
Local client
------------
  Administrative configurations:
    Mode:              passive
    Unidirection:      not supported
    Link monitor:      supported (on)
    Remote loopback:   not supported
    MIB retrieval:     not supported
    Mtu size:          1500

  Operational status:
    Port status:       operational
    Loopback status:   no loopback
    PDU revision:      0

Remote client
-------------
  MAC address: 5254.000f.d2cf
  Vendor(oui): 00000C(cisco)

  Administrative configurations:
    PDU revision:      0
    Mode:              active
    Unidirection:      not supported
    Link monitor:      supported
    Remote loopback:   not supported
    MIB retrieval:     not supported
    Mtu size:          1500

  • This command shows the MTU of the remote partner which can be handy.

The only capability supported right now (and by default) is link monitor. This is also seen the output of show ethernet oam summary.

CE1#show ethernet oam summary 
Symbols:          * - Master Loopback State,  # - Slave Loopback State
                  & - Error Block State
Capability codes: L - Link Monitor,  R - Remote Loopback
                  U - Unidirection,  V - Variable Retrieval

  Local                       Remote
Interface       MAC Address    OUI    Mode    Capability

  Gi1           5254.000f.d2cf 00000C active   L

If we enable support for remote loopback on PE1, this is seen as a capability of the partner on CE1. The show ethernet oam summary output is information about the neighbor.

PE1(config)#int Gi1
PE1(config-if)#ethernet oam remote-loopback supported

CE1#show ethernet oam summary 
Symbols:          * - Master Loopback State,  # - Slave Loopback State
                  & - Error Block State
Capability codes: L - Link Monitor,  R - Remote Loopback
                  U - Unidirection,  V - Variable Retrieval

  Local                       Remote
Interface       MAC Address    OUI    Mode    Capability

  Gi1           5254.000f.d2cf 00000C active   L R

Using the show ethernet oam status command we can see timeout values for various parameters configured locally.

CE1#show ethernet oam status 
GigabitEthernet1
General
-------
  Admin state:           enabled
  Mode:                  passive
  PDU max rate:          10 packets per second
  PDU min rate:          1 packet per 1000 ms
  Link timeout:          5000 ms
  High threshold action: no action
  Link fault action:     no action
  Dying gasp action:     no action
  Critical event action: no action

Link Monitoring
---------------
  Status: supported (on)

  Symbol Period Error
    Window:              100 x 1048576 symbols
    Low threshold:       1 error symbol(s)
    High threshold:      none

  Frame Error
    Window:              10 x 100 milliseconds
    Low threshold:       1 error frame(s)
    High threshold:      none

  Frame Period Error
    Window:              1000 x 10000 frames
    Low threshold:       1 error frame(s)
    High threshold:      none

  Frame Seconds Error
    Window:              100 x 100 milliseconds
    Low threshold:       1 error second(s)
    High threshold:      none

  Receive-Frame CRC Error
    Window:              10 x 100 milliseconds
    Low threshold:       10 error frame(s)
    High threshold:      none

  Transmit-Frame CRC Error: Not Supported

By default OAMPDUs are sent once per second. This is the min rate you see above and is the lowest this timer can go.

CE1(config-if)#ethernet oam min-rate ?
  <1-10>  Set the rate in number of seconds

The min-rate is used for the OAMPDU interval under normal operations. The max-rate is only used for PDU flooding during critical events. This rate can go no quicker than 10 frames per second (once per 100ms), and the slowest it can go is 1 frame per second.

CE1(config-if)#ethernet oam max-rate ?
  <1-10>  Set the rate in number of packets per second

So by default, the OAMPDU intervals are at the quickest interval you can configure. The timeout value is the only setting that you can make more aggresive. By default it is 5 seconds and you can set it as low as 2 seconds.

CE1(config-if)#ethernet oam timeout ?
  <2-30>  Set timeout value (2-30) in number of seconds.

OAM Features

Link Monitoring

Link monitoring is used to monitor the quality of the link. If certain errors are detected, and the “low” threshold is crossed, the device sends an Event Notification PDU to the OAM peer. If the “high” threshold is crossed, you can errdisable the interface. By default, there is no high threshold setting. You can configure the various settings under ethernet oam link-monitor. By default link-monitor is already on. show ethernet oam status shows the configured parameters for each setting.

CE1(config-if)#ethernet oam link-monitor ?               
  frame           Errored frames in a period measured by time
  frame-period    Errored frames in a period measured by number of frames
  frame-seconds   Errored frame seconds in a period
  high-threshold  Additional high threshold configuration
  on              Enable link monitoring on the interface
  receive-crc     Received frames with CRC errors in a period
  supported       Support link monitoring on the interface
  symbol-period   Errored symbols in a period

Remote Failure Indication (RFI)

Ethernet OAM provides a way to indicate to a peer that local faults have occured. These faults include a link fault or a critical event (which is vendor specific). My favorite fault is called the “dying gasp.” This indicates that an unrecoverable condition has occured. I like to imagine the device literally dying and in its final gasp for air, revealing to its peer that something bad has happened and saying its final goodbyes. A dying gasp is sent when you gracefully shutdown an interface via the shutdown command.

A device can take action upon receiving any of these three RFIs. The only action available is to err-disable the interface.

CE1(config-if)#ethernet oam remote-failure critical-event action ?
  error-disable-interface  Error-disable the interface

CE1(config-if)#ethernet oam remote-failure dying-gasp action ?    
  error-disable-interface  Error-disable the interface
  <cr>                     <cr>

CE1(config-if)#ethernet oam remote-failure link-fault action ?
  error-disable-interface  Error-disable the interface
  <cr>                     <cr>

I’ve set the dying-gasp action to errdisable on PE1, and shutdown Gi1 on CE1.

CE1 immedaitely sends the OAMPDU with the dying gasp flag set:

PE1 sees this and immedaitely errdisables the interface:

*Oct 31 14:39:04.925: %ETHERNET_OAM-6-DYING_GASP: The client on interface Gi1 has received a Dying Gasp indication from its remote peer (failure reason = remote client link shutdown, action = error-disable)
*Oct 31 14:39:05.919: %ETHERNET_OAM-6-EXIT_SESSION: The client on interface Gi1 has left the OAM session.
PE1#
*Oct 31 14:39:06.927: %LINK-5-CHANGED: Interface GigabitEthernet1, changed state to administratively down
*Oct 31 14:39:07.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1, changed state to down

Remote Loopback

This is a very handy feature that allows one device to turn the remote device’s port into a loopback. By default this capability is turned off. To enable it you must use the following command on both devices:

ethernet oam remote-loopback supported

Only a device in active mode can initate a remote loopback session. This means that CE1 cannot initiate the session. Only PE1 can initiate the session and turn CE1 Gi1 into a loopback.

Start the remote loopback session with the following command:

PE1#ethernet oam remote-loopback start interface Gi1
PE1#
*Oct 31 14:57:50.065: %ETHERNET_OAM-6-LOOPBACK: Interface Gi1 has entered the master loopback mode.

*Oct 31 14:57:50.128: %ETHERNET_OAM-6-LOOPBACK: Interface Gi1 has entered the slave loopback mode.

On CE2 (the Z end of the E-Line service in this lab) I run a ping to CE1. The ARP messages are literally looped back and seen twice on a pcap:

Besides the syslog messages, you can also see that the loopback session is running from show ethernet oam summary and show ethernet oam discovery.

PE1#show ethernet oam sum                    
Symbols:          * - Master Loopback State,  # - Slave Loopback State
                  & - Error Block State
Capability codes: L - Link Monitor,  R - Remote Loopback
                  U - Unidirection,  V - Variable Retrieval

  Local                       Remote
Interface       MAC Address    OUI    Mode    Capability

* Gi1           5254.0019.86c2 00000C passive  L R

PE1#show ethernet oam discovery | in Loopback
    Loopback status:   remote loopback


CE1#show ethernet oam summary 
Symbols:          * - Master Loopback State,  # - Slave Loopback State
                  & - Error Block State
Capability codes: L - Link Monitor,  R - Remote Loopback
                  U - Unidirection,  V - Variable Retrieval

  Local                       Remote
Interface       MAC Address    OUI    Mode    Capability

# Gi1           5254.000f.d2cf 00000C active   L R

CE1#show ethernet oam discovery | in Loopback
    Loopback status:   local loopback

Run this command to stop the loopback session:

PE1#ethernet oam remote-loopback stop interface Gi1
PE1#
*Oct 31 15:02:05.439: %ETHERNET_OAM-6-LOOPBACK: Interface Gi1 has exited the master loopback mode.

Further Reading

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/cether/configuration/xe-3s/asr903/16-12-1/b-ce-xe-16-12-asr900/b-ce-xe-16-12-asr900_chapter_00.pdf

https://sudonull.com/post/126874-What-is-Slow-Protocols

Previous802.1ah (MAC-in-MAC)Next802.1ag (CFM)

Last updated