We saw in the previous article, that if there is a break anywhere in an LSP, the LSP ping will not succeed. However the LSP ping will not tell us exactly where the break is.
LSP traceroute uses the same MPLS echo requests and replies as LSP ping. LSP traceroute increments the TTL value, starting at 1, just like the regular traceroute utility. The big difference between LSP traceroute and LSP ping is that LSP traceroute is also concerned with knowing each hop’s downstream mapping. Think of the downstream mapping as the outgoing label stack and forwarding information. You may seen downstream mapping also called DSMAP. Each hop that generates an MPLS echo reply TTL exceeded message includes its own downstream mapping information. This allows you as the operator to visually see the MTU and label stack information at each hop in your traceroute results.
Lab
We’ll use the same topology as before:
Currently our LSP is fully functional. Let’s run an mpls traceroute to see what the expected behavior is. Just like mpls ping, you must specify a target FEC, not an IP address.
PE1#traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.1.3.1 MRU 1500 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 14 ms
L 2 10.3.4.4 MRU 1500 [Labels: implicit-null Exp: 0] 5 ms
! 3 10.2.4.2 8 ms
MPLS traceroute shows the 0th hop, which is the local router’s downstream mapping information. MRU stands for Maximum Receive Unit. This is the max size of a labeled packet that can be forwarded, including the label headers. The penultimate hop should have a label value of “implicit-null” which explicitly shows us that it is preforming PHP.
We can see that this router’s own outgoing label is 19 at index=0. The 1st hop (index=1) is P3, and its own FIB entry swaps 19 for 24003. So its downstream mapping is label 24003, which we see in the traceroute output.
P3#show mpls forwarding-table labels 19
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
19 24003 2.2.2.2/32 65249 Gi1 10.3.4.4
When we get to PE2 at 10.2.4.2, it is the originator of 2.2.2.2/32, so it has no downstream mapping. The traceroute stops at that point.
Here is a capture of the MPLS echo request with TTL set to 1, captured at the PE1-P3 link:
Just like the LSP ping, the IPv4 destination is 127.0.0.1
The MPLS TTL is 1
The own router’s downstream mapping is included in the Echo Request
This is included so that the receiving router can verify it received the packet on the correct interface and with the correct label for the FEC. Basically it allows the next-hop router to verify that it is the intended node.
This changes as the TTL is incremented. When TTL=1, the downstream mapping is the local router’s own mapping. The first hop router replies with its own downstream mapping to the local router. The local router then uses this received downstream mapping and uses it for the echo request with TTL=2. That way the second hop router receives the request with the first hop router’s downstream mapping, and the second hop router can verify it is the intended node for the LSP. This process continues as the TTL increments.
Here is a capture of the MPLS echo reply from P3 to PE1:
P3 includes its own downstream mapping information (MTU, label, next-hop router 10.3.4.4)
The reply is IP forwarded, with the source interface set to the interface address that received the TTL=1 packet. Just like LSP ping, LSP traceroute only verifies the unidirectional LSP.
Breaking the LSP
Let’s bring LDP down between P4 and PE2 again, and test LSP traceroute when the LSP is broken.
#P4, PE2
mpls ldp
no int Gi0/0/0/1
PE1#traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.1.3.1 MRU 1500 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 4 ms
B 2 10.3.4.4 MRU 1500 [No Label] 14 ms
B 3 10.3.4.4 MRU 1500 [No Label] 5 ms
. 4 *
B 5 10.3.4.4 16 ms
B 6 10.3.4.4 9 ms
B 7 10.3.4.4 8 ms
B 8 10.3.4.4 6 ms
B 9 10.3.4.4 6 ms
B 10 10.3.4.4 6 ms
B 11 10.3.4.4 5 ms
B 12 10.3.4.4 4 ms
B 13 10.3.4.4 5 ms
B 14 10.3.4.4 4 ms
B 15 10.3.4.4 6 ms
B 16 10.3.4.4 5 ms
B 17 10.3.4.4 6 ms
B 18 10.3.4.4 5 ms
B 19 10.3.4.4 5 ms
B 20 10.3.4.4 5 ms
B 21 10.3.4.4 4 ms
B 22 10.3.4.4 4 ms
B 23 10.3.4.4 5 ms
B 24 10.3.4.4 4 ms
B 25 10.3.4.4 4 ms
B 26 10.3.4.4 4 ms
B 27 10.3.4.4 4 ms
B 28 10.3.4.4 4 ms
B 29 10.3.4.4 4 ms
B 30 10.3.4.4 3 ms
Here we see one annoying thing about LSP traceroute. If the LSP is broken, the traceroute will continue until hop 30 by default. We can specify a max TTL value to avoid this:
PE1#traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1 ttl 5
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.1.3.1 MRU 1500 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 17 ms
B 2 10.3.4.4 MRU 1500 [No Label] 5 ms
B 3 10.3.4.4 MRU 1500 [No Label] 8 ms
. 4 *
B 5 10.3.4.4 10 ms
Why is every last hop P4? This is because P4 has an unlabelled entry for 2.2.2.2/32, so it continually returns an error code no matter how high the TTL goes.
Here is P4’s Echo Reply:
You can see that there is no label information under the downstream mapping section
LSP traceroute tells us where the LSP is broken, which we could not figure out with LSP ping alone. We see that the 0th hop and 1st hop have label values. However the 2nd hop, P4 shows no label, and a code of B indicated unlabaled output interface. With LSP ping, we saw five Bs but no indication of which hop sent the error. With LSP traceroute we can see that the first hop has code L (labeled output interface) as normal, and P4 specifically is the one sending the B error code.
Using LSP ping to verify MTU
Let’s re-enable LDP between P4 and PE2 again.
#P4, PE2
mpls ldp
int Gi0/0/0/1
With LSP traceroute, you cannot specify the size, I believe largely because there is so much information contained in the DSMAP field.
However, with LSP ping we can verify end-to-end MTU. Currently all links have the default MTU of 1500. What is the maximum LSP ping size that will succeed?
The answer is 1496. The four bytes of the MPLS header count towards the total MTU.
PE1#ping mpls ipv4 2.2.2.2/32 size 1496 source 1.1.1.1
Sending 5, 1496-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/7/9 ms
Total Time Elapsed 54 ms
PE1#ping mpls ipv4 2.2.2.2/32 size 1497 source 1.1.1.1
Sending 5, 1497-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
QQQQQ
Success rate is 0 percent (0/5)
Total Time Elapsed 15 ms
The Q error code indicates request not sent. PE1 did not even attempt to send the packet out its egress interface, and the packet size is too large. Let’s set MTU on PE1 to 1508, which will allow two MPLS headers, and see what the error code is in that case. We must also set MTU on P3 so that OSPF stays up.
#PE1, P3
int Gi2
mtu 1508
end
!
#PE1
PE1#ping mpls ipv4 2.2.2.2/32 size 1497 source 1.1.1.1
Sending 5, 1497-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
.....
Success rate is 0 percent (0/5)
Total Time Elapsed 9499 ms
The LSP pings are simply timing out now. If we run an LSP traceroute, we can easily see the MTU of every outgoing interface in the path.
PE1#traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.1.3.1 MRU 1508 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 20 ms
L 2 10.3.4.4 MRU 1500 [Labels: implicit-null Exp: 0] 6 ms
! 3 10.2.4.2 12 ms
There is one interesting difference in MTU behavior with IOS-XR. When you set the MTU to a value that is not 1500, the router appears to subtract 14 bytes, the size of an ethernet header, from the MRU value.
To see this, set P4 and PE2 to mtu 1508 on their Gi0/0/0/1 links:
#P4, PE2
int Gi0/0/0/1
mtu 1508
If we run the LSP traceroute again, the MTU at the P4 hop is 1494 (1508-14).
PE1#traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.1.3.1 MRU 1508 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 6 ms
L 2 10.3.4.4 MRU 1494 [Labels: implicit-null Exp: 0] 12 ms
! 3 10.2.4.2 8 ms
Interestingly, the LSP ping with size 1496 no longer works. The largest ping size is now 1494.
PE1#ping mpls ipv4 2.2.2.2/32 size 1496 source 1.1.1.1
Sending 5, 1496-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
.....
Success rate is 0 percent (0/5)
Total Time Elapsed 9412 ms
PE1#ping mpls ipv4 2.2.2.2/32 size 1494 source 1.1.1.1
Sending 5, 1494-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/9/14 ms
Total Time Elapsed 65 ms
PE1#ping mpls ipv4 2.2.2.2/32 size 1495 source 1.1.1.1
Sending 5, 1495-byte MPLS Echos to 2.2.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
<snip>
Type escape sequence to abort.
.....
Success rate is 0 percent (0/5)
Total Time Elapsed 9711 ms
This is just something to be aware of when operating IOS-XR routers in your core. In IOS-XR, the mtu command configures the MTU including the L2 header. This is why the router subtracts 14 bytes from the MRU. In IOS-XE, the mtu command configures the MTU without including the L2 header. The default interface MTU in IOS-XR is actually 1514.
RP/0/0/CPU0:P4#show int Gi0/0/0/0 | in MTU
MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
RP/0/0/CPU0:P4#show run int gi0/0/0/0
interface GigabitEthernet0/0/0/0
ipv4 address 10.3.4.4 255.255.255.0
!
What we really wanted to do when we set MTU to 1508 is actually set the mpls mtu to 1508. This allows an IPv4 packet up to 1500 bytes with two mpls header. In order to accomplish this on IOS-XR it is a little confusing. You have to set the interface overall MTU to 1500 + 14 + 8 (ethernet header plus two MPLS headers), and then manually reduce the ipv4 MTU down to 1500. Otherwise IPv4 packets could take up that 8 bytes we are allocating for MPLS.
#P4, PE2
int Gi0/0/0/1
mtu 1522
ipv4 mtu 1500
#P4
show mpls int gi0/0/0/1 private location 0/0/CPU0
Wed Aug 31 14:08:16.270 UTC
Interface IFH MTU Label Stack Depth
Prim Bkup Srte
-------------- ---------- ----- -------------------
Gi0/0/0/1 0x00000040 1508 1 3 10
#PE1
traceroute mpls ipv4 2.2.2.2/32 source 1.1.1.1
Tracing MPLS Label Switched Path to 2.2.2.2/32, timeout is 2 seconds
<snip>
Type escape sequence to abort.
0 10.1.3.1 MRU 1508 [Labels: 19 Exp: 0]
L 1 10.1.3.3 MRU 1500 [Labels: 24003 Exp: 0] 6 ms
L 2 10.3.4.4 MRU 1500 [Labels: implicit-null Exp: 0] 4 ms
! 3 10.2.4.2 10 ms
Strangely the MRU is still 1500 in the LSP traceroute. This may be an issue with XRv.
In this article we explored LSP traceroute and saw how it provides downstream mapping information at each hop. It uses the same MPLS request and reply messages as LSP ping, but adds additional information which helps with troubleshooting. With LSP traceroute, you can see the exact hop that an LSP fails, and you see an error code that tells you why it failed. Additionaly the outgoing interface MTU is included at each hop, enabling you to troubleshot end-to-end LSP MTU. The outgoing mapping information generated at each hop is based on that hop’s outgoing interface if it were to continue to send the packet down the LSP.
