CSC (Carrier Supporting Carrier)

CSC (Carrier Supporting Carrier) is a design which allows one ISP to provide transit MPLS service for another ISP. This is sometimes called hierarchical L3VPN.

An example will help explain this. Imagine ISP1 wants to provide service in a new area but they cannot obtain their own connectivity in order to connect the new PoP to their existing IGP. In order to offer L3VPN service (or really any service for that matter) they need to connect the router to their internal MPLS network.

Usually ISP1 will order L2VPN service, but sometimes this is not possible. For example, if an ethernet handoff is not available at the new PoP, then Layer 3 might be the best option.

So ISP1 orders L3VPN service from ISP2, but ISP2 cannot treat ISP1 like a regular L3VPN customer. ISP2 needs to exchange labels with ISP1 so that ISP1 can send its own labeled traffic through ISP2. The two routers connecting ISP1 and ISP2 are actually both PEs. From ISP2’s perspective, ISP1 is a CE, but from ISP1’s perspective, that device is a PE which will connect to other CEs.

ISP2 is called the backbone carrier or core carrier, because it is offering backbone service to ISP1.

ISP1 is called the customer carrier because they connect directly to the customer. The backbone carrier provides MPLS L3VPN transit service to the customer carrier.

SP2’s (backbone carrier) PE is called the CSC_PE. ISP1’s (customer carrier) PE is called the CSC_CE, and connects to real CEs of the customers. The CSC_CE is named as such because it acts like the CE from the backbone carrier’s perspective.

There are two options for exchanging labels between the CSC_PE and CSC_CE: LDP/IGP or BGP-LU. In the following lab we will use both options. LDP is not available on VRFs in IOS-XR, so we will use LDP and OSPF on the XE CSC_PE and CSC_CE devices, and BGP-LU on the XR CSC_PE and CSC_CE devices.

The customer carrier needs to be able to send labeled traffic to the backbone carrier with the top label representing the loopback IP of another CSC_CE. So all the CSC_CEs advertise their loopbacks to the CSC_PEs with a label, and the CSC_PEs advertise other CSC_CE loopbacks to the connected CSC_CE with a label. The backbone carrier only needs to be concerned with the customer carrier CSC_CE loopback IPs, and does not care at all about the actual Customer L3VPNs behind the CSC_CEs.

As you go through the lab this will make much more sense. In a way, CSC is just regular L3VPN but enabling labeled traffic at the CSC_PE/CSC_CE peering point.

Lab

Notice in the diagram below how CSC creates a heirarchy. SP2 is the backbone carrier at the top of the heirarchy. It treats SP1 as a normal L3VPN customer except it also exchanges labels for the /32 loopback prefixes that SP1 advertises.

The Customer devices (CE1 and CE2) see SP1 as PE devices.

Here are the startup configs. I have configured everything as a “normal” L3VPN service.

#CE1
hostname CE1
!
line con 0
 logging sync
!
int Gi1
 ip address 100.64.0.2 255.255.255.252
 no shut
!
int lo0
 ip address 10.1.1.1 255.255.255.0
!
router bgp 65000
 neighbor 100.64.0.1 remote-as 100
 network 10.1.1.0 mask 255.255.255.0

#CE2
hostname CE2
!
line con 0
 logging sync
!
int Gi1
 ip address 100.64.0.6 255.255.255.252
 no shut
!
int lo0
 ip address 10.1.2.1 255.255.255.0
!
router bgp 65001
 neighbor 100.64.0.5 remote-as 100
 network 10.1.2.0 mask 255.255.255.0

#CSC_CE1
hostname CSC_CE1
!
line con 0
 logging sync
!
vrf definition CUSTOMER
 rd 100:1
 route-target both 100:1
 address-family ipv4 unicast
 exit
!
int Gi1
 vrf forwarding CUSTOMER
 ip address 100.64.0.1 255.255.255.252
 no shut
!
int Gi2
 ip address 100.64.100.2 255.255.255.252
 no shut
!
int lo0
 ip address 100.1.1.1 255.255.255.255
!
router bgp 100
 address-family ipv4 unicast vrf CUSTOMER
  neighbor 100.64.0.2 remote-as 65000

#CSC_PE1
hostname CSC_PE1
!
line con 0
 logging sync
!
vrf definition CUSTOMER_ISP1
 rd 200:1
 route-target both 200:1
 address-family ipv4 unicast
 exit
!
int Gi1
 vrf forwarding CUSTOMER_ISP1
 ip address 100.64.100.1 255.255.255.252
 no shut
!
int Gi2
 ip address 10.1.3.1 255.255.255.0
 ip ospf network point-to-point
 mpls ip
 no shut
!
int lo0
 ip address 200.1.1.1 255.255.255.255
!
router ospf 1
 network 0.0.0.0 255.255.255.255 area 0
!
router bgp 200
 neighbor 200.3.3.3 remote-as 200
 address-family vpnv4
  neighbor 200.3.3.3 activate

#ISP2_P3
hostname ISP2_P3
!
line con 0
 logging sync
!
int Gi1
 ip address 10.1.3.3 255.255.255.0
 ip ospf network point-to-point
 mpls ip
 no shut
!
int Gi2
 ip address 10.2.3.3 255.255.255.0
 ip ospf network point-to-point
 mpls ip
 no shut
!
int lo0
 ip address 200.3.3.3 255.255.255.255
!
router ospf 1
 network 0.0.0.0 255.255.255.255 area 0
!
router bgp 200
 neighbor 200.1.1.1 remote-as 200
 neighbor 200.1.1.1 update-source lo0
 neighbor 200.2.2.2 remote-as 200
 neighbor 200.2.2.2 update-source lo0
 address-family vpnv4
  neighbor 200.1.1.1 activate
  neighbor 200.1.1.1 route-reflector-client
  neighbor 200.2.2.2 activate
  neighbor 200.2.2.2 route-reflector-client

#CSC_PE2
hostname CSC_PE2
!
vrf CUSTOMER_ISP1
 address-family ipv4 unicast
  import route-target 200:1
  export route-target 200:1
!
int Gi0/0/0/0
 vrf CUSTOMER_ISP1
 ip address 100.64.100.5 255.255.255.252
 no shut
!
int Gi0/0/0/1
 ip address 10.2.3.2 255.255.255.0
 no shut
!
int lo0
 ip address 200.2.2.2 255.255.255.255
!
router ospf 1
 area 0
  int Gi0/0/0/1
   network point-to-point
  int lo0
!
mpls ldp
 int Gi0/0/0/1
!
router bgp 200
 address-family vpnv4 unicast
 address-family ipv4 unicast
 neighbor 200.3.3.3
  remote-as 200
  update-source lo0
  address-family vpnv4 unicast

#CSC_CE2
hostname CSC_CE2
!
vrf CUSTOMER
 address-family ipv4 unicast
  import route-target 100:1
  export route-target 100:1
!
int Gi0/0/0/0
 vrf CUSTOMER
 ip address 100.64.0.5 255.255.255.252
 no shut
!
int Gi0/0/0/1
 ip address 100.64.100.6 255.255.255.0
 no shut
!
int lo0
 ip address 100.2.2.2 255.255.255.255
!
router bgp 100
 address-family vpnv4 unicast
 address-family ipv4 unicast
 vrf CUSTOMER
  rd 100:1
  address-family ipv4 unicast
  neighbor 100.64.0.6
   remote-as 65001
   address-family ipv4 unicast
    route-policy PASS in
    route-policy PASS out
!
route-policy PASS
 pass
end-policy

Right now the CSC_CEs are learning routes from the CEs, but traffic is not working. The two CSC_CEs will need to peer with each other over vpnv4. Let’s first enable LDP between the two IOS-XE routers:

#CSC_CE1
int Gi2
 mpls ip
 mpls ldp discovery transport-address interface

#CSC_PE1
int Gi1
 mpls ip
 ! The discovery address will be the interface address since no other IP addresses
 !  exist for the VRF

When you run LDP on an interface in a VRF, the transport address is automatically the interface. This is the case on CSC_PE1, because int Gi1 is part of the CUSTOMER_ISP1 VRF.

On the interface in the global VRF, which is Gi2 on CSC_CE1, we need to force the transport-address to be the interface address. Otherwise the transport address is Lo0 (the RID), which CSC_PE1 cannot reach right now. See the article “LDP Router-ID Issues” for more on this topic.

We’ll now run OSPF between CSC_CE1 and CSC_PE1 so that CSC_PE1 can learn CSC_CE1’s loopback and advertise this into the L3VPN, by redistributing OSPF into BGP. This configuration is regular OSPF as a PE-CE protocol.

#CSC_CE1
router ospf 1
 network 100.64.100.0 0.0.0.3 area 0
 network 100.1.1.1 0.0.0.0 area 0

#CSC_PE1
! We must use OSPF 10 because OSPF 1 is use in the global VRF
router ospf 10 vrf CUSTOMER_ISP1
 network 100.64.100.0 0.0.0.3 area 0
 redistribute bgp 200
!
router bgp 200
 address-family ipv4 vrf CUSTOMER_ISP1
  redistribute ospf 10

IOS-XR does not have the ability to run LDP in a VRF. Instead we will use BGP labeled unicast to exchange label information for the ISP1 loopbacks at the CSC_PE2-CSC_CE2 peering point.

#CSC_CE2
router bgp 100
 address-family ipv4 unicast
  network 100.2.2.2/32
  allocate-label all
 neighbor 100.64.100.5
  remote-as 200
  address-family ipv4 labeled-unicast
   route-policy PASS in
   route-policy PASS out

#CSC_PE2
router bgp 200
 vrf CUSTOMER_ISP1
  rd 200:1
  address-family ipv4 unicast
   allocate-label all
  neighbor 100.64.100.6
   remote-as 100
   address-family ipv4 labeled-unicast
    route-policy PASS in
    route-policy PASS out
!
route-policy PASS
 pass
end-policy

As we saw in Option C, we also need to create a static /32 route on the XR routers in order to send labeled traffic to a nexthop.

#CSC_CE2
router static address-family ipv4 unicast
 100.64.100.5/32 Gi0/0/0/1

#CSC_PE2
router static vrf CUSTOMER_ISP1 address-family ipv4 unicast
 100.64.100.6/32 Gi0/0/0/0

CSC_CE1 and CSC_CE2 now have reachability between their loopbacks. Not only that, but they have an LSP between them.

CSC_CE1#show mpls forwarding-table 
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
16         21         100.2.2.2/32     0             Gi2        100.64.100.1

RP/0/0/CPU0:CSC_CE2#show mpls forwarding 
Tue Aug  9 13:46:53.929 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes       
Label  Label       or ID              Interface                    Switched    
------ ----------- ------------------ ------------ --------------- ------------
24000  Unlabelled  10.1.2.0/24[V]     Gi0/0/0/0    100.64.0.6      0           
24001  Aggregate   100.64.100.0/24    default                      0           
24002  24001       100.1.1.1/32                    100.64.100.5    0           
24003  24005       100.64.100.0/30                 100.64.100.5    0

CSC_CE2#ping 100.1.1.1 source lo0      
Tue Aug  9 13:55:12.605 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 9/17/49 ms

All we need to do in order to enable the L3VPN service for CE1 and CE2 is run vpnv4 between the ISP1 PEs.

#CSC_CE1
router bgp 100
 neighbor 100.2.2.2 remote-as 100
 neighbor 100.2.2.2 update-source lo0
 address-family vpnv4 
  neighbor 100.2.2.2 activate

#CSC_CE2
router bgp 100
 neighbor 100.1.1.1
  remote-as 100
  update-source lo0
  address-family vpnv4 unicast

The CEs now have reachability between themselves:

CE1#ping 10.1.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.2.1, timeout is 2 seconds:
Packet sent with a source address of 10.1.1.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 9/17/46 ms

Test your knowledge:

  • How many labels deep is the stack at ISP2_P3 as it receives the ICMP request from CSC_PE1?

  • What is the AS path of the route 10.1.2.0/24 as seen on CE1?

Answers:

The label stack at ISP2_P3 is three labels deep:

  1. The transport label representing CSC_PE2 (200.2.2.2/32)

  2. The service label CSC_PE2 allocated for 100.2.2.2/32

  3. The VPN label that CSC_CE2 allocated for 10.1.2.0/24

The AS path for 10.1.2.0/24 on CE1 is 100 65001 i. The route is advertised directly from CSC_CE2 to CSC_CE1. ISP2 plays no role in routing for the customer VRF. It only provides transit service to ISP1 for its own loopbacks.

Next, examine a traceroute from CE1 to CE2 and see if you can identify the labels:

CE1#traceroute 10.1.2.1 source lo0 probe 1
Type escape sequence to abort.
Tracing the route to 10.1.2.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.64.0.1 41 msec
  2 100.64.100.1 [MPLS: Labels 21/24001 Exp 0] 24 msec
  3 10.1.3.3 [MPLS: Labels 17/24006/24001 Exp 0] 32 msec
  4 10.2.3.2 [MPLS: Labels 24006/24001 Exp 0] 13 msec
  5 100.64.100.6 [MPLS: Label 24001 Exp 0] 14 msec
  6 100.64.0.6 101 msec

  • Hop 2: 21 is the label assigned by CSC_PE1 for CSC_CE2’s loopback, and 24001 is the service label assigned by CSC_CE2 for 10.1.2.0/24. This label (24001) will be carried throughout the entire path.

  • Hop 3: 17 represents CSC_PE2 and 24006 is the service label assigned by CSC_PE2 for CSC_CE2’s loopback.

Conclusion

CSC sounds complex, but once you actually configure it I hope you find it easier than you thought it would be. The problem that CSC solves, is that in order for a backbone carrier to provide L3 MPLS service to a customer carrier, the CSC_PE and CSC_CE need to be able to send labeled traffic to each other. This allows the customer ISP to carry a service label end-to-end from one CSC_CE to another, all the way through the backbone ISP.

This means that CSC is essentially just heirarchical L3VPN service, but with some type of label exchange (BGP-LU or LDP) running between the backbone carrier and customer carrier.

The only information that the customer carrier needs to advertise to the backbone carrier is the loopback IPs which are used for BGP peering internally. This enables the CSC_CEs to form LSPs between each other through the backbone carrier.

Further Reading

RFC4364 Section 9 “Carrier’s Carriers” https://www.rfc-editor.org/rfc/rfc4364#section-9

https://orhanergun.net/carrier-supporting-carrier-csc/

If you are interested in something a little more complex, check out Nick Russo’s fantastic white paper, Global MPLS Design Using CSC http://njrusmc.net/pub/csc_optc.pdf

PreviousInter-AS L3VPN Pt. 4, Option CNextPE NAT

Last updated