QoS Shaping and Policing (Part 5)
Both shaping and policing set a speed limit to traffic ingressing or egressing an interface.
Shaping delays traffic by holding it in a queue and de-queing packets at the dictated rate. Shaping can only be used for traffic egressing an interface.
Policing drops any traffic that exceeds the CIR. Policing can be used for traffic both ingressing or egressing an interface, but is typically used for ingress traffic.
Shaping
Terminology
First we need to look at the terminology used to describe shaping.
Tc (time interval)
Time interval in msec, over which the Bc (committed burst) is sent.
Bc (Committed Burst)
Bc = committed burst size in bits. This is the amount of traffic to send in the interval (Tc). The reason it is considered a “burst” is because the bits flow at the line rate out the interface, “bursting” above the CIR. However it is a committed burst because it will still conform to the CIR.
CIR (Committed Information Rate)
The CIR defines the “slower” rate the administrator whishes to send traffic at, out the interface.
Be (Excess Burst)
Be = Excess burst in bits. The number of bits beyond Bc that can be sent after some period of inactivity, allowing you to burst over the normal Bc.
Tc, Bc, and CIR are all related. The CIR divided into each Tc defines the amount of data that can be sent (Bc). If you have a serial interface with a clock rate of 512Kb/s but a CIR of 256Kb/s, at a 1/8 time interval you can send 1/8 of 256Kb/s which is 32Kb. If you send 32Kb eight times at an 1/8 second interval, you’ve sent 256Kb in one second.
Token Bucket System
Instead of basing the sending interval on a timer, IOS uses a token bucket system. The Bc for each interval is placed in a “token bucket” with one token per bit. As the router sends traffic, it removes a token per bit that is sent. When the bucket is empty, the router cannot send traffic any longer and queues and delays the traffic. The token bucket is refilled at each Tc (time interval) with new tokens to fill the bucket to the Bc again.
Burst excess is used when there is a period of inactivity. In this case, you can briely send over the Bc in the first time interval without exceeding the overall CIR. The amount you can burst over the Bc is the Be.
When using Be, the token bucket now has a capacity of Bc + Be. Bc is added at every time interval, and Be is added whenever there is a period of inactivity.
An example will help you understand how this works.
Example (Shaping at 100Mb/s on a 1Gb/s interface)
Let’s say you are a customer for an EPL which has a contracted rate of 100Mb/s with the provider. The provider will police traffic inbound to their device at 100Mb/s. In order to prevent packet loss, you wish to shape traffic out this gigabit interface at 100Mb/s.
To do this, the router must send at line rate (1Gb/s) but only for a tenth of the time. In a given second, if the router sends at 1Gb/s for .1 seconds, and does not send anything for .9 seconds, you have achieved 100Mb sent in one second.
To keep the math easy, let’s say the Tc (time interval) is .1 second. Every .1 second, the router will fill the Bc bucket with 10Mbs of tokens. The router can now send 10Mb of traffic which will take .01 seconds at 1Gbps. The router now waits .09 seconds until the token bucket is filled again with 10Mb worth of tokens. The process repeats.
In the diagram above, at every .1 second interval, 10Mbit of coins are placed in the bucket. As soon as the coins are available, 10Mbits of traffic are sent at line rate (1Gbps) for .01 seconds. This “spends” coins in the bucket. The router then waits .09 seconds until more coins are added to the bucket.
Adding Excess Burst (Be)
Without excess burst, if there is no traffic for a period of time, the excess coins are just wasted.
At .2 seconds, there was no more data to send, and coins were added to the bucket. The bucket is now full. At .3seconds, more coins are added, but the bucket is already full, so the excess coins are wasted. At .3 seconds, the router can only send 10Mbit of traffic. This is because the bucket is only the size of Bc, so no excess burst is allowed.
If we make the bucket twice as big, we can “bank” the excess and send twice as much traffic in the next interval:
Each interval still fills Bc, but now we can exceed Bc and fill the bucket up to Be if we don’t spend all our coins. At time interval .3 secconds, the router now sends 20Mbit of traffic instead of just 10Mbit.
Shaping Configuration
The operation of shaping is fairly complex, but the configuration is quite simple.
The most basic configuration is below:
Besides using a rate in bits per second, you can also use a percentage of the link bandwidth:
By default, IOS will calculate the Bc and Be value. On the CSR1000v the Bc and Be appear to be based on a 4msec interval. The Be equals the Bc, effectively creating a bucket that is two times Bc.
If for some reason you want to adjust the Tc yourself, you have to do this by specifying the Bc and Be values. Remeber, Tc = Bc/CIR, so adjusting Bc adjusts the Tc value. You normally shouldn’t need to do this. Cisco recommends that you let the router calculate this automatically. But if you want to, the configuration is below. Setting the Bc lower decreases the time interval, while a large Bc increases the time interval and increases delay for packets waiting in the queue.
If you don’t specify Be, it is automatically equal to Bc.
Using queuing with shaping
The simple examples above only use the class-default. All traffic egressing the interface is shaped at the rate. But what if you want to create multiple queues and give preferential treatment to voice traffic, all while shaping overall traffic to the CIR?
To do this, you nest policy-map maps inside each other. One policy-map defines the queuing, and one defines the shaping and applies the queuing policy-map.
When the shaping policy-map is applied outbound on an interface, two queues are created: the voip LLQ queue, and the class-default which uses WFQ. The queuing controls removing packets from the two queues, and placing them into the overall queue for the shaping policy-map, which is shaped at 100m. By creating two queues, you allow voice traffic to “cut infront” of class-default traffic. It is a bit like the fast-pass lane in Disneyland. The fast-pass and general admission lanes are separate at the beginning when you get in line, but the lanes eventually funnel into one single lane once you get closer to getting on the ride.
This ability to nest policy-maps within eachother is called hierarchical QoS, or HQF (hierarchical QoS framework). This simply allows you to manage QoS at multiple levels. In this example we managed QoS for the interface itself (the 100m shaper), plus the QoS for each class (voip and class-default) within that shaper.
Policing
The simplest policer has only two categories for traffic: conforming and exceeding. Either the traffic conforms to the CIR, or it exceeds the CIR. If it exceeds the policer can drop or just re-mark the traffic. This is called a two color policer. It uses a single token bucket.
A policer can also allow for burst. In this case there are three categories: conforming, exceeding, and violating. In the exceeding state, the packet is using up the burst. In the violate state, the packet is outside the CIR plus burst rate. This is called a three-color policer, and it uses two token buckets.
Single rate, two-color
A policer uses a similar token bucket concept as seen with shaping. When a packet arrives at the policer, if there are available tokens in the bucket, the policer “spends” the tokens (removes them from the bucket) and the packet is conforming. If there aren’t enough tokens in the bucket, the packet is exceeding and tokens are not removed. (There is no negative balance concept in the bucket).
The way tokens are replenished in the bucket is slightly different from shaping. In policing, everytime a packet arrives, tokens are replenished in the bucket. The amount of tokens added to the bucket is based on the difference in time from when the previous packet arrived. So if the previous packet arrived 100ms ago, then 1/10 of the tokens for a full second are added to the bucket. The formula is the following:
The police rate is in bits per second, and the token bucket for policing is in bytes. So you divide this result by 8 to determine the number of bytes to place in the bucket. Essentially this means that the policer puts a proportional amount of tokens into the bucket based on the variable Tc (time interval). The Tc is dynamically calculated every time a packet arrives.
Using this definition, you can now see how policing and shaping are so different. With policing, it simply forwards the packet or discards it. With shaping, packets are queued up and slowly sent out the interface to conform to a certain speed. Policing results in much more packet loss than shaping, although shaping introduces delay.
A note on bursting
Bursting (going above the policed CIR for a brief moment) in policing is controlled by the size of the Bc bucket. A bigger bucket allows you to “bank” more tokens, and then spend them all at once if a lot of traffic comes in too quickly. A very shallow bucket only allows you to bank a very low amount of tokens, preventing any bursting. By default IOS will calculate the size of the Bc bucket for you, using the CIR in bytes over a .25 second period. The CIR is in bits per second, so the formula for the default Bc is as follows:
Single-rate, three-color
A three-color policer uses a dual token bucket system to categorize traffic into either conforming, exceeding, or violating.
In this system, one bucket is used for Bc and one bucket is used for Be. The policer replenishes the Bc bucket as with the previous two-color policer - when packets arrive. However, if the replenishment is more than the size of the Bc bucket, the extra “spills over” into the Be bucket.
If the packet does not conform (there are not enough tokens in the Bc bucket), but there are enough tokens in the Be bucket, the packet is now exceeding and the policer removes tokens from the Be bucket.
If there are not enough tokens in the Bc bucket nor the Be bucket, the packet is now violating.
Note that the size of the Be bucket uses the same formula by default in IOS: Be = CIR / 32.
You may want to take the following actions for each category:
conforming = transmit
exceeding = transmit but mark down
A device further upstream may discard this traffic in the case of congestion. If no congestion is present upstream, this traffic gets through.
violating = discard
Dual-rate, three-color
With dual-rate, you can set two completely different sustained rates of traffic, instead of a byte amount of burst excess. You configure a CIR for conforming traffic, and a PIR (Peak Information Rate) for exceeding traffic.
There are still two buckets, but they have different names. There is a CIR bucket and a PIR bucket. This time, the CIR bucket does not “spill over” into the PIR bucket. They are two separate buckets.
The PIR bucket is bigger because the PIR > CIR
Both buckets are filled by the arrival of a single packet, using the same formula as before. A packet arrives, and each bucket is filled proportionally based on the time delta of the previous packet arrival and the CIR/PIR rate. The PIR is larger (higher bandwidth) than the CIR, so the PIR is always filled with more tokens.
When a packet arrives, if there are tokens available in the CIR bucket, tokens are removed from both the CIR and PIR bucket and the packet is conforming.
If there are no tokens available in the CIR bucket, but there are available tokens in the PIR bucket, tokens are only removed from the PIR bucket and the packet is exceeding.
This process is quite intuitive when you realize that the CIR and PIR buckets are completely separate. In a way it is like running two separate single-rate two color processes at the same time.
Policing Configuration
Just like shaping, policing configuration can be extremely simple.
The above configuration uses a single rate two-color policer. By default, the conform-action is transmit, and the exceed action is drop. You can confirm this using the below show command:
As you can see, the Bc was automatically calculated. The Bc is the size of the Bc bucket. A larger size allows for more burst. By default it uses the formula Bc = CIR/32 because it is calculated using a 250msec interval, and the bucket is in bytes, not bits. (The Bc and Be values are in bytes, while the CIR is in bits).
To turn this into a single-rate, three-color policer, you simply add a violate action.
Notice that there is now a Be bucket. This is a single-rate three color policer, so excess tokens from the Bc bucket “spill into” the Be bucket. By default the conform-action and exceed-action are still transmit and drop, respectively.
To configure a dual-rate three-color policer, you add the pir keyword:
Notice that we didn’t have to specify a violate-action. Since we specified the PIR, IOS knows that this has to be a three-color policer, so all three actions are present with their default values. The violate-action is the same as the exceed-action by default.
If you want to transmit but mark-down exceeding traffifc you use the following configuration:
Notice that the violate-action took the exceed-action by default after we changed it. To keep violate as drop we need to set it explicitly.
You can also police only a single class of traffic. This works the same way when policing using LLQ. The below policer only polices scavenger traffic. No action is taken on any other traffic.
Lab
We will re-use the lab from the previous article, but this time use iperf3 to test throughput.
All policy-maps from the previous article have been deleted on R2.
R2 wishes to limit traffic received on Gi2 to 1mbps. To do this, a policer is applied as follows:
I’ll now run iperf server mode on the Server and connect to it from HostA. In iperf, the “client” uploads traffic to the “server.” HostA will send traffic towards the Server.
Notice that 55 retransmits occured within the first second. Checking the policy-map counters on Gi2 (below) we can see that the 71 retransmits directly correlated with the 71 packets dropped by the policer:
To improve preformance, let’s add a shaper to Gi2 of R1. R1 will shape traffic out Gi2 at 1Mbps so that traffic won’t exceed the policer CIR on R2.
We run the iperf test again, and see less retransmissions:
We still see some retransmissions at the 6-7 second interval because iperf tried to go way over the contracted rate at 3.65Mbps.
If we check R1, we can see that these drops were actually caused by the shaper, not the policer:
To improve preformance, let’s lengthen the queue length.
Interestingly this just delayed the point at which iperf ran into retransmissions. Now packet loss happens at the last 9-10 second interval:
Packet drops increased even more on the shaper:
Testing policing/shaping using VyOS
CSR1000v limits throughput to 2Mbps without a license, which makes it very hard to test policing and shaping. I setup a simple three-node VyOS lab and preformed policing and shaping on the middle node, while running iperf between the edge nodes. VyOS is FOSS (free open source software).
With a simple policing policy at 100mbps, the iperf test suffers horribly from the packet loss. The packet loss prevents the sliding window from getting big enough to take advantage of the bandwidth.
If we delete the policer and implement a 100m shaper outbound on the other interface on the middle node, the test is much better:
As you can see, the low bandwidth of 1mbps when testing using CSR1000v hid the problems that policing causes. This is why you want to shape outbound traffic towards a provider that is implementing a policer.
Further Reading
Cisco QoS Exam Guide, Wendell Odom and Michael Vacanaugh, Ch. 6
See additional sections in this “book”
Last updated
