Cisco Express Forwarding (2023 Edition)

Cisco Express Forwarding (2023 Edition)

CEF Deep Dive (commands) for IOS-XE

C9300 used as example

Model: C9300L-24UXG-4X

Control Plane (CP) = R0

Data Plane (DP) = F0, UADP/ASIC

CEF Background: 

Why Cisco Express Forwarding (CEF)? 

“Necessity is the mother of invention” -proverb

Let’s take a trip in the networking time machine back to simpler times (late 80s, early 90s).

I’m going to start with the fundamentals of software-based forwarding and all the pipeline steps involved. We didn’t have fancy ASICs like we do today, so everything was done with 100% raw CPU horsepower. This was known as “process switching” and was performed on EVERY SINGLE packet.  🥲

  1. I’m a router and receive an ingress frame, is the frame check sequence (FCS) good?
    • Bad FCS = bad hardware, duplex/speed mismatch, misbehaving NIC/drivers, etc
  2. Extract the IP packet
  3. IP packet check: IPv4 checksum | IPv6 skip checksum
  4. Is this IP destination (DST) for me?
    • Yes  = we have reached the final destination, No = continue to step 5
  5. Check TTL: > 1, continue to step 6 | <= 1, drop and send ICMP time exceeded to source (SRC)
  6. Check Routing Information Base (RIB) for DST longest prefix match (LPM)
  7. I need to build a new frame for forwarding, what’s the next hop?
    • *VERY IMPORTANT step in the context of CEF*
  8. Packet is ready to go, need to reduce the TTL by “1” in the IP header and create a new checksum (IPv4)
  9. Send it! Encapsulate packet inside new data link header/trailer and off we go!
    • At this point in the packets journey through this router, the packet is encapsulated into a frame and “on-the wire” in the egress direction.

Enter “Fast Switching” in IOS version 10.x. Which is an optimization feature to “cache” previous process switched packets for subsequent use, thus reducing the CPU/processing.

As a result, the CLI command looks something like this, “ip route-cache” since that’s what the feature of fast switching was doing, caching entries to be used in the future. 

As we all know, caching as a method of optimization has tradeoffs like EVERYTHING we do in engineering. This “route once, switch many times” approach had some glaring issues.  

  1. EVERY SINGLE new packet (unique destination) is process switched, so a large burst can bring the CPU to its knees, begging for mercy.
  2. Stall cache entries needed to be purged and refreshed, this would impact the CPU as the 1st few packets were process switched to create the fresh cache entries 
  3. Load sharing issues since the cache entry only used the 1st outgoing interface and polarized traffic to that single interface
  4. Convergence and new paths

I like to think of web caching as having the same challenges (stale data) to what we experienced with “fast switching”. I’m sure there’s other tradeoffs that I forgot about, however this was an improvement nonetheless vs pure process-based forwarding. 

Enter 1996 and the birth of CEF. 

To further optimize forwarding, we had to focus on overcoming the limitations of “fast switching” and its cached entries. What part of the process is the MOST processor intensive and requires the most computational cycles? 

The answer? Finding the best path for destination networks, the outbound interface to reach that destination and finally build the new data link header (frame rewrite, step #7 above in the process switch pipeline).

This was a very linear process to scan the RIB and find the LPM. Now imagine looking through a 1M entry RIB top to bottom, only to find the 0/0 is the only valid path to the destination and it was at the very end of the lookup? There’s invariably room for improvement and this was the focus of CEF.

Let’s start with the major improvements with CEF

  1. CEF Forwarding Information Base (FIB): Fast lookups using the FIB, derived from the RIB. The FIB is built to optimize fast retrieval of information (prefixes) contained in the RIB and targets LPM match lookups. These prefixes can be stored in software or in the hardware. This information (prefixes) is structured in a tree, focused on optimal data retrieval when talking about software/CP. This information (prefixes) is also optimized for optimal data retrieval/pointers when talking about hardware/DP (TCAM/ASIC).
  2. CEF Adjacency Table: A router could have a million plus routes in its RIB, but only a handful of neighbors (next hops) to reach those destinations. Subsequently, we end up using common next-hop, outgoing interfaces, and frame rewrites for these DSTs.

In process switching this happened for EVERY routed packet, for fast switching per-DST entries were cached. Enter the Adjacency Table and the concept of an “intelligent” router. We can “look ahead” with CEF by dipping into the RIB/ARP table and build the complete forwarding information BEFORE the packets start flowing across the digital highway. Then this state information is preemptively programed into the hardware for forwarding. That’s the magic of CEF!

Summary: 

The CEF FIB and CEF adjacency table are the key components of CEF and have further evolved beyond process and fast switching. 

  • The FIB contains all known IP prefixes from the RIB, organized for rapid lookups via a tree/trie data structure in the control plane (CP).
  • A lookup in the FIB using the packet’s DST IP address as a key produces a pointer into the adjacency table that contains pre-built frame headers for individual next hops, so that they can be instantly applied to the packet. 
  • The state is programed into the hardware/ASIC and then used for optimal forwarding and provides relief to the CP processor (CPU) so that it can preform other duties.
  • This combination of software (CP) and hardware (DP) working in harmony is why CEF is still used 27 years later in the year of our Lord, 2023.

Now, let’s dive into some the details.

On Cisco IOS-XE devices such as the ASR 1000s or Catalyst 9000s, the control plane (CP/General Purpose CPU) is separate from the data plane (DP/ASIC). Whenever an update needs to be passed from the control plane to the data plane, it has to go through the data flow shown in the flow diagram inline.

For example (ASR1000), in case of CEF whenever any prefix is learned on the control plane, this update passes from the control plane (IOSd) to the forwarding manager of the control plane (FMAN-RP). The forwarding manager on the control plane uses kernel utilities like lsmpi, Hyper-transport (HT) links, and so on in order to… <highlighted in BLUE, control plane/software>

pass the update to data plane forwarding manager (FMAN-FP). The forwarding manager sends the update to the Quantum Flow Processor (QFP) ASIC which programs QFP microcode in order to finally program the QFP subsystem which does the actual forwarding of packets in Cisco Aggregation Services Router (ASR 1000) devices. <highlighted in YELLOW, data plane/hardware>

Now it’s time to roll up our sleeves and do some CLI interrogation.

There are various commands you can use to check the CEF state and we will explore some of these for the Catalyst 9000. 

First let’s make sure CEF is enabled (enabled by default).

Let’s look at the CEF entry for “10.0.0.1/32 and 0.0.0.0/0” respectively. <CP/Software>

NOTE: This is how we actually view the Forwarding Information Base (FIB), if you want to view the entire FIB it’s just “show ip cef” <CP/Software>

I will use these two sources (RIB and ARP) to create my OWN CEF adjacency table

  • (RIB) n3tArk_C9300#sh ip route 0.0.0.0 

Routing entry for 0.0.0.0/0, supernet

  Known via “static”, distance 1, metric 0, candidate default path

  Routing Descriptor Blocks:

  * 10.100.200.1  Next Hop 

      Route metric is 0, traffic share count is 1

  • (ARP) n3tArk_C9300#sh ip arp 10.100.200.1

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  10.100.200.1            0   ac17.c85c.bcdc  ARPA   Vlan777

Adjacency Entry

As you can see, I have information in the control plane (CP) and now let’s look at how CEF takes this information and uses it to create an adjacency entry to be used in forwarding for the data plane (DP). <Still CP/Software>

  • n3tArk_C9300#show platform software ip switch 1 r0 cef prefix 0.0.0.0/0 detail 

Forwarding Table

0.0.0.0/0 -> OBJ_ADJACENCY (0x22), urpf: 36

Prefix Flags: Default

OM handle: 0x348036a748

OK, so in my CEF CP side we are good, we have an entry with an adjacency index/ID of “0x22”

Let’s check to see what will be programmed into the data plane at the later stage of the forwarding manager (FMAN) process. Keep in mind that you will see references to “QFP”. This is the ASIC in the ASR1000 platforms, in the Cat9K we use an ASIC called UADP/Doppler, but functional it’s the same. 

Reference: “show platform software” is still technically “control plane”, but represents that the HW handle was created and will be programmed into the forwarding path.

  • n3tArk_C9300#show platform software ip switch 1 f0 cef prefix 0.0.0.0/0 detail 

Forwarding Table

0.0.0.0/0 -> OBJ_ADJACENCY (0x22), urpf: 36

Prefix Flags: Default

aom id: 231, HW handle: (nil) (created)

Great! We can assume this CEF entry is programed into the HW/forwarding path!

It’s out of scope for this blog, but there may be commands to dive deeper into the hardware forwarding programming such as “show platform hardware xxx” or “show ip cef platform xxx”, but you get the idea here.

Interested in what the “re-write” data looks like? 

Take the index ID (0x22) from above and run the following command

  • show platform software adjacency switch 1 f0 index 0x22 #REMEMBER TO USE HEX AND NOT DECIMAL#

Number of adjacency objects: 41

Adjacency id: 0x22 (34)

  Interface: Vlan777, IF index: 40, Link Type: MCP_LINK_IP

  Encap: ac:17:c8:5c:bc:dc:5c:a6:2d:c4:c0:7b:8:0

#FRAME REWRITE: DST MAC|SRC MAC|ETHERTYPE#

  Encap Length: 14, Encap Type: MCP_ET_ARPA, MTU: 1500

  Flags: no-l3-inject

  Incomplete behavior type: None

  Fixup: unknown

  Fixup_Flags_2: unknown

  Nexthop addr: 10.100.200.1

  IP FRR MCP_ADJ_IPFRR_NONE 0

  aom id: 579, HW handle: (nil) (created)

Now let’s look at VLAN 777 adjacency entry!

  • n3tArk_C9300#show adjacency vlan777 detail                          

Protocol Interface                 Address

IP       Vlan777                   10.100.200.1(12)

                                   0 packets, 0 bytes

                                   epoch 0

                                   sourced in sev-epoch 2

                                   Encap length 14

                                   AC17C85CBCDC5CA62DC4C07B0800

# FRAME REWRITE: DST MAC|SRC MAC|ETHERTYPE #

                                   L2 destination address byte offset 0

                                   L2 destination address byte length 6

                                   Link-type after encap: ip

                                   ARP

In closing, we have evolved from process switching, to fast switching, to CEF. Always remember that everything we do in engineering has tradeoffs. CEF is great, but nothing is free and the cost for CEF is memory. In modern router/switch platforms this is not as much of a concern as it was in the past. I just priced out a Catalyst 8300 Edge (I know, it’s a router with “switch” naming nomenclature) and could upgrade the ram to 32GB!

Also, I’m sure there are folks reading this that remembers syslog messages that looked something like this “failed to program the forwarding table”. That’s what happens with memory (TCAM/FIB or RAM/RIB) exhaustion.

Also, there’s been some confusion about the use of actual “switching” or pure Layer 2 frame switching with regards to current CEF implementations.

I think Peter Paluch masterfully summarizes the confusion around this subject. 

Additional commands for reference: 

  • n3tArk_C9300#show ip cef adjacency vlan 777 10.100.200.1 detail 

IPv4 CEF is enabled for distributed and running

VRF Default

 53 prefixes (52/1 fwd/non-fwd)

 Table id 0x0

 Database epoch:        3 (53 entries at this epoch)

0.0.0.0/0, epoch 3, flags [default route]

  recursive via 10.100.200.1

    attached to Vlan777

10.0.0.1/32, epoch 3

  recursive via 10.100.200.1

    attached to Vlan777

10.100.200.1/32, epoch 3, flags [attached]

  Adj source: IP adj out of Vlan777, addr 10.100.200.1 7F200CAF3BE8

    Dependent covered prefix type adjfib, cover 10.100.200.0/24

  1 RR source [no flags]

  attached to Vlan777

  • n3tArk_C9300#show ip cef adjacency vlan 777 10.100.200.1 platform 

0.0.0.0/0

  Platform adj-id: 0x22, 0x0, tun_qos_dpidx:0

10.0.0.1/32

  Platform adj-id: 0x22, 0x0, tun_qos_dpidx:0

10.100.200.1/32

  Platform adj-id: 0x22, 0x0, tun_qos_dpidx:0

Alternatively, you can use this command on IOS-XE/C9300s. 

  • n3tArk_C9300#sh ip cef platform

NOTE: The output of this command can take some time to generate depending on the CEF table size.

Example Output: 

0.0.0.0/0

  Platform adj-id: 0x22, 0x0, tun_qos_dpidx:0

0.0.0.0/8

0.0.0.0/32receive

10.0.0.1/32

  Platform adj-id: 0x22, 0x0, tun_qos_dpidx:0

Also, I want to personally thank Peter P. and Daniel D. for peer reviewing this post. What a great community we have!

References:

Cisco Systems. (2017). (Web Page) Understand and Troubleshoot CEF on Cisco IOS XE Routers
https://www.cisco.com/c/en/us/support/docs/ip/express-forwarding-cef/211377-Understand-and-Troubleshoot-CEF-on-IOS-X.html

D. Peñaloza. (2019). (Blog) Demystifying CEF
https://learningnetwork.cisco.com/s/blogs/a0D3i000002SKKGEA4/demystifying-cef

P. Paluch & D. Dib (2023). (Twitter)

Peter Paluch: Twitter

Daniel Dib: Twitter

David Peñaloza: Twitter

Written By: Shaun Gomez
01.17.2023 Rev3

Comments are closed.