It’s been way too long since I posted on my blog. Well, I have been studying for the CCIE Data Center lab and wanted to pass on some very critical information on fabric extender (FEX) configurations. One of the most common questions that our Cisco friends ask me is “Why do I need to create a port channel for the FEX-Fabric links?”. Well let’s dive into the WHY, and then explore the HOW.

Let’s first start with a foundation on what FEX is and how it works. FEX is a highly scalable, low latency data center access layer solution. What makes it so awesome? The fact that is managed as a line card vs. a separate ToR/EoR/MoR switch. Take your Nexus 7000 (core) or 5000 (agg/access) and they play the role of the “PARENT” switch. The FEX 2000 is essentially a remote line card to the parent switch. The FEX supports 1/10G and FCoE for consolidated I/O.

Nexus 2000 Comparison:

2248-TP (E) : This is the most common FEX, 48 ports of 100/1000 BASE-T host interfaces and 4 x 10G (SFP+) fabric uplinks. The (E) varient has additional shared buffer (32MB) locally vs. using what’s on the parent switch.

2224-TP: Same as above just 24 host ports instead of 48.

2148T: First generation FEX. The host interface can only operate at 1G. Not recommended any longer, go with the 2248-TP instead.

2232PP:  This is our de-facto 10G FEX. It has 32 x 1/10G and FCoE host interfaces (SFP+) and 8 fabric uplinks (SFP+). This switch also supports DCB.

2232TM: There is also a 10G BASE-T varient of the above FEX. THIS ONE DOES NOT SUPPORT FCoE, but does support DCB. 

OK, now that that is out of the way, let’s get back to the question at hand. How do I configure the connectivity to the parent switch?

I’ll get stright to the point. Use EtherChannel for the fabric uplink interfaces.

So, what are my options anyways?

Static Pinning and EtherChannel are your two options.

Why do we like EtherChannel fabric interfaces anyways?

Well the bottom line is that with Static Pinning you do not have automatic failover capability. Sure it’s deterministic, but if one of those fabric links goes down, so do the associated host interfaces. Let’s say your using two links and set the ‘pinning max-links 2’, half of the FEX host interfaces are mapped to one fabric uplink and the other half of the host ports is mapped to the other fabric uplink. Let’s look at a visual representation of static pinning.

The major issue with this configuration is that if the fabric uplink goes down, so do the ports associated to that interface. There is NO failover. This is WHY we want to use EtherChannel instead of static pinning.

Now let’s talk about the other (preferred) solution. EtherChannel is 110% the way to go. Let’s look at the visual representation.

As you can see from this diagram, we are load balancing the host interfaces (HIF) across all the fabric uplinks. Here is a note from the Cisco Configuration Guide.

When you configure the Fabric Extender to use an EtherChannel fabric interface connection to its parent switch, the switch load balances the traffic from the hosts that are connected to the host interface ports by using the following load-balancing criteria to select the link:

• For a Layer 2 frame, the switch uses the source and destination MAC addresses.
• For a Layer 3 frame, the switch uses the source and destination MAC addresses and the source and destination IP addresses.
  And finally the configuration (HOW) for all this awesomeness.
  
  
  1) ENABLE FEX GLOBALLY:
  N5K(config)# feature fex
  2) Configure the member interfaces for FEX connectivity:
  N5K(config)# interface e1/1, e1/2
  N5K(config-if)# switchport mode fex-fabric
  N5K(config-if)# fex associate 100
  N5K(config-if)# channel-group 100
  3) Create the EtherChannel Interface: This probably is done already based on the previous command, but it doesn’t hurt to make sure.
  N5K(config)# interface po100
  N5K(config-if)# switchport mode fex-fabric
  N5K(config-if)# fex associate 100
  4) Create the FEX:
  N5K(config)# fex 100
  N5K(config-fex)# description FEX_100<<PO100>>
  N5K(config-fex)# pinning max-links 1 (This must 1 for an EtherChannel configuration, if you change this to any other number you cannot use EC and your using static pinning instead)
  FOUR EASY STEPS!!!!
  Here are some good troubleshooting commands.
  “sh fex”
  “sh fex detail” – This command will show the HIF to Fabric Uplink Mappings and the version of code running on the FEX.
  “sh interface fex-fabric”  – This command will display all the FEX units attached to the parent switch.
  “sh inventory fex xxx” Display the inventory information about a specific FEX.
  “show diagnostic result fex 100″ – Display diagnostic test results for a specific FEX.

NOTE: Nexus 7000 (Parent) FEX

If your trying to use FEX on the N7K be certain to issue the following commands or FEX WILL NOT BE ENABLED.

In the DEFAULT VDC issue this command “install feature-set fex”
Now switch to the VDC that you want to enable FEX on and issue this command “feature-set fex”

Your all set now.

Cisco.com Configuration Guide for Nexus 2000:
http://tinyurl.com/36uojrv