We ended yesterday being confused about how to make the LAN connection between our Cisco 6500 core swicth and the 6120 Fabric Interconnect. The physical layout is 6500 has 2 – 4 port 10 G modules using the XenPack-10GB-SR model (there are many Xenpack options, make sure you pick the correct one for your environment). We installed 50 micron laser optimized fiber jumpers between the gear with an LC connector for the 6120 side and a SC connector for the 6500 side (again know which connectors you need for both ends, if you have the wrong one it can be a drag). We setup an Etherchannel by using 2 – 10 G connections for each fabric interconnects. The physical connections were the easy part. The 6500 is a classic Ethernet switch with a lot of functionality. The 6120 Fabric Interconnect is a layer 2 switch (based on the Nexus 5000 line) plus other things like a fiber channel switch and all the management and smarts for the UCS system. So is the 6120 handled like a Ethernet switch? But servers or blades do not connect directly to the 6120 because they plug into the fabric-extenders contained in the chassis. We were trying to wrap our heads around this new way of approaching LAN connectivity while using what we normally do as a frame of reference. Here is what we figured out (others who were there can help correct me if I am off based with some of what I am saying . . .)
The 6500 is the core layer 3 switch and we want to trunk many VLANs to the 6120 system. We also use VLAN 1 on our LAN as a valid network, which is a little different. 6500 terms: On a trunk port the native VLAN is untagged. All other VLANs that are trunk are tagged. native = untagged
The 6120 has a physical connection to the LAN via the northbound 6500 switch. Southbound, the 6120 is connected to the chassis via the fabric-extenders. The blades are connected to the fabric-extenders. 6120 terms: In the LAN (global) setting you define all the VLANs that have been trunked from the 6500 to the 6120 and you define one VLAN as “default”. Note, “default” can be any VLAN ID number, it does not have to be 1. What you have defined as “default” will be handled as untagged traffic, all other VLANs will be passed with a tag. On the LAN tab global setting “default” = untagged
Each blade in our configuration has an I/O interface card (CNA) with 2 Ethernet and 2 HBA ports. We use a service profile, created via the UCS manager (running on the 6120) to define vNICs (as well as other things associated with a blade). Service Profile terms: In the Service Profile’s for the vNICs, if you want to trunk many VLANs (like you would for a VMWare ESX host) you can select any number of VLANs that have been defined in the global setting and then you select one VLAN to be “native”. In the Service Profile the VLAN set to “native” is the VLAN the blade will boot on and it will send traffic untagged. In the Service Profile vNIC the VLAN selected as “native” = untagged
How to Configure the LAN:
After some trial and error (strange results) with different combinations on each peice of equipment regarding “native”, “default”, etc. and talking with Cisco engineers, we determined the following correct configuration for VLANs. 6500 we trunk 15 VLANs and set VLAN 1 as “native” (there is real traffic on VLAN 1 in our enviroment). 6120 we defined all 15 VLANs that are trunked from the 6500 and set VLAN 1 to be ”default”. Service Profile for a VMWare ESX host, we select the 15 VLANs that are needed for guest servers and then select the “ESX Service Console” VLAN as “native”.
How does this work?
When the VMWare ESX host boots using the above Service Profile it will boot on what is defined as the “native” VLAN. This “native” VLAN is known to the 6120 as VLAN 778 and knows to trunk that VLAN northbound to the 6500 switch. The 6500 then is able to route that traffic appropriately and the return traffic will come back via VLAN 778.
6500 “native” = 6120 “default” = Service Profile “native”