If you have multiple datacenters or a multi tenant fibre channel environment and you’re using Cisco FC switches, it’s a best practice to use VSANs to separate the configurations of each location / tenant. To allow storage arrays and / or hosts in different VSANs to communicate with each other Inter VSAN Routing needs to be used.
If you need to have 2 EMC VNX storage arrays “talk” to each other for MirrorView for example over 2 or more datacenters (for data replication purposes that is) or hosts in one DC talk to storage in another DC, using transit VSANs (and therefore IVR) will keep your VSANs with equipment indoors and the slightly more vulnerable VSAN outdoors. If some farmer with his tractor rips your single mode fiber, only the outdoor VSAN will be fractured and the indoor VSANs remain unharmed. And of course communication between the remote sites is interrupted, but the indoor VSANs / fabrics remain unchanged.
In order to use Inter VSAN Routing (IVR) you need IVR capable switches, like the MDS 92xx series, 95xx series or 97xx series OR the new MDS 9148(S) switches and with the 9124 and 9134 going EOL now ALL Cisco MDS switches can be made IVR capable by adding the right license. Assuming you already have some sort of IVR in place, what happens when you want to add new or replace existing IVR switches?
In the configuration of these switches a thing called IVR topology is present. This IVR topology is simply put a list of switches and the VSANs they are connected to. In a way it’s a list of wwns and the VSANs they represent on their side of the topology.
Furthermore you can have automatic IVR topology or manual IVR topology. The easiest is the automatic, since it will automatically recognise any VSAN in the topology and if an IVR capable switch can get to that VSAN, that VSAN will be added to the list of VSANs that switch can route to. Now that was a tongue breaker, but it’s not that difficult to understand. Let’s show you an example of what such a config looks like:
Run the command:
FC-CORE-01# sh run | i topology n 5
(this means do a complete listing of the running config and only show the word “topology” and the next 5 lines for each time that word shows up.
This is the output in my example:
ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:0e:ec:82:62:c0 vsan-ranges 1-5,50
autonomous-fabric-id 1 switch-wwn 20:00:00:2b:6b:b4:81:40 vsan-ranges 50,101-105
ivr vsan-topology auto
zone mode enhanced vsan 1
zone mode enhanced vsan 2
zone mode enhanced vsan 3
zone mode enhanced vsan 4
zone mode enhanced vsan 5
This means that the IVR VSAN topolgy is set to auto according to the line “ivr vsan-topology auto”
Also the switch with wwn “20:00:00:0e:ec:82:62:c0” (FC-CORE-01) can see VSANs 1 to 5 and 50, where I must add that 50 is the VSAN connecting this switch to the other IVR capable switch. This VSAN 50 is the so called transit VSAN.
The switch with wwn “20:00:00:2b:6b:b4:81:40” can see VSANs 50 and 101 to 105 and here VSAN 50 is again that so called transit VSAN connecting both IVR switches.
In this configuration any IVR zone with a wwpn from any of the VSANs 1 to 5, 50 and 101 to 105 will be routed by one of the two switches. So if you have an EMC VNX using MirrorView in VSAN 2 which needs to replicate its data to another VNX in VSAN 102, the first IVR switch will route traffic to VSAN 50 and the second IVR switch will route it to VSAN 102. Another possibility is to configure these MirrorView ports in VSAN 50, so no routing will have to take place at all, but that’s another tweak I’ve been using for some time now.
Now to add a new IVR capable switch to the topology and delete the old one, follow these steps:
FC-CORE-01# config t
Enter configuration commands, one per line. End with CNTL/Z.
FC-CORE-01(config)# ivr vsan-topology database
FC-CORE-01(config-ivr-topology-db)# show ivr
Inter-VSAN Routing is enabled
Inter-VSAN enabled switches
—————————
AFID VSAN DOMAIN CAPABILITY SWITCH WWN
——————————————————————-
1 1 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
1 2 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
1 3 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
1 4 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
1 5 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
1 50 0x78(120) 0000001f 20:00:00:2b:6b:b4:81:40
1 50 0x82(130) 0000001f 20:00:00:2c:6a:b4:84:b0 *
Total: 7 IVR-enabled VSAN-Domain pairs
Inter-VSAN topology status
————————–
Current Status: Inter-VSAN topology is ACTIVE, AUTO Mode
Last activation time: Mon Apr 27 11:07:52 2015
Inter-VSAN zoneset status
————————-
name : IVR_ZONESET
state : activation success
last activate time : Mon Apr 27 11:08:43 2015
Fabric distribution status
———————–
fabric distribution enabled
Last Action Time Stamp : Mon Apr 27 11:02:45 2015
Last Action : Distribution Enable
Last Action Result : Success
Last Action Failure Reason : noneInter-VSAN NAT mode status
————————–
FCID-NAT is enabled
Last activation time : Mon Apr 27 11:07:51 2015
Now that you have confirmed that the local switch can actually see VSANs 1 to 5 and 50 and that the remote switch can see VSAN 50 as well, you can continue the actual configuration:
FC-CORE-01(config-ivr-topology-db)# autonomous-fabric-id 1 switch-wwn 20:00:00:2c:6a:b4:84:b0 vsan-ranges 1-5,50
fabric is now locked for configuration. Please ‘commit’ configuration when done.
FC-CORE-01(config-ivr-topology-db)# ivr commit
commit initiated. check ivr status
FC-CORE-01(config)# sh run | i topology n 5
ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:0e:ec:82:62:c0 vsan-ranges 1-5,50
autonomous-fabric-id 1 switch-wwn 20:00:00:2b:6b:b4:81:40 vsan-ranges 50,101-105
autonomous-fabric-id 1 switch-wwn 20:00:00:2c:6a:b4:84:b0 vsan-ranges 1-5, 50
ivr vsan-topology auto
zone mode enhanced vsan 50
zone mode enhanced vsan 101
zone mode enhanced vsan 102
zone mode enhanced vsan 103
zone mode enhanced vsan 104
zone mode enhanced vsan 105
So we’ve added the new switch with wwn “20:00:00:2c:6a:b4:84:b0” to the config and we’ve verified that this topology now has 3 switches, one of which is the old to be deleted switch (wwn “20:00:00:0e:ec:82:62:c0”).
Now let’s delete that old switch from the configuration:
FC-CORE-01(config)# ivr vsan-topology database
FC-CORE-01(config-ivr-topology-db)# no autonomous-fabric-id 1 switch-wwn 20:00:00:0e:ec:82:62:c0
fabric is now locked for configuration. Please ‘commit’ configuration when done.
FC-CORE-01(config-ivr-topology-db)# ivr commit
commit initiated. check ivr status
FC-CORE-01(config)# sh run | i topology n 5
ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:2b:6b:b4:81:40 vsan-ranges 50,101-105
autonomous-fabric-id 1 switch-wwn 20:00:00:2c:6a:b4:84:b0 vsan-ranges 1-5,50
ivr vsan-topology auto
zone mode enhanced vsan 50
zone mode enhanced vsan 1
zone mode enhanced vsan 2
zone mode enhanced vsan 3
zone mode enhanced vsan 4
zone mode enhanced vsan 5
By adding the word “no” to the “autonomous”-fabric-id” line, this line (switch) will be deleted from the configuration as soon as it’s committed and that’s exactly what we did by typing “ivr commit”. The check of the config indeed now shows only 2 switches are left in the configuration and the new local switch is showing VSANs 1-5 and 50.
If you want more control of the VSANs that can be routed, you need to set the IVR VSAN-topology to manual by using the activate command. This will activate the local configuration and negate the auto setting:
FC-CORE-01(config)# ivr vsan-topology database
FC-CORE-01(config)# ivr vsan-topology activate
fabric is now locked for configuration. Please ‘commit’ configuration when done.
FC-CORE-01(config)# ivr commit
commit initiated. check ivr status
FC-CORE-01(config)# sh run | i topology n 5
ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:2b:6b:b4:81:40 vsan-ranges 50,101-105
autonomous-fabric-id 1 switch-wwn 20:00:00:2c:6a:b4:84:b0 vsan-ranges 1-5,50
zone mode enhanced vsan 50
zone mode enhanced vsan 1
zone mode enhanced vsan 2
zone mode enhanced vsan 3
zone mode enhanced vsan 4
zone mode enhanced vsan 5
That’s it! The line “ivr vsan-topology auto” is now gone, which means it’s set to manual in this topology (so also for the remote switch).
Happy routing!!
0 Comments.