Cisco Nexus – Ping Sweep

The following script can be utilized to execute a ping sweep of a /24 network on Cisco Nexus switch. This has been tested and verified on Cisco Nexus 7K running 6.2.8a code version.

Actual script:

tclsh
for {set i 1} {$i < 255} {incr i} {
cli "ping 192.168.1.$i count 2 time 1 vrf VRF_A"
}

Script execution on a Cisco Nexus 7000 switch:

N7K_SW1# tclsh
N7K_SW1-tcl# for {set i 1} {$i < 255} {incr i} {  
> cli "ping 192.168.1.$i count 2 time 1 vrf VRF_A"
> }

Similar script but copy/pasting specific IP addresses:

tclsh
foreach address {
> 10.90.55.97
> 10.90.55.98
> } { cli ping $address count 2 time 1 vrf VRF_A }

F5 TMM Crash

We have a pair of F5 Viprions that are connected to Cisco Nexus 7K (Aggr A & B) switch as shown here:

network_diagram_1_1

TMM Crash:

The TMM Crashed in one of the F5 Viprions as the following conditions were met:

  1. Your BIG-IP system is processing a large amount of active connections.
  2. You attempt to display the connection table using the tmsh show sys connection command.
  3. You then attempt to cancel the tmsh show sys connection command by using the Ctrl+C key sequence while the command is still in the process of displaying the connection table.

SOL15246

When the Viprion is handling hundreds of thousands of connections and the “show sys  connection” is executed and subsequently cancelled with “Ctrl+C” before the connections are displayed, it will cause the TMM to crash. This is common to multi-blade system  like Viprion and single units.

BugID: 595773

For the Viprions, apart from the TMM crash, the “Ctrl+C” is not propagated to all the blades in the multi-blade chassis Viprion. This has been identified as BugID: 595773. This has been fixed in 11.5.6 code and it may be retroactively fixed in 11.5.4 + HF2 (not sure).

BugID: 579284

Under certain conditions, memory within mcpd can be corrupted. This memory corruption within mcpd has been identified as BugID: 579284. The previously stated BugID: 595773 will trigger BugID: 579284 resulting in memory corruption within mcpd.

The memory corruption was serious enough to cause loss of inter-blade connectivity and thus each blade was acting as a stand-alone system and this caused the packets to loop within the network.

This bug will probably be fixed in 12.x code.

Logs from the Viprion:

 May  3 16:20:15 slot1/LB1-domain.com err tmsh[29166]: 01420006:3: operation canceled
 May  3 16:20:31 slot3/LB1-domain.com crit tmm6[17982]: 01010020:2: MCP Connection aborted, exiting
 May  3 16:20:31 slot4/LB1-domain.com info bcm56xxd[9563]: 012c0012:6: Reprogram vDAG cmp state to 0xb for vtrunk default (previous state 0xf)
 May  3 16:20:31 slot3/LB1-domain.com info bcm56xxd[9919]: 012c0012:6: Reprogram vDAG cmp state to 0xb for vtrunk default (previous state 0xf)
 May  3 16:20:31 slot1/LB1-domain.com info bcm56xxd[8234]: 012c0012:6: Reprogram vDAG cmp state to 0xb for vtrunk default (previous state 0xf)
 ...
 May  3 16:20:31 slot4/LB1-domain.com info bcm56xxd[9563]: 012c0012:6: Reprogram vDAG cmp state to 0x2 for vtrunk default (previous state 0xa)
 May  3 16:20:31 slot1/LB1-domain.com info bcm56xxd[8234]: 012c0012:6: Reprogram vDAG cmp state to 0x2 for vtrunk default (previous state 0xa)
 May  3 16:20:31 slot4/LB1-domain.com info bcm56xxd[9563]: 012c0012:6: FFP HDAG installed for default (cmp state 0x2)
 May  3 16:20:31 slot1/LB1-domain.com info bcm56xxd[8234]: 012c0012:6: FFP HDAG installed for default (cmp state 0x2)
 
 ... and the blade logs a restart.

The following logs were identified in the Cisco Nexus 7K that was connected to the Viprion:

2016 May  3 16:20:26 switch-1 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 4111.3111.abc1 among ports Po66 and Po11 on vlan 100 - Disabling dynamic learning notifications for a period between 120 and 240 seconds on vlan 100
2016 May  3 16:20:33 switch-1 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 4111.3111.a6c1 among ports Po11 and Po66 on vlan 200 - Disabling dynamic learning notifications for a period between 120 and 240 seconds on vlan 200

Summary of the 2 conditions that we hit:

  1. TMM Crash because of the “Ctrl+C” used to break “show sys conn” command.
  2. Ctrl+C does not propagate to all the blades causing memory corruption resulting in loss of inter-blade connectivity and thus making the multi-blade Viprion to create a closed loop.

 

 

Identifying the Right Module – Cisco Nexus

N7K.LON# locator-led ?
chassis               Blink chassis led
fan                      Blink Fan led
module              Blink module led
powersupply   Blink powersupply led
xbar                   Xbar

When you are in a remote location and the data center tech can’t identify the right module, “locator-led” command in Nexus 7000 can be used to identify the right module. In other platforms, “blink” command is utilized.

Automating Cisco Switch Changes

Recently, I was involved in a project to re-architect the VLANs for 1000+ Servers. The idea was to move about 1000+ servers to their new VLANs in batches of about 250 servers. This process involved creating:

  • New VLANs
  • Configuring the ports of multiple switches

Each server was dual-homed to 2 switches (odd & even switches – switch1 & switch2) in different cabinets for redundancy. Some of the batches required making changes to 100+ switches at a time.

I used the following scripts to make the changes:

  • Bash Script to obtain the TACACS username/password in order to log into the switch.
  • Expect Script to create new VLANs.
  • Expect Script to configure the ports.

This is the bash script that is utilized to obtain the TACACS username/password from the user and to  log into each switch in the “switch-list.txt”. “switch-list.txt” is a normal file that contains the list of switches that require configuration changes.

VLAN Configuration Changes:

  1. Create a list of switches that needs to be changed. I utilized “vi” edit tool to open up a file “switch-list.txt” and paste the list of switch names. You can also use the IP addresses of the switches.

Example of switch-list.txt:


 $ cat switch-list.txt
 switch1
 switch2
 switch3
 switch4
 

2. Create a bash script that will use the “switch-list.txt” file, obtain the username/password from the user and utilize the VLANConfig.exp expect script to create VLANs in the switches in the “switch-list.txt” file.


 #!/bin/bash
 # Collect the current user's ssh and enable passwords
 echo -n "Enter the SSH password for $(whoami) "
 read -s -e password
 echo -ne '\n'
 echo -n "Enter the Enable password for $(whoami) "
 read -s -e enable
 echo -ne '\n'
 # Feed the expect script a device list & the collected passwords
 for device in `cat ~/switch-list.txt`; do
 ./VLANConfig.exp $device $password $enable ;
 done
 

3. This “VLANConfig.exp” Expect Script will create the right VLANs and this expect script is utilized in the bash script.


#!/usr/bin/expect -f
# Set variables
 set hostname [lindex $argv 0]
 set username $env(USER)
 set password [lindex $argv 1]
 set enablepassword [lindex $argv 2]
# Log results
 log_file -a ~/results-VLAN1300.log
# Announce which device we are working on and at what time
 send_user "\n"
 send_user ">>>>> Working on $hostname @ [exec date] <<<<" {
 send "enable\n"
 expect "*assword"
 send "$enablepassword\n"
 expect "*#"
 }
 }
# Configuration Changes
 send "conf t\n"
 expect "(config)#"
 send "vlan 1300\n"
 expect "(config-vlan)#"
 send "name VLAN-WEB-1300\n"
 expect "(config-vlan)#"
 send "end\n"
 expect "#"
 send "write mem\n"
 expect "#"
 send "exit\n"
 expect ":~\$"
 exit

PORT Configuration Changes:

  1. This is the bash script that is utilized to collect the username/password and feed it to the expect script that will change the ports.

 #!/bin/bash
 # Collect the current user's TELNET and enable passwords
 echo -n "Enter the TELNET password for $(whoami) "
 read -s -e password
 echo -ne '\n'
 echo -n "Enter the Enable password for $(whoami) "
 read -s -e enable
 echo -ne '\n'
 # Feed the expect script passwords
 ./PORTConfig.exp $password $enable

2. This “PORTConfig.exp” Expect Script will change the relevant ports. In this script, the array contains the switch name and the ports that needs to be changed.


#!/usr/bin/expect -f
# Set variables
 set username $env(USER)
 set password [lindex $argv 0]
 set enablepassword [lindex $argv 1]
# Log results
 log_file -a ~/results-port.log
#Add switch & interfaces
 array set interface {
 switch1 "int range g1/3"
 switch2 "int range g1/3"
 switch3 "int range g1/6, g1/8, g1/10, g1/12"
 switch4 "int range g1/6, g1/8, g1/10, g1/12"
 }
foreach hostname [array names interface] {
 # Announce which device we are working on and at what time
 send_user "\n"
 send_user ">>>>> Working on $hostname @ [exec date] <<<<" {
 send "enable\n"
 expect "*assword"
 send "$enablepassword\n"
 expect "*#"
 }
 }
# Configuration Changes
 send "conf t\n"
 expect "(config)#"
 send "$interface($hostname)\n"
 expect "(config-if-range)#"
 send "switchport access vlan 1300\n"
 expect "(config-if-range)#"
 send "end\n"
 expect "#"
 send "write mem\n"
 expect "#"
 send "exit\n"
 expect ":~\$"
 }
 exit

Reference: Blog

Nexus – Virtual Port Channel (vPC)

Port Channel:

Multiple ports bundled together such that they appear as a single port. Port Channel runs between 2 devices. Port Channel utilizes LACP (Standard) or PAgP (Cisco Proprietary).

Virtual Port Channel:

Multiple ports bundled together such that they appear as a single port. In the case of vPC, the links are connected to two different Cisco Nexus devices and appear as a single Port Channel to a 3rd device.

A simple vPC setup is shown here:

vpc_2

This diagram provides an introduction to vPC terminology:

vPC – Port Channel between vPC peer and the downstream device.

vPC Domain – Pair of vPC Switches.

vPC Peer – A vPC switch in a common vPC Domain.

vPC Member Port – One of a set of ports to form vPC.

vPC Peer Link – Link used to synchronize state between vPC peer devices, must be 10G

vPC Peer Keepalive Link – Keepalive link between vPC peer devices.

vPC VLAN – Any of the VLANs carried over the vPC Peer Link.

Orphan Device – A device on a vPC VLAN but only connected to only one vPC Peer and not both.

Orphan Port – An orphan port is a interface that connects to an Orphan Device.

Reference: 

Cisco Whitepaper on vPC

Nexus – IO Module

Nexus IO Module has 2 main classification – M & F Series.

M Series:

M1 – 1/10 G

M2 – 10/40/100 G

F Series:

F1 – 1G (L2)

F2 – 10G (L2/L3)

F2E – 10G (L2/L3)

F3 – 40G – Closes feature gap between M & F, with fewer exceptions.

As a rule of thumb, M series is used for L2/L3/L4 with large forwarding table requirements. F Series provides high performance, low latency, low cost & low power with mainly L2 features. F2 & later versions provide more of L3 features.

F1 & F2 requires its own VDC (Virtual Device Context). F2E has the ability to share VDC context with other M & F IO Modules. Recommended good practice is to make sure that each VDC has its own IO Module.

Common M/F Features:

Classic L2/L3

QoS Feature Set

VDC

Security Feature Set

Multicast Feature Set

M-Specific Features:

MPLS, OTV & LISP

F-Specific Features:

Fabricpath & FCoE

In the coming days, the functionality that is unique to M-Series will be replicated on the F-Series. Cisco’s comparison can be found here for Nexus 7000.