F5 Error – No such file or directory

Dec 12 20:57:36 lb-A err statsd[6711]: 011b0203:3:
Error ‘No such file or directory’ opening file /sys/block/sdb/stat

Dec 12 20:57:36 lb-A err statsd[6711]: 011b0900:3:
TMSTAT error max disk stat: read failed.

The error is because the F5 is trying to query for stats from a removed hard drive or USB. Bugid 441400 is associated with this error. This has been fixed post 11.6 code version and for older code versions, reboot fixes this error.

F5 – big3d restarting

I ran into an issue where the big3d daemon was restarting continuously on an F5 running LTM only (No GTM). The following article details the steps that were taken to solve the restart issue. The solution was achieved after raising a support case with F5 Networks.

I tried to restart the daemon (big3d & httpd)

(tmos)#restart /sys service big3d

(tmos)#restart /sys service httpd

and tried stopping and starting using the following commands but it did not help in preventing the continuous restarts:

(tmos)#stop /sys service httpd 

(tmos)#stop /sys service big3d

(tmos)#start /sys service httpd

(tmos)#start /sys service big3d

I ran the following command from tmsh:

(tmos)#load sys config

and it resulted in the following error:

Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration …
BIGpipe unknown operation error:
01070920:3: Application error for confpp: Syntax OK
The certificate does not match the key.  To change them try ‘bigpipe httpd { sslcertfile /etc/httpd/conf/ssl.crt/server.crt sslcertkeyfile /etc/httpd/conf/ssl.key/server.key }’
*************************************************************
Sep  9 22:56:52 localhost confpp[9878]: syntax check command FAILURE for unix_config_httpd returned: ‘2304’
Restarting syslog-ng:
Shutting down syslog-ng: [  OK  ]
Starting syslog-ng: [  OK  ]
Shutting down ntpd: [  OK  ]
Starting ntpd: [  OK  ]
[FAILED]ing httpd: [FAILED]

VERIFY Device Cert & Key:

As seen in the output of the “load sys config” command, the cert & key did not match. The following command (run from bash) is utilized to verify if the cert & key match or not:

openssl rsa -in /etc/httpd/conf/ssl.key/server.key -modulus -noout | openssl md5

openssl x509 -in /etc/httpd/conf/ssl.crt/server.crt -modulus -noout | openssl md5

The md5 hash was different indicating that the cert & key did not match. I used the following solution guide in order to generate new cert/key pair: SOL9114

  1. Log in to the bash command line.
  2. Generate the new device certificate and key using the following syntax:

    openssl req -x509 -nodes -days <# of days> -newkey rsa:<keysize> -keyout /config/httpd/conf/ssl.key/server.key -out /config/httpd/conf/ssl.crt/server.crt

    Note: Replace <# of days> with the number of days in year increments for which you want the certificate to be valid. I used 3650 days (10 years).

  3. Enter the certificate attributes.
  4. Restart the httpd process by typing the following command:

    bigstart restart httpd

  5. Copy the new self-signed certificate to the trusted device certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/big3d/client.crt

    Note: Alternatively, you can add the new certificate to the trusted device certificate file and remove all old certificates by running the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/big3d/client.crt

  6. (BIG-IP GTM and BIG-IP Link Controller) Copy the new self-signed certificate to the trusted server certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/gtm/server.crt

    Note: Alternatively, you can add the new certificate to the trusted server certificate file and remove all old certificates by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/gtm/server.crt

After creating the right cert/key pair, the non-stop restarts of big3d stopped. After preventing the restarts, I had issues with GUI not loading the Virtual Server option and had to perform a full box reboot in order for the GUI to function without any issues.

Reference:

SOL10999

SOL13444

Identifying the Viprion Blade

F5 Viprions have different blade hardware versions. There isn’t a simple command that will help you identify the blade model. I have found this to be useful:


(/Common)(tmos)# show /sys hardware field-fmt | grep -e platform -e marketing
sys hardware platform {
marketing-name BIG-IP VPR-C2400
platform A112


Platform A112 is B2250 blade

Platform A113 is B2150 blade

Platform A109 is B2100 blade

Reference for platform:

https://support.f5.com/kb/en-us/products/big-ip-afm/releasenotes/product/relnote-afm-11-4-1.print.html

https://devcentral.f5.com/questions/f5-viprion-blade-model

Automating Cisco Switch Changes

Recently, I was involved in a project to re-architect the VLANs for 1000+ Servers. The idea was to move about 1000+ servers to their new VLANs in batches of about 250 servers. This process involved creating:

  • New VLANs
  • Configuring the ports of multiple switches

Each server was dual-homed to 2 switches (odd & even switches – switch1 & switch2) in different cabinets for redundancy. Some of the batches required making changes to 100+ switches at a time.

I used the following scripts to make the changes:

  • Bash Script to obtain the TACACS username/password in order to log into the switch.
  • Expect Script to create new VLANs.
  • Expect Script to configure the ports.

This is the bash script that is utilized to obtain the TACACS username/password from the user and to  log into each switch in the “switch-list.txt”. “switch-list.txt” is a normal file that contains the list of switches that require configuration changes.

VLAN Configuration Changes:

  1. Create a list of switches that needs to be changed. I utilized “vi” edit tool to open up a file “switch-list.txt” and paste the list of switch names. You can also use the IP addresses of the switches.

Example of switch-list.txt:


 $ cat switch-list.txt
 switch1
 switch2
 switch3
 switch4
 

2. Create a bash script that will use the “switch-list.txt” file, obtain the username/password from the user and utilize the VLANConfig.exp expect script to create VLANs in the switches in the “switch-list.txt” file.


 #!/bin/bash
 # Collect the current user's ssh and enable passwords
 echo -n "Enter the SSH password for $(whoami) "
 read -s -e password
 echo -ne '\n'
 echo -n "Enter the Enable password for $(whoami) "
 read -s -e enable
 echo -ne '\n'
 # Feed the expect script a device list & the collected passwords
 for device in `cat ~/switch-list.txt`; do
 ./VLANConfig.exp $device $password $enable ;
 done
 

3. This “VLANConfig.exp” Expect Script will create the right VLANs and this expect script is utilized in the bash script.


#!/usr/bin/expect -f
# Set variables
 set hostname [lindex $argv 0]
 set username $env(USER)
 set password [lindex $argv 1]
 set enablepassword [lindex $argv 2]
# Log results
 log_file -a ~/results-VLAN1300.log
# Announce which device we are working on and at what time
 send_user "\n"
 send_user ">>>>> Working on $hostname @ [exec date] <<<<" {
 send "enable\n"
 expect "*assword"
 send "$enablepassword\n"
 expect "*#"
 }
 }
# Configuration Changes
 send "conf t\n"
 expect "(config)#"
 send "vlan 1300\n"
 expect "(config-vlan)#"
 send "name VLAN-WEB-1300\n"
 expect "(config-vlan)#"
 send "end\n"
 expect "#"
 send "write mem\n"
 expect "#"
 send "exit\n"
 expect ":~\$"
 exit

PORT Configuration Changes:

  1. This is the bash script that is utilized to collect the username/password and feed it to the expect script that will change the ports.

 #!/bin/bash
 # Collect the current user's TELNET and enable passwords
 echo -n "Enter the TELNET password for $(whoami) "
 read -s -e password
 echo -ne '\n'
 echo -n "Enter the Enable password for $(whoami) "
 read -s -e enable
 echo -ne '\n'
 # Feed the expect script passwords
 ./PORTConfig.exp $password $enable

2. This “PORTConfig.exp” Expect Script will change the relevant ports. In this script, the array contains the switch name and the ports that needs to be changed.


#!/usr/bin/expect -f
# Set variables
 set username $env(USER)
 set password [lindex $argv 0]
 set enablepassword [lindex $argv 1]
# Log results
 log_file -a ~/results-port.log
#Add switch & interfaces
 array set interface {
 switch1 "int range g1/3"
 switch2 "int range g1/3"
 switch3 "int range g1/6, g1/8, g1/10, g1/12"
 switch4 "int range g1/6, g1/8, g1/10, g1/12"
 }
foreach hostname [array names interface] {
 # Announce which device we are working on and at what time
 send_user "\n"
 send_user ">>>>> Working on $hostname @ [exec date] <<<<" {
 send "enable\n"
 expect "*assword"
 send "$enablepassword\n"
 expect "*#"
 }
 }
# Configuration Changes
 send "conf t\n"
 expect "(config)#"
 send "$interface($hostname)\n"
 expect "(config-if-range)#"
 send "switchport access vlan 1300\n"
 expect "(config-if-range)#"
 send "end\n"
 expect "#"
 send "write mem\n"
 expect "#"
 send "exit\n"
 expect ":~\$"
 }
 exit

Reference: Blog

F5 SNMP Problems

F5 was being polled by a server. Some of the OIDs were working while others didn’t work. Restarting snmpd didn’t help. Restarting both snmpd & subsnmpd solved the problem and all the OIDs were working again.

(tmos)# restart sys service snmpd
(tmos)# restart sys service subsnmpd

(tmos)# show sys service snmpd subsnmpd
snmpd run (pid 31980) 3 minutes, 4 starts
subsnmpd run (pid 927) 2 minutes, 1 restart

SOL8035 has information on the BigIP Daemons.

RST & FIN Out of Order

There was a constant increase in “overrun” and “input errors” on the Cisco ASA Interface. Upon examination, using “show asp drop“, “tcp-rstfin-ooo” & “tcp-3whs-failed” were constantly increasing.

ASP-DROP

Using the following to capture real-time traffic, the IP addresses and the ports can be identified:

# capture ASP type asp-drop tcp-rstfin-ooo buffer 2048 real-time

In this case, we were able to isolate port 5666 for Nagios servers as a culprit in sending RST after FIN and this was breaking the TCP protocol. This is an environment with 100s of Servers that was monitored by Nagios. When 100s of Servers end up sending RST simultaneously, it can turn out to be a mini-self-DOS. With older Firewalls & Code Versions, this can cause reboots. When we searched online, we were able to identify the following bugs on Nagios:

https://bugs.launchpad.net/ubuntu/+source/nagios-nrpe/+bug/989156

http://tracker.nagios.org/view.php?id=305

F5 CLI

F5 provides 3 different CLI navigation option:

TMSH  (tmos)#

BASH   #

bpsh     >

TMSH or Traffic Management Shell is the newer shell that is utilized to manage the F5 via CLI. BASH is used for running linux like commands with “b” as the 1st letter (#b pool show)

With the newer v11 code version, F5 is moving more towards the TMSH and has stopped developing bpsh. If you are looking to learn CLI, it is recommended that you learn TMSH on F5 rather than BASH or bpsh

To move into TMSH, type “tmsh” from BASH or bpsh

To move into BASH from TMSH, type “run util bash”

To move into bpsh from BASH, type “bpsh”

Nexus – Virtual Port Channel (vPC)

Port Channel:

Multiple ports bundled together such that they appear as a single port. Port Channel runs between 2 devices. Port Channel utilizes LACP (Standard) or PAgP (Cisco Proprietary).

Virtual Port Channel:

Multiple ports bundled together such that they appear as a single port. In the case of vPC, the links are connected to two different Cisco Nexus devices and appear as a single Port Channel to a 3rd device.

A simple vPC setup is shown here:

vpc_2

This diagram provides an introduction to vPC terminology:

vPC – Port Channel between vPC peer and the downstream device.

vPC Domain – Pair of vPC Switches.

vPC Peer – A vPC switch in a common vPC Domain.

vPC Member Port – One of a set of ports to form vPC.

vPC Peer Link – Link used to synchronize state between vPC peer devices, must be 10G

vPC Peer Keepalive Link – Keepalive link between vPC peer devices.

vPC VLAN – Any of the VLANs carried over the vPC Peer Link.

Orphan Device – A device on a vPC VLAN but only connected to only one vPC Peer and not both.

Orphan Port – An orphan port is a interface that connects to an Orphan Device.

Reference: 

Cisco Whitepaper on vPC

Nexus – IO Module

Nexus IO Module has 2 main classification – M & F Series.

M Series:

M1 – 1/10 G

M2 – 10/40/100 G

F Series:

F1 – 1G (L2)

F2 – 10G (L2/L3)

F2E – 10G (L2/L3)

F3 – 40G – Closes feature gap between M & F, with fewer exceptions.

As a rule of thumb, M series is used for L2/L3/L4 with large forwarding table requirements. F Series provides high performance, low latency, low cost & low power with mainly L2 features. F2 & later versions provide more of L3 features.

F1 & F2 requires its own VDC (Virtual Device Context). F2E has the ability to share VDC context with other M & F IO Modules. Recommended good practice is to make sure that each VDC has its own IO Module.

Common M/F Features:

Classic L2/L3

QoS Feature Set

VDC

Security Feature Set

Multicast Feature Set

M-Specific Features:

MPLS, OTV & LISP

F-Specific Features:

Fabricpath & FCoE

In the coming days, the functionality that is unique to M-Series will be replicated on the F-Series. Cisco’s comparison can be found here for Nexus 7000.