F5 – Automating CLI Execution

Purpose:
This is a really simple way to automate CLI command execution on multiple F5 devices using Bash & TCL scripting. The scripts have been tested on a linux and a mac machine.

How to use it:
There is a bash script (F5_Bash_v1) that is utilized to collect the username/password for F5 access. A text file (F5_Host.txt) that stores the management IP address of multiple F5 devices and a TCL script (F5_Out_v1.exp) that is used to execute CLI commands on the F5 devices.

The bash script is the master script that obtains the username/password and executes the TCL script for multiple F5 devices.

Setup:
On a linux machine that is utilized to connect to the F5 device:

#Create a directory
mkdir F5_Check

Within the “F5_Check” directory, create the following 3 files:
F5_Host.txt
F5_Bash_v1
F5_Out_v1.exp

File Content: F5_Host.txt contains the management IP of the F5 devices.
Example:

$ cat F5_Host.txt
10.12.12.200
10.12.12.201
10.12.12.202
10.12.12.203

File Content: F5_Bash_v1

#!/bin/bash
# Collect the username and password for F5 access
echo -n "Enter the username "
read -s -e user
echo -ne '\n'
echo -n "Enter the password "
read -s -e password
echo -ne '\n'

# Feed the expect script a device list & the collected username & passwords
for device in `cat ~/F5_Check/F5_Host.txt`; do
./F5_Out_v1.exp $device $password $user ;
done

File Content: F5_Out_v1.exp

#!/usr/bin/expect -f

# Set variables
set hostname [lindex $argv 0]
set password [lindex $argv 1]
set username [lindex $argv 2]

# Log results
log_file -a ~/F5_Check/F5LOG.log

# Announce which device we are working on and the time
send_user "\n"
send_user ">>>>>  Working on $hostname @ [exec date] <<<<<\n"
send_user "\n"

# SSH access to device
spawn ssh $username@$hostname

expect {
"no)? " {
send "yes\n"
expect "*assword: "
sleep 1
send "$password\r"
}
"*assword: " {
sleep 1
send "$password\r"
}
}

expect "(tmos)#"
send "sys\n"
expect "(tmos.sys)#"

send "show software\n"
expect "#"
send "exit\n"
expect "#"
send "quit\n"

expect ":~\$"
exit

F5 v11.x Device Trust Group

A week ago, I was upgrading HA F5 pair from 11.5.1 to 11.5.3 and noticed the existence of default “device_trust_group” in sync-only mode in GUI. I did not create it but it just showed up and there wasn’t a way to delete it. Apparently, this always existed in the background but was exposed via GUI in the later 11.x versions. Based on my experience, it wasn’t exposed via GUI in 11.5.1 but was exposed via GUI from 11.5.6

Device_Trust_Group

Reference: DevCentral

F5 Pool & Nodes

A Node is an IP address. Example: 10.10.10.10

A Pool Member is an IP Address + Port. Example: 10.10.10.10:8080

A Pool is a collection of Pool Members.

If you are managing an enterprise grade F5 infrastructure, there may come a time when you may have to replace a specific IP address with another IP address or replace multiple IP addresses in an F5 or multiple F5 devices.

This is a quick one-liner that will help you to identify all the pools that contain an IP address:

tmsh -q list ltm pool one-line | grep -E '($node_hostname|$node_ip)' | awk '{ print $3 }'

The above command should be run from “bash”.

Accessing F5’s bash:

root@LB1(/S1-green-P:Active)(tmos)# run util bash

[root@LB1:/S1-green-P:Active] ~ #

 NOTE: The “list ltm pool one-line” is available in 11.x code and not available in 10.x code. The command will list each pool in a single line.

An Example:

[root@LB1:/S1-green-P:Active] ~ # tmsh -q list ltm pool one-line | grep -E '10.10.10.19' | awk '{ print $3 }'
 POOL_ta_lt_http_private
 POOL_ta_lt_private
 POOL_ta_lt_public
 POOL_ta_lt-maintainance
 POOL_ta_lt-private
 POOL_ta_lt-public

Reference: Devcentral – Pool

What if you want the pool member alongside the pool ?

tmsh -q list ltm pool one-line | egrep -E "$check:[0-9]+"  | while read line; do myipport=$(echo $line | egrep -oE "$check:[0-9]+"); echo $line | awk '{printf "%s ",$3}'; echo "$myipport "; done

In the above line, replace “$check” with the IP Address that you are checking.

[root@LB1:/S1-green-P:Active] ~ #tmsh -q list ltm pool one-line | egrep -E "10.10.10.19:[0-9]+"  | while read line; do myipport=$(echo $line | egrep -oE "10.10.10.19:[0-9]+"); echo $line | awk '{printf "%s ",$3}'; echo "$myipport "; done

POOL_ta_lt_http_private 10.10.10.19:10542 
POOL_ta_lt_public 10.10.10.19:10253 
POOL_ta_lt_maintainance 10.10.10.19:10251 
POOL_ta_lt_private 10.10.10.19:10092 
POOL_ta_lt_public 10.10.10.19:10093

F5 CLI Display Length

While running cli commands on F5, you may run into display length issues:

(tmos.ltm)# show pool members | grep "10.10.10.10:"
Display all 158 items? (y/n) y

If you are executing the script on the F5 to obtain data, the "Display all xxx items? (y/n)" could be a problem. We can alter the display threshold using the following command:

(/Common)(tmos)# modify cli preference pager disabled display-threshold ?

Specifies the maximum number of objects that tmsh will display without requiring a user response to the question, "Display all <number> items? (y/n)". You can specify from 0 (zero) through 4,294,967,265 objects. 0 (zero) specifies that tmsh will display any number of objects without the warning.

(/Common)(tmos)#  modify cli preference pager disabled display-threshold 0

 

F5 – big3d restarting

I ran into an issue where the big3d daemon was restarting continuously on an F5 running LTM only (No GTM). The following article details the steps that were taken to solve the restart issue. The solution was achieved after raising a support case with F5 Networks.

I tried to restart the daemon (big3d & httpd)

(tmos)#restart /sys service big3d

(tmos)#restart /sys service httpd

and tried stopping and starting using the following commands but it did not help in preventing the continuous restarts:

(tmos)#stop /sys service httpd 

(tmos)#stop /sys service big3d

(tmos)#start /sys service httpd

(tmos)#start /sys service big3d

I ran the following command from tmsh:

(tmos)#load sys config

and it resulted in the following error:

Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration …
BIGpipe unknown operation error:
01070920:3: Application error for confpp: Syntax OK
The certificate does not match the key.  To change them try ‘bigpipe httpd { sslcertfile /etc/httpd/conf/ssl.crt/server.crt sslcertkeyfile /etc/httpd/conf/ssl.key/server.key }’
*************************************************************
Sep  9 22:56:52 localhost confpp[9878]: syntax check command FAILURE for unix_config_httpd returned: ‘2304’
Restarting syslog-ng:
Shutting down syslog-ng: [  OK  ]
Starting syslog-ng: [  OK  ]
Shutting down ntpd: [  OK  ]
Starting ntpd: [  OK  ]
[FAILED]ing httpd: [FAILED]

VERIFY Device Cert & Key:

As seen in the output of the “load sys config” command, the cert & key did not match. The following command (run from bash) is utilized to verify if the cert & key match or not:

openssl rsa -in /etc/httpd/conf/ssl.key/server.key -modulus -noout | openssl md5

openssl x509 -in /etc/httpd/conf/ssl.crt/server.crt -modulus -noout | openssl md5

The md5 hash was different indicating that the cert & key did not match. I used the following solution guide in order to generate new cert/key pair: SOL9114

  1. Log in to the bash command line.
  2. Generate the new device certificate and key using the following syntax:

    openssl req -x509 -nodes -days <# of days> -newkey rsa:<keysize> -keyout /config/httpd/conf/ssl.key/server.key -out /config/httpd/conf/ssl.crt/server.crt

    Note: Replace <# of days> with the number of days in year increments for which you want the certificate to be valid. I used 3650 days (10 years).

  3. Enter the certificate attributes.
  4. Restart the httpd process by typing the following command:

    bigstart restart httpd

  5. Copy the new self-signed certificate to the trusted device certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/big3d/client.crt

    Note: Alternatively, you can add the new certificate to the trusted device certificate file and remove all old certificates by running the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/big3d/client.crt

  6. (BIG-IP GTM and BIG-IP Link Controller) Copy the new self-signed certificate to the trusted server certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/gtm/server.crt

    Note: Alternatively, you can add the new certificate to the trusted server certificate file and remove all old certificates by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/gtm/server.crt

After creating the right cert/key pair, the non-stop restarts of big3d stopped. After preventing the restarts, I had issues with GUI not loading the Virtual Server option and had to perform a full box reboot in order for the GUI to function without any issues.

Reference:

SOL10999

SOL13444

F5 Upgrade from v10 to v11 – Lessons Learned

Pre-Maintenance Checks:

Make sure that the F5 is running in “Volume Partition” mode. “lvscan” within bash should provide output like this:

config # lvscan
ACTIVE ‘/dev/vg-db-sda/dat.share.1’ [30.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.log.1’ [7.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.swapvol.1’ [1.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1.root’ [392.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._usr’ [2.48 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._config’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.2.root’ [256.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._usr’ [1.34 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._config’ [512.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.3.root’ [256.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._usr’ [1.34 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._config’ [512.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.maint.1’ [300.00 MB] normal

Make sure there are no HTTP Classes in your configuration, other than the default “httpclass” by checking Local Traffic  ››  Profiles : Protocol : HTTP Class from the F5 GUI.

Make sure that there are no spaces in profile naming. SOL15144

As a precaution, go through your configuration and remove any unwanted/unused configuration elements like a “Test Virtual Server” or configuration from the past that is not in use at the moment.

Load the code version to the F5 LTM via GUI, SCP or any other preferred method.

Maintenance:

Before performing any F5 code upgrade, make sure that the “Service Check Date” on the device is AFTER the License Check Date for the new code version as listed here in SOL7727

If not, the maintenance would include a license re-activation step before proceeding with code upgrade. This step would take about 10-20 minutes.

cpcfg to the new code version location – Example: cpcfg HD1.2

Although “cpcfg HD1.x” has worked most of the times, I would recommend backing up the .UCS file in a remote location and also saving a copy in “/shared/tmp/<UCS File>“. After saving the UCS file in the “/shared/tmp/” location, you can utilize “load /sys ucs <path/to/UCS> no-license” to load the configuration as noted in SOL12880

Reboot. This will take about 20 minutes for the device to load the new configuration and come back up. If you are using HA F5, upgrade the Standby F5 first. It will take a few minutes for the Standby F5 to become “Active”. So, be patient.

Conservative estimate for the maintenance window is about 1 hour. I would recommend giving yourself 90 minutes, if you are not familiar with F5 code upgrade. Downtime can be minimized if you have BigIP F5 in High Availability Active/Standby Pair.

F5 Email Alerts

F5 has the ability to send out an email for various events like the failure of a pool member. For 10.x code version, Postfix within F5 can be utilized to send out emails. Beginning in 11.x code version, Postfix is not utilized. We would have to utilize SSMTP with mail-server: SOL13180

A Simple Step-by-Step Configuration Process (10.x Code Version):

  • Save the Existing Configuration & Create Configuration Archive before making any changes to the existing configuration.

Get CLI-TMSH access to the F5 device:

(tmos)#save sys config partitions all

  • Make sure F5 is configured with DNS Servers:

(tmos)#list sys dns

If there are no DNS Servers configured, add your DNS Servers and make sure F5 can access them:

(tmos)#modify sys dns name-servers add {1.1.1.1}

  • Make sure that PostFix is running from tmsh:

show sys service postfix
start sys service postfix
modify sys service postfix add
list sys service | grep regexp Boot postfix

  • Run a test email from F5 from bash:

echo test | mail
postfix flush
mailq

  • Using SNMP Traps:

Default SNMP traps are available under /etc/alertd/alert.conf

User Configured Alerts are set up here: /config/user_alert.conf

NEVER change the default SNMP traps file – /etc/alertd/alert.conf

The following default SNMP trap for pool member going down/up has been copied from the /etc/alertd/alert.conf file and then copied to /config/user_alert.conf file

cat /config/user_alert.conf will provide the following result:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
snmptrap OID=”.1.3.6.1.4.1.3375.2.4.0.10″;
email toaddress=”f5notifications@gmail.com”
fromaddress=”root”
body=”Member Status Change Notification – DOWN”
}
alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP {
snmptrap OID=”.1.3.6.1.4.1.3375.2.4.0.11″;
email toaddress=”f5notifications@gmail.com”
fromaddress=”root”
body=”Member Status Change Notification – UP”
}

In the above alerts, “email toaddress” has been changed to the email address that is expected to receive the alerts from F5.

“fromaddress” is F5’s email address. Leave this at default settings “root”

“body” can be set to any information that needs to be conveyed.

After making the above changes, restart the alertd daemon from bigip bash & save the configuration:

bigstart restart alertd
b load
b save

In the above example, if you want to create customer alerts based on the logs: /var/log/ltm:

alert  "" {
   snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.XXX"
}

In the above configuration, “log message” can be obtained from /var/log/ltm that you want to match in order to trigger the email. The SNMP trap OID with “xxx” at the end has to have a number greater than 300 for customer SNMP alerts for F5.

Gotchas:

Make sure to create relevant PTR record for the F5 in order to prevent the emails from being marked as SPAM

Emails can sometimes get queued for a long time on the F5 and consume all the memory on the F5. In order to prevent this, the “queue_minfree” has to be changed to something else from the default zero:

Default is no memory limitation:

# postconf -d | grep queue_minfree

queue_minfree = 0

10M Bytes of storage is provided by the following command:

# echo “queue_minfree = 10000000” >> /etc/postfix/main.cf postconf -e “queue_minfree = 10000000”

Reference:

SOL3727

F5 Error – bcm56xxd

Oct 20 10:08:16 local/lb01 err bcm56xxd[3506]: 012c0011:3: I2C SFP operation failed: dev:0xee at :228, errmsg(Operation timed out)

This specific error is benign. bcm56xxd has a thread that checks every 5 seconds to see if there is any units plugged or unplugged on all the SFP and XFP ports by accessing the GPIO pins attached to these SFP/XFP ports via the I2C bus.

The plug check does not disrupt the traffic flow on an existing port.

If the system is busy some I2C operation may time out as the I2C bus becomes overloaded.

bcm56xxd – The switch daemon configures and controls the Broadcom 56xx switch chips.

Reference:

SOL13444

BigIP F5 – Poodle Vulnerability

Poodle was initially targeted against SSLv3. Quite a few websites fixed this issue at the server and client side by disabling SSLv3. There is a variation of Poodle for TLS with the following CVE ID: CVE-2014-8730. For a brief description of the issue: Poodle on TLS

This is known to affect load balancers like F5. F5 recommends a code upgrade. As of now (Dec 09, 2014), it is recommended that the code is upgraded to at least 10.2.4 with HotFix 10 if you are running 10.x code version and one of the 11.x code versions that is not vulnerable as per F5 documentation: SOL15882

F5 has stated that the code upgrade is the best possible option available. 

BigIP F5:

If you are running F5 LTM on 11.x code version, using 11.5.1 with HF6 and above is recommended.

If you are running F5 LTM on 10.x code version and want to stay in 10.x code version, you would have to perform the following tasks in order to have a “A-” rating in Qualys SSL test:

  • Upgrade code to 10.2.4 HF10
  • After code upgrade to 10.2.4 HF10, complete the following steps in order to remove RC4.
(tmos.ltm)# create profile client-ssl PARENT-SSL-SECURE ciphers 'HIGH:MEDIUM:!SSLv3:!RC4'

(tmos.ltm)# modify profile client-ssl CUSTOM-CLIENT-SSL defaults-from PARENT-SSL-PROFILE-SECURE

Qualys Rating is “B” after code upgrade and “A-” after performing code upgrade and removing RC4.

After making the code upgrade & removing RC4 cipher, it is recommended that you test your site for any vulnerabilities at the Qualys Site.

The above commands will create a “Parent” SSL Client profile – “PARENT-SSL-SECURE” that will disable SSLv3, RC4 and order the ciphers from High to Medium strength. Please, note that any client initiating connection on SSLv3 will be dropped. Usually clients running Windows XP and IE6 will initiate SSLv3 connections.

In order to know the total SSLv3 connections, you can run this command:

(tmos.ltm)# show profile client-ssl CUSTOM-CLIENT-SSL | grep Protocol
 Protocol
 SSL Protocol Version 2                       0
 SSL Protocol Version 3                      156
 TLS Protocol Version 1.0                   16.1K
 TLS Protocol Version 1.2                   11.6K
 DTLS Protocol Version 1                      0

Based on the above output, it can be seen that the client SSL profile configured on the Virtual Server handling SSL traffic has received 156 SSLv3 connection requests.

The stats can be cleared using this command:

(tmos.ltm)# reset-stats profile client-ssl

(tmos.ltm)# show profile client-ssl CUSTOM-CLIENT-SSL | grep Protocol
 Protocol
 SSL Protocol Version 2                       0
 SSL Protocol Version 3                       0
 TLS Protocol Version 1.0                     0
 TLS Protocol Version 1.2                   11.6K
 DTLS Protocol Version 1                      0

Note: The “reset-stats” command will NOT clear the stats for TLS1.2  in 10.2.3 code version. F5 is aware of this bug. I was able to clear the stats for 10.2.4 code version. I am not sure if this bug exists for 10.2.3 and lower code versions.

The cipher suite that is being utilized, after the change can be identified by running the following command in bash:

[root@lbal1:Active] config # tmm --clientciphers 'HIGH:MEDIUM:!SSLv3:!SSLv2:!RC4:!COMPAT:!EXP'

Cipher-Suite-F5-TMM

NOTE:

After making the above changes, some of the older browsers may not be able to connect to your website on HTTPS as the older browsers don’t support TLS1.2. For a list of browsers and supported protocols, see here.

Having dealt with SSL/TLS vulnerabilities, it looks like TLS1.2 with Ephemeral Diffie-Hellman is the most secure current version. Anything before TLS1.1 would be insecure and would have to be avoided, if possible. The main problem is that there are quite a few client browsers that are just incompatible with TLS1.2 and this has to be factored before making any changes.

GHOST Vulnerability:

This is a recent vulnerability – CVE-2015-0235

According to SOL16057, if you want to stay with 10.x code version on F5 LTM, it is better to use 10.2.4 + HF11. For 11.x code version, use at least 11.5.1 + HF8.

Update:

For a great Qualys grade see the following link.

!SSLv2:!EXPORT:!DHE+AES-GCM:!DHE+AES:!DHE+3DES:ECDHE+AES-GCM:ECDHE+AES:RSA+AES-GCM:RSA+AES:ECDHE+3DES:RSA+3DES:-MD5:-SSLv3:-RC4:@STRENGTH