Dec 12 20:57:36 lb-A err statsd[6711]: 011b0203:3:
Error ‘No such file or directory’ opening file /sys/block/sdb/stat
Dec 12 20:57:36 lb-A err statsd[6711]: 011b0900:3:
TMSTAT error max disk stat: read failed.
The error is because the F5 is trying to query for stats from a removed hard drive or USB. Bugid 441400 is associated with this error. This has been fixed post 11.6 code version and for older code versions, reboot fixes this error.
I ran into an issue where the big3d daemon was restarting continuously on an F5 running LTM only (No GTM). The following article details the steps that were taken to solve the restart issue. The solution was achieved after raising a support case with F5 Networks.
I tried to restart the daemon (big3d & httpd)
(tmos)#restart /sys service big3d
(tmos)#restart /sys service httpd
and tried stopping and starting using the following commands but it did not help in preventing the continuous restarts:
(tmos)#stop /sys service httpd
(tmos)#stop /sys service big3d
(tmos)#start /sys service httpd
(tmos)#start /sys service big3d
I ran the following command from tmsh:
(tmos)#load sys config
and it resulted in the following error:
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration …
BIGpipe unknown operation error:
01070920:3: Application error for confpp: Syntax OK
The certificate does not match the key. To change them try ‘bigpipe httpd { sslcertfile /etc/httpd/conf/ssl.crt/server.crt sslcertkeyfile /etc/httpd/conf/ssl.key/server.key }’
*************************************************************
Sep 9 22:56:52 localhost confpp[9878]: syntax check command FAILURE for unix_config_httpd returned: ‘2304’
Restarting syslog-ng:
Shutting down syslog-ng: [ OK ]
Starting syslog-ng: [ OK ]
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
[FAILED]ing httpd: [FAILED]
VERIFY Device Cert & Key:
As seen in the output of the “load sys config” command, the cert & key did not match. The following command (run from bash) is utilized to verify if the cert & key match or not:
openssl rsa -in /etc/httpd/conf/ssl.key/server.key -modulus -noout | openssl md5
openssl x509 -in /etc/httpd/conf/ssl.crt/server.crt -modulus -noout | openssl md5
The md5 hash was different indicating that the cert & key did not match. I used the following solution guide in order to generate new cert/key pair: SOL9114
openssl req -x509 -nodes -days <# of days> -newkey rsa:<keysize> -keyout /config/httpd/conf/ssl.key/server.key -out /config/httpd/conf/ssl.crt/server.crt
Note: Replace <# of days> with the number of days in year increments for which you want the certificate to be valid. I used 3650 days (10 years).
bigstart restart httpd
cat /config/httpd/conf/ssl.crt/server.crt >> /config/big3d/client.crt
Note: Alternatively, you can add the new certificate to the trusted device certificate file and remove all old certificates by running the following command:
cat /config/httpd/conf/ssl.crt/server.crt > /config/big3d/client.crt
cat /config/httpd/conf/ssl.crt/server.crt >> /config/gtm/server.crt
Note: Alternatively, you can add the new certificate to the trusted server certificate file and remove all old certificates by typing the following command:
cat /config/httpd/conf/ssl.crt/server.crt > /config/gtm/server.crt
After creating the right cert/key pair, the non-stop restarts of big3d stopped. After preventing the restarts, I had issues with GUI not loading the Virtual Server option and had to perform a full box reboot in order for the GUI to function without any issues.
Reference:
F5 Viprions have different blade hardware versions. There isn’t a simple command that will help you identify the blade model. I have found this to be useful:
(/Common)(tmos)# show /sys hardware field-fmt | grep -e platform -e marketing
sys hardware platform {
marketing-name BIG-IP VPR-C2400
platform A112
Platform A112 is B2250 blade
Platform A113 is B2150 blade
Platform A109 is B2100 blade
Reference for platform:
https://support.f5.com/kb/en-us/products/big-ip-afm/releasenotes/product/relnote-afm-11-4-1.print.html
https://devcentral.f5.com/questions/f5-viprion-blade-model
This post provides some tmsh commands that has helped me in saving time:
Set connection limit to all pool members in a pool:
(tmos.ltm)#modify pool POOL_WEB members modify { all { connection-limit 300 } }
F5 was being polled by a server. Some of the OIDs were working while others didn’t work. Restarting snmpd didn’t help. Restarting both snmpd & subsnmpd solved the problem and all the OIDs were working again.
(tmos)# restart sys service snmpd
(tmos)# restart sys service subsnmpd
(tmos)# show sys service snmpd subsnmpd
snmpd run (pid 31980) 3 minutes, 4 starts
subsnmpd run (pid 927) 2 minutes, 1 restart
SOL8035 has information on the BigIP Daemons.
F5 provides 3 different CLI navigation option:
TMSH (tmos)#
BASH #
bpsh >
TMSH or Traffic Management Shell is the newer shell that is utilized to manage the F5 via CLI. BASH is used for running linux like commands with “b” as the 1st letter (#b pool show)
With the newer v11 code version, F5 is moving more towards the TMSH and has stopped developing bpsh. If you are looking to learn CLI, it is recommended that you learn TMSH on F5 rather than BASH or bpsh
To move into TMSH, type “tmsh” from BASH or bpsh
To move into BASH from TMSH, type “run util bash”
To move into bpsh from BASH, type “bpsh”
Pre-Maintenance Checks:
Make sure that the F5 is running in “Volume Partition” mode. “lvscan” within bash should provide output like this:
config # lvscan
ACTIVE ‘/dev/vg-db-sda/dat.share.1’ [30.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.log.1’ [7.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.swapvol.1’ [1.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1.root’ [392.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._usr’ [2.48 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._config’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.1._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.2.root’ [256.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._usr’ [1.34 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._config’ [512.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.2._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.3.root’ [256.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._usr’ [1.34 GB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._config’ [512.00 MB] normal
ACTIVE ‘/dev/vg-db-sda/set.3._var’ [3.00 GB] normal
ACTIVE ‘/dev/vg-db-sda/dat.maint.1’ [300.00 MB] normal
Make sure there are no HTTP Classes in your configuration, other than the default “httpclass” by checking Local Traffic ›› Profiles : Protocol : HTTP Class from the F5 GUI.
Make sure that there are no spaces in profile naming. SOL15144
As a precaution, go through your configuration and remove any unwanted/unused configuration elements like a “Test Virtual Server” or configuration from the past that is not in use at the moment.
Load the code version to the F5 LTM via GUI, SCP or any other preferred method.
Maintenance:
Before performing any F5 code upgrade, make sure that the “Service Check Date” on the device is AFTER the License Check Date for the new code version as listed here in SOL7727
If not, the maintenance would include a license re-activation step before proceeding with code upgrade. This step would take about 10-20 minutes.
cpcfg to the new code version location – Example: cpcfg HD1.2
Although “cpcfg HD1.x” has worked most of the times, I would recommend backing up the .UCS file in a remote location and also saving a copy in “/shared/tmp/<UCS File>“. After saving the UCS file in the “/shared/tmp/” location, you can utilize “load /sys ucs <path/to/UCS> no-license” to load the configuration as noted in SOL12880
Reboot. This will take about 20 minutes for the device to load the new configuration and come back up. If you are using HA F5, upgrade the Standby F5 first. It will take a few minutes for the Standby F5 to become “Active”. So, be patient.
Conservative estimate for the maintenance window is about 1 hour. I would recommend giving yourself 90 minutes, if you are not familiar with F5 code upgrade. Downtime can be minimized if you have BigIP F5 in High Availability Active/Standby Pair.
when HTTP_REQUEST {
set URI [string tolower [HTTP::uri]]
if {$URI starts_with "/m/" }{
set NEW_URI [string range [HTTP::uri] 2 end]
HTTP::respond 301 Location http://www.domain.com$NEW_URI
}
}
$ curl -Ik http://10.10.10.10/m/URI
HTTP/1.0 301 Moved Permanently
Location: http://www.domain.com/URI
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
Replacing
set NEW_URI [string range [HTTP::uri] 2 end]
with
set NEW_URI [string trimleft [HTTP::uri] /m/]
will provide the following output:
$ curl -Ik http://10.10.10.10/m/URI
HTTP/1.0 301 Moved Permanently
Location: http://www.domain.comURI
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
Note the lack of “/” in the “Location” header, just before the start of URI.
In order to retain the “/”, I did a string trimleft with “/m” instead of “/m/” i.e.,
set NEW_URI [string trimleft [HTTP::uri] /m]
instead of
set NEW_URI [string trimleft [HTTP::uri] /m/]
However, the result was the same.
Based on this devcentral article, it looks like “string range” is a better option as “string trim” tends to trim individual characters as instead of the “string of characters”.
Another issue to remember is that if we use “string trim”, we get the following output:
$ curl -Ik http://10.10.10.10/m/mURI
HTTP/1.0 301 Moved Permanently
Location: http://www.domain.comURI
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
and “string range” provides the following:
$ curl -Ik http://10.10.10.10/m/mURI
HTTP/1.0 301 Moved Permanently
Location: http://www.domain.com/mURI
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
So, if you are just looking to remove specific “string of characters”, “string range” is a better option based on testing on 11.x code version.
when HTTP_REQUEST {
set HOST [string tolower [HTTP::host]]
if { ($HOST eq "domain.com" || $HOST eq "www.domain.com" ) and
([HTTP::header User-Agent] contains "Chrome/33.0.175") and
([HTTP::header User-Agent] contains "(X11; Linux x86_64)") } {
reject
}
}
The above iRule will enable the F5 to send a TCP RST (reset) to the client for a specific domain and specific User-Agent information within the incoming HTTP Request.
F5 performs health-checks or monitors by utilizing the status code or the content in the response from the servers to mark the server as UP or DOWN.
It is relatively easy to mark the server UP/DOWN by using a single status code in the “Receive String” attribute within the Monitor:
200 OK
If you want to mark a server up for multiple status code like using 200-209, you can use this:
HTTP/1.[01] [2]0[0-9]
In order to mark the server UP for status code 200-209 or 401
HTTP/1.[01] ([2]0[0-9]|401)
