F5 logs can be seen here: /var/log/ltm
From TMSH, run the following command to move to bash:
root@lbal1(Active)(tmos)# run util bash
[root@lbal1:Active] ~ # cd /var/log
[root@lbal1:Active] log # cat ltm
/var/log/ltm has the logs for the past 24 hours. The logs available are rotated every 24 hours and past logs are stored under /var/log as: ltm.1.gz, ltm.2.gz etc
If you want to view the past logs, you can run the following command:
[root@lbal1:Active] log # zcat ltm.1.gz
The above GIF shows a simplified Round-Robin load balancing method. In this load balancing method, the load balancer doesn’t care about the number of connections that are being handled by the server. The 1st connection is sent to the 1st server, 2nd connection to 2nd server and 3rd connection to 3rd server without any regard to the pre-existing connections.
Least Connections:
The above GIF shows a simplified Least-Connections load balancing method. As the name suggests, the Load Balancer will send the connections to the server with the least number of connections.
In the initial state of the GIF, the servers, have 5,4,3 connections as noted. The 1st connection is sent to the server with the least number of connections – S3. The 2nd connection is sent to S2 as S1 has 5 and S2 & S3 are tied for number of connections at 4. The 3rd connection is sent to S3 as S1 & S2 have more number of connections (5) than S2 (4). In the last image, all the servers have equal number of connections at 5.
When using CSW Rule with POST method and load balancing based on incoming URI, you could run into a defect that prevents the right configuration. By default, “case-insensitive” CSW Rules would have to be added to “case-insensitive” CSW Policies. However, “case-insensitive” key word at the end of the CSW Rule for HTTP Method is NOT recognized by the Brocade ADX. Most browsers send the HTTP Method in Uppercase and hence, Brocade Engineers did not design the CSW Rule for HTTP Method to work with “case-insensitive” keyword.
Virtual ADX(config)#csw-rule “POST_Method” method eq “POST” case-insensitive
Virtual ADX(config)#csw-policy “CSW_POLICY” case-insensitive
Virtual ADX(config-CSW_POLICY)#match “POST_Method” forward 1
Rule “POST_Method” does not match policy “CSW_POLICY” case sensitivity type
What if you want to combine HTTP Method related CSW Rule (case-sensitive) with normal L7 load balancing rules based on case-insensitive URI ?
In this case, you can only create “case-sensitive” CSW Policy and not “case-insensitive” CSW Policy as the CSW Rule for Method doesn’t have a “case-insensitive” option. In effect, the URI that is utilized within the CSW Rule has to be “case-sensitive” and won’t match any case.
I wanted this configuration (case-insensitive):
csw-rule “POST_Method” method eq “POST” case-insensitive
csw-rule “URI_ADMIN” url pattern “/ADMIN/” case-insensitive
csw-rule “URI_STAGE” url pattern “/STAGE/” case-insensitive
csw-policy “CSW_POLICY” case-insensitive
match “URI_ADMIN” forward 1026
match “URI_STAGE” forward 1026
match “POST_Method” forward 1027
Settled For This Configuration (case-sensitive):
csw-rule “POST_Method” method eq “POST”
csw-rule “URI_ADMIN” url pattern “/admin/”
csw-rule “URI_STAGE” url pattern “/stage/”
csw-policy “CSW_POLICY”
match “URI_ADMIN” forward 1026
match “URI_STAGE” forward 1026
match “POST_Method” forward 1027
Issue seen: 12.4D code.
Create a new persistence profile with cookie name “MYCOOKIE”
ltm persistence cookie PROF_MYCOOKIE {
cookie-name MYCOOKIE
defaults-from cookie
}
Use the following iRule with the Virtual Server:
when CLIENT_ACCEPTED {
set COOKIE_NAME "MYCOOKIE"
set PASSPHRASE "Secret1234"
}when HTTP_RESPONSE {
if { [HTTP::cookie exists $COOKIE_NAME] } {
HTTP::cookie encrypt $COOKIE_NAME $PASSPHRASE
}
}when HTTP_REQUEST {
if { [HTTP::cookie exists $COOKIE_NAME] } {
set DECRYPTED [HTTP::cookie decrypt $COOKIE_NAME $PASSPHRASE]
if { ($DECRYPTED eq "") } {
# Cookie wasn't encrypted, delete it
HTTP::cookie remove $COOKIE_NAME
}
}
}
It is essential to understand that the cookie encryption is useful in masking the information about the F5 Load Balancer and the real-servers serving the content. If someone can capture the encrypted cookie, they can still use it for malicious purposes even without decrypting it.
You can utilize a different cookie name and pass phrase in the above iRule. Just remember to use the same cookie name in the iRule and in the cookie persistence profile
There are ways to implement cookie persistence using the HTTP profile in the F5 LTM. However, there can be issues with utilizing this as there is a bug that prevents proper functioning when the cookie name as a "." (period) in it.
Reference:
With the newer Brocade ADX code versions (12.4G+), it is possible to alter the TCP profile settings that is associated with the Virtual Server.
Why Change the TCP Profile:
I have run into issues when there is slowness in data transfer between the client and the Brocade ADX. Usually, I have seen this happen under 2 configuration setting:
When we enable CSW or terminate SSL on the Brocade ADX, it tends to act as a “Full Proxy” device with TCP stack for Client-Side connection and another TCP stack for Server-Side connection. Somehow this setting will result in sub-optimal performance.
A sample TCP Profile:
tcp profile CLIENT_TCP
nagle off
delayed-ack off
rxbuf-size 524288
txbuf-size 2516544
tcp-wnd-scale 6
tcp profile SERVER_TCP
nagle off
rxbuf-size 2516544
txbuf-size 524288
tcp-wnd-scale 6
SSH@vADX(config)#server virtual VS-1.1.1.1
SSH@vADX(config-vs-VS-1.1.1.1)#port http tcp-proxy CLIENT_TCP SERVER_TCP
“CLIENT_TCP” profile changes the TCP profile setting that is client facing and the “SERVER_TCP” profile changes the TCP profile setting that is server facing. We can utilize the same TCP profile settings for both client and the server side, if required.
Ideally, I would recommend using 12.4U code version with the following TCP profile:
tcp profile TCP-PROFILE
delayed-ack off
rxbuf-size 1024000
txbuf-size 1024000
tcp-wnd-scale 4
tcp-timestamp on
tcp-sack on
The last 2 options for “timestamp” and “sack” are provided in 12.4U code version and are not available in earlier code versions.
The TCP profile can also be altered at the SSL profile level as noted here.
ssl profile star.domain.com
keypair-file star.domain.com-key16
certificate-file star.domain.com-crt16
cipher-suite all-cipher-suites
enable-certificate-chaining
tcp-profile TCP-PROFILE
session-cache off
In the above case, the TCP profile is added to the SSL profile and this will be attached to the Virtual Server. With newer code version, we can directly attach to the TCP profile to the Virtual Server’s port using the “tcp-proxy” keyword as illustrated earlier.
F5 has the ability to send out an email for various events like the failure of a pool member. For 10.x code version, Postfix within F5 can be utilized to send out emails. Beginning in 11.x code version, Postfix is not utilized. We would have to utilize SSMTP with mail-server: SOL13180
A Simple Step-by-Step Configuration Process (10.x Code Version):
Get CLI-TMSH access to the F5 device:
(tmos)#save sys config partitions all
(tmos)#list sys dns
If there are no DNS Servers configured, add your DNS Servers and make sure F5 can access them:
(tmos)#modify sys dns name-servers add {1.1.1.1}
show sys service postfix
start sys service postfix
modify sys service postfix add
list sys service | grep regexp Boot postfix
echo test | mail
postfix flush
mailq
Default SNMP traps are available under /etc/alertd/alert.conf
User Configured Alerts are set up here: /config/user_alert.conf
NEVER change the default SNMP traps file – /etc/alertd/alert.conf
The following default SNMP trap for pool member going down/up has been copied from the /etc/alertd/alert.conf file and then copied to /config/user_alert.conf file
cat /config/user_alert.conf will provide the following result:
alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
snmptrap OID=”.1.3.6.1.4.1.3375.2.4.0.10″;
email toaddress=”f5notifications@gmail.com”
fromaddress=”root”
body=”Member Status Change Notification – DOWN”
}
alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP {
snmptrap OID=”.1.3.6.1.4.1.3375.2.4.0.11″;
email toaddress=”f5notifications@gmail.com”
fromaddress=”root”
body=”Member Status Change Notification – UP”
}
In the above alerts, “email toaddress” has been changed to the email address that is expected to receive the alerts from F5.
“fromaddress” is F5’s email address. Leave this at default settings “root”
“body” can be set to any information that needs to be conveyed.
After making the above changes, restart the alertd daemon from bigip bash & save the configuration:
bigstart restart alertd
b load
b save
In the above example, if you want to create customer alerts based on the logs: /var/log/ltm:
alert "" {
snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.XXX"
}
In the above configuration, “log message” can be obtained from /var/log/ltm that you want to match in order to trigger the email. The SNMP trap OID with “xxx” at the end has to have a number greater than 300 for customer SNMP alerts for F5.
Gotchas:
Make sure to create relevant PTR record for the F5 in order to prevent the emails from being marked as SPAM
Emails can sometimes get queued for a long time on the F5 and consume all the memory on the F5. In order to prevent this, the “queue_minfree” has to be changed to something else from the default zero:
Default is no memory limitation:
# postconf -d | grep queue_minfree
queue_minfree = 0
10M Bytes of storage is provided by the following command:
# echo “queue_minfree = 10000000” >> /etc/postfix/main.cf postconf -e “queue_minfree = 10000000”
Reference:
Universal or UIE Persistence is one of the great features that is available with F5 devices. UIE stands for Universal Inspection Engine. Universal Persistence provides us with the ability to load balance traffic based on the information available in the HEADER or the CONTENT of the incoming packet.
When to use it:
Usage Example:
Sometimes, you may want to have persistence to exist based on a “Session Cookie” that is inserted by the Server. This “Session Cookie” could be utilized within the Header or URI of incoming packet and this may also be utilized within the content, when you are using the HTTP POST method.
Configuration Steps:
A simple example based on X-Forwarded-For header:
when HTTP_REQUEST {
if {[HTTP::header X-Forwarded-For] != ""}{
persist uie [HTTP::header X-Forwarded-For] 600
}
}
Use the above iRule within a UIE persistence profile for persistence based on the incoming X-Forwarded-For header value. “600” denotes the timeout in seconds and this can also be set in the UIE persistence profile.
Reference:
The following iRule taken from devcentral.f5.com was utilized to insert the “Secure” tag to all the cookies within the Response Header. Note that some part of the iRule has been “deactivated” as this part involves adding the “HTTPOnly” cookie tag which isn’t required for this customer. I was able to verify its functionality in 10.x code version (also works in 11.x):
when HTTP_RESPONSE {
if { [catch { set CK_VALUE [HTTP::header values "Set-Cookie"]
HTTP::header remove "Set-Cookie"
foreach value $CK_VALUE {
if { "" != $value } {
set TEST_VALUE [string tolower $value]
set LENGTH_VALUE [string length $value]
switch -glob $TEST_VALUE {
"*;secure*" -
"*; secure*" { }
default { set value "$value; Secure"; }
}
switch -glob $TEST_VALUE {
"*;httponly*" -
"*; httponly*" { }
default { set value "$value; HttpOnly"; }
}
HTTP::header insert "Set-Cookie" $value
}
}
} ] } {
log local0. "Exception thrown, client_addr=[client_addr] HttpOnly and/or Secure cookie attributes may not have been set"
}
}
For 11.x code version, there is a simpler way of achieving this functionality as noted here: https://devcentral.f5.com/wiki/iRules.HTTP__cookie.ashx
HTTP::cookie httponly <name> [enable|disable] HTTP::cookie secure <name> [enable|disable]
When the “HTTP::cookie” rules were tried out in a lab environment using the following iRule:
when HTTP_RESPONSE {
HTTP::cookie version BigIP_Cookie 1
HTTP::cookie httponly BigIP_Cookie enable
}
I got the following error log:
TCL error: /Common/iRule-HTTPOnly-Secure <HTTP_RESPONSE> – Illegal argument (line 1) invoked from within “HTTP::cookie version BigIP_Cookie 1”
So, I removed the “cookie version” command:
when HTTP_RESPONSE {
HTTP::cookie httponly BigIP_Cookie enable
}
and got the following error:
TCL error: /Common/iRule-HTTPOnly-Secure – Improper version (line 1) invoked from within “HTTP::cookie httponly BigIP_Cookie enable”
It looks like the “HTTP::coookie version” function isn’t working as expected in 11.x code version. F5 Bug-id for this issue 338981.
This is an alternate iRule that seems to be working in 11.x code version
when HTTP_RESPONSE {
set COOKIE_VAL [HTTP::header values "Set-Cookie"]
HTTP::header remove "Set-Cookie"
foreach COOKIE_NAME $COOKIE_VAL {
HTTP::header insert "Set-Cookie" "${COOKIE_NAME}; Secure; HttpOnly"
}
}
Reference: Devcentral.
Oct 20 10:08:16 local/lb01 err bcm56xxd[3506]: 012c0011:3: I2C SFP operation failed: dev:0xee at :228, errmsg(Operation timed out)
This specific error is benign. bcm56xxd has a thread that checks every 5 seconds to see if there is any units plugged or unplugged on all the SFP and XFP ports by accessing the GPIO pins attached to these SFP/XFP ports via the I2C bus.
The plug check does not disrupt the traffic flow on an existing port.
If the system is busy some I2C operation may time out as the I2C bus becomes overloaded.
bcm56xxd – The switch daemon configures and controls the Broadcom 56xx switch chips.
Reference:
Poodle was initially targeted against SSLv3. Quite a few websites fixed this issue at the server and client side by disabling SSLv3. There is a variation of Poodle for TLS with the following CVE ID: CVE-2014-8730. For a brief description of the issue: Poodle on TLS
This is known to affect load balancers like F5. F5 recommends a code upgrade. As of now (Dec 09, 2014), it is recommended that the code is upgraded to at least 10.2.4 with HotFix 10 if you are running 10.x code version and one of the 11.x code versions that is not vulnerable as per F5 documentation: SOL15882
F5 has stated that the code upgrade is the best possible option available.
BigIP F5:
If you are running F5 LTM on 11.x code version, using 11.5.1 with HF6 and above is recommended.
If you are running F5 LTM on 10.x code version and want to stay in 10.x code version, you would have to perform the following tasks in order to have a “A-” rating in Qualys SSL test:
(tmos.ltm)# create profile client-ssl PARENT-SSL-SECURE ciphers 'HIGH:MEDIUM:!SSLv3:!RC4' (tmos.ltm)# modify profile client-ssl CUSTOM-CLIENT-SSL defaults-from PARENT-SSL-PROFILE-SECURE
Qualys Rating is “B” after code upgrade and “A-” after performing code upgrade and removing RC4.
After making the code upgrade & removing RC4 cipher, it is recommended that you test your site for any vulnerabilities at the Qualys Site.
The above commands will create a “Parent” SSL Client profile – “PARENT-SSL-SECURE” that will disable SSLv3, RC4 and order the ciphers from High to Medium strength. Please, note that any client initiating connection on SSLv3 will be dropped. Usually clients running Windows XP and IE6 will initiate SSLv3 connections.
In order to know the total SSLv3 connections, you can run this command:
(tmos.ltm)# show profile client-ssl CUSTOM-CLIENT-SSL | grep Protocol Protocol SSL Protocol Version 2 0 SSL Protocol Version 3 156 TLS Protocol Version 1.0 16.1K TLS Protocol Version 1.2 11.6K DTLS Protocol Version 1 0
Based on the above output, it can be seen that the client SSL profile configured on the Virtual Server handling SSL traffic has received 156 SSLv3 connection requests.
The stats can be cleared using this command:
(tmos.ltm)# reset-stats profile client-ssl (tmos.ltm)# show profile client-ssl CUSTOM-CLIENT-SSL | grep Protocol Protocol SSL Protocol Version 2 0 SSL Protocol Version 3 0 TLS Protocol Version 1.0 0 TLS Protocol Version 1.2 11.6K DTLS Protocol Version 1 0
Note: The “reset-stats” command will NOT clear the stats for TLS1.2 in 10.2.3 code version. F5 is aware of this bug. I was able to clear the stats for 10.2.4 code version. I am not sure if this bug exists for 10.2.3 and lower code versions.
The cipher suite that is being utilized, after the change can be identified by running the following command in bash:
[root@lbal1:Active] config # tmm --clientciphers 'HIGH:MEDIUM:!SSLv3:!SSLv2:!RC4:!COMPAT:!EXP'
NOTE:
After making the above changes, some of the older browsers may not be able to connect to your website on HTTPS as the older browsers don’t support TLS1.2. For a list of browsers and supported protocols, see here.
Having dealt with SSL/TLS vulnerabilities, it looks like TLS1.2 with Ephemeral Diffie-Hellman is the most secure current version. Anything before TLS1.1 would be insecure and would have to be avoided, if possible. The main problem is that there are quite a few client browsers that are just incompatible with TLS1.2 and this has to be factored before making any changes.
GHOST Vulnerability:
This is a recent vulnerability – CVE-2015-0235
According to SOL16057, if you want to stay with 10.x code version on F5 LTM, it is better to use 10.2.4 + HF11. For 11.x code version, use at least 11.5.1 + HF8.
Update:
For a great Qualys grade see the following link.
!SSLv2:!EXPORT:!DHE+AES-GCM:!DHE+AES:!DHE+3DES:ECDHE+AES-GCM:ECDHE+AES:RSA+AES-GCM:RSA+AES:ECDHE+3DES:RSA+3DES:-MD5:-SSLv3:-RC4:@STRENGTH