Tag: F5

F5 – RST or ICMP Packet Rate

You can follow SOL13151 in order to increase the packets/sec value. However, I would caution against doing it or at least recommend keeping the value smaller. The default setting is in place to prevent the F5 from overwhelming its resources by sending out RST. This could potentially end up being a self-inflicted DoS. So, either don’t change the value or increase the value in minimal steps like +50 to 300 packets/sec.

tmsh
modify sys db tm.maxrejectrate value 300
save sys config

You may have to use SOL13223 in order to identify the reason for RST. SOL9812 provides reasons for which the F5 sends RST.

In order to understand which monitor is marking the pool member down, see this SOL13898. If you are using 11.4 and after code version, the monitor that triggered the failure should be auto-displayed as per K14407

K12531 is a good reference for troubleshooting monitors in F5.

F5 iControl REST

F5 utilizes iControl REST API as part of their automation toolkit. REST API is a powerful way to automate F5 management. iControl REST API was introduced by F5 in 11.5 code version. 11.6 code version is the first major code version with a relatively stable release. However, 11.6 does not support remote authentication like TACACS+. For iControl REST API with remove authentication, it is important to utilize 12.x code version. F5 programmability training documentation and related information are available here.

F5 – SSL Cert Expiration

K14318 – Identifying expired certs and certs about to expire in 30 days.

K15288 – Email reminder for cert expiration.

A few one-liners from bash to identify the cert expiration date:

Identifying the expiration date from the certificate name:

~ # tmsh list sys file ssl-cert domain.crt | grep expiration
    expiration-date 1505951999
    expiration-string "Sep 20 23:59:59 2017 GMT"

 

Identifying the Client SSL profile for a certificate:

~ # tmsh list ltm profile client-ssl one-line | grep domain.crt | awk '{print $3,$4}'
    client-ssl CLIENTSSL-domain.com

 

Identifying the Virtual Server from Client SSL profile:

~ # tmsh list ltm virtual one-line | grep CLIENTSSL-domain.com | awk '{print $2,$3}'
    virtual VS-10.10.10.10-Public

 

Identifying the expiration date for cert associated with VS:

~ # echo | openssl s_client -connect 10.10.10.10:443 2> /dev/null | openssl x509 -noout -dates
notBefore=Nov 21 00:00:00 2016 GMT
notAfter=Nov 22 23:59:59 2017 GMT

 

F5 Virtual Server – Order of Precedence

The VS order of predence differs with code version and the tm.continuematching db variable. This tm.continuematching db variable is set to false by default and hence, a lower predence VS does not handle the traffic if there exists an higher predence VS in a disabled state. If the traffic has to be handled by lower precedence VS when the higher precedence VS is disabled, we would have to set this db variable as true:

11.x Code Version:

(tmos)# modify /sys db tm.continuematching value true
(tmos)# save /sys config

9.x – 10.x Code Version:

bigpipe db TM.ContinueMatching true
bigpipe save all

The order of predence for VS processing for different code version is provided below.

Order of Precedence for code version: 9.2 – 11.2.x

<address>:<port>
 <address>:*
 <network>:<port>
 <network>:*
 *:<port>
 *:*

Order of Precedence for code version: 11.3 and later

Order Destination Source Service port
1 <host address> <host address> <port>
2 <host address> <host address> *
3 <host address> <network address> <port>
4 <host address> <network address> *
5 <host address> * <port>
6 <host address> * *
7 <network address> <host address> <port>
8 <network address> <host address> *
9 <network address> <network address> <port>
10 <network address> <network address> *
11 <network address> * <port>
12 <network address> * *
13 * <host address> <port>
14 * <host address> *
15 * <network address> <port>
16 * <network address> *
17 * * <port>
18 * * *

F5 – Bleeding Active Connections

Scenario:

A Virtual Server is load balancing connections to a pool with 2 pool members. During maintenance window, one of the two pool members is disabled and maintenance is completed followed by the other pool member.

However, as the users make continuous API calls every 5 seconds, the existing TCP connection never bleeds out. Even after waiting for 24 hours, connections still exist on the disabled pool member.

Solution:

By default, F5 makes load balancing decision when the 1st HTTP request within a TCP connection is received. Subsequent HTTP request within the TCP connection are sent to the same pool member as the very 1st HTTP request.

By enabling OneConnect profile with a /32 netmask (255.255.255.255), we were able to force the F5 to make load balancing decision for every HTTP request instead of its default behavior.

The OneConnect profile used along with disabled or forced-offline setting will move the connection from the failed pool member to the active pool member.

Reference Link.

OneConnect & HTTP Requests

This is a copy/paste of a Q&A in devcentral. I didn’t change it as it is quite descriptive and gets the point across.

Current Setup:

We are using Cookie Insert method for session persistence. So LTM adds “BigipServer*” Cookie in the http response header with value as encoded IP address and port name. Subsequent requests from the client (in our case browser) will have this cookie in the request header and this helps LTM to send the request to same server. This LTM cookie’s expiry is set to session, so this cookie will be cleared when we close the browser or we expire it using iRule.

Use Case:

We have set of servers configured as pool members serving traffic to users who are logged in. During release time, we will release the code to new set of servers and add those servers also to the LTM pool. LTM will now have servers with both old code as well as new code. We disable all servers which has the old code so that LTM routes only the requests which already has “BigipServer*” Cookie value pointing to those servers. This will not interrupt the users who are already logged in and doing some work. All new requests (new users) will be load balanced to any of the active servers which has new code. We will ask our already logged in users to logout and login back again once they are done with the current work. We have an iRule configured to expire the LTM cookie during logout, so our expectation is that users will be connected to new servers when they are logging in again.

Problem:

Even though iRule expires the LTM cookie during logout and the cookie is not present the request header of login, users are still routed to the same disabled server when they are logging in again. Ideally, LTM should have load balanced the request to any of the active servers.

Root Cause:

Upon analyzing this further with network traffic, we found that, whenever the browser has a persistent TCP connection open with LTM after logout, browser uses that existing TCP connection for sending the login request. LTM routes this login request to the same disabled server which handled the previous request even though LTM cookie is not present in the request header. If we close the TCP connection manually after logout (using CurrPosts or some other tool), the browser establishes a new connection with LTM during login and LTM load balances this requests to any active server. One option for us is to send “Connection: close” in the response header during logout, but the browser may hold multiple persistent TCP connections (I have seen browser holding even three connections) and hence closing a single TCP connection will not help. Other option is to close the browser, but we don’t have that choice for reasons I cannot explain here (trust me).

SOLUTION:

Try using the following:

  1. OneConnect Profile in VS with netmask of /32.
  2. Action on Service Down in the Pool set to Reselect.

(1) will force the load balancing decision to be made for every HTTP request instead of the the default of lb decision being made only for the 1st HTTP request within a TCP connection.

(2) will force the HTTP Request to be sent to a new pool member when the selected member is down as the load balancing decision is made for every HTTP request instead of the very 1st HTTP request within a persistent/keep-alive connection.

Keep-alive Connection (also referred to as Persistent Connection) is used to refer to the same feature provided by HTTP1.1 where you can utilize a single TCP connection in order to send multiple HTTP requests within a single TCP connection.

 

F5 – Automating CLI Execution

Purpose:
This is a really simple way to automate CLI command execution on multiple F5 devices using Bash & TCL scripting. The scripts have been tested on a linux and a mac machine.

How to use it:
There is a bash script (F5_Bash_v1) that is utilized to collect the username/password for F5 access. A text file (F5_Host.txt) that stores the management IP address of multiple F5 devices and a TCL script (F5_Out_v1.exp) that is used to execute CLI commands on the F5 devices.

The bash script is the master script that obtains the username/password and executes the TCL script for multiple F5 devices.

Setup:
On a linux machine that is utilized to connect to the F5 device:

#Create a directory
mkdir F5_Check

Within the “F5_Check” directory, create the following 3 files:
F5_Host.txt
F5_Bash_v1
F5_Out_v1.exp

File Content: F5_Host.txt contains the management IP of the F5 devices.
Example:

$ cat F5_Host.txt
10.12.12.200
10.12.12.201
10.12.12.202
10.12.12.203

File Content: F5_Bash_v1

#!/bin/bash
# Collect the username and password for F5 access
echo -n "Enter the username "
read -s -e user
echo -ne '\n'
echo -n "Enter the password "
read -s -e password
echo -ne '\n'

# Feed the expect script a device list & the collected username & passwords
for device in `cat ~/F5_Check/F5_Host.txt`; do
./F5_Out_v1.exp $device $password $user ;
done

File Content: F5_Out_v1.exp

#!/usr/bin/expect -f

# Set variables
set hostname [lindex $argv 0]
set password [lindex $argv 1]
set username [lindex $argv 2]

# Log results
log_file -a ~/F5_Check/F5LOG.log

# Announce which device we are working on and the time
send_user "\n"
send_user ">>>>>  Working on $hostname @ [exec date] <<<<<\n"
send_user "\n"

# SSH access to device
spawn ssh $username@$hostname

expect {
"no)? " {
send "yes\n"
expect "*assword: "
sleep 1
send "$password\r"
}
"*assword: " {
sleep 1
send "$password\r"
}
}

expect "(tmos)#"
send "sys\n"
expect "(tmos.sys)#"

send "show software\n"
expect "#"
send "exit\n"
expect "#"
send "quit\n"

expect ":~\$"
exit