Tag: F5

F5 BIG-IP iControl REST vulnerability CVE-2022-1388

The vulnerability CVE-2022-1388 has a score of 9.8. Code upgrade to mitigate this vulnerability has been provided in the F5 knowledge article in the reference. The most important step is to restrict access to the F5 Self and Management IP addresses. Other than the mitigation provided in the K23605346, these configuration steps will help to reduce the attack surface:

This will allow the self-IP addresses on the F5 devices to listen on udp:1026 and tcp:4353 which are required for configuration sync and failover between F5 devices. In order to implement this, you should not be using the self-IP addresses for any management access (22/80/443 ports).

list net self all allow-service

Configuration Changes:

modify net self all allow-service replace-all-with { udp:1026 tcp:4353 }

This will limit access to httpd from source IP address of 192.168.0.0/255.255.0.0. As a quick note, this does not mitigate the vulnerability but it is a good practice to limit access to httpd only from specific internal IP addresses.

list /sys httpd allow

Configuration Changes:

modify sys httpd allow add { 192.168.0.0/255.255.0.0 }

The following set of configuration will create a set of filters on the management IP address and limits access to the management IP address for specific services like dns, ntp, snmp, https and ssh. This will reduce the attack surface on the management IP address.

list security firewall management-ip-rules

Configuration Changes:

modify /security firewall management-ip-rules rules add { mgmt-udp-access { action accept destination { ports add { 53 161 123 1026 } } ip-protocol udp log yes place-before first source { addresses add { 192.168.0.0/255.255.0.0} } status enabled uuid auto-generate } }

modify /security firewall management-ip-rules rules add { mgmt-tcp-access { action accept destination { ports add { 22 53 161 443 4353 } } ip-protocol tcp log yes place-before first source { addresses add { 192.168.0.0/255.255.0.0 } } status enabled uuid auto-generate } }

modify /security firewall management-ip-rules rules add { mgmt-deny-all { action drop destination {} source {} log yes place-after last status enabled uuid auto-generate }}

Reference:

https://support.f5.com/csp/article/K23605346

https://www.bleepingcomputer.com/news/security/cisa-tells-federal-agencies-to-fix-actively-exploited-f5-big-ip-bug/

F5 Failover in AWS

F5 requires IMDSv1 in order to initiate failover between two F5 devices. IMDSv1 is susceptible to SSRF vulnerabilities as indicated in the AWS document. If IMDSv1 is disabled in AWS environment for security reasons, F5 failover will not be seamless and the F5 logs will have errors like this:

err logger[15542]: /usr/libexec/aws/aws-failover-tgactive.sh (traffic-group-1): Instance sanity check failed with error:

F5 is tracking support for IMDSv2 in AWS internally using ID 968657

F5 – RST or ICMP Packet Rate

You can follow SOL13151 in order to increase the packets/sec value. However, I would caution against doing it or at least recommend keeping the value smaller. The default setting is in place to prevent the F5 from overwhelming its resources by sending out RST. This could potentially end up being a self-inflicted DoS. So, either don’t change the value or increase the value in minimal steps like +50 to 300 packets/sec.

tmsh
modify sys db tm.maxrejectrate value 300
save sys config

You may have to use SOL13223 in order to identify the reason for RST. SOL9812 provides reasons for which the F5 sends RST.

In order to understand which monitor is marking the pool member down, see this SOL13898. If you are using 11.4 and after code version, the monitor that triggered the failure should be auto-displayed as per K14407

K12531 is a good reference for troubleshooting monitors in F5.

F5 iControl REST

F5 utilizes iControl REST API as part of their automation toolkit. REST API is a powerful way to automate F5 management. iControl REST API was introduced by F5 in 11.5 code version. 11.6 code version is the first major code version with a relatively stable release. However, 11.6 does not support remote authentication like TACACS+. For iControl REST API with remove authentication, it is important to utilize 12.x code version. F5 programmability training documentation and related information are available here.

F5 – SSL Cert Expiration

K14318 – Identifying expired certs and certs about to expire in 30 days.

K15288 – Email reminder for cert expiration.

A few one-liners from bash to identify the cert expiration date:

Identifying the expiration date from the certificate name:

~ # tmsh list sys file ssl-cert domain.crt | grep expiration
    expiration-date 1505951999
    expiration-string "Sep 20 23:59:59 2017 GMT"

 

Identifying the Client SSL profile for a certificate:

~ # tmsh list ltm profile client-ssl one-line | grep domain.crt | awk '{print $3,$4}'
    client-ssl CLIENTSSL-domain.com

 

Identifying the Virtual Server from Client SSL profile:

~ # tmsh list ltm virtual one-line | grep CLIENTSSL-domain.com | awk '{print $2,$3}'
    virtual VS-10.10.10.10-Public

 

Identifying the expiration date for cert associated with VS:

~ # echo | openssl s_client -connect 10.10.10.10:443 2> /dev/null | openssl x509 -noout -dates
notBefore=Nov 21 00:00:00 2016 GMT
notAfter=Nov 22 23:59:59 2017 GMT

 

F5 Virtual Server – Order of Precedence

The VS order of predence differs with code version and the tm.continuematching db variable. This tm.continuematching db variable is set to false by default and hence, a lower predence VS does not handle the traffic if there exists an higher predence VS in a disabled state. If the traffic has to be handled by lower precedence VS when the higher precedence VS is disabled, we would have to set this db variable as true:

11.x Code Version:

(tmos)# modify /sys db tm.continuematching value true
(tmos)# save /sys config

9.x – 10.x Code Version:

bigpipe db TM.ContinueMatching true
bigpipe save all

The order of predence for VS processing for different code version is provided below.

Order of Precedence for code version: 9.2 – 11.2.x

<address>:<port>
 <address>:*
 <network>:<port>
 <network>:*
 *:<port>
 *:*

Order of Precedence for code version: 11.3 and later

Order Destination Source Service port
1 <host address> <host address> <port>
2 <host address> <host address> *
3 <host address> <network address> <port>
4 <host address> <network address> *
5 <host address> * <port>
6 <host address> * *
7 <network address> <host address> <port>
8 <network address> <host address> *
9 <network address> <network address> <port>
10 <network address> <network address> *
11 <network address> * <port>
12 <network address> * *
13 * <host address> <port>
14 * <host address> *
15 * <network address> <port>
16 * <network address> *
17 * * <port>
18 * * *

F5 – Bleeding Active Connections

Scenario:

A Virtual Server is load balancing connections to a pool with 2 pool members. During maintenance window, one of the two pool members is disabled and maintenance is completed followed by the other pool member.

However, as the users make continuous API calls every 5 seconds, the existing TCP connection never bleeds out. Even after waiting for 24 hours, connections still exist on the disabled pool member.

Solution:

By default, F5 makes load balancing decision when the 1st HTTP request within a TCP connection is received. Subsequent HTTP request within the TCP connection are sent to the same pool member as the very 1st HTTP request.

By enabling OneConnect profile with a /32 netmask (255.255.255.255), we were able to force the F5 to make load balancing decision for every HTTP request instead of its default behavior.

The OneConnect profile used along with disabled or forced-offline setting will move the connection from the failed pool member to the active pool member.

Reference Link.

OneConnect & HTTP Requests

This is a copy/paste of a Q&A in devcentral. I didn’t change it as it is quite descriptive and gets the point across.

Current Setup:

We are using Cookie Insert method for session persistence. So LTM adds “BigipServer*” Cookie in the http response header with value as encoded IP address and port name. Subsequent requests from the client (in our case browser) will have this cookie in the request header and this helps LTM to send the request to same server. This LTM cookie’s expiry is set to session, so this cookie will be cleared when we close the browser or we expire it using iRule.

Use Case:

We have set of servers configured as pool members serving traffic to users who are logged in. During release time, we will release the code to new set of servers and add those servers also to the LTM pool. LTM will now have servers with both old code as well as new code. We disable all servers which has the old code so that LTM routes only the requests which already has “BigipServer*” Cookie value pointing to those servers. This will not interrupt the users who are already logged in and doing some work. All new requests (new users) will be load balanced to any of the active servers which has new code. We will ask our already logged in users to logout and login back again once they are done with the current work. We have an iRule configured to expire the LTM cookie during logout, so our expectation is that users will be connected to new servers when they are logging in again.

Problem:

Even though iRule expires the LTM cookie during logout and the cookie is not present the request header of login, users are still routed to the same disabled server when they are logging in again. Ideally, LTM should have load balanced the request to any of the active servers.

Root Cause:

Upon analyzing this further with network traffic, we found that, whenever the browser has a persistent TCP connection open with LTM after logout, browser uses that existing TCP connection for sending the login request. LTM routes this login request to the same disabled server which handled the previous request even though LTM cookie is not present in the request header. If we close the TCP connection manually after logout (using CurrPosts or some other tool), the browser establishes a new connection with LTM during login and LTM load balances this requests to any active server. One option for us is to send “Connection: close” in the response header during logout, but the browser may hold multiple persistent TCP connections (I have seen browser holding even three connections) and hence closing a single TCP connection will not help. Other option is to close the browser, but we don’t have that choice for reasons I cannot explain here (trust me).

SOLUTION:

Try using the following:

  1. OneConnect Profile in VS with netmask of /32.
  2. Action on Service Down in the Pool set to Reselect.

(1) will force the load balancing decision to be made for every HTTP request instead of the the default of lb decision being made only for the 1st HTTP request within a TCP connection.

(2) will force the HTTP Request to be sent to a new pool member when the selected member is down as the load balancing decision is made for every HTTP request instead of the very 1st HTTP request within a persistent/keep-alive connection.

Keep-alive Connection (also referred to as Persistent Connection) is used to refer to the same feature provided by HTTP1.1 where you can utilize a single TCP connection in order to send multiple HTTP requests within a single TCP connection.