Category: Application Delivery Controller

F5 BIG-IP iControl REST vulnerability CVE-2022-1388

The vulnerability CVE-2022-1388 has a score of 9.8. Code upgrade to mitigate this vulnerability has been provided in the F5 knowledge article in the reference. The most important step is to restrict access to the F5 Self and Management IP addresses. Other than the mitigation provided in the K23605346, these configuration steps will help to reduce the attack surface:

This will allow the self-IP addresses on the F5 devices to listen on udp:1026 and tcp:4353 which are required for configuration sync and failover between F5 devices. In order to implement this, you should not be using the self-IP addresses for any management access (22/80/443 ports).

list net self all allow-service

Configuration Changes:

modify net self all allow-service replace-all-with { udp:1026 tcp:4353 }

This will limit access to httpd from source IP address of 192.168.0.0/255.255.0.0. As a quick note, this does not mitigate the vulnerability but it is a good practice to limit access to httpd only from specific internal IP addresses.

list /sys httpd allow

Configuration Changes:

modify sys httpd allow add { 192.168.0.0/255.255.0.0 }

The following set of configuration will create a set of filters on the management IP address and limits access to the management IP address for specific services like dns, ntp, snmp, https and ssh. This will reduce the attack surface on the management IP address.

list security firewall management-ip-rules

Configuration Changes:

modify /security firewall management-ip-rules rules add { mgmt-udp-access { action accept destination { ports add { 53 161 123 1026 } } ip-protocol udp log yes place-before first source { addresses add { 192.168.0.0/255.255.0.0} } status enabled uuid auto-generate } }

modify /security firewall management-ip-rules rules add { mgmt-tcp-access { action accept destination { ports add { 22 53 161 443 4353 } } ip-protocol tcp log yes place-before first source { addresses add { 192.168.0.0/255.255.0.0 } } status enabled uuid auto-generate } }

modify /security firewall management-ip-rules rules add { mgmt-deny-all { action drop destination {} source {} log yes place-after last status enabled uuid auto-generate }}

Reference:

https://support.f5.com/csp/article/K23605346

https://www.bleepingcomputer.com/news/security/cisa-tells-federal-agencies-to-fix-actively-exploited-f5-big-ip-bug/

F5 Failover in AWS

F5 requires IMDSv1 in order to initiate failover between two F5 devices. IMDSv1 is susceptible to SSRF vulnerabilities as indicated in the AWS document. If IMDSv1 is disabled in AWS environment for security reasons, F5 failover will not be seamless and the F5 logs will have errors like this:

err logger[15542]: /usr/libexec/aws/aws-failover-tgactive.sh (traffic-group-1): Instance sanity check failed with error:

F5 is tracking support for IMDSv2 in AWS internally using ID 968657

Serial Number – Viprion Blades and Chassis

# clsh tmsh show sys hardware | grep 'Host Board Serial'

The above command is run from bash shell on the F5 in order to identify the serial number of all the blades in an F5 Viprion device.

# tmsh show sys hardware | grep 'Chassis Serial'

The above command is run from bash shell on the F5 in order to identify the serial number of the chassis in an F5 Viprion device.

F5 – RST or ICMP Packet Rate

You can follow SOL13151 in order to increase the packets/sec value. However, I would caution against doing it or at least recommend keeping the value smaller. The default setting is in place to prevent the F5 from overwhelming its resources by sending out RST. This could potentially end up being a self-inflicted DoS. So, either don’t change the value or increase the value in minimal steps like +50 to 300 packets/sec.

tmsh
modify sys db tm.maxrejectrate value 300
save sys config

You may have to use SOL13223 in order to identify the reason for RST. SOL9812 provides reasons for which the F5 sends RST.

In order to understand which monitor is marking the pool member down, see this SOL13898. If you are using 11.4 and after code version, the monitor that triggered the failure should be auto-displayed as per K14407

K12531 is a good reference for troubleshooting monitors in F5.

F5 iControl REST

F5 utilizes iControl REST API as part of their automation toolkit. REST API is a powerful way to automate F5 management. iControl REST API was introduced by F5 in 11.5 code version. 11.6 code version is the first major code version with a relatively stable release. However, 11.6 does not support remote authentication like TACACS+. For iControl REST API with remove authentication, it is important to utilize 12.x code version. F5 programmability training documentation and related information are available here.

GTM Code Upgrade

These are a few quick checks as part of the GTM code upgrade maintenance that will be useful.  As part of the maintenance preparatory work, check the license “service check date” as per K7727

Before starting the code upgrade and after the code upgrade, the following can be utilized to check the status of the devices:

From tmsh:

(/Common)(tmos)# show sys software
(/Common)(tmos)# show gtm server | grep -e "Gtm::" -e "Availability" -e "State"

From bash:

/shared/bin/big3d –v

From another client machine:

dig @<GTM1_IP> <WIP_FQDN>
dig @<GTM2_IP> <WIP_FQDN>

Just after the code is upgraded, make sure to run the big3d_install commands as per K13312. This will help to make sure that all the devices run the latest big3d version.

F5 GTM – DNS Query Processing Order

When a DNS query arrives at a F5 GTM/DNS, this is the processing order for the DNS query.

1 – DNS Query is processed by the Listener.

2- If Recursion Desired (RD) flag is set in the incoming query and if the DNS Profile associated with the Listener has “Process Recursion Desired” enabled, the following is done:

a. DNS iRule

b. DNSSEC Key Processing

c. DNS Express

d. DNS Profiles

3 – If Recursion Desired (RD) flag is set in the incoming query and if the DNS Profile associated with the Listener has “Process Recursion Desired” disabled, the query is considered “Un-handled” and dispatched according to “Unhanded Query Action” set in DNS Profile.

4 – DNS Cache is used to handle any DNS query that doesn’t match Big-IP GTM/DNS or DNS Express Records.

Reference: K14510

F5 Logs

F5 logs are available under /var/log/ directory.

If you are looking for LTM traffic related logs, look under  /var/log/ltm

If you are looking for GTM traffic related logs, look under /var/log/gtm

GTM has been renamed as DNS from 12.x code version.

The logs are rotated every 24 hours. There will be one ltm file that contains the latest logs and multiple ltm.gz files like: ltm.1.gz, ltm.2.gz etc. The ltm.1.gz files can be opened by using the following command: zcat ltm.1.gz

    (tmos)# run util bash
    ~# cd /var/log
    ~# cat ltm

Other such logs are outlined in this article – K16197

Thoughts on F5 Deployment

This is a simplified check list for GTM & LTM deployment based on my experience.

Don’t deploy GTM in HA pair in a single DC:

GTM devices work in a synchronization group across geographic regions. If you deploy GTM-1 in DC-1 and GTM-2 in DC-2, these 2 GTM devices will serve as Active-Active HA Pair for most deployments.

There is no reason to have GTM-1A & GTM-1B in DC-1 and GTM-2A & GTM-2B in DC-2, where the A & B devices are part of an HA pair. Option-B design is just an overkill.

F5_GTM_Deployment

GTM to LTM VS Health Monitors:

Let the LTM monitor its pool members. The GTM will obtain the status of the LTM pool members from the LTM. GTM directly monitoring the LTM pool members or LTM VS is not required in most cases.

Don’t deploy LTM in Active-Active state:

There is a lot of room for flexibility with this statement. I haven’t found the Active-Active pair to be reliable in production environment. Such deployments are rare in the industry and hence, if there are any bugs with Active-Active pair, there is a greater chance that your deployment could be affected by the software bug than the Active-Standby setting which is widely deployed and stable.

Code Upgrade Checklist:

  1. Check license reactivation date before code upgrade.
  2. Check to make sure that the code you are utilizing is not End of Support.
  3. Save the configuration on the box and outside the box whenever upgrading code.
  4. Save the license number on the box in a /var/tmp file and off the box.

Code Version:

You are better off picking one of the Long Term Stable code version releases as provided in this K5903.

Route Domains:

Route domains are F5’s version of VRF. Although they are easy to deploy, they can be a pain to troubleshoot with the “%” addition to IP address.

iRule Vs LTM Policy:

My personal preference is always iRule as it provides greater granular control and flexibility. However, LTM Policy may be better optimized and can provide lower latency for the same function. LTM Policy can be an effective substitute for simple iRules.

Pool & Pool Members:

As a rule of thumb, all the members in a pool should have the same port. For example: POOL_WEB_80. This naming convention tells you that this is a pool of web servers that are listening on port 80 even without looking into the individual pool members. Having multiple members listening on a common port as part of a single pool helps in the long run to scale the configuration without adding significant operational complexity.

Naming Convention:

While creating custom VS, Pool, Monitor etc, I recommend starting with capital letters. For example:

POOL_Web_80
VS_domain.com_80
VS_domain.com_443
MON_HTTP_80

The default configuration elements like tcp, http monitor tends to be lower case in F5. The capital letters helps to distinguish between user-created (custom profiles) and default profiles which are all lower-case.

Custom & Default profiles:

NEVER use default configuration elements. Create custom configuration elements with the default configuration elements as parent profile.Whenever configuration change is required, always change only the custom profiles. This will prevent someone from accidentally changing the default profiles while working on the F5.

F5 TMM Crash

We were using a DNS VS listening on port 53 but configured to handle TCP protocol as shown here:

ltm virtual /Common/VS_DNS {
 destination /Common/10.10.10.10:53
 ip-protocol tcp
 mask 255.255.255.255
 pool /Common/pool_dns
 profiles {
 /Common/tcp { }
 }
 source 0.0.0.0/0
 source-address-translation {
 pool /Common/SNAT-10.10.10.10
 type snat
 }
 translate-address enabled
 translate-port enabled
 }

An iRule was using RESOLV::lookup against the configured TCP VS. RESOLV::lookup uses UDP requests and since the VS was configured to handle only TCP, the F5 crashed with core file generated.

According to F5 Engineers, bug alias 570575 is associated with this condition where RESOLV::lookup against a TCP Virtual Server causes the F5 to crash generating a core file in /var/core/

The workaround involved using “ip-protocol” as “any” and “profiles” as “fastL4” for a configuration that looks like this:

ltm virtual /Common/VS_DNS {
 destination /Common/10.10.10.10:53
 ip-protocol any
 mask 255.255.255.255
 pool /Common/pool_dns
 profiles {
 /Common/fastL4 { }
 }
 source 0.0.0.0/0
 source-address-translation {
 pool /Common/SNAT-10.10.10.10
 type snat
 }
 translate-address enabled
 translate-port enabled
 }