F5 – big3d restarting

I ran into an issue where the big3d daemon was restarting continuously on an F5 running LTM only (No GTM). The following article details the steps that were taken to solve the restart issue. The solution was achieved after raising a support case with F5 Networks.

I tried to restart the daemon (big3d & httpd)

(tmos)#restart /sys service big3d

(tmos)#restart /sys service httpd

and tried stopping and starting using the following commands but it did not help in preventing the continuous restarts:

(tmos)#stop /sys service httpd 

(tmos)#stop /sys service big3d

(tmos)#start /sys service httpd

(tmos)#start /sys service big3d

I ran the following command from tmsh:

(tmos)#load sys config

and it resulted in the following error:

Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration …
BIGpipe unknown operation error:
01070920:3: Application error for confpp: Syntax OK
The certificate does not match the key.  To change them try ‘bigpipe httpd { sslcertfile /etc/httpd/conf/ssl.crt/server.crt sslcertkeyfile /etc/httpd/conf/ssl.key/server.key }’
*************************************************************
Sep  9 22:56:52 localhost confpp[9878]: syntax check command FAILURE for unix_config_httpd returned: ‘2304’
Restarting syslog-ng:
Shutting down syslog-ng: [  OK  ]
Starting syslog-ng: [  OK  ]
Shutting down ntpd: [  OK  ]
Starting ntpd: [  OK  ]
[FAILED]ing httpd: [FAILED]

VERIFY Device Cert & Key:

As seen in the output of the “load sys config” command, the cert & key did not match. The following command (run from bash) is utilized to verify if the cert & key match or not:

openssl rsa -in /etc/httpd/conf/ssl.key/server.key -modulus -noout | openssl md5

openssl x509 -in /etc/httpd/conf/ssl.crt/server.crt -modulus -noout | openssl md5

The md5 hash was different indicating that the cert & key did not match. I used the following solution guide in order to generate new cert/key pair: SOL9114

  1. Log in to the bash command line.
  2. Generate the new device certificate and key using the following syntax:

    openssl req -x509 -nodes -days <# of days> -newkey rsa:<keysize> -keyout /config/httpd/conf/ssl.key/server.key -out /config/httpd/conf/ssl.crt/server.crt

    Note: Replace <# of days> with the number of days in year increments for which you want the certificate to be valid. I used 3650 days (10 years).

  3. Enter the certificate attributes.
  4. Restart the httpd process by typing the following command:

    bigstart restart httpd

  5. Copy the new self-signed certificate to the trusted device certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/big3d/client.crt

    Note: Alternatively, you can add the new certificate to the trusted device certificate file and remove all old certificates by running the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/big3d/client.crt

  6. (BIG-IP GTM and BIG-IP Link Controller) Copy the new self-signed certificate to the trusted server certificate file by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt >> /config/gtm/server.crt

    Note: Alternatively, you can add the new certificate to the trusted server certificate file and remove all old certificates by typing the following command:

    cat /config/httpd/conf/ssl.crt/server.crt > /config/gtm/server.crt

After creating the right cert/key pair, the non-stop restarts of big3d stopped. After preventing the restarts, I had issues with GUI not loading the Virtual Server option and had to perform a full box reboot in order for the GUI to function without any issues.

Reference:

SOL10999

SOL13444