There was a constant increase in “overrun” and “input errors” on the Cisco ASA Interface. Upon examination, using “show asp drop“, “tcp-rstfin-ooo” & “tcp-3whs-failed” were constantly increasing.
Using the following to capture real-time traffic, the IP addresses and the ports can be identified:
# capture ASP type asp-drop tcp-rstfin-ooo buffer 2048 real-time
In this case, we were able to isolate port 5666 for Nagios servers as a culprit in sending RST after FIN and this was breaking the TCP protocol. This is an environment with 100s of Servers that was monitored by Nagios. When 100s of Servers end up sending RST simultaneously, it can turn out to be a mini-self-DOS. With older Firewalls & Code Versions, this can cause reboots. When we searched online, we were able to identify the following bugs on Nagios:
https://bugs.launchpad.net/ubuntu/+source/nagios-nrpe/+bug/989156
http://tracker.nagios.org/view.php?id=305