You can follow SOL13151 in order to increase the packets/sec value. However, I would caution against doing it or at least recommend keeping the value smaller. The default setting is in place to prevent the F5 from overwhelming its resources by sending out RST. This could potentially end up being a self-inflicted DoS. So, either don’t change the value or increase the value in minimal steps like +50 to 300 packets/sec.
tmsh
modify sys db tm.maxrejectrate value 300
save sys config
You may have to use SOL13223 in order to identify the reason for RST. SOL9812 provides reasons for which the F5 sends RST.
In order to understand which monitor is marking the pool member down, see this SOL13898. If you are using 11.4 and after code version, the monitor that triggered the failure should be auto-displayed as per K14407
K12531 is a good reference for troubleshooting monitors in F5.