Re: The state server has closed an expired TCP/IP connection
Dec 07, 2018 04:12 PM|Koenemoes|LINK
Of course the question as to why the server closed your open connection on you is the mystery. Perhaps there is something in the mysql server logs to indicate why? Perhaps if you could include a packet capture of the transaction I can confirm that this module did indeed see the connection close, though that doesn't answer the why, only confirms. In the normal case, each side terminates its end of the connection by sending a special message with the FIN (finish) bit set. This message, sometimes called a FIN, serves as a connection termination request to the other device, while also possibly carrying data like a regular segment. Go to PuTTy options – Connection. Change the default value for “Seconds between keepalives (0 to turn off)”: from 0 to 600 (10 minutes) — in my case taking 30 secound That means it sends a “ping” every 30 seconds to prevent the connection from timing out. Check the “Enable TCPkeepalives (SOKEEPALIVE option)” check box. On the discover server that is running the scan: Download and install the Microsoft JDBC driver jar version mssql-jdbc-8.4.1.jre8.jar or later into the C: Program Files Symantec DataLossPrevention DetectionServer Protect lib jdbc directory.
We have had this issue for a couple of months at seemingly random times and levels of severity. Last week I finally got the extra hint from a seemingly unrelated event.
We also saw a 100% CPU spike on at least 1 IIS app pool which brings down the whole server after a while.
IISRESET was our only solution and this would work.
Long story short: it happened during WSUS updates of our dc's. The AD resync after rebooting them took longer than the configured time window of 1 hour.
So the situation could arise that both DNS servers were offline -> so also no AD
State server was configured as 'localhost', 127.0.0.1 would not have had this issue. I check by disassembling webengine4.dll. It returns something like 'connection closed by remote host' HRESULT 72746.
But system.web interprets this as the session being locked by a concurrent query and keeps retrying at full speed. This created an infinite loop of failing socket connections PER SESSION!
And that is why cpu got to 100%. It fills up the request queue, still waiting for the session data and then fajils completely with http 503 for all appools.
Our logs also showed connection issues to our sql because ad was down.
We were so focussed on the 100% cpu being the cause, but it was only a symptom. The fact that this error logged in event viewer was very misleading in my case.
In the end it was a quick 1 minute fix of the DC. -> no more emergency wake up calls for me :)
This was a perfect storm of timing events and small config mistakes which led to misleading logs and cpu spike by a asp.net bug...? Why keep retrying to read the session data if it won't work due to dns. Just end the webrequest there and log sometinhg like: NameResolution Failed instead of connection timeout
Learn how to configure the timeout settings for your Load Balancer resource.
Server Closed Connection
Oracle Cloud Infrastructure load balancers support connection multiplexing. The load balancer can route many incoming requests from multiple clients to the destination backend server through a few (one or multiple) backend connections.
After your load balancer connects a client to a backend server, the connection can be closed because of inactivity. Also, you can configure load balancer listeners to control the maximum idle time allowed during each TCP connection or HTTP request and response pair. Oracle recommends that you do not allow your backend servers to close connections to the load balancer.
Transmit Server Closed Connection
The following timeout settings affect your load balancer's behavior:
Keep-alive setting between the load balancer and the client
The Load Balancing service sets the keep-alive value to maintain the connection for 10,000 transactions or until it has been idle for 65 seconds, whichever limit occurs first.
Keep-alive setting between the load balancer and backend server
The load balancer closes backend server connections that are idle for more than 300 seconds. See Keep-Alive Settings for more information.
Idle timeout
You can set the duration of the idle timeout when you create a listener. This setting applies to the time allowed between two successive receive or two successive send network input/output operations during the HTTP request-response phase. See Idle Timeout Settings for more information.
Keep-Alive Settings
Learn how the Load Balancer resource sets the keep-alive value to maintain a connection with backend servers.
The Load Balancing service does not honor keep-alive settings from backend servers. The load balancer closes backend server connections that are idle for more than 300 seconds. Oracle recommends that you do not allow your backend servers to close connections to the load balancer. To prevent possible 502 errors, ensure that your backend servers do not close idle connections in less than 310 seconds.
The Load Balancing service sets the keep-alive value to maintain the connection for 10,000 transactions or until it has been idle for 65 seconds, whichever limit occurs first.