wiki

The application on AIX is closing the idle connection sooner than expected. How to check the connection idle timeout used by the application? Question & Answer

Question

The application on AIX is closing the idle connection sooner than expected. How to check the connection idle timeout used by the application?

Answer

There are 3 network options that control the connection idle timeout on AIX. The “no -a” command lists these options. These are system-wide options for all connections.

tcp_keepidle is the time after which an idle connection is started to be probed. Default Value: 14400 Unit: halfsecond Change: only effective for future connections.

tcp_keepintvl is the duration between two probes. Default Value: 150 Unit: halfsecond Change: only effective for future connections.

tcp_keepcnt is the number of time the connection is probed after tcp_keepidle is reached. Default Value: 8 Unit: numeric Change: dynamic

Once the connection is idle for tcp_keepidle time, the number of probes set by tcp_keepcnt are sent to the remote host, each with a time interval set by tcp_keepintvl. For default values, if tcp connection is idle for 14400 halfseconds (7200 seconds), then 8 keepalive probes are sent, each with 150 halfseconds (75 seconds) interval. If there is no reply from the remote host, then the connection will be closed. This means idle connection will close after 2 hrs and 10 minutes (7200 + (75 * 8) = 7800 seconds = 2 hours and 10 minutes). If the remote host replies to keepalive probe, then idle timeout will reset to 14400 halfseconds.

Application can overwrite these no options by using setsockopt(). These options can be hard coded in the application or can be configurable in the application’s configuration file. setsockopt() uses option SO_KEEPALIVE to enable keepalives, TCP_KEEPIDLE to set the value of tcp_keepidle, TCP_KEEPINTVL to set the value of tcp_keepintvl and TCP_KEEPCNT to set the value of tcp_keepcnt.

If idle connections are closing sooner than the value set by no option tcp_keepidle, then the application is overwriting the options or firewall/switch/gateway is closing the connection.

For example, when tcp_keepidle, tcp_keepintvl and tcp_keepcnt are set to default values 14400, 150 and 8 respectively, then the idle connection will close after 2 hours and 10 minutes. But if any application connections are closing sooner, then use the following steps to check the tcp_keepidle set for the connection.

(1) Open two windows and log into the AIX host.

(2) Start a new application connection.

(3) In the first window, run “netstat -Aan” to find the PCB of the connection started in step 2. In this example, the server application is running on port 1521 on host with ipaddress 10.0.0.57.

# netstat -Aan | grep 10.0.0.82.54498

PCB/ADDR                         Proto    Recv-Q     Send-Q     Local Address               Foreign Address                      (state)
f1001000003e33c0       tcp4          0                0            10.0.0.57.1521            10.0.0.82.54498              ESTABLISHED

(4) On the second window, as a root run sockinfo inside the kdb to display the details about PCB/ADDR “f1001000003e33c0“ from step 3.

# kdb
           START              END <name>
0000000000001000 0000000007150000 start+000FD8
F00000002FF47600 F00000002FFE1000 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F1001104C0000000 F1001104D0000000 pvproc+000000
F1001104D0000000 F1001104D8000000 pvthread+000000

(0)> sockinfo f1001000003e33c0 tcpcb
---- TCPCB ----(@ F1001000003E33C0)----
    seg_next......@F1001000003E33C0  seg_prev......@F1001000003E33C0  
    t_softerror... 00000000 t_state....... 00000004 (ESTABLISHED)
    t_timer....... 00000000 (TCPT_REXMT)
    t_timer....... 00000000 (TCPT_PERSIST)
    t_timer....... 00000E10 (TCPT_KEEP) <=== E10 halfseconds in hex is 3600 halfseconds in decimal which is 30 minutes.
    t_timer....... 00000000 (TCPT_2MSL)
    t_rxtshift.... 00000000 t_rxtcur...... 00000003 t_dupacks..... 00000000
    t_maxseg...... 000005A8 t_force....... 00000000
    t_flags....... 3E0803E4 (NODELAY|RFC1323|SENT_WS|RCVD_WS|SENT_TS|RCVD_TS|LARGESEND|VIRTUAL_LARGESEND|SENT_LS|RCVD_LS|COPYFLAGS)

(5) This shows that the connection uses a tcp_keepidle of 30 minutes set by the application instead of AIX default value of 2 hours (14400 halfseconds). Hence, idle connection closes sooner.

It is important to note that TCPT_KEEP a dynamic counter. When the connection gets established, it is set to the value of tcp_keepidle set by no option or application. When the connection is idle, it starts ticking down and as soon as a packet is received from a remote host, then it resets to tcp_keepidle. Currently, the sockinfo does not list the values of tcp_keepintvl and tcp_keepcnt.