Reading TCP buffers on Linux

A slow transfer is easy to blame on a small buffer. Start by separating the application queue, the receive window, and the amount of data the network can carry in flight.

Three different limits

TCP presents an ordered byte stream. The send buffer holds data accepted from the application, including bytes still awaiting acknowledgement. The receive buffer holds incoming data until the application reads it. Neither buffer turns individual writes into message boundaries; the receiving program still needs its own framing.

The receiver advertises a window that limits how much unacknowledged data the sender may have outstanding. Congestion control imposes a separate limit based on network conditions. The effective allowance depends on both. A large advertised window cannot remove congestion or make a busy receiving process consume data faster.

Observe before changing

On an established connection, ss -tinm combines TCP statistics with socket memory information. Recv-Q describes data waiting for the application; Send-Q includes data not yet acknowledged. Listening sockets use different queue meanings, so compare connections in the same state. Take several samples during the transfer rather than relying on one quiet moment.

ss -tinm
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
sysctl net.core.rmem_max net.core.wmem_max

A steadily growing receive queue suggests that the application is falling behind. A persistent send queue can reflect loss, a slow reader at the other end, or an application producing data faster than the connection can deliver it. Check round trip time and retransmissions alongside queue length.

Keep the units straight

The bandwidth delay product estimates the bytes needed in flight to fill a path. At 100 megabits per second and a 40 millisecond round trip, the estimate is about 500,000 bytes. This is a planning calculation, not a recommendation to assign that exact buffer size to every socket.

Linux normally adjusts TCP buffers within configured limits. The three values in tcp_rmem and tcp_wmem describe minimum, default, and maximum settings. Explicit application buffer requests interact with separate socket limits. Linux also doubles values requested through SO_RCVBUF and SO_SNDBUF for bookkeeping; the reported allocation is not simply available payload space.

Record the workload and current settings before experimenting. Change one limit at a time, then compare throughput, latency, and memory use. If the receiving process is CPU bound, buffer growth may only postpone the same bottleneck.