SSH: The Secure Shell (The Definitive Guide)
Barrett, Silverman, & Byrnes / O’Reilly

SSH Frequently Asked Questions

Sometimes my SSH connection hangs when exiting — the shell (or remote command) exits, but the connection remains open, doing nothing.

Quick Fix

You're probably using the OpenSSH server, and started a background process on the server which you intended to continue after logging out of the SSH session. Fix: redirect the background process stdin/stdout/stderr streams (e.g. to files, or /dev/null if you don't care about them). For example, this hangs:
client% ssh server
server% xterm &
server% logout
but this behaves as expected:
client% ssh server
server% xterm < /dev/null >& /dev/null &
server% logout
SSH session terminates

Short Explanation

This problem is usually due to a feature of the OpenSSH server. When writing an SSH server, you have to answer the question, "When should the server close the SSH connection?" The obvious answer might seem to be: close it when the server-side user program started by client request (shell or remote command) exits. However, it's actually a bit more complicated; this simple strategy allows a race condition which can cause data loss (see the explanation below). To avoid this problem, sshd instead waits until it encounters end-of-file (eof) on the pipes connecting to the stdout and stderr of the user program.

This strategy, however, can have unexpected consequences. In Unix, an open file does not return eof until all references to it have been closed. When you start a background process from the shell on the server, it inherits references to the shell's standard streams. Unless you prevent this by redirecting these, or the process closes them itself (daemons will generally do this), the existence of the new process will cause sshd to wait indefinitely, since it will never see eof on the pipe connecting it to the (now defunct) shell process — because that pipe also connects it to your background process.

This design choice has changed over time. Early versions of OpenSSH behaved as described here. For some time, it was changed to exit immediately upon exit of the user program; then, it was changed back when the possibility of data loss was discovered.

Race Condition Details

As an example, let's take the simple case of:
ssh server cat foo.txt
This should result in the entire contents of the file foo.txt coming back to the client — but in fact, it may not. Consider the following sequence of events: This sequence of events can, for example, cause file truncation when using scp.