A firewall blocking the connection by explicitly closing it.
VNC Server's security is not properly configured.
Ensure that you have selected an authentication method, set a password if required, and generated secure keys for the server if required.
VNC Server is crashing for some reason.
If this is the case, then the VNC Server Service will no longer be "running" on the server machine.
Under Windows NT based platforms, including Windows XP, you can find out why VNC Server closed the connection by looking at the server's Application Event Log using the Event Viewer administrative tool. VNC Server will log information under the tag "WinVNC4" when running as a service. Log entries starting "Connections: closed" will include a brief description of the reason for the connection closing. If you don't see any such messages at all then VNC Server is not even receiving the connections - this can happen if a firewall is blocking them, or if you are connecting to the wrong computer.
On Unix platforms, VNC Server will normally log information to a ".log" file in the user's ".vnc" directory. As under Windows, log entries starting "Connections: closed" will include a brief description of the reason for the connection closing.