Getting Connection Reset error from sftp server frequently with parallel threads trying to put file on sftp

Question

I have a piece of multithreaded code which has 22 threads running in parallel and trying to put files on sftp server.

But I keep getting Connection Reset error intermittently in my logs and few of the records fail because of that.

On initial analysis, I found out that the size of the sftp server was t2.small and CPU utilization was going to 92%.

Considering this as I reason I changed the server to c5n.xlarge, now the error is coming less frequently but still, I am getting it at times even when the maximum CPU utilization goes to 63%.

I am not able to find anything different in sftp server logs at /var/log/secure.

Below is the piece of code being used to put file, every thread creates a new session and closes it.

JSch ssh = new JSch();
            // ssh.setKnownHosts("/path/of/known_hosts/file");
            java.util.Properties config = new java.util.Properties();
            config.put("StrictHostKeyChecking", "no");
            // Use key authentication if it is set, else use password auth
            if (mpServerDetails.get(SftpFile.SFTP_USERKEY) != null
                    && mpServerDetails.get(SftpFile.SFTP_USERKEY) != "") {
                    File userKeyFile = new File(mpServerDetails.get(SftpFile.SFTP_USERKEY).toString());
                if (userKeyFile == null || !userKeyFile.exists()) {
                    throw new NonRetriableException(
                            "Key file " + mpServerDetails.get(SftpFile.SFTP_USERKEY).toString() + "not found.");
                }
                ssh.addIdentity(userKeyFile.getAbsolutePath());
                session = ssh.getSession(mpServerDetails.get(SftpFile.SFTP_USERNAME).toString(),
                        mpServerDetails.get(SftpFile.SFTP_HOSTNAME).toString());
            } else if (mpServerDetails.get(SftpFile.SFTP_PASSWORD) != null) {
                session = ssh.getSession(mpServerDetails.get(SftpFile.SFTP_USERNAME).toString(),
                        mpServerDetails.get(SftpFile.SFTP_HOSTNAME).toString());
                session.setPassword(mpServerDetails.get(SftpFile.SFTP_PASSWORD).toString());
            }
            session.setConfig(config);
            session.connect();
            if (session != null && !session.isConnected()) {
                logger.warn("**session is not connected going to connect the sftp session ** {} ", session.getHost());
                session.connect();
            }
            channel = (ChannelSftp) session.openChannel("sftp");
            if (channel != null && !channel.isConnected()) {
                logger.warn("**channel is not connected going to connect the sftp channel ** {} ",
                        channel.getSession().isConnected());
                channel.connect();
            }
            channel.put(file.getAbsolutePath(), dest.getConfig().get(TransporterFileConstants.SFTP_DIRECTORY).toString()
                    + File.separatorChar + dest.getFileName(), new SystemOutProgressMonitor());

        }
        catch (NonRetriableException e) {
            throw new NonRetriableException(e);
        }
        catch (Exception e) {
            logger.error(
                    "Error occured while uploading file having name " + dest.getFileName() + " from remote directory:"
                            + dest.getConfig().get(TransporterFileConstants.SFTP_DIRECTORY).toString(),
                    e);
            logger.error("SFTP Exception : ", e);
            throw new RetriableException(e);
        }
        finally {
            if (null != channel && channel.isConnected()) {
                try {
                    channel.disconnect();
                }
                catch (Throwable e) {
                    logger.error("Error while disconnecting channel : ", e);
                }
            }
            if (null != session) {
                try {
                    session.disconnect();
                }
                catch (Throwable e) {
                    logger.error("Error while returning object to sftp pool : ", e);
                }
            }
        }

Can someone help me understand why I might be getting this exception?

SFTP server configurations are

MaxSessions 50
Capacity - 25 GB
4 core server with 10 GB Ram

A snippet of error message

com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset
    at com.jcraft.jsch.Session.connect(Session.java:558) ~[honeybee-engine.jar:na]

If this would keep coming, my data processing would not be consistent.

Answer 1

MaxSessions 50

The SSH server MaxSessions parameter limits the number of "sessions" that can run through a single SSH connection. You're only running one session--the SFTP session--through each connection, so the MaxSessions limit isn't particularly relevant to you.

Your problem may be with the MaxStartups setting:

MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100....

Basically, if there are too many clients connected to the server which haven't authenticated yet, the server will drop some of those connections. If your application is opening too many connections to the server at the same time, the server may be dropping some of those connections. The solution here is to adjust the value of MaxStartups, or change your application not to open so many connections at once.

There is also an operating system limit called the listen backlog. Basically, the operating system will only hold on to a certain number of pending TCP connections. If enough connection attempts come in at the same time, and the ssh server process isn't fast enough at accepting them, then the OS will drop some of the connection requests. The SSH server requests a backlog of 128 connections, but the OS may be capping the backlog at a lower value. If your SSH server is busy enough, you may be running into this limit.