hg clone hangs at sending between command

Question

I'm trying to clone a Mercurial repository hosted on BitBucket via SSH from the Windows command line.

When I add the --debug flag I can see that it never gets past the 'sending between command'

When I ctrl+c, it just shows interrupted! without any other error messages.

I'm also looking at the Resource Monitor and it's not showing any related network traffic after the first minute or so.

I have tried adding the --noupdate option and the --uncompressed option with the same results.

Windows Server 2012 with just Mercurial - No TortoiseHg

Any ideas?

Is there a reason the 'sending between command' would take awhile? The repository is about 150MB

Answer 1

I know I'm late to the party, since Bitbucket has long dropped the Mercurial support, still this might be relevant to those setting it up on Windows to use with any other service or in a self-hosted environment.

I assume that you've set plink as your SSH client. If not, this can be easily done by adding the ssh parameter in your Mercurial configuration. Here is what my mercurial.ini file's contents look like:

>hg config --edit
[ui]
ssh="C:\programs\putty\plink.exe" -ssh -2

I spent like two hours tinkering around with my setup before I had an Aha! moment and tried to connect to the server with plink to see if the problem lies there:

>plink.exe hg-ssh@hg.example.com -noagent id
The server's host key is not cached in the registry. You
have no guarantee that the server is the computer you
think it is.
The server's ssh-ed25519 key fingerprint is:
ssh-ed25519 255 90:8d:b0:26:c5:6e:74:2a:6c:3d:80:42:54:9e:15:47
If you trust this host, enter "y" to add the key to
PuTTY's cache and carry on connecting.
If you want to carry on connecting just once, without
adding the key to the cache, enter "n".
If you do not trust this host, press Return to abandon the
connection.
Store key in cache? (y/n)

Bingo!

As you can see here, plink prompts for the input, and we have to provide it before it can proceed any further. So this is what makes hg hang indefinitely if we don't.

You can also download Process Explorer, open it and choose Show Process Tree (Ctrl+T) from the View menu, and search for the process plink. You can see that this process is running and is a child of hg.

I answered "y", and the key was cached in a registry entry under the

Computer\HKEY_CURRENT_USER\SOFTWARE\SimonTatham\PuTTY\SshHostKeys

The new entry contains the remote server name and the key itself.

Once that was out of the way, I could go about my business using clone, push, pull, as well as other commands, like in and out:

>hg clone ssh://hg-ssh@hg.example.com/review
destination directory: review
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 4 changes to 4 files
new changesets f82779871203:e9b1a2b79f98 (1 drafts)
updating to branch default
4 files updated, 0 files merged, 0 files removed, 0 files unresolved

It looks like hg messes with the input and the output of the SSH client, and you never get to see what is going on there, nor can you interact with it. And specifying the --debug option makes no difference.

You can also supply the -sshlog option to plink to save the log protocol details of a session to a file:

[ui]
ssh="C:\programs\putty\plink.exe" -ssh -2 -sshlog plink.log

Here is a portion of the log, where it says that a host key is unknown to the client:

Incoming packet #0x2, type 21 / 0x15 (SSH2_MSG_NEWKEYS)
Event Log: Server also has ecdsa-sha2-nistp256/ssh-rsa host keys,
           but we don't know any of them
Event Log: Host key fingerprint is:
Event Log: ssh-ed25519 255 90:8d:b0:26:c5:6e:74:2a:6c:3d:80:42:54:9e:15:47

And it is getting stuck around that spot waiting for the user input.

Out of curiosity I downloaded the Mercurial source code and tracked the debug messages down to the method _performhandshake in hg/mercurial/sshpeer.py:

$ hg clone https://www.mercurial-scm.org/repo/hg

$ grep -R 'sending between command' hg

$ less hg/mercurial/sshpeer.py
ui.debug(b'sending hello command\n')
ui.debug(b'sending between command\n')

Full sshpeer.py source code.

You can see up the call stack there's a call inside the instance to the _makeconnection, which in turn calls the procutil.popen4, which then uses subprocess.Popen class constructor to spawn a new process, and subprocess.PIPE is used as a value of the stdin, stdout, and stderr arguments.

And from the subprocess manual, you can see that:

PIPE indicates that a new pipe to the child should be created.

So this pretty much explains why you cannot interact with the SSH client, as all its standard input, standard output, and standard error are eaten up by Mercurial.

Anyone reading this answer, give this solution a try, and let us know if it was helpful!