CODEWITHSUNDEEP
pythonunit-testingmockingparamikoremote-server
Chris Sewell
Chris Sewell

Reputation: 719

Mock a Remote Host in Python

I am writing some functions, using paramiko, to execute commands and create files on a remote host. I would like to write some unit tests for them, but I don't know what would be the simplest way to achieve this? This is what I envisage as being an example outline of my code:

import os
import paramiko
import pytest

def my_function(hostname, relpath='.', **kwargs):
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname, **kwargs)
    sftp = ssh.open_sftp()
    sftp.chdir(relpath)
    stdin, stdout, stderr = ssh.exec_command("echo hallo > test.txt")

@pytest.fixture("module")
def mock_remote_host():
    # start a remote host here with a local test path
    try:
        yield hostname, testpath, {"username":"bob", "password":"1234"}
    finally:
        # delete the test path
        # close the remote host

def test_my_function(mock_remote_host):
    hostname, dirpath, kwargs = mock_remote_host
    my_function(hostname, **kwargs)
    filepath = os.path.join(dirpath, 'test.txt')
    assert os.path.exists(filepath)

I have had a look at the paramiko test modules, but they seem quite complex for my use case and I'm not sure how to go about simplifying them.

Upvotes: 6

Views: 13334

Answers (5)

Asger Svenning
Asger Svenning

Reputation: 23

I didn't find any good solutions to my use-case in any of the threads I could find. This one seemed like the most relevant one, so I will post my solution here. Obviously, it is a bit different since it seems that the thread here only relates to an SSH host, but that should just be simpler. Basically the only difference should be to use a different Docker Image.

I am/was developing an LFTP-wrapper for Python, to facility on-the-fly datastreaming with high-level code (this is just a minimal example I pulled from the project README):

# Set the environment variables (only necessary in a non-interactive setting)
# If you are simply running this as a Python script, 
# you can omit these lines and you will be prompted to set them interactively
import os
os.environ["PYREMOTEDATA_REMOTE_USERNAME"] = "username"
os.environ["PYREMOTEDATA_REMOTE_URI"] = "storage.example.com"
os.environ["PYREMOTEDATA_REMOTE_DIRECTORY"] = "/MY_PROJECT/DATASETS"
os.environ["PYREMOTEDATA_AUTO"] = "yes"

from pyremotedata.implicit_mount import IOHandler

handler = IOHandler()

with handler as io:
    print(io.ls())

# The configuration is persistent, but can be removed using the following:
from pyremotedata.config import remove_config
remove_config()

However, writing unit tests for this code, was pretty hard, since in order to create the most realistic scenario, I felt I needed to mock an SFTP server. Well long story short, I found the easiest way was to start a Docker Container from atmoz/sftp and provide it with a temporary SSH key generated and removed during testing. If anyone else needs this for their project, it can be done like this:

# SSH setup
mkdir -p /tmp # Ensure /tmp exists, mostly for sanity
ssh-keygen -t rsa -b 4096 -f /tmp/temp_sftp_key -q -N "" # Create a temporary SSH key
eval $(ssh-agent -s) # Start the ssh-agent
ssh-add /tmp/temp_sftp_key # Add the temporary SSH key to the ssh-agent
# echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> $GITHUB_ENV # Needed for GitHub Actions, not for local testing

# Mock SFTP server setup
mkdir -p /tmp/upload # Create a temporary directory which will serve as the root of the mock SFTP server file system
chmod ugo+rwx /tmp/upload # Set flexible permissions to avoid invalid permissions issues, which shouldn't arise with a properly set up production SFTP server
sudo docker run --name mock_sftp_server -p 0.0.0.0:2222:22 -d \ # Basic Docker Container setup
    -v /tmp/temp_sftp_key.pub:/home/foo/.ssh/keys/temp_sftp_key.pub:ro \ # Add the temporary key to authorized_keys on the mock SFTP server
    -v /tmp/upload:/home/foo/upload \ # Add the temporary SFTP server directory for file manipulation on the mock SFTP server
    atmoz/sftp foo::1001 # Get the atmoz/sftp image and add a user "foo" without a password
for i in {1..10}; do # Tries to add the SSH fingerprints to known_hosts a few times, since the Docker Container may need a little time to come up and running properly
    ssh-keyscan -p 2222 -H 0.0.0.0 >> ~/.ssh/known_hosts && break
    echo "Waiting for SFTP server to be ready..."
    sleep 1 # wait for 1 second before retrying
done

# Run your tests here
python3 -m unittest discover tests # or however else you want to run your tests

# Clean the environment
ssh-keygen -R 0.0.0.0 # Remove the SSH fingerprints from known_hosts
sudo docker stop mock_sftp_server # Stop the Docker Container
sudo docker rm mock_sftp_server # Remove the Docker Container
ssh-add -d /tmp/temp_sftp_key # Remove the temporary SSH key from the ssh-agent
rm /tmp/temp_sftp_key /tmp/temp_sftp_key.pub # Delete the temporary SSH key
eval "$(ssh-agent -k)" # Kill the ssh-agent
rm -rf /tmp/upload # Remove the temporary directory

Upvotes: 0

Jess
Jess

Reputation: 3725

I agree with HraBal, because of "Infrastructure as code". You can treat virtual machine as a block of code.

For example:

  1. you can use vagrant or docker to initialize a SSH server and then, and modify your DNS configuration file. target domain 127.0.0.1
  2. put application into the server. and run paramiko to connect target domain and test what you want.

I think it is the benefit that you can do this for all programming languages and not need to reinvent the wheel . In addition, you and your successors will know the detail of the system.

(My English is not very good)

Upvotes: 0

Chris Sewell
Chris Sewell

Reputation: 719

To answer my own question, I have created: https://github.com/chrisjsewell/atomic-hpc/tree/master/atomic_hpc/mockssh.

As the readme discusses; it is based on https://github.com/carletes/mock-ssh-server/tree/master/mockssh with additions made (to implement more sftp functions) based on https://github.com/rspivak/sftpserver

The following changes have also been made:

  • revised users parameter, such that either a private_path_key or password can be used
  • added a dirname parameter to the Server context manager, such that the this will be set as the root path for the duration of the context.
  • patched paramiko.sftp_client.SFTPClient.chdir to fix its use with relative paths.

See test_mockssh.py for example uses.

Upvotes: 2

running.t
running.t

Reputation: 5709

I think what you really need to mock is paramiko.SSHClientobject. You are unittesting your function my_function, you can assume paramiko module works correctly and the only thing you need to unit test is if my_function calls methods of this paramiko.SSHClient in correct way.

To mock paramiko.SSH module you can use unittest.mock and decorate your test_my_function function with @mock.patch.object(paramiko.SSHClient, sshclientmock). You have to define sshclientmock as some kind of Mock or MagicMock first.

Also in python 2.7 there is some equivalent of unittest.mock but I dont remember where to find it exactly.

EDIT: As @chepner mentioned in comment. For python 2.7 you can find mock module in pypi and install it using pip install mock

Upvotes: 5

Hrabal
Hrabal

Reputation: 2523

If you want to test remote connectivity, remote filesystem structure and remote path navigation you have to set-up a mock host server (a VM maybe). In other words if you want to test your actions on the host you have to mock the host.

If you want to test your actions with the data of the host the easiest way seems to proceed as running.t said in the other answer.

Upvotes: 0

Related Questions