Python and .Net Integration options

Question

I want to integrate Python with C#. I found two approaches using Interprocess communication and IronPython

Interprocess communication requires Python.exe to be installed on all client machines so not a viable solution.

We started using IronPython, but it only supports 2.7 python version for now. We are using 3.7 version.

Following code we tried using IronPython:

private void BtnJsonPy_Click(object sender, EventArgs e)
    {
        // 1. Create Engine
        var engine = Python.CreateEngine();

        //2. Provide script and arguments
        var script = @"C:\Users\user\source\path\repos\SamplePy\SamplePy2\SamplePy2.py"; // provide full path
        var source = engine.CreateScriptSourceFromFile(script);

        // dummy parameters to send Python script 
        int x = 3;
        int y = 4;

        var argv = new List<string>();
        argv.Add("");
        argv.Add(x.ToString());
        argv.Add(y.ToString());

        engine.GetSysModule().SetVariable("argv", argv);

        //3. redirect output
        var eIO = engine.Runtime.IO;

        var errors = new MemoryStream();
        eIO.SetErrorOutput(errors, Encoding.Default);

        var results = new MemoryStream();
        eIO.SetOutput(results, Encoding.Default);

        //4. Execute script
        var scope = engine.CreateScope();

        var lib = new[]
        {
            "C:\\Users\\user\\source\\repos\\SamplePy\\CallingWinForms\\Lib",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9\\lib",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.StdLib.2.7.9"
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32\\Lib",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\python.exe",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\DLLs"
        };

        engine.SetSearchPaths(lib);
        engine.ExecuteFile(script, scope);
        //source.Execute(scope);

        //5. Display output
        string str(byte[] x1) => Encoding.Default.GetString(x1);

        Console.WriteLine("Errrors");
        Console.WriteLine(str(errors.ToArray()));
        Console.WriteLine();
        Console.WriteLine("Results");
        Console.WriteLine(str(results.ToArray()));

        lblAns.Text = str(results.ToArray());
    }

The problem sometimes is, to do heavy Machine Learning programming we need to add "Modules". These Modules would be dependent on other modules. This increases point 4, Execute Scripts part of code, as more data path of that module has to be given here var lib = new[] and also some modules are not supported with Iron Python as well (for e.g. modules concerning OCR operations etc.)

Due to these limitations I found Pythonnet which also helps in integrating .net applications with Python. But I am new to it, so want some ideas on implementing the same, and code samples available, and is it feasible or recommended to use with Python 3.7

I checked that setting up Pythonnet is cumbersome initially, so want help or steps on how to set up the same. Also would like to know if in future Iron Python would support Python 3.X as well or not.

Answer 1

I am not familiar with IronPython, but I use pythonnet quite a lot for the same purpose - integrate Python with C#, so I can elaborate on that.

The advantage of using pythonnet for your purposes is having all the CPython packages available for you to use (numpy, scipy, pandas, Theano, Keras, scikit-learn etc), but avoiding the overhead of calling python.exe as separate process (pythonnet works by loading pythonXY.dll into your process).

Pay attention that pythonnet also requires to have stand-alone Python availiable, but you can use Embeddable Python package which is very light-weight and can be distributed with your application.

pythonnnet supports Python 3.7, but the published NuGet packages are only for Python 3.5. You have several choices to obtain pythonnet for Python 3.7:

1. Download pythonnet wheel package from PyPi and extract Python.Runtime.dll from it
2. Download NuGet package from pythonnet appveyor build artifacts, as advised on pythonnet installation wiki
3. Build from sources

Important note: pythonnet version has to match your Python version and bitness. For example, if you are using Python 3.7 32-bit, download pythonnet-2.4.0-cp37-cp37m-win32.whl. If your Python is 64-bit, download pythonnet-2.4.0-cp37-cp37m-win_amd64.whl. Your C# project platform target also has to match (x86 for 32-bit or x64 for 64-bit).

Code sample with similar functionality to what you have posted, using pythonnet (tested with Python 3.7.4 on Windows 7 and pythonnet NuGet from latest build artifacts):

        private void Test()
        {
            // Setup all paths before initializing Python engine
            string pathToPython = @"C:\Users\user\AppData\Local\Programs\Python\Python37-32";
            string path = pathToPython + ";" +
                          Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
            Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
            Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);

            var lib = new[]
            {
                @"C:\Users\user\source\path\repos\SamplePy\SamplePy2",
                Path.Combine(pathToPython, "Lib"),
                Path.Combine(pathToPython, "DLLs")
            };

            string paths = string.Join(";", lib);
            Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);

            using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
            {
                try
                {
                    // import your script into the process
                    dynamic sampleModule = Py.Import("SamplePy");

                    // It is more maintainable to communicate with the script with
                    // function parameters and return values, than using argv 
                    // and input/output streams.

                    int x = 3;
                    int y = 4;
                    dynamic results = sampleModule.sample_func(x, y);
                    Console.WriteLine("Results: " + results);
                }
                catch (PythonException error)
                {
                    // Communicate errors with exceptions from within python script -
                    // this works very nice with pythonnet.
                    Console.WriteLine("Error occured: ", error.Message);
                }
            }
        }

SamplePy.py:

def sample_func(x, y):
    return x*y