Resizing the image in C# and sending it to OpenCV results in distorted image

Question

This is a follow-up question related to this one. Basically, I have a DLL which uses OpenCV to do image manipulation. There are two methods, one accepting an image-Path, and the other one accepting a cv::Mat. The one working with image-path works fine. The one that accepts an image is problematic.

Here is the method which accepts the filename (DLL):

CDLL2_API void Classify(const char * img_path, char* out_result, int* length_of_out_result, int N)
{
    auto classifier = reinterpret_cast<Classifier*>(GetHandle());

    cv::Mat img = cv::imread(img_path);
    cv::imshow("img recieved from c#", img);
    std::vector<PredictionResults> result = classifier->Classify(std::string(img_path), N);
    std::string str_info = "";
    //...
    *length_of_out_result = ss.str().length();
}

Here is the method which accepts the image (DLL):

CDLL2_API void Classify_Image(unsigned char* img_pointer, unsigned int height, unsigned int width, 
                              int step, char* out_result, int* length_of_out_result, int top_n_results)
    {
        auto classifier = reinterpret_cast<Classifier*>(GetHandle());
        cv::Mat img = cv::Mat(height, width, CV_8UC3, (void*)img_pointer, step);
        std::vector<Prediction> result = classifier->Classify(img, top_n_results);

        //...
        *length_of_out_result = ss.str().length();
    }

Here is the code in C# application: DllImport:

[DllImport(@"CDll2.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
        static extern void Classify_Image(IntPtr img, uint height, uint width, int step, byte[] out_result, out int out_result_length, int top_n_results = 2);

The method which sends the image to the DLL:

private string Classify_UsingImage(Bitmap img, int top_n_results)
{
    byte[] res = new byte[200];
    int len;
    BitmapData bmpData;
    bmpData = img.LockBits(new Rectangle(0, 0, img.Width, img.Height), ImageLockMode.ReadOnly, img.PixelFormat);
    Classify_Image(bmpData.Scan0, (uint)bmpData.Height, (uint)bmpData.Width, bmpData.Stride, res, out len, top_n_results);
    //Remember to unlock!!!
    img.UnlockBits(bmpData); 
    string s = ASCIIEncoding.ASCII.GetString(res);
    return s;
}

Now, this works well when I send an image to the DLL. if I use imshow() to show the received image, the image is shown just fine.

The Actual Problem:

However, when I resize the very same image and send it using the very same method above, the image is distorted.

I need to add that, If I resize an image using the given C# method below, then save it, and then pass the filename to the DLL to be opened using Classify(std::string(img_path), N); it works perfectly.

Here is the screenshot showing an example of this happening:
Image sent from C` without being resized:

When The same image is first resized and then sent to the DLL:

Here the image is first resized (in C#), saved to the disk and then its filepath sent to the DLL:

This is the snippet responsible for resizing (C#):

/// <summary>
/// Resize the image to the specified width and height.
/// </summary>
/// <param name="image">The image to resize.</param>
/// <param name="width">The width to resize to.</param>
/// <param name="height">The height to resize to.</param>
/// <returns>The resized image.</returns>
public static Bitmap ResizeImage(Image image, int width, int height)
{
    var destRect = new Rectangle(0, 0, width, height);
    var destImage = new Bitmap(width, height);

    destImage.SetResolution(image.HorizontalResolution, image.VerticalResolution);

    using (var graphics = Graphics.FromImage(destImage))
    {
        graphics.CompositingMode = CompositingMode.SourceCopy;
        graphics.CompositingQuality = CompositingQuality.HighQuality;
        graphics.InterpolationMode = InterpolationMode.HighQualityBicubic;
        graphics.SmoothingMode = SmoothingMode.HighQuality;
        graphics.PixelOffsetMode = PixelOffsetMode.HighQuality;

        using (var wrapMode = new ImageAttributes())
        {
            wrapMode.SetWrapMode(WrapMode.TileFlipXY);
            graphics.DrawImage(image, destRect, 0, 0, image.Width, image.Height, GraphicsUnit.Pixel, wrapMode);
        }
    }

    return destImage;
}

and this is the original image

This is the Classify method which uses the filepath to read the images:

std::vector<PredictionResults> Classifier::Classify(const std::string & img_path, int N)
{
    cv::Mat img = cv::imread(img_path);
    cv::Mat resizedImage;
    std::vector<PredictionResults> results;
    std::vector<float> output;

   // cv::imshow((std::string("img classify by path") + type2str(img.type())), img);
    if (IsResizedEnabled())
    {
        ResizeImage(img, resizedImage);
        output = Predict(resizedImage);
    }
    else
    {
        output = Predict(img);
        img.release();
    }

    N = std::min<int>(labels_.size(), N);
    std::vector<int> maxN = Argmax(output, N);
    for (int i = 0; i < N; ++i)
    {
        int idx = maxN[i];
        PredictionResults r;
        r.label = labels_[idx];
        r.accuracy = output[idx];
        results.push_back(r);
    }

    return results;
}

And this is the ResizeImage used in the method above :

void Classifier::ResizeImage(const cv::Mat & source_image, cv::Mat& resizedImage)
{
    Size size(GetResizeHeight(), GetResizeHeight());
    cv::resize(source_image, resizedImage, size);//resize image

    CHECK(!resizedImage.empty()) << "Unable to decode image ";
}

Problem 2:
Distortion after resizing aside, I am facing a discrepancy between resizing in C# and resizing using OpenCV itself.
I have created another method using EmguCV (also given below) and passed the needed information and did not face any such distortions which happen when we resize the image in C# and send it to the DLL.
However, this discrepancy made me want to understand what is causing these issues.
Here is the method which uses EmguCV.Mat is the code that works irrespective of resizing:

private string Classify_UsingMat(string imgpath, int top_n_results)
{
    byte[] res = new byte[200];
    int len;

    Emgu.CV.Mat img = new Emgu.CV.Mat(imgpath, ImreadModes.Color);

    if (chkResizeImageCShap.Checked)
    {
        CvInvoke.Resize(img, img, new Size(256, 256));
    }

    Classify_Image(img.DataPointer, (uint)img.Height, (uint)img.Width, img.Step, res, out len, top_n_results);

    string s = ASCIIEncoding.ASCII.GetString(res);
    return s;
}

Why do I care?
Because, I get a different accuracy when I use OpenCV resize (both when I use EMguCV's CvInvoke.resize() and cv::resize()) than what I get from resizing the image in C#, saving it to disk and send the image path to the openCV.
So I either need to fix the distortion happening when I deal with images in C#, or I need to understand why the resizing in OpenCV has different results than the C# resizing.

So to summarize issues and points made so far:

1. All situations intact, If we resize the image inside C# application, and pass the info normally as we did before, the image will be distorted (example given above)
2. If we resize the image, save it to the disk, and give its filename to the OpenCV to create a new cv::Mat, it works perfectly without any issues.

3. If I use EmugCV and instead of working with Bitmap, use Emug.CV.Mat and send the needed parameters using mat object from C#, no distortion happens.

   However, the accuracy I get from a resized image from C# (see #2), is different than the one I get from the resized image using OpenCV. This doesn't make any difference if I resize the image before hand using CvInvoke.Resize() from C#, and send the resulting image to the DLL, or send the original image (using EmguCV) and resizing it in the C++ code using cv::resize(). This is what prevents me from using the EmguCV or passing the image original image and resizing it inside the DLL using OpenCV.

Here are the images with different results, showing the issues:

--------------------No Resizing------------------------------
1.Using Bitmap-No Resize, =>safe, acc=580295
2.Using Emgu.Mat-No Resize =>safe, acc=0.580262
3.Using FilePath-No Resize, =>safe, acc=0.580262
--------------------Resize in C#------------------------------
4.Using Bitmap-CSharp Resize, =>unsafe, acc=0.770425
5.Using Emgu.Mat-EmguResize, =>unsafe, acc=758335
6.**Using FilePath-CSharp Resize, =>unsafe, acc=0.977649**
--------------------Resize in DLL------------------------------
7.Using Bitmap-DLL Resize, =>unsafe, acc=0.757484
8.Using Emgu.DLL Resize, =>unsafe, acc=0.758335
9.Using FilePath-DLL Resize, =>unsafe, acc=0.758335

I need to get the accuracy which I get at #6. as you can see the EmguCV resize and also the OpenCV resize function used in the DLL, act similar and don't work as expected (i.e. like #2)!, The C# resize method applied on the image is problematic, while if it is resized, saved and the filename passed, the result will be fine.

You can see the screen shots depicting different scenarios here: https://i.sstatic.net/LnMMu.jpg

Answer 1

What I did was to use imdecode as EdChum suggested. This is how the functions in the DLL and C# look now:

#ifdef CDLL2_EXPORTS
#define CDLL2_API __declspec(dllexport)
#else
#define CDLL2_API __declspec(dllimport)
#endif

#include "classification.h" 
extern "C"
{
    CDLL2_API void Classify_Image(unsigned char* img_pointer, long data_len, char* out_result, int* length_of_out_result, int top_n_results = 2);
//...
}

The actual method:

CDLL2_API void Classify_Image(unsigned char* img_pointer, long data_len,
                              char* out_result, int* length_of_out_result, int top_n_results)
{
    auto classifier = reinterpret_cast<Classifier*>(GetHandle());
    vector<unsigned char> inputImageBytes(img_pointer, img_pointer + data_len);
    cv::Mat img = imdecode(inputImageBytes, CV_LOAD_IMAGE_COLOR);

    cv::imshow("img just recieved from c#", img);

    std::vector<Prediction> result = classifier->Classify(img, top_n_results);
    //...
    *length_of_out_result = ss.str().length();
}

Here is the C# DllImport:

[DllImport(@"CDll2.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
static extern void Classify_Image(byte[] img, long data_len, byte[] out_result, out int out_result_length, int top_n_results = 2);

and this is the actual method sending the image back to the DLL:

private string Classify_UsingImage(Bitmap image, int top_n_results)
{
    byte[] result = new byte[200];
    int len;
    Bitmap img;

    if (chkResizeImageCShap.Checked)
        img = ResizeImage(image, int.Parse(txtWidth.Text), (int.Parse(txtHeight.Text)));
    else
        img = image;

    //this is for situations, where the image is not read from disk, and is stored in the memort(e.g. image comes from a camera or snapshot)
    ImageFormat fmt = new ImageFormat(image.RawFormat.Guid);
    var imageCodecInfo = ImageCodecInfo.GetImageEncoders().FirstOrDefault(codec => codec.FormatID == image.RawFormat.Guid);
    if (imageCodecInfo == null)
    {
        fmt = ImageFormat.Jpeg;
    }

    using (MemoryStream ms = new MemoryStream())
    {
        img.Save(ms, fmt);
        byte[] image_byte_array = ms.ToArray();
        Classify_Image(image_byte_array, ms.Length, result, out len, top_n_results);
    }

    return ASCIIEncoding.ASCII.GetString(result);
}

By doing this after resizing the image from C#, we don't face any distortions at all.

I couldn't, however, figure out why the resize on OpenCV part wouldn't work as expected!