How to position Landmarks over xamarin forms Image view?

Question

I'm using a face detection service that given a picture returns me face landmarks positions. My goal is to position these landmarks on the picture I'm showing in the screen.

My idea is to use an AbosluteLayout with the Image view and position the Landmarks over the picture. The overlapping works good. The problem is that the points of the landmark refer to the original picture size coordinates and the rendered image has completely different size. I'm using AspectFit for the moment, so full picture is shown in the AbsoluteLayout preserving the aspect ratio, usually with letterboxing.

What I've tried is to take in account the screen density and apply it to the original landmark points like that:

rightEyePoint.X = Convert.ToInt32(rightEyePoint.X / displayDensity);

and it get's closer but not on all pictures (differs from landscape/portrait).

Then I thought that I might want to know the scale factor of the rendered image considering the original one, and by that I could transform all the Landmark points but I don't know how to do that. I tried to create a custom image view and renderer to try to get the scale calculations from it, but wasn't successful as I didn't get to interfere on where the calculations for the native image component are made :

public class CustomImageView : Image
{
    protected override SizeRequest OnMeasure(double widthConstraint, double heightConstraint)
    {
        return base.OnMeasure(widthConstraint, heightConstraint);


    }

    protected override void OnSizeAllocated(double width, double height)
    {
        base.OnSizeAllocated(width, height);

    }

    protected override void OnPropertyChanged([CallerMemberName] string propertyName = null)
    {
        base.OnPropertyChanged(propertyName);
    }
}

public class CustomImageViewRenderer : ImageRenderer
{
    public CustomImageViewRenderer(Context context) : base(context) { }

    protected override void OnElementChanged(ElementChangedEventArgs<Image> e)
    {
        base.OnElementChanged(e);
        if(e.NewElement != null)
        {
            var image = Control as ImageView;

        }
    }
    protected override void OnSizeChanged(int w, int h, int oldw, int oldh)
    {
        base.OnSizeChanged(w, h, oldw, oldh);
    }

    protected override Task TryUpdateBitmap(Image previous = null)
    {
        return base.TryUpdateBitmap(previous);
    }
}

Then I thought I could calculate myself the current rendered image size but I can't seem to get it. So I only get the xamarin forms Image component size, but not the image rendered in it. At this point I thought that I could caculate, depending on the aspect ratio, the size of the rendered image, as I'm using AspectFit but got stuck in the process.

The axml (sample):

<AbsoluteLayout BackgroundColor="Fuchsia" HorizontalOptions="Fill" VerticalOptions="Fill">
    <Controls:PinchToZoomContainer HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand" CurrentPosition="{Binding CurrentPhotoPosition}" CurrentScale="{Binding CurrentPhotoScale}" AbsoluteLayout.LayoutBounds="0, 0, 1, 1" AbsoluteLayout.LayoutFlags="All" IsEnabled="{ Binding CurrentStep, Converter={StaticResource checkIntToBooleanValueConverter}, ConverterParameter=2}">
        <Image BackgroundColor="Blue" Source="{Binding CurrentPhotoPath}">
              <Image.Behaviors>
                  <behaviors:EventToCommandBehavior 
                                                EventName="SizeChanged" 
                                                Command="{Binding ImageSizeChangedCommand}"/>
                                        </Image.Behaviors>
                                    </Image>
                                    <!--<Controls:CustomImageView Source="{Binding CurrentPhotoPath}" BackgroundColor="Purple" SizeChanged=""/-->
                                </Controls:PinchToZoomContainer>

                                <!-- calculated landmarks -->
                                <Controls:LandmarkView 
                                Padding="2" 
                                DragMode="TouchAndRelease" 
                                LBounds="{Binding HairStartLandmarkBounds, Mode=TwoWay}" 
                                DragDirection="Vertical"
                                HorizontalOptions="FillAndExpand"
                                VerticalOptions="Start"
                                IsVisible="{ Binding CurrentStep, Converter={StaticResource checkIntToBooleanValueConverter}, ConverterParameter=3}"
                                x:Name="HairStartLandmark" 
                                ToggleDraggingCommand="{Binding HairStartLandmarkTouchedCommand}">
                                    <Controls:LandmarkView.Content>
                                        <AbsoluteLayout>
                                            <Image BackgroundColor="Red" 
                                                   WidthRequest="{Binding LandmarkLineWidth}" 
                                                   AbsoluteLayout.LayoutBounds= "{Binding LandmarkLineXPos}"
                                                   AbsoluteLayout.LayoutFlags="YProportional,WidthProportional"/>
                                            <ffimageloading:CachedImage 
                                                WidthRequest="{Binding HairStartLandmarkIconSize}" 
                                                HeightRequest="{Binding HairStartLandmarkIconSize}"
                                                Source="{Binding HairStartLandmarkIcon, Converter={StaticResource SvgImageSourceConverter}}">
                                            </ffimageloading:CachedImage>
                                        </AbsoluteLayout>
                                    </Controls:LandmarkView.Content>
                                </Controls:LandmarkView>
</AbsoluteLayout>

So my question is, how can I easily get the rendered image coordinates system origin and scale factor to apply to the landmark points coordinates and position them correctly on the rendered image?

UPDATE:

I add an image of the current status of what I have and what I think I need. The blue background is the Xamarin.Forms Image view background:

Thank you very much

Answer 1

In the end I used a simpler system. Here's the code of the method I created to get the rendered image bounds.

private void CalculateRendredImageBounds()
    {
        Point origin = new Point();

        _renderedImageRatio = CurrentPhotoSize.Width / CurrentPhotoSize.Height;

        if (CurrentPhotoSize.Height > CurrentPhotoSize.Width)
        {
            //case height greater than width (portrait)
            _renderedPhotoWidth = (CurrentPhotoSize.Width * RenderedImageContainerSize.Height) / CurrentPhotoSize.Height;
            origin.X = Convert.ToInt32((RenderedImageContainerSize.Width - _renderedPhotoWidth) / 2);
            origin.Y = 0;

            _renderedImageBounds = new Rectangle(origin.X, origin.Y, _renderedPhotoWidth, RenderedImageContainerSize.Height);
        }
        else
        {
            //case width greater than height (landscape)
            _renderedPhotoHeight = (CurrentPhotoSize.Height * RenderedImageContainerSize.Width) / CurrentPhotoSize.Width;
            origin.X = 0;
            origin.Y = Convert.ToInt32((RenderedImageContainerSize.Height - _renderedPhotoHeight) / 2);

            _renderedImageBounds = new Rectangle(origin.X, origin.Y, RenderedImageContainerSize.Width, _renderedPhotoHeight);
        }
    }

Then when you want to find a point in the container coordinate system you can use the following

X Landscape:

Convert.ToInt32((originalXPos * RenderedImageContainerSize.Width) / CurrentPhotoSize.Width);

Y Landscape:

Convert.ToInt32((originalYPos * _renderedPhotoHeight) / CurrentPhotoSize.Height + (RenderedImageContainerSize.Height - _renderedPhotoHeight) / 2);

X Portrait:

Convert.ToInt32((originalXPos * _renderedPhotoWidth) / CurrentPhotoSize.Width + (RenderedImageContainerSize.Width - _renderedPhotoWidth) / 2);

Y Portrait:

Convert.ToInt32((originalYPos * RenderedImageContainerSize.Height) / CurrentPhotoSize.Height);

Note: I don't show the private properties declaration, but I think the code it's readable enough.

Answer 2

Some math incoming.

Xamarin Forms uses Device-Independent Pixels, so my guess is what you need is the ratio of your image to render inside the frame. Let's say its A1:A2 where one unit is x long. The ratio of your frame is given, let's say B1:B2 where one unit is y long. Our problem is that x is unknown. So we need to get it out of the equation (literally :D)

First, you need to determine which is bigger, A1 or A2. Why? Because the side that is longer, will be the side that will fit entirely in the frame. (E.g. if your image is wider than higher, than you will have letterboxes at the top and bottom. If your picture is higher than wider, it will have letterboxes at the left and right sides.) Let's say A2 is bigger (meaning that your picture's height is bigger than its width).

In that case, A2 * x should be equal to B2 * y. So, x = B2 / A2 * y

Your picture will be positioned in the middle of your frame, so the other side of the frame will be like:

[{z} wide blue background] - [image width] - [{z} wide blue background]

Now we only need to get the length of z. Which is (B1 * y - (B2 / A2 * y) * A1) / 2.

---

Now to make it more understandable with numbers:

Size of background frame: 600 x 400 (width x height) - can be retrieved from code - means that B1 = 3, B2 = 2 and y = 200.

Ratio of image: 4:5 - can be retrieved from original image size - means that A1 = 4, A2 = 5, and we do not know what x is. (Since that is part of what you need.)

This results in x = 2 / 5 * 200 = 80.

Which ultimately means, that z is (3 * 200 - 80 * 4) / 2 = (600 - 320) / 2 = 140.

Which means, that in this case, your image's first dip is at (140, 0) position inside your background frame.

I hope this somewhat helped and lets you create your own calculation method.