How would I achieve this in opencv with an affine transform?

Question

I was wondering how I would replicate what is being done in this image:

To break it down:

1. Get facial landmarks using dlib (green dots)
2. Rotate the image so that the eyes are horizontal
3. Find the midpoint of the face by averaging the left most and right most landmarks (blue dot) and center the image on the x-axis
4. Fix the position along the y-axis by by placing the eye center 45% from the top of the image, and the mouth center 25% from the image

Right now this is what I have:

I am kind of stuck on step 3, which I think can be done by an affine transform? But I'm completely stumped on step 4, I have no idea how I would achieve it.

Please tell me if you need me to provide the code!

EDIT: So after look at @Gal Dreiman's answer I was able to center the face perfectly so that the blue dot in the the center of my image.

Although when I implemented the second part of his answer I end up getting something like this:

I see that the points have been transformed to the right places, but it's not the outcome that I had desired as it is sheered quite dramatically. Any ideas?

EDIT 2:

After switching the x, y coordinates for the center points around, this is what I got:

Answer 1

As I see you section 3, the easiest way to do that is by:

1. Find the faces in the image:

   faces = faceCascade.detectMultiScale(
       gray,
       scaleFactor=1.1,
       minNeighbors=5,
       minSize=(30, 30),
       flags = cv2.cv.CV_HAAR_SCALE_IMAGE
   )
   

2. For each face calculate the midpoint:

   for (x, y, w, h) in faces:
       mid_x = x + int(w/2)
       mid_y = y + int(h/2)
   

3. Affine transform the image to center the blue dot you already calculated:

   height, width = img.shape
   x_dot = ...
   y_dot = ...
   
   dx_dot = int(width/2) - x_dot
   dy_dot = int(height/2) - y_dot
   
   M = np.float32([[1,0,dx_dot],[0,1,dy_dot]])
   dst = cv2.warpAffine(img,M,(cols,rows))
   

Hope it was helpful.

Edit:

Regarding section 4: In order to stretch (resize) the image, all you have to do is to perform an affine transform. To find the transformation matrix, we need three points from input image and their corresponding locations in output image.

    p_1 = [eyes_x, eye_y]
    p_2 = [int(width/2),int(height/2)] # default: center of the image
    p_3 = [mouth_x, mouth_y]

    target_p_1 = [eyes_x, int(eye_y * 0.45)]
    target_p_2 = [int(width/2),int(height/2)] # don't want to change
    target_p_3 = [mouth_x, int(mouth_y * 0.75)]

    pts1 = np.float32([p_1,p_2,p_3])
    pts2 = np.float32([target_p_1,target_p_2,target_p_3])

    M = cv2.getAffineTransform(pts1,pts2)

    output = cv2.warpAffine(image,M,(height,width))

To clear things out:

1. eye_x / eye_y is the location of the eye center.
2. The same applies on mouth_x / mouth_y, for mouth center.
3. target_p_1/2/3 are the target points.

Edit 2: I see you are in trouble, I hope this time my suggestion will work for you:

There is another approach I can think of. you can perform sort of "crop" to the image by pointing on 4 points, lets define them as the 4 points which wrap the face, and change the image perspective according to their new position:

up_left = [x,y]
up_right = [...]
down_left = [...]
down_right = [...]

pts1 = np.float32([up_left,up_right,down_left,down_right])
pts2 = np.float32([[0,0],[300,0],[0,300],[300,300]])

M = cv2.getPerspectiveTransform(pts1,pts2)

dst = cv2.warpPerspective(img,M,(300,300))

So all you have to do is to define those 4 points. My suggestion, calculate the conture around the face (which you already did) and after that add delta_x and delta_y (or subtract) to the coordinates.