CODEWITHSUNDEEP
pythonmappingmeshblendingmediapipe
LoopingDev
LoopingDev

Reputation: 864

how to create blendshapes from mediapipe facemesh

I need to know how to create and store blendshapes in a certain position of each part of the face (ex: left eye, left eyebrow, right eye, right eyebrow, nose, upper lip, lower lip, left cheek, right cheek )

using the landmarks output from the camera feed?

example in the documents:
app.py

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
mp_face_mesh = mp.solutions.face_mesh

# For webcam input:
drawing_spec = mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
cap = cv2.VideoCapture(0)
with mp_face_mesh.FaceMesh(
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5) as face_mesh:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
      print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
      continue

    # To improve performance, optionally mark the image as not writeable to
    # pass by reference.
    image.flags.writeable = False
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = face_mesh.process(image)

    # Draw the face mesh annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_face_landmarks:
      for face_landmarks in results.multi_face_landmarks:
        mp_drawing.draw_landmarks(
            image=image,
            landmark_list=face_landmarks,
            connections=mp_face_mesh.FACEMESH_TESSELATION,
            landmark_drawing_spec=None,
            connection_drawing_spec=mp_drawing_styles
            .get_default_face_mesh_tesselation_style())
        
    # Flip the image horizontally for a selfie-view display.
    cv2.imshow('MediaPipe Face Mesh', cv2.flip(image, 1))
    if cv2.waitKey(5) & 0xFF == 27:
      break
cap.release()
enter code here

the result would be something like this

what I'm trying to do is to create some blendshapes for each part of the face as I've mentioned earlier

how to create blendshapes by (ex: pressing space ) while the loop is running and the points keeps on updating? ( for example while default pose, smiling, frowning, angry .. etc )

how to store it in landmarks points array or mesh/es or other files?

Upvotes: 0

Views: 2297

Answers (1)

Bryan Pratte
Bryan Pratte

Reputation: 31

This is actually a very hard problem and the FM points alone are not going to get you where you need to be to have blendshapes that are uniform across a broad and diverse set of faces. We have been building and collecting training data for this problem for over a year.

We have an SDK that is in beta you might want to try https://joinhallway.com/

Note we do not allow its use in collecting training data but if you have an application that just needs shapes such as face tracking you could use this instead of training your own model from scratch.

Upvotes: 0

Related Questions