Creating different synthetic Body Shapes for one specific Human Pose for Training Pose classification

Question

I'm working on my master's thesis, which involves tracking singing poses to provide feedback on posture. Specifically, I am using pose images, such as those derived from keypoint detection models like YOLOv8 Pose, to classify various singing poses.

One of my goals is to generate synthetic data with diverse body shapes for each pose (each frame in one video), so I can train my model on various types of people—different genders, heights, body types (e.g., fat or thin)—and make the model more robust to differences in body shape.

Example:

Does anyone have experience or ideas on how to create synthetic body shapes based on pose images? Ideally, I would like to be able to:

1. Adjust body shape (e.g., fat, thin).
2. Change height and body proportions
3. Simulate different genders.

I looked into Blender, but I'm not sure if it provides a way to load an image of a specific pose and convert it into a synthetic body shape with that exact pose for changing body shape, gender, etc.

In my project, each movement in a singing pose can be very subtle, so I need highly accurate synthetic poses with different body shapes. Therefore, a tool with a pose estimation feature before generating the body shape would be more suitable.

Answer 1

I will broadly cover the necessary steps. Each and every one of them needs lengthy explanations but thanks to huge community of Blender, you'll be able to find tutorials explaining all of them in depth.

Body: There are two ways to approach this problem

1. Creating blendshapes to change body types & proportions.
2. Training a model to create body meshes based on your parameters. Microsoft has a paper on that, only for faces though but will give you an idea. Authors' one of the previous papers delves into the specifics more

Pose: After extracting the pose (preferably 3D pose), apply the positions of keypoints using inverse kinematics to the body. I am not sure but pose could be a parameter for generating body models to assume that pose using the second method I mentioned above.

If you are willing to use another tool to generate your data, check Unity's synthetic humans repo.