Blender and MediaPipe

and one big issue with it

I’m currently working on a project of mine that I call Window to the Past. Its premise is to have a window-like display that displays a scene from the past/future from the point of view of the viewer, a sort of Augmented Reality without the need for glasses.

I have never used the UnReal engine before so the video below is quite an achievement for me. It shows in some detail the experience and functionality of the Window. ahem, the video could benefit from some re-editing to shorten it but, f it, it is what it is.

The next step was to make some sort of minimal viable demo, and that was done in three.js and opencv.js. The choice fell on web architecture as its platform agnostic and could be easily maintained but it turned out it’s not mature enough, especially opencv.js, and is prone to crashes.

The next attempt is using MediaPipe in Blender. Blender supports the use of Python as its scripting language. I must admit it was not my first choice but the ease of use of Python made it a no-brainer.

Now, what is MediaPipe? MediaPipe is a Google-trained AI for body pose/face marker detection. Its primary purpose is to create face filters for mobile apps but lately has seen its use in motion capture. You can find an open-source project that can translate hand movement from sign language to spoken word in real-time, or a cheap mocap solution for your animations.

First off we need to install dependencies, OpenCV, and Mediapipe. Blender uses its own Python subsystem and does not go too much into it; the easiest way to install them is to create a small Blender Python script and run it. It will ensure you have Pip installed, updated, and OpenCV and MediaPipe components installed.

import subprocess
import sys

subprocess.check_call([sys.executable, "-m", "ensurepip"])
subprocess.check_call([sys.executable, "-m", "pip", "install", "--upgrade", "pip"])
subprocess.check_call([sys.executable, "-m", "pip", "install", "opencv-python"])
subprocess.check_call([sys.executable, "-m", "pip", "install", "mediapipe"])

If this does not work try this:

Locate the Blender installation directory. Inside this directory, navigate to <blender_version>\python\lib\site-packages. In the site-packages directory of Blender's Python environment, create a new text file named global_packages.pth. Open the file in a text editor and paste in the full path to your globally installed python, eg:

C:\Users\<userName>\AppData\Roaming\Python\Python310\site-packages

This will make Blender's Python aware of the global Python site-packages directory and should allow Blender to use all packages that are visible to normal python installation.

I’ve prepared a Blender file you can download on Github and see the complete script and play with it.

The interfacing with MediaPipe and running the script are rather straightforward but what I found is that Blender is not the right platform to do this project.

Blender Python is not threaded so when running the script entire Blender is locked for user input. There is a way around that but I’m not interested in workarounds that’s why I’m going to Unreal Engine. The idea was as I’m more familiar with Blender I could get away with it but after playing with it a bit it’s just not a good match.

But as a test that MediaPipe is working as advertised, so that’s good.