Post
🚀 IT WORKS!!!! 🚀
ARKit iPhone face tracking, without an iPhone. Just a webcam on any Mac.
All the blend shapes. All the features. No magic camera required. It just works.
Now I just need to plug this into VBridger/VTS and clean things up a bit ✨
Logs: https://gist.github.com/hoshinolina/ba7548a50563146fa1caf3d2bd8ea3be
@lina The computer built in webcams aren’t that terribly good either afaik.
Sounds like it’s software more than hardware.
Ever tried lowering the quality from the camera through software to see how “bad” it can get before it stops working well enough? <— no practical reason for this, just curious, don’t burn a lot of time on my behalf :)
@yon Recent Mac webcams are actually using the same sensors and processing as some iPhone front cams!
Resolution wise, the face tracking uses a tiny 128x128 crop, so you don't need much. I'm running the webcam at 640x480 here, and it's just about right for sitting a fair distance away from the laptop without losing face bounding box quality (my face is larger than 128 pixels across in the frame).
A good webcam helps with low light performance though. That's important unless you have consistently good lighting.
@lina That's amazing! Congrats! :)
This is the exact same code and machine learning model that ARKit uses under the hood on iPhones, with the same quality.
Apple doesn't publicly support it on macOS through ARKit... but they've been shipping it secretly as a private API.
BTW, it seems to be using around 65% CPU time (of one M2 core), or 80% if I tell it to force CPU-only processing.
My code can probably be optimized, but yeah, it's not a very heavy model or anything. Doesn't even need a GPU/NPU really I think.
Install bonfire.cafe
Get the full app experience
