Ooooh, this is so cool. I didn’t realize I was seeing projected ‘makeup’ on faces until they showed some lower latency prior work. Once I saw it, I could see occasional misalignment. But, this is just an awesome and fun thing. I can imagine it being just amazing for live theater costuming and story telling.
That said, it looks like the tech currently works at 1-2m range. I’d guess this is because that’s an easy range for a projector to work at and provide high res, in which case there’s no reason you couldn’t get a zoom setup going.
Perhaps some enterprising director of the next Disney theater production will put the money in to get this working reliably on stage.
bhouston 1 days ago [-]
Neat stuff. I wonder if one could use a neural network to try to do prediction as well.
Human movements are super stereotyped in the face and you could likely predict with decent accuracy the next frame given the previous frames.
tsumnia 1 days ago [-]
Check out a lot of DisneyResearchHub's videos on YouTube. They are regularly posting research on object/face tracking and projections.
That said as you mentioned faces are pretty stereotyped. We "solved" face mapping two decades ago with Tim Cootes and Paul Ekman's work. We're able to quickly map rough estimates using traditional haar cascade classifiers and Viola-Jones with AdaBoost. Neural Networks may help, but we have other solutions that also handle the problem with "relative" ease (ignoring lighting, occlusion, etc.)
addandsubtract 1 days ago [-]
What is the SotA OSS for face tracking these days? I've never been able to make face tracking work without having alignment issues during movement.
zipy124 1 days ago [-]
I don't know if it's SoTa but previously in a joint academic+industrial project I've had very good success with faceboxes for face tracking and then adaptive Wing loss for facial alignment. With c++ and an ONNX runtime, I achieved very fast and accurate results, being robust to lighting and harsh angles.
The artistic possibilities of this are really fascinating. Looking forward to what people come up with! Projection mapping interactive art is always mesmerizing
Is good but when you need that extra lens shift camera, it goes from mass market to super niche.
echelon 1 days ago [-]
It seems like they've done two things here - the physical optical apparatus setup and the algorithm.
I'm wondering why the physical alignment is so important. Are camera distortion models and mapping, view projection, etc. just too slow or low quality to run?
I suppose I'll have to take a look at their paper later.
croes 1 days ago [-]
> to create immersive experiences that have been used in the makeup and entertainment industries.
Or for fake propaganda videos.
Dangerous time for these kinds of progress.
Hugsun 1 days ago [-]
What sort of propaganda do you forsee this enabling?
croes 1 days ago [-]
„Leaked“ videos of political enemies saying abysmal things
The Protocols of the Elders of Zion on steroids.
grayhatter 1 days ago [-]
Uh... this is a white paper about projecting an image onto a moving face. Not super imposing, but projecting... with a projector... using light... so what exactly are you worried about with this tech? Or is it just that you're afraid of face tracking in general? And it's not really about this topic specifically?
croes 1 days ago [-]
It’s about high speed face tracking.
Can you imagine other use cases than projection?
HanayamaTriplet 1 days ago [-]
You don't need high speed face tracking to make video edits, those are not going to be done in real time.
grayhatter 1 days ago [-]
> Can you imagine other use cases than projection?
yes
vtubers
human interface device for quadriplegics
automated camera tracking/zoom for video calls
emotion recognition support software for people with difficulty reading or understanding emotions (ASD)
truly smart, smart home that can track and respond to gaze, e.g. "can you turn off that light" and it will turn off the light they're looking at
sleep detection for smart locks
liveness detection for identify verification systems
vr tracking for interactions that both escape the uncanny valley, and feel more real time
need more?
croes 1 days ago [-]
Yes, list the malicious ones too.
grayhatter 1 days ago [-]
impersonation?
are there others?
Rendered at 22:51:42 GMT+0000 (Coordinated Universal Time) with Vercel.
That said, it looks like the tech currently works at 1-2m range. I’d guess this is because that’s an easy range for a projector to work at and provide high res, in which case there’s no reason you couldn’t get a zoom setup going.
Perhaps some enterprising director of the next Disney theater production will put the money in to get this working reliably on stage.
Human movements are super stereotyped in the face and you could likely predict with decent accuracy the next frame given the previous frames.
That said as you mentioned faces are pretty stereotyped. We "solved" face mapping two decades ago with Tim Cootes and Paul Ekman's work. We're able to quickly map rough estimates using traditional haar cascade classifiers and Viola-Jones with AdaBoost. Neural Networks may help, but we have other solutions that also handle the problem with "relative" ease (ignoring lighting, occlusion, etc.)
https://github.com/protossw512/AdaptiveWingLoss
https://github.com/sfzhang15/FaceBoxes
try mediapipe w/ gpu support
Is the source code available?
https://cycling74.com/forums/n4m-facemesh-handpose-google-me...
I'm wondering why the physical alignment is so important. Are camera distortion models and mapping, view projection, etc. just too slow or low quality to run?
I suppose I'll have to take a look at their paper later.
Or for fake propaganda videos. Dangerous time for these kinds of progress.
The Protocols of the Elders of Zion on steroids.
Can you imagine other use cases than projection?
yes
vtubers
human interface device for quadriplegics
automated camera tracking/zoom for video calls
emotion recognition support software for people with difficulty reading or understanding emotions (ASD)
truly smart, smart home that can track and respond to gaze, e.g. "can you turn off that light" and it will turn off the light they're looking at
sleep detection for smart locks
liveness detection for identify verification systems
vr tracking for interactions that both escape the uncanny valley, and feel more real time
need more?
are there others?