"...even the movement of our nostrils will be translated. What a time to be alive!" - Dr. Károly Zsolnai-Fehér, nose enthusiast

@user-wq9mw2xz3j

Жыл бұрын

or a time to not be alive...

@4hm35319hd0h5

Жыл бұрын

@@user-wq9mw2xz3j To phrase it slightly less darkly, 'What a time to be virtually alive!'.

@raelrs

Жыл бұрын

Please let it not track what’s coming out of the nose during winter… I’d prefer to be blissfully ignorant of that at least in VR! ;)

@emhgarlyyeung

Жыл бұрын

I just hope this technique won't be weaponize by some evil gov to make fake thing to smear others, instead, should use it in a good & fun way.

@rakijr9176

Жыл бұрын

So that's how his name is spelled.

Is there a better channel than demonstrates bleeding edge technology? No, this is it. If you’re not subscribed, you’re missing out.

@TwoMinutePapers

Жыл бұрын

You are too kind Phil, thank you so much! 🙏

@dundermifflinity

Жыл бұрын

@@TwoMinutePapers no, thank you. I mean it. Nobody else does it like Two Minute Papers.

@Rodutchi

Жыл бұрын

Absolutely correct

@lubbyLB

Жыл бұрын

But he didn't talk about Stable Diffusion yet.

@dannyarcher6370

Жыл бұрын

I have not seen one. If you want to get ahead of the competition, especially when it comes to multimedia, just follow this channel to see what's coming down the pike.

3:38 I like how confused the AI is when the lips are behind the tea cup. Input Video: "Siiip" AI: "schlop schlop schlop?"

Wow, can you imagine movies or games based on historical figures being reproduced by photo real 3d actors? This will be wild. Also terrifying that AI programs will probably be used for people to recreate and interact with digital versions of their dead spouses or children, this might mess some peoples heads up.

@Rctdcttecededtef

Жыл бұрын

Imagine if we could some how train AI based popular literature and have it recreate the authors personality

@Numbabu

Жыл бұрын

@@Rctdcttecededtef I mean it can already imitate the writing style they used in their books, if you have a bunch of personal writings from a person I don’t see a reason you couldn’t generate text that matches them.

@GreatMCGamer

Жыл бұрын

If you ask me... In order to lose your sanity due to virtual world, it must already be lost. Most commonly, being superstitious and or religious.

@Rctdcttecededtef

Жыл бұрын

@@Numbabu that's a much better explanation of what I was imagining

@EliIud

Жыл бұрын

@@GreatMCGamer that's cool but what if you're brought up under this tech since 0 years of age

Love this channel. Wish more were as upbeat as yours ... Cheers.

@rick_vito

Жыл бұрын

make one!

@aronhighgrove4100

Жыл бұрын

It sounds slightly hyperbolic and too whispery/aspirated at times though. It's a bit distracting when you constantly feel like it's a bit ad like.

If only this was on the market last year when I spent 6 intense months tracking dots on faces on mo-cap actors for a video game! There was so much manual cleanup and fixing because the captured facecam data wasn't always great, it was a lot like 2:19

@spork4966

Жыл бұрын

I want to do what you do when I grow up, but I don't know what you are doing and where I can start 😂

@janimelender2674

Жыл бұрын

Well here's your chance to do something with this paper. Create a company and offer your skills in facial tracking. If you're cheaper than you are right now, you'll make big bucks! In time you could become very proficient in this type of motion capture. Let's say it's weird with capturing blinking. Well "just" write an algo that detects when the top and bottom of the eye-lids are close to each other, in order to detect blinking. Then delete the blinking data from ~0.3 seconds before that detection until ~0.3 seconds after, and replace it with a standard blinking motion. Stuff like that will be worth gold, since you'll be able to do so much more work so much faster than industry standards.

5:40 you could use this to insert yourself into a game and actually talk as yourself (not just for VR)... Imagine having virtual conversations with other players in a 3rd personal online game.. where other players talk realistically

@Jack-vv7zb

Жыл бұрын

Star citizen already does this. Called face over IP (FOIP)

@maiskorrel

Жыл бұрын

I can guarantee you that people will just meme the shit out of each other

@sevens3383

Жыл бұрын

Player customization will be more advanced soon hopefully

@knockdoun

Жыл бұрын

Star Citizen has this. Though its its a little outdated but they could totally update it with an improved version of something like this. current face tracking is goofy but does work

@Hundr_

Жыл бұрын

Wow a new tool to commit fraud with!

We'll need a minimum of 10 more years of training for it to capture Jim Carrey's expressions accurately. It'll most likely be way to confused

@Dilligff

Жыл бұрын

He'll be the stress test for the technology. Kind of like the meme 'can it run Cryengine?', but it'll be 'but can it do Jim Carrey?'.

“But can it track Jim Carry” feels like the “but can it run Crysis” of face tracking solutions lol

possible uses: motion capture facial animation facial sentiment detection (embedding, augmentation) facial recognition automated lip reading (embedding, augmentation) speech audio-to-animation (data augmentation) eye tracking

This could take animation to another level. Nvidia's Audio2Face can already take text and generate emotions. Avatars that are indistinguishable from humans is probably not more than a year or two down the line.

@theamici

Жыл бұрын

indistinguishable through a casual look, sure, but algorithmically generated faces also generate data artifacts which can be traced by software

@MikkoRantalainen

Жыл бұрын

@@theamici Yes, totally perfect deep fakes are not yet close but the "video feed" you send to that Teams meeting can be easily virtual presentation of yourself in perfect lighting in a studio and nice clothing where ever you're actually attending from.

@JuanSanchez-qp1xp

Жыл бұрын

@@MikkoRantalainen or a president is seen on video doing something or saying something he didn’t. Scary Barry

@MikkoRantalainen

Жыл бұрын

@@JuanSanchez-qp1xp Yes, the general public should be educated that a video can lie worth 10000 words.

What a time to be *AI*

This is literally another one of those papers that I've been waiting for I do work in animation i got my degree in 2d animation and this tool along with various other things that are coming out are going to revolutionize the animation more than it already is

@AhmedAli-kt1ez

Жыл бұрын

How does a 3D tool help with 2D? On a side note I doubt anything made by A.I will ever look as good as Akira or Ghibli stuff.

@cjeff99

Жыл бұрын

@@AhmedAli-kt1ez 3D models made to look 2D

@litjellyfish

Жыл бұрын

@@AhmedAli-kt1ez the data can easily be flattened to 2D. Actually they already are in the end

@lovejsouvzacne

Жыл бұрын

is there a way to export blender files from it or do you know how to use it ? :)

@litjellyfish

Жыл бұрын

@@lovejsouvzacne this is a real-time image to facial feature. I don’t think it handle the mesh generation. Also if it would it would be to industry standard format most likely and not Blender format

This is a huge W for the gaming industry.

I can't wait until you can walk around in an open-world game map in VR and engage any NPC in conversation and the AI will generate in it an actual personality with unique character traits - though the player will be able to set parameters barring types they don't want to run into!

@muhammedkoroglu6544

Жыл бұрын

Definitely what I hope will happen too!

@proloycodes

Жыл бұрын

so basically an echo chamber?

@Zebred2001

Жыл бұрын

@@proloycodes If you want to set your parameters for provocative conversations or constant arguments go right ahead!

@joevaghn457

Жыл бұрын

No, if the AI is smart enough to do that, you’ll have absolutely zero say in what it’s personality is. Lol

@mhb11

Жыл бұрын

Ladies and gentlemen, an ERP junkie.

Mind blowing.

This tech would be extremely useful for streaming: from Vtubers to facial tracking for green screens! I could also see this being used as a kind of encoder-decoder pair for telepresence applications, where one’s head and shoulders are sent as position data rather than a high resolution image, and then reconstructed on the receiver side for high speed, high fidelity, lighting insensitive video calls!

@livedandletdie

Жыл бұрын

It would also be useful to be used in the reverse for more accurate facial expressions of characters in video games. Collecting enough data, and turning it all into vector space, and thus being able to perform more human expressions. There have been a lot of games throughout the year that have promised "realistic" facial expressions, but none of them have been up to par, with this they could quite easily use the data to improve that. Now the only thing that needs work is lighting of non-static objects.

@Cube_Box

Жыл бұрын

nice idea

@dundermifflinity

Жыл бұрын

That teleconference idea exists already - I think, if I remember correctly, by Nvidia. This channel did a video on it a couple of years ago.

@dundermifflinity

Жыл бұрын

Here it is kzread.info/dash/bejne/loqVk9qLkau6cZM.html

@etaashmathamsetty7399

Жыл бұрын

@@livedandletdie oh ur 100% right about that But we probably won't see it any time soon

The face tracking technology (without tracking points) is getting wild! The digital animation workflow is going to be sped up two fold in the next year! (Or at least, I think)

1:15 Baby wake up, a new NFT collection just dropped!

Always love the reveal with "hold onto your papers!" or even the SqUeEzE your papers on this one. :) Great content as always, thank you!

Combine this tech with 3D animation, Voice synthesis, AI script writing, AI story creation, AI generated artwork, targeted advertising, cameras in theatres to gage response and interest. We can have "Hollywood in a box" and potentially add the tech of Brain Machine Interfacing,(the first of which has become commercially available very recently) we could even eliminate theatres and get better response from the audience. Of course, we all know what this will ACTUALLY be used for...

@snaphaan5049

Жыл бұрын

Add to that CRISPR babies and the enormous benefits it has and in a couple of generations human beings as we know them will be a thing of the past. It's weird, almost every tech development , from self driving cars, boston dynamic's robots, midjourney AI, 3D printing etc is replacing physical labour. It's like man's answer to the Garden of Eden. We don't need God to go back there. We can counter "in the sweat of your brow" with and machine server power. The writing is on the wall. I mean, when you have silicon valley intrepreneurs waving normal human beings as a burden and glorifying a "new designed haminity" then we are pretty much damned.

This is going to be great for XR applications!

I wonder if expensive stuff you need for mocap will have their prices go down once these AIs reach a point where they’re on par and available to the general public

@danielmonge2318

Жыл бұрын

It's going to happen the same thing that happened with the GPS market. It used to be very expensive to own one and there was a yearly fee for updated maps. Now we've got Google Maps for free on our phones!

Wow, that’s amazing. ARKit still does some things better, because of the depth data. For example, in those side-by-side videos, you can see that ARKit better tracks the lips, around the eyes, and nose/nostrils. It’s the “nooks & crannies” of our faces that depth sensing can better interpret. The color algorithms are getting better & better though! They are still just a little more jiggly compared to color + depth. Thanks for sharing this cool video!

Omg seeing Mona Lisa move like that was awesome! I love technology ❤️

Really love your video summaries. Excellent work. Thanks for posting!

The fact, that news outlets are using this technology, is troubling.

Let be honest folks: I know that y'all wanna a massive open world RPG like Skyrim where every single NPC is lifelike and totally unique and will pass the Turing test blindfolded.

@elichapin3366

Жыл бұрын

your completely right

@michaelleue7594

Жыл бұрын

I mean, yes, I do, but we are so far from that. Look at Evermore: it's basically a real life version of this, with real actors playing the parts of NPCs, and you *still* can't get anything approaching an authentic experience. Even *actual people* are no good at passing the Turing test in a fictional setting.

You know what would be really cool on a testing program like this? Use some glass materials to block people's faces in various ways, and film them in visible light and in IR, and then let the AI work with the IR footage to figure things out, meaning that the person's face is occluded, but then you would have visual footage from the same angle that you could overlay, so that you could see what it got right and got wrong perfectly.

@maiskorrel

Жыл бұрын

I suppose that footage could also be used for training the AI to correct it's mistakes.

@mhb11

Жыл бұрын

And then measure the gaps for every data-point it got wrong, and then optimize those gaps via feeding them as training data.

VR team meetings where you don't need a fake avatar.

I see what pushes you forward, the rewrite of the Game of Thrones final series! I cannot wait

Facial recognition The dream tool of all governments....

I'm excited to see this easier&higher quality capture in Vtubing

Even slight misplacement of virtual face from real face is noticed by our brains

Nice Nice, seems like 2020-2030 will be a decade where AI will become common, in a big way, in software runing on desktop, laptops, smartphones. It's time for a lot of new cool software that will increase productivity and enable us to make new things that were previusly imposibile to make.

@rettenthetetlen8759

Жыл бұрын

Great recession with global unemployment.

@zdenekburian1366

Жыл бұрын

@@rettenthetetlen8759 exactly, and ultimately a big war, but there is a hope, a communist revolution making obsolete money, market, wage work, exploitment, capital

Your enthusiasm is infectious.

Said it before, we're right in the age that's making all the building blocks for what will one day be common a place Holodeck-like technology.

6:23 in sync with the narrator! :D Amaaaazing! lol

This is great because image based tracking is always cheaper then any special sensors. Now nobody will need expensive headsets or iphone sensors for good face tracking.

0:33 This makes me think that we are living in a simulation. Dang.

In the 1970's German television had a movie "Welt am Draht" (World on a wire), where computer experts created a virtual world and were able to enter this. One of the virtual beings found a way in the real world and said, that this isn't the real world yet, finally it came, that the computer experts of the real world gave the technology to the virtual people. At least, the concept of virtual world is something, where we go to far.

This is outstanding, can't wait to try this new thing.

because this works so well with obstructions, this might be useful for vrchat in the future...

You make me squeeze the papers so hard they turned into Diamane

I've thought for years that AI will eventually be able to break down movies and TV shows into it's elements, including actor appearances, voice and acting styles. This could be used as part of that by capturing the facial expressions of actors. One thing I see that could be improved is by identifying expression anchor points of different faces and people. Topographical maps isn't enough when it comes to faces. People typically have different points on their faces where expressions seem to radiate from. It's probably tied to either muscle anchor points or just how different people learn to control their facial muscles as they grow up.

Signed up on Runway, excited to see what it can do.

Whenever I watch one of these with virtual humans in it, I can't help feel disapointed that we'll still have to wait so much longer for the same to be possible with virtual not-humans. One day though we will be able to get an AI to generate animated or realistic characters of any kind from humans to fantasy creatures to aliens and beyond all with the same algorithm. And maybe ten papers down the line that same AI would be able to study a video of godzilla and create him. And one day all of this will lead to the destination I dream of.

@litjellyfish

Жыл бұрын

This is already possible today. It’s just what mesh you want to use as output. Instagram etc snapshat filters do this.

@joevaghn457

Жыл бұрын

Yeah… Those with the craziest fetishes will definitely be satisfied.

Started with the uncanny valley in the simulations and then somehow got every more unsettling with the wireframe robo-masks

It's still super weird to me that you can train and AI on computer-generated data... it's like the computers can teach themselves based off of their own imaginations... Love it! and thank you!

@spideyninja

Жыл бұрын

It's important to mention that the entire dataset is procedurally generated, but not via an AI. Thus it's 100% precise.

@yorgle

Жыл бұрын

@@spideyninja Fair enough. :)

@cube2fox

Жыл бұрын

AlphaZero (Go software with superhuman performance) was also completely trained on synthetic data.

Let's use it for VR remote conferences.

Dr! I just saw you are presenting at GTC this year! Signed up immediately for your presentation when I saw your name

We need good vr hand tracking!!

If it can track Jim Carrey, it can track everyone

HALF OF REAL TIME?! NICE! All vector everything!

As I make characters in Character Creator.. lol this is amazing.

5:24 "Squeeze that paper" ...... When holding on to your papers just doesn't cut it anymore.

Regarding deep-fakes-.. Imagine this, having a photo of a relative or someone who's passed, or a dog, and preset movements- or ai to make them more life-like. Sortof like the pictures and paintings in Harry Potter. Digital frames already exist- but "magical" moving people frames don't. Fun idea for anyone savvy out there!

extremely impressive. is this facial performance capture now good enough for real movie-making? also it would be very useful to see the motion being targeted onto animations, especially animals.

@markusgp

Жыл бұрын

Probably not if you can't tweak the results but it won't be long

@dan_loup

Жыл бұрын

It's actually even more impressive than what was stated in the video, because it's not only running on the CPU, it's running on one CPU core

@fabriperoconalgomasytodojunto

Жыл бұрын

If they gave it depth too then pretty sure it would

@jensenraylight8011

Жыл бұрын

the result from motion capture often are jagged, and the result will be baked on every frame, it's hard to clean it up , unless if they somehow invent the way to clean up the jagged in mocap.

@juliandarley

Жыл бұрын

@@jensenraylight8011 yes, cleaning up mocap has always been a total nightmare - very expensive and very tedious. if AI can clean up mocap and get rid of the unnatural jinks and jumps then that would be a huge leap forwards. recently, i have seen improvements, especially in fixing footsliding. maybe karoly find a paper that finally really fixes messy mocap.

Elképesztő lett a kontent, csak így tovább!❤

@Sekir80

Жыл бұрын

Nem simán kontent! Valódi mondanivaló. ;)

You could use this for real time augmented reality layovers of someone's virtual avatar over their face, that could be pretty cool.

I think we all know what people are going to use this for

@Rctdcttecededtef

Жыл бұрын

?

@brandenwoolf5075

Жыл бұрын

Big facts 😂🤌

Wonderful as always

"what a time to be alive!" is my favorite catchphrase

love your videos! amazing tech!

The next step would be to train a neural network for the typical motion and acceleration patterns of each landmark so that these abrupt movements do not occur when certain parts are obscured. 7:12

I'm gonna use this in the Metaverse.

c o o l but i like light transport papers more

Look forward to converting scanned faces to a miniature model, so that players can literally play as themselves in DnD

I wasn't ready for the Two Minute Papers face reveal, but I'm glad i now have a face to the soothing voice!

I must say that ideas like using data generation from virtual environment like using footage from virtual environment for training driving AI is just something that I haven't heard of but it so much on the surface, considering that you get both structured and unstructured data distributions at the same time, and very little work is needed by human. If someone knows papers on successful application of virtual environment used for training (outside of RL of course, I am talking more CV), please link it here, would really appreciate!

wow, i was looking in all internet some appropiate topic regarding virtual costumes for filmmakers... wow, it's insane... this will be a serios plus for filmmakers's buget... Waiting for unreal engine to implement all this stuff soon

This is the synthetic Data revolution

can't believe humans are real now. terrifying.

08:09 "instantly remove unwanted objects" removes humans 😂

This is really a behind the scenes for Grand Theft Auto 6

When I've started producing I personally have downloaded most of the plugins from there, never had problems with virus, malwares and

Lets hope this is open source or it somehow finds its way over to the boys at Epic. This algorithm needs to be implemented in live link face like, yesterday. Imagine metahumans with this level of mocap fidelity.

@joevaghn457

Жыл бұрын

It’s Microsoft, so who knows. They probably patented it, and likely won’t open source the most critical parts

Naon jó lett.Csak így tovább. Imadom a videoidat.

Apple’s ARKIT face tracking does not use the depth sensor. The depth sensor would run way too hot when used for long periods.

Oh my god the implications for adult entertainment...celebrities and Ip is about to get defamed on a whole other level in the near future.

When I see this sort of thing I immediately think of the benefits it would have for visual effects in movies, specifically in regards to augmenting an actor's face with some CG _without even needing to place dots on their faces anymore!_ Think of the old TV shows like Star Trek The Next Generation where characters like Worf, the Klingon, had to endure 3 or more hours of makeup every single day. Using the above technology, Michael Dorn could just come into work wearing his Starfleet costume and the rest would be done in post! And the best part is that it would actually look BETTER. Even up close shots would likely be perfect, especially once this technology is further developed. What a time to be alive!

Amazing work. I love it :*

Synthetic data is by far the only way for the future of computer vision (and AI generally)🤩

Imagine having to undergo surgery. Your body get scanned, as well as your brain under different circumstances like joy, rest, pain etc. Your full body scan including all the interior tissue gets fed into an AI operated robot surgeon. It analyses all the data, figures out the best procedure for this surgery to be a succes, with minimal incisions, movements and risks. The AI then explains the whole procedure to the doctors involved, so they understand what it is going to do. It monitors your brain patterns to be sure you're actually under anesthesia during the procedure. Difficult procedures that once took many hours and involved huge risks, could now be done in 45 minutes, with next to no risk involved. Doctors would still be in the room, to intervein when necessary, as one of many failsafe systems of course.

This could be very useful for diagnosing subtle facial palsies. It could also be an important tool in the simulation of plastic surgery results. It also be an important tool and planning plastic surgery. Many implications here.

I love your delivery!

So will we be seeing cameras auto-orienting a person's avatar so that they look like their staring into a camera when it's actually off-center? I could see the system calculating the center of a screen-partner's face as the center of where a camera should be, and adjusting the avatar of the caller (so it seems to be oriented as though the camera were placed there). It could take the communication awkwardness out of video calls and teleconferences. Moreover, I think these intereractions are like talking to a person who avoids eye contact, essentially undermining confidence and hindering the adoption of video conferencing at a psychological level.

@MikkoRantalainen

Жыл бұрын

If that's the only think you want to fix, half mirror in front of your camera and teleprompter-like screen setup is all you need. However, if you want to show yourself in perfect lighting and some specific clothing and background, then you need the tech from this video.

Great work sir! Though I’m from finance and accounting background I still find your videos very amazing love from 🇮🇳 India

They should have added some stabilization against jitter in uncertain areas.

FAN DUBS THAT CAN LIP SYNCH ANY MOVIE

Id love to use it for facial tracking simulations, where the face is a real persons, but Ai virtual. Im sure that could be a method that would create better facial capture tracks.

I can't hold onto my papers! My papers flew out of my hands before the video even started!

As a game and educatuon dev, most definitely I would use it for motion capture of actors for humans and monsters.

So many applications. Knowing the universe of human expression, AI can label and predict the range of human emotions; truth, lying, happiness, caution, fear, aggression, defecation, love and awe. All this can be transferred to robots, game characters, fakes, chat videos, arousal, doppelganger searches and criminal intent prediction systems (shoplifters, thieves, assassins). We're entering the post-human age.

7:13 There may be a technique for guessing the position of the jaw through the movement of the cheek muscles, but it seems that it is not yet.

Doc, I don't know why but every time I see you in your videos I am thrown off. I just cant correlate your voice with your image. I hope you are a real person 😂

I would express when my face is realy tired. Or generate oc for art reftences.

this is good for video compression and realtime video chats

The only channel that makes me feel upbeat about the impending dystopia! 😅

Hi Karoly, I have a degree in maths and went into teaching. What would you recommend to someone looking to get into AI? I'm hoping to transition but have little coding experience. I find every video you produce fascinating and want to get on the other side of the curtain. Thank you for your efforts. Ben

@slangster42

Жыл бұрын

Unfortunately, I think that coding is required to work in AI related fields. Personally, I'd recommend you to read a few technical books about AI, just to make sure that you really want to switch from teaching to AI, and to get some basics in programming. I think that "Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow" by Aurelien Geron is a great start. Just be sure to have the second edition and not the first, because the first edition uses an outdated python library. In any case, you'll also have to learn to code in python, and there are probably a lot of books or tutorials online for that as well.

@bencrossley647

Жыл бұрын

@@slangster42 Lovely reply. Thank you. I've started teaching myself python by doing the first 20 ish Euler problems. Thank you for the recommendation. I'll check it out :)