This microscope uses touch
Ғылым және технология
Visit janestreet.com/amp to find out more about their Academy of Math and Programing
Gelsight is a microscope the presses gel into the object of study.
Thanks to The Sempre Group for showing me their machines! Find out more about them here:
And here's a direct link to GelSight (for non-UK people this is a more useful link):
You can buy my books here:
You can support me on Patreon and get access to the exclusive Discord:
just like these amazing people:
Tim S Root
Lizzy and Jack
Buy nerdy maths things: mathsgear.co.uk
Пікірлер: 2 689
Would *you* call it a microscope? The sponsor is Jane Street. Find out about their Academy of Math and Programing here: janestreet.com/amp
4 күн бұрын
I would say: This is a precursor to robot skins And that you like to use "weird" a lot, like you're trying to induce people to subtrans state or something, plus flashing lights, red pipe, flags all around
10 күн бұрын
22 күн бұрын
I see special applications. Couple it with a surface duplication device, and it could be used to allow a blind person to "see" the tiny details they would not normally be able to detect by touch. For surface duplication; en.m.wikipedia.org/wiki/Pin_Art A modified "Pin Art" board, with micro solenoid control for each individual pin, and a similar membrane over the top. The "imager" could then be pressed onto a tiny object, and a much larger magnified "image" replicated using the PinArt-based tactile "display".
I would call it the world's most accurate reverse engineering device.
I'd love to know more about the gel they are using, It seems absolutely incredible that a gel would conform to such small details so perfectly and yet have no memory in that it returns to it's original shape.
King of the Koopas
22 күн бұрын
I love how someone who actually works for the manufacturer is completely ignored, while he’s stating what your asking
To be clear, they do have to replace these pads with varying frequency depending on what the application is. They wear and could potentially leave behind debris on the part being measured.
And what the gel tastes like
Really? It's kinda natural and comment sense that a semi solid/liquid would form to anything that is placed on it with pressure or it pressed into something with pressure... What's so "unforeseen" to you?
I could probably watch about 12 hours of Steve pressing various objects into the gel and just describing what they are before moving onto the next one.
Right? I wonder what various kinds of feathers would look like - flight feathers vs down, etc
The wierd neon-colored image you can see used to create 3D model is called a normal map and it's meant to hold information about height of the object using different colours for different light angles. It is used widely in video games to create detail in low poly objects by changing how light reflects off of the surface. I find it fascinating they used normal map and not a regular height map but given the technology of 6 light sources it makes all the sense
Height maps contain less information than normal maps. Each pixel's color channel in a normal map contains information on the normal (orientation) of the surface, whereas a height map only contains direct vertical information.
It's actually nuts that you're getting such a deep depth of field on such small things, and you can even move them around a few millimeters and still retain focus. And you can capture it all with video and then share it with other people. Amazing.
24 күн бұрын
That's a really good point. My experience with regular microscopes is that the focus is touchy and the depth of field is very shallow, unlike here where the whole object is in focus.
This is insane. We are getting to watch the first steps of an entire new tool that will change entire fields of study. It is amazing. I'm probably way more hyped about this then I should
Not at all! We're curious beings and seeing new innovations inspires our mindset!
We could actually use that almost every day at work to measure engravings on the toolings for chocolate moulds. Would safe us a lot of time. Super interesting 🧐👌🏼
I still saw a mountain even with the image turned upside down. Didn't see a crater-
2 күн бұрын
Same. With the numbers I could see the difference though
6 күн бұрын
7 күн бұрын
Just imagine a light source from the upper left corner, that way the shadows make it look like mountain/crater.
15 күн бұрын
I can't even tell what it is 💀
21 күн бұрын
I also saw the lettering as indented in both images
Was really wondering about how easily it was contaminated and how frequently you would need to replace the gel, how expensive is the replacement and how easy is the process? It looked like everything you had on there had some dust or hairs, it seemed like it wasn't like you were constantly scanning dirty pieces but instead that the microscope itself had issue. Incredibly cool technology, I even want one but the dirt issue seems hard to deal with.
This has WILD implications as a mobile highly accurate surface measurer. Damn, I want one just to have. I bet they're expensive. Industry really needs to have this. Feels revolutionary.
I would love a larger scale of this, it would be amazing for modelling small electronics where the manufacturer doesn't provide 3d files
I work with 3D making games for a living. Seeing those 3D models being created was so cool. Essentially using the same information that's used to generate and render normal and displacement maps.
Ok, Steve, I'm so glad that youtube thought to put your video in my path. That's an odd microscope indeed, and the gel is strangely sensitive. Removing color does indeed leave us with form, texture, and shadow. Photometric Stereo? Ok then. I wonder what insects would look like with this system? Does that gel wear out? So it has a quality control inspection use, also interesting. Jane Street Academy... sounds excellent. Thank you for all of this information. Well done. But then you already know that.
im a 3d artist and i want one of these to make instant normalmaps from objects ,these look great
You can take a bunch (like 50 to 100) photos of an object and load them in to 'Meshroom'. It's a photogrammetry program that gives you a 3D model that includes colour. You can then load that model in to blender or some other program. It does have a hard time with verry reflective and translucent objects though.
@Multiarray Haha did it?! xD
@Asu Kan it would complicate the process tremendously if there were more lights. If what you are scanning is relatively small then an older chunkier flatbed scanner can be used as they have some depth focus and a light that's offset to the side. Just scan 4 times, rotating 90 degrees each time
@Martin Baadsgaard Does it have to be in a studio setting i.e., *only* light from the sides and dark from everywhere else or just increased light from the sides?
I would use it for displacement rather. Normal maps are useful only for realtime rendering.
So cool. Its like making a normal map of a object 😍
what a *fascinating* tool. I personally don't have a lot of use for it, but I can just imagine machinists being absolutely delighted at the details and measurements you can get from it.
Steve: this needs a sound effect Me: schlorp Steve: schlorp
I work at a jet engine company and we use these all the time! So cool to see!
to what end? wear inspection?
I imagine this could be incredibly useful for making normal maps for added detail in 3d modeling
As a 3D artist myself this was one of the most fascinating things I have ever seen in a long while. If produced massively across the globe. It could be really helpful in 3D Film Industry.
@Alan Berger you can only sell something for what the buyer is willing to pay in a free society, have you forgotten?
@torhgrim 😂😂😂 LOL Yup. I want this so desperately. making 3d visuals for my channel.lol would love to see your feedback.
You can bet your ass the artists who worked on the Lego movie would have killed to get one of these. I wonder if they could make a giant one and go press it on peoples face to extract a bump map :D
@Joseph Still $500 is for the hobby "mini" version. Steve was using the professional version.
Never knew such thing existed. But this is really a valuable tool for a lot of people who do precision work. So thank you for brining it to my attention it deserves that.
This multi-light setup that generates the depth and normal maps is the same method that is used to generate many textures and material used in games and film. You can do it yourself with a camera, a light and either free software or Adobe Substance. You can also remove the reflection from the images using cross polarisation by putting a polarising gel on your light source and in a different orientation on your camera.
I could imagine a larger version of this combined with a traditional 3d scanner to very quickly generate texture maps for game design
Sounds like it could be a major improvement to a thumbprint scanner. I’m a mechanic and get cuts and callouses on my hands every day. My thumbprint usually only lasts 2 days on my iPhone before I have to rescan it.
That mystery item looks like a watch battery and the surrounding parts, with a lot of overlapping layers of parts held in by small screws, I'm going to guess it's a complex mechanism... so, a watch? The movement of an analog quartz wristwatch? Edit: I realized after posting that I betrayed my knowledge of watchmaking with the terminology. For those who don't know, the "movement" of a watch refers to the mechanism, specifically the moving parts that make it work, the gears, the springs, and so on. In a quartz watch, it'd also include the battery and other electronic parts. The hands are also included but obviously not shown here.
This would be awesome to generate height maps for making smart materials in substance. Just noticed the height map at 2:41 and wanted to plug it into substance. Mistook height map and normal map as the same thing by mistake here. Thankfully people seem to get what I meant
Смерть в бахилах
& 6:30 profit 2 it’s a pity that the aftor didn’t put the tip of a ballpoint pen there or traces of the electronic board or damage on the wires
@Paco That might be tricky to pull off. The slightest movement between shots will ruin the necessary alignment.
2 ай бұрын
@Esger My engineer thesis was really similar. But I used 0,5x0,5x0,5m frame, 4 sources of lights and camera. Results were miserable but really good for a price I made that device (I think it was about 30$ not counting my own 150$ camera) I worked with textures but never came up with to merge that two ideas :D
2 ай бұрын
I was just saying "it looks like a height map" and then I scroll down and see this. Artists thinks alike
2 ай бұрын
That's what they've been doing AFAIK
This reminds me of when my dad, a master carpenter, taught me to feel if 2 pieces are level on a small scale with my fingers instead of looking because it's way more accurate.
We have a 3d scanner at work that works on a similar, off less accurate, principle of shining a light onto something and taking a picture. It is amazing how it will confuse printed text as raised or lowered surface. This seems to be a great workaround the normal "spray paint it all grey" method we have to use. about a decade ago i worked with a charity to try and replicate a giant version of a medal. I tried photogrammetry to get a scan of a plaster cast of the medal (again, all uniform no shiny colour). it ended up terrible. Something like this would have worked amazing.
I'm now wondering: Isn't this microscope still optical since it uses light to create the shadows that form the picture?
I have 16 years as a US military aircraft structural mechanic. I love your presentation style. It's light and engaging. I have never seen our nondestructive technicians make a cast of an indentation. We use ultrasound, edy current, sometimes x-rays, and more rarely these days optical micrometers. Never the less. I need one of these, I would love to compare it to currently accepted techniques.
I was expecting a video on an Atomic Force Microscope, but this is far more fascinating! What a novel way to collect 3d data!
Heck out breaking taps channel
2 ай бұрын
As a microscope enthusiast and former microscope student i agree this is really cool
2 ай бұрын
haha I'm surprised to see someone other than myself comment on this within the top comments =D I hope he does check out or try to make his own atomic force microscope hehe (well, something analogous to one)
2 ай бұрын
2 ай бұрын
I was also expecting an AFM, but I found this less fascinating. The magnification is just very minimal, you can see all the details here with the naked eye.
This is like real life normal mapping. Super cool
Nailed it! I recognized the battery from the mystery device after a bit of pondering, and then just guessed the most common application of that kind of battery.
I replaced a watch battery a month ago and remembered that. Lol
Got to play with one last year. They're 40k. We were using it to look for pinholes in our product, but I mostly used it for coins
I've been following this company for many years. Their technology is highly interesting. I'm mostly interested for bio-inspired designs and collecting surface profilometry of scales, scutes, skin, etc. A disadvantage of the method is that the surface of the gel degrades with use. Perhaps they are trying to improve this design, but I think it is probably unavoidable to some extent. That consumable cost is kind of expensive for casual use, thus restricting this method to only industrial uses.
There seems to be some limitations when used for surface features with high aspect ratio. For example, it wasn't able to discern the vertical fall off from the edges of the Lego brick pegs But none the less very interesting and could be very useful in a lot of scenarios By the way, what's the MTBF of the gel? (i.e. how many times can it be used)
I'd probably call it a profilometer more than a microscope. But then again it can be one
Oh... you mentioned it towards the end of the video... hahaha
Its incredible how well that gel conforms to objects
@MEMES FROM DEEP SPACE He actually did say that in the video, on the part with the stickiness. He said that it becomes sticky and needs replacing :)
MEMES FROM DEEP SPACE
2 ай бұрын
I bet the gell have "working life time" ..... The gell Will be the thing Will replace regulary ..... 🤔🤔🤔
Ali Devrim OGUZ
2 ай бұрын
I think it is the 80% of that technology.
Does the gel leave a residue behind to the point to where you have to refill the gel over time?
The same technology is used in AOI machines worldwide to provide 3d models of the inspected board. Such precision is used to find out if components were assembled wrong, have defects got dislodged while going through the oven. Edit: those don't use gel though.
Thanks for using an actual stereographic video and making it easy to cross my eyes to see that sea cucumber swim in 3D.
My past 5+ years of EM experience using various electron detectors approves this video on a fundamental level.
I've always had an issue viewing the crater of Neal Armstrong's bootprint on the moon. I always see the boot print as raised above the surface. From the knowledge gained in this video, I flipped the image 180 degrees, and voila, now I see the imprint instead. Amazing!
That zigzag motion of the print head is "boustrophedonic". It is a literal reference to the motion of an ox ploughing a field, and you still usually see it in patents to describe print heads, scanners, etc.
I learned what that word meant in the context of writing systems.
2 ай бұрын
Space filling curves
2 ай бұрын
I'm here to learn where you learned this
2 ай бұрын
This concludes my night, I've learnt enough. Thank you for that nugget of information.
2 ай бұрын
I'm here to learn words like boustrophedonic
It's surprising that the gel doesn't capture more bubbles of air
Your prints are pretty good :D Nice layers no gap, no loose belts, nice
If you're interested in looking at other unique sensors, Event-Based Cameras (also called neuromorphic cameras or silicon retinas) are an interesting rabbit hole to go down. They use a unique approach to electro-optical sensing to get high fidelity of certain information in a very efficient manner.
As someone who has taken apart watches, I knew exactly what that mystery object was 😂
A note on the ML side - the thing Steve refers to is called "top down" reasoning, where the later "high-level" layers representing object types feed back into the earlier "low-level" layers representing shapes and edges. This is fantastically computationally expensive, and is one indicator of just how sophisticated the human visual system is.
This would be an incredible medium to make some kind of animated movie, even just a short little thing. What i'm imagining is in the same vein as the "A Boy And His Atom" animation made with individual atoms.
@JustOneAsbesto You aren't very bright huh?
2 ай бұрын
@JustOneAsbesto I mean isn't technically everything that it sees made of atoms?
2 ай бұрын
@Nick Coleman yo thats perfect!
2 ай бұрын
‘A Boy and His Micron’ maybe?
2 ай бұрын
@JustOneAsbesto I was not suggesting that it did, thanks though.
That is a super cool microscope. I had the watch innards picked pretty much immediately. Watch battery with no Hg, crystal resonator and plenty of mechanical bits that aren't in digital watches.
This company presented us this technology at work, it was satisfying and playful to try 😀
The crater illusion is tripping me out because of how inconsistent I'm finding it between people. The crater image only ever looked like a crater to me, while many commenters had the opposite experience. Then with the indented lettering, it looked indented to me the first way, and like it was sticking out while Steve is saying it obviously looks indented now XD That's the weird part to me.
5:00 Thanks for making the left/right arrangement compatible with the cross-eye technique! 😸👍
Would probably be good for 3d meshes in video game development
I work with lots of 3D LIDAR scans and one huge issue for us is the reflection off of reflective surfaces such as mirrors. Interseting how this gets rid of that, at least on a micro scale.
2 ай бұрын
Mirrors look really interesting in a 3D picture. They're kind of like a hole.
2 ай бұрын
Would training an algorithm be easier using this technique as a source of truth? Small images, and maybe large images later, could be developed accurately with this, but would that information from this technique help you train a model faster? And further, would information about small things like the quarter or matchhead be useful for larger scale applications of LIDAR such as autonomous vehicles?
2 ай бұрын
Just tape a gel cube to the sensor and drive into things!
I’m surprised they don’t use vacuums or air pressure to get the gel to conform more to the objects
The "crater" looked like a mountain to me no matter the angle. Even after you said it was a crater and flipped it, my brain couldn't un-mountain it. :D
Makes it look like some sort of height map you can generate from 3D objects
There was NO way i expected for that to be handheld! Thats NUTS
Coolest technology I've seen in years. Makes you think about what else might be out there and kept from the public.
No money in keeping it from the public.
Nearly every mundane object under that type of imaging looks quite mesmerizing! I actually thought your stubble was one of the more interesting ones. It really shows just how cleanly the blades cut the hairs at the ends. The draping problem was the first thing that came to mind as someone who has done a lot of vacuum forming. But that’s very interesting that it’s not much of an issue as long as they can get the depth value. And as long as it’s not deeper than it is wide.
6:07 "This image looks like it's sticking out because the shadows are on the bottom, but by flipping the light source you see the reality that they're actually indented" That was the exact opposite for me. The first instance was clearly indented while the second was and emboss.
IAs VFX artist I love that it creates normal textures. That´s a totally cool device for scanning hightmaps and normal maps. Especially for fabrics and general surface imperfections. Exciting technology
This is a gel sight. When you do analyze you can also create a 3d printable image enhancement.
This could be an award winning video for the “oddly satisfying” category, i just can’t stop looking at it
Loving the expensive high-tech microphone mounting system you have set up! Hey it works lol
With the crater illusion, I saw it as a mountain the whole time, even after seeing the shadows falling on it like a crater - the look of the inside of the crater looks so plateu like. Same with the indented text, in fact I think the top light source makes it stronger, and I did eventually recognise the crater, but I had to look closely at the shadows, and looking at the light side makes it look like a mountain no matter what.
He's attributing way too much importance to the "lit from above" assumption. We are all very used to seeing images shot from above where that doesn't apply. I saw the crater as a plateau in both orientations, and saw the print as indented / raised opposite to what he thought we should.
Death of all things potato
@LeoDaFinchy Or neurological. I'm autistic.
So, these replies: do the commenters regularly view satellite imagery? Are those images more typically from the northern or southern hemisphere? Do the commenters work in specialist lighting conditions? I'm sure someone could wring a doctorate out of studying the various ways humans interpret relief patterns and sociological effectors on that.
Yes, same! All my life, many photos of the surfaces of other planets or their moons have looked to me like they’re covered in mountains, not craters.
Yeah it was a mountain for me both directions. Then after reading this comment I went back up and rewound the video to when it was supposed to be in crater mode, and... it was a crater. I let it flip back to mountain mode, and it was still a crater. So for me at least, it seems to get "locked" as how I last saw it, regardless of rotation.
We use similar methods in the game industry to extract height and normal maps using photogrammetry.
This is the real life equivalent of a matcap or clay render in 3D rendering.
The crator illusion worked opposite for me. Looked indented when lit from below and sticking out from the top
I'd love to see a record under this
The gel seriously had me tricked that it was water. So glad I clicked on this video!
4:37 That's called monocular depth estimation and it very much already exists. There are quite a few open-source models that can estimate depth fairly well from a single flat image, and if you have multiple images from different angles there are neural networks that can build up an accurate model of the entire scene, sufficient to move a virtual "camera" around and through the scene and create new views in the process.
Derick Wong Friedrich
But how reliable and accurate are they? Probably not good enough for the purposes these camera gel deforming sensing things are bought for?
as a watch guy int interesting to see a miyota(owned by citizen watch company) quartz movement inside of a casio watch who can and does make their own quartz modules
This would be insanely valuable for creating texture depth maps for 3d models in like a video game or something.
My favorite map in Halo 7 is Terrain Designer's Left Eyebrow
This would be great for looking at error coins under the microscope !!
I don't know about a microscope, but we'd surely hope to see this tech being used in creating 3D models
Great!! Anyone aware of any similar sensors from other brands that use the same technology?
@05:38 I see a mountain in both cases. The shading in the detail around the perimeter comes from bulging shapes, not depressions. Also, if cast shadows are present in such cases, they clarify the situation.
@Elfin Stuff >"the source image is literally a crater" Proof, please.
But the source image is literally a crater, so your mind is clearly being tricked.
2 ай бұрын
The credit card was very easy for me to see, but yeah, I kept seeing a mountain, no matter how I looked at it.
2 ай бұрын
2 ай бұрын
then you see wrong, in real life, this is a crater
when you were explaining the optical illusions of indented and sticking out parts, my brain works exactly opposite for some reason. the ones you called indented looked sticking out to me, and the ones you called sticking out looked indented. interesting...
So far my biggest problem with AI, is it seems to be the new fantasy we cling to about salvation. Also for all the problems ai creates the offered solution is often just more ai lol
Just the first minute and a half makes this microscope seems interesting asf for me, since this is in my inexperienced short life (atm) very unique, interesting, and weird
If possible, it would be interesting to see an example of how the algorithm would interpret a color image. It would demonstrate the gap between how these algorithms interpret images compared to our brains.
I think my reaction to your example at 6:00 brings up an interesting point. See, I DIDNT experience the crater illusion when you used the touch microscope, and I think it's because humans are VERY good at context clues and learning; Because you'd shown me other visuals of the touch microscope, I was able to tell from the patterning around the letters that a flat, intended surface was pressed against the microscope, and that the letters were smooth because nothing was touching the gel surface. If they really had been protruding from the block, it would be the letters that had texturing, not the background.
This is honestly amazing! You're so lucky to be able to play with. Is there any information on whether this is going to be sold commercially? I'd love to get my hands on this.
@Barrie Shepherd i mean , that happens with pretty much everything, the Spot robot of Boston Dynamics is 75k meanwhile the chinese knockoffs are way less expensive, even if both were of the same Quality , the original Spot would cost more due to all of the R&D involved , in the other hand the knockoff doesn't has that much R&D into it , and is less expensive due to that
2 ай бұрын
@mustardofdoom Oh man, I guess us common folk won't be able to use it :/
2 ай бұрын
I once received a quote for one about 4 years ago. It is in the tens of thousands to own. They offered a rental arrangement that is in the thousands. I don't know how much the gel pads cost, but they are consumable.
2 ай бұрын
@randallrun i would, if i had half the resources they have
Photometric Stereo is the proper name for the algorithm used like Steve said (comes from the original 1980 paper), but the name can be misleading, because it does not use stereo camera system (e.i. a camera pair), only a single camera. It is based upon the better named Shape From Shading algorithm because it reconstructs the depth information from shading, as opposed to camera motion, positional disparity between cameras, etc. Saying that this microscope uses touch is a bit of a stetch (I think it's arguable either way), this device is mostly optical, but interestingly there ARE microscopes which directly measure by touching the object called scanning probe microscopes. There's a neat video about an atomic force microscope on the Breaking Taps channel.
Omg this is UN REAL! So many applications. It’s like…an “Approach-a-scope” so sick! 🎉
Apart from the extremely interesting features and applications of this idea, one of my takeaway messages is the nth confirm that Hexagons are the Bestagons 🤓
Steve. Look up NeRFs! You can do 3d photogrammetry and capture reflective objects. It's a bit different than you described doing but still relevant I think. Unless you mentioned them and I missed it.
06:55 On an aircraft "Traditionally you would create a cast of the scratch and then measure the cast in the lab" - I never saw that. We could use ultrasonic inspection or other forms of non-destructive testing in-situ to measure the depth of the scratch. In any case, best to smooth it out to avoid stress concentration and the risk of crack initiation - re-checking to ensure that the panel thickness is within tolerance or safe limits.
Mould is on another level with his science videos. He always finds the most interesting and obscure stuff.
Interesting, the images that the 3d scanner (I think that's maybe more apt than a microscope) puts out already reminded me a lot of normal maps, tech we use in games to cheaply render 3d geometry on a flat polygonal surface. Which actually look exactly like the kind of purple image you see at 2:42 . Funny to see this tech show up in unexpected places
I REALLY want one. I'm a guitar luthier and this would be an INCREDIBLY useful tool day in, day out.
Metrology has to be some of the most fascinating disciplines
I ran atomic force microscope (AFM) that used a tiny needle to bounce along the surface and generate images in the nanoscale.
Very interesting. I can visualize this technology being useful in coin collecting. Replacing an individual's perception on flaws or abnormalities that make a coin valuable, a definitive definition would bring consistency.
2 ай бұрын
Wow this is an interesting use-case, but who would use it? regular collectors or like curating organizations? I don't know the field, so I'm not entirely sure if there's like a central authority organization or the like
put a sphere painted in vantablack (or something close) in that machine. i really want to see how the 3d scan turns out
I work with Casio watches replacing their batteries, so the instant I saw it in the microscope I nearly jumped up, "I know that! That's the inside of a watch!"
i replace all watch batteries and i also immediately knew lol
Nice video, I taught they will use ultrasonic imaging medical sensor to see the depth.
this is really interesting! I just learned about vanta-black objects. humans cant exactly figure out what shape vanta black is just by looking at it so it would be nice seeing vanta-black objects being used