I built my own memory profiler (in Rust, on Linux)

Ғылым және технология

Пікірлер: 115

  • @fasterthanlime
    @fasterthanlime Жыл бұрын

    To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/fasterthanlime The first 200 of you will get 20% off Brilliant’s annual premium subscription.

  • @mzg147

    @mzg147

    Жыл бұрын

    Yes!! Go get that sponsors boyy

  • @PaulFisher

    @PaulFisher

    Жыл бұрын

    Stack that paper baby

  • @aleksandermirowsky7988
    @aleksandermirowsky7988 Жыл бұрын

    Honestly, every one of your videos always feel like a blessing. It's reminiscent of a sage descending from the mountains to share his sacred knowledge with the village folk.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I see myself more like a fool tumbling down the hill every so often to bestow the curse of knowledge upon others, but I think the idea is roughly the same!

  • @charetjc

    @charetjc

    Жыл бұрын

    @@fasterthanlime You have an amazing ability to digest all you learned into a concise, interesting, and easy-to-follow manner. This video is excellent.

  • @JheregJAB
    @JheregJAB Жыл бұрын

    every one of your videos up until now has been right on the perfect edge of "I understand, but we're stretching it". I've got to be honest, this one managed to get over my head. That's not bad, its a challenge. Gonna have to rewatch it a couple times and do some extra research on top of it, I think.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Thanks for the feedback! That video definitely warrants at least a couple follow-ups, I have unscripted ideas that I think could be neat. Gotta change it up a little, those videos are an enormous amount of work.

  • @abiiranathan
    @abiiranathan Жыл бұрын

    PHD-esque stuff. Mind-blowing 🤯

  • @dnullify100
    @dnullify100 Жыл бұрын

    Your content is the absolute nexus of production quality, content quality, and approachability. The thoughtful and smooth crescendo of complexity that keeps one engaged for the whole video/story.

  • @multispectrum
    @multispectrum Жыл бұрын

    This was absolutely fantastic. A follow up video or resources to explain things in more detail for people who don't have much low level experience would also be great.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I have a couple in mind, but can you give me a few areas in particular you'd like cleared up?

  • @multispectrum

    @multispectrum

    Жыл бұрын

    ​@@fasterthanlime Again, fantastic work, you're a really great teacher. Watching it again, I wrote down things that I have questions on. There are definitely resources on most of things available and I have a good place to start learning from now. The largest gap in available resources to me seems like eBPF. What defines the memory sections (ro, rw, re)? (Making our own executable packer is on my list to read yet.) TLB makes sense for virtual->physical, but what does walking the page table mean? (Also how does the TLB cache recent memory accesses? Is there a large performance hit to TLB misses? And can the TLB only work for a single process at a time? How does flushing work?) What exactly is a page fault? What is stack/heap, and how to they relate to syscalls (break, mmap, madvise, etc)? Does madvise just allow the kernel to remove pages from the resident memory, but they're still in the virtual memory? What is malloc, malloc_trim, calloc, realloc, free, and how do they relate to syscalls, heap, virtual address space, etc? With your mention of eBPF I lost all understanding of what eBPF is. I thought that it was only for packet manipulation and statistics collection on network traffic. A video on eBPF would be awesome. Everything after ~18:00 I was completely lost. That could probably take a decade of learning. (ptrace, userfaultfd, what is a fd?, what's a setuid binary, self-tracing processes, PID namespaces, seccompbpf?)

  • @FryuniGamer
    @FryuniGamer Жыл бұрын

    "My previous video about this was, in fact, correct. I was as shocked as you are" I wish I had that confidence, but even more I wish I was at the level of saying such a thing.

  • @walksanator

    @walksanator

    2 ай бұрын

    Thats not co fidence though...

  • @meegleeto
    @meegleeto Жыл бұрын

    I can't believe the "this video is sponsored by" bit actually paid off

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    It was all just practice 🧐

  • @SamSargent-kh7gl
    @SamSargent-kh7gl Жыл бұрын

    It's such a relief that people like you make content for us like this. This is exceptional, cheers.

  • @louisfifteen
    @louisfifteen Жыл бұрын

    "This town aint big enough for both of us" by "The Sparks" The one playing the piano is what this guy reminds me off. They were brilliant and so is he.

  • @driedurchin
    @driedurchin Жыл бұрын

    This video exposes a lot of "false dogma" IMO. You traditionally hear the simple explanation of the cost of dynamic vectors, usually saying something like "if a push exhausts the capacity, you have to reallocate and copy everything over", which is *morally* true, and maybe was *actually* true 20 years ago. Nowadays, on modern OSes its interesting to see how it's not the case. Yet another reason to never preemptively optimize. Measure first!

  • @sambazeley5366
    @sambazeley53662 ай бұрын

    I watched this video all the way through assuming that this was another big programming channel. I was surprised to see how small your channel is. There is so much quality!

  • @ahuggingsam
    @ahuggingsam Жыл бұрын

    as always I'm gonna have to go over some of this stuff multiple times, but I'm learning so much. Thanks for making these videos Amos, I doubt I could clearn about this stuff without someone like you making it engaging.

  • @kehaarable
    @kehaarable Жыл бұрын

    Your videos are great. I was looking for some rust videos when I stumbled upon them. I've been programming for about 20 years so most videos are painfully slow. Your videos always touch on subjects that I've never had to deal with. I've not learnt a huge amount of rust, but my understanding of how systems actually work has grown a lot, and enticed me to think more about the tools I take for granted every day and to think about in earnest how I would build them. So thank you!

  • @nindalf
    @nindalf Жыл бұрын

    Fantastic video, thank you for making it. I'd love to see a video of using mevi to optimise existing programs.

  • @lejee_
    @lejee_ Жыл бұрын

    great video, as always :) btw. nice seeing you have a (real) sponsor now!

  • @zperk13
    @zperk13 Жыл бұрын

    Congrats on getting sponsored!

  • @artifishul
    @artifishul Жыл бұрын

    Love this content. Thanks for all the work you put in to explain this so well

  • @Dygear
    @Dygear Жыл бұрын

    Crushed it again Amos! So many things to try this out on!

  • @nikandfor
    @nikandfor Жыл бұрын

    Super cool! I love the observability, and this is the next level!

  • @desuburinga
    @desuburinga Жыл бұрын

    Amazing video, like everyone else mentioned, absolutely mind blowing stuff! Keep up the great work and congrats on the sponsorship!

  • @appelnonsurtaxe
    @appelnonsurtaxe Жыл бұрын

    That was both interesting and very enjoyable to watch! Thanks a lot for making these videos.

  • @Onkoe
    @Onkoe Жыл бұрын

    I love these videos! I’d happily watch a one or two hour version. I have ADHD, so your articles are a lot harder for me, but these videos are perfect!!!

  • @employee9911
    @employee9911 Жыл бұрын

    Really enjoy content like this

  • @jorgeosorio1613
    @jorgeosorio161311 ай бұрын

    this video was great, I loved the introduction.

  • @OhadLutzky
    @OhadLutzky Жыл бұрын

    Excellent job, and congrats on the sponsor!

  • @kickeddroid
    @kickeddroid Жыл бұрын

    What's crazy is I looked this up earlier this morning, taking a break now and watching youtube I see this lmao

  • @patryk6769
    @patryk6769 Жыл бұрын

    Thank you Brilliant for sponsoring this video ❤

  • @fluffy_tail4365
    @fluffy_tail4365 Жыл бұрын

    Incredible video as usual

  • @asadullahfarooqi254
    @asadullahfarooqi254 Жыл бұрын

    yo brah it was amazing, can't wait to see the eBPF :)

  • @craigcraig6248
    @craigcraig6248 Жыл бұрын

    Wow, great video. Im glad i found this channel.

  • @MrMariy0
    @MrMariy0 Жыл бұрын

    very interesting, as always!

  • @bakabaaka2681
    @bakabaaka2681 Жыл бұрын

    Damn... Great video man.

  • @flippert0
    @flippert06 ай бұрын

    While way over my head for now, this feels amazing! One day, I might be able to understand what's going under the hood.

  • @clo4
    @clo4 Жыл бұрын

    Yooo get that bag! Good job Amos!!

  • @chylex
    @chylex Жыл бұрын

    I love the visualization of memory allocations

  • @gpugpugpu
    @gpugpugpu9 ай бұрын

    Another fire fasterthanlime video let’s GOOOOOOOOO

  • @jackevansevo
    @jackevansevo Жыл бұрын

    Blown away by every single one of these videos

  • @flyingsquirrel3271
    @flyingsquirrel3271 Жыл бұрын

    I'm so happy that you casually use the term "kibibytes" :D (I had this discussion a few times with unteachable people...)

  • @gruelingpine1857
    @gruelingpine18578 ай бұрын

    this is pretty cool! Your profile pic reminds me of the scrap metal artist from the Iron Giant

  • @musikosv
    @musikosv Жыл бұрын

    Very interesting, thanks for sharing!"

  • @bowarc
    @bowarc Жыл бұрын

    Rly cool, keep up the content, it's really interesting l!

  • @zperk13
    @zperk13 Жыл бұрын

    Thanks for captions!

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Thanks! I used Whisper for these but even then it's a solid 2.5x-the-runtime of manual cleanup/sync work :)

  • @TobiasFrei
    @TobiasFrei Жыл бұрын

    Oh, these great contributions of yours. You *do* have an impact on the way I think about- and write Rust. Looking forward to some eBPF. Will you go the Aya way? 🤓🐝

  • @TobiasFrei

    @TobiasFrei

    Жыл бұрын

    Or no. It won't be the Aya way. Just as likely you're gonna roll it yourself 😋

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I've naturally looked at Aya since I've used it before for networking, but I'm unsure whether it'll do the job for that project. Time will tell!

  • @johanngambolputty5351
    @johanngambolputty5351 Жыл бұрын

    Not sure I understood any of that, but I was impressed ;)

  • @Leik000
    @Leik000 Жыл бұрын

    Going from "sponsored by coffee" to "sponsored by Brilliant". KZread achievement unlocked.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    All that caffeine finally paid off!

  • @__mrmino__
    @__mrmino__ Жыл бұрын

    Technically sections in an ELF don't need padding. The section header contains the size field, the loader in the kernel can just load the non-zero stuff and leave the rest. Filling ELF section with zeroes ensures that the page is also filled with zeroes, instead of garbage. It has nothing to do with page alignment. The virtual *adresses* still need to be page aligned though, as most MMUs don't support setting memory attributes on anything other than a whole page.

  • @simonfarre4907
    @simonfarre4907 Жыл бұрын

    You haven't looked at RR code base before? A lot of this stuff's in there, although, your project I think is a better source to learn this stuff from since it's probably smaller in scope (and it's in Rust, RR is written in C++). But you've got the sending of fd's over a socket, the injection of syscalls, etc and all that really cool Linux stuff.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I've never cracked rr open (but I've used it a couple times!) but I 100% trust you on that. mevi definitely makes for a smaller surface to study, and you can always drop into some of the crates it uses if you're interested in a specific abstraction!

  • @oyewodayo
    @oyewodayo Жыл бұрын

    You're a good teacher. You explain things differently, even a 2 years old kid will understand anything you explain.

  • @bxnny0374
    @bxnny0374 Жыл бұрын

    Have nothing to say, just commenting to help this video out so more people can see it :) insanely high quality

  • @JustAnotherLight
    @JustAnotherLight Жыл бұрын

    At the beginning of the video, I thought Brilliant's starting a Rust course!

  • @YTCrazytieguy
    @YTCrazytieguy Жыл бұрын

    Yay sponsorship!

  • @Vampirat3
    @Vampirat3 Жыл бұрын

    @fasterthanlime , you get mad respect. great video thank you.

  • @ExCyberino
    @ExCyberino Жыл бұрын

    rm -rf = rename --recursive --flawlessly Basically solves the slash problem in all files

  • @MateuszWykurzIsAwesome
    @MateuszWykurzIsAwesome Жыл бұрын

    Thanks!

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Thank you for your support !

  • @pkoch
    @pkoch Жыл бұрын

    If only my OS class was this good. ❤

  • @user-ed5qx1ih3i
    @user-ed5qx1ih3i Жыл бұрын

    Your doctor is great!

  • @oyewodayo
    @oyewodayo Жыл бұрын

    Please do a series on Axum indepth.

  • @DarrienGlasser
    @DarrienGlasser Жыл бұрын

    Buses and trains don’t exist in that universe I didn’t know you visited America recently :o

  • @Otakutaru
    @Otakutaru Жыл бұрын

    I didn't expect the linux kernel to be so fiddly. Is everything related to low level OS that challenging? For example, memory management on windows? Because I got scared

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I think the short answer is yes. But the good news is that it's "knowable", and then you slowly change from feeling intimidated by it to feeling irritated by it at times 🙃

  • @CramBL
    @CramBL Жыл бұрын

    Great video. What's the website you use for making those sketches at @3:11 for example?

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    That's draw.io / diagrams.net!

  • @heyiamigor
    @heyiamigor Жыл бұрын

    Could you please share your code editor setup? I saw you using VSCode and vim extension, what other configs/extensions do you use? Thx!

  • @brazni
    @brazni Жыл бұрын

    17:29 Don't tell me what to do magic mind reading slides!

  • @EricTorreborre
    @EricTorreborre Жыл бұрын

    Your videos are always mind-blowing, not only for their content but also for the quality of the presentation. Kudos!

  • @kowalkem
    @kowalkem Жыл бұрын

    Missed opportunity here - should have been built on Gentoo or at least Arch 😉 In all seriousness - massive thumbs up 👍

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Oh I've served my time in both the Gentoo & ArchLinux jails, I'm free now, I can use Ubuntu and still have street cred 😎

  • @Verrisin
    @Verrisin Жыл бұрын

    damn, ok, growing a Vec looks a LOT nicer than something like ArrayList in JVM, which I think DOES copy the array. - I might be wrong and it does some magic behind the scenes, but this looks beautiful ...

  • @focusEngineered
    @focusEngineered Жыл бұрын

    AMAZING

  • @sammay1540
    @sammay1540 Жыл бұрын

    Gold

  • @kasvith
    @kasvith Жыл бұрын

    Can you do Zig vids as well

  • @fulgorete
    @fulgorete Жыл бұрын

    This is not a trivial subject. Good work overall

  • @Verrisin
    @Verrisin Жыл бұрын

    so, do we see if Firefox is leaking memory in the "third way" ? That would be a cool thing to find out after the effort.

  • @m4rt_
    @m4rt_ Жыл бұрын

    could you make a thing like RemedyBG... but for linux?

  • @thepetesmith
    @thepetesmith Жыл бұрын

    I’d like to see faster than lime’s car.

  • @as-qh1qq
    @as-qh1qq Жыл бұрын

    This is way above my pay grade.

  • @mint_fresch4329
    @mint_fresch43298 ай бұрын

    imagin u#r talking about linux kernel very basic/complex stuff and get an normal sponsor xD, bro u made it xD

  • @jeffg4686
    @jeffg4686 Жыл бұрын

    nice

  • @versacebroccoli7238
    @versacebroccoli7238 Жыл бұрын

    It's so crazy that the minimum memory a program can allocate (4k) used to be sold as high end ram sticks. I was reading a comment on some other tech video and a person said their computer could have up to 20 4k ram cards. But that would take all their expansion slots.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    It was another time, for sure! From devblogs.microsoft.com/oldnewthing/20200728-00/?p=104012 Huge pointers could refer to memory blocks larger than 64KB by adjusting the segment whenever the offset overflowed.¹ Pointer arithmetic with huge pointers was computationally expensive, so you didn’t use them much.²

  • @Dragon-xd4ru
    @Dragon-xd4ru Жыл бұрын

    Keep making more videos pls 🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏

  • @gakman
    @gakman Жыл бұрын

    lol "Stop reading these slides and pay attention"

  • @Verrisin
    @Verrisin Жыл бұрын

    I wonder if this needed so much unsafe code, it would have been easier to write in Zig ? Probably not, but I wonder. (and yes, it would probably not share code with browser) - Also, I have zero clue how one would even research all the documentation to make all this crazy stuff work ... - How to even figure out all the magic kernel stuff? And how to do it from something other than C ? EDIT: 22:22 - I see. So that's how! Naruhodo.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I've seen that same "we switched from unsafe Rust to Zig" article that you have, but in that case, no, Rust was plenty comfortable, thanks to a wealth of crates that exposed all the system interfaces I wanted :)

  • @Verrisin

    @Verrisin

    Жыл бұрын

    @@fasterthanlime hehe, yup. - Awesome to know! - It's good to know it can be abstracted away into a safe crate.

  • @sudiproy4413
    @sudiproy44139 ай бұрын

    😨 18:13

  • @rotteegher39
    @rotteegher39 Жыл бұрын

    Everytime you say "Mevy" I hear "Bevy" instead (rust game engine)

  • @xmeo209
    @xmeo2096 ай бұрын

    I have no idea what this man says, but its very uhm.. interesting?

  • @muneebdev
    @muneebdev Жыл бұрын

    "a lesser man"

  • @justaguy1299
    @justaguy1299 Жыл бұрын

    This made me think about a video about the same topic I saw a while ago, there they made a tool that compacts the fragmented memory pages of a process. Might be an interesting watch kzread.info/dash/bejne/lWWJpKyblrWTeJM.html

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Yes! I've been meaning to talk about MESH - anything Emery Berger is involved in is sure to be fascinating :)

  • @motbus3
    @motbus32 ай бұрын

    For a month? Hummm...

  • @kaioh33
    @kaioh33 Жыл бұрын

    Pagliaci joke😂😂😂

  • @victorfds
    @victorfds Жыл бұрын

    I don't understand 7 in each 10 words you say.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    Is my pronunciation that bad?

  • @victorfds

    @victorfds

    Жыл бұрын

    @@fasterthanlime absolutely not. I have a little bit knowledge about low level programming. The topics looks so deep that I can't get it.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I have quite a few videos about that but I've been slowly figuring out how to do more accessible videos with a tighter scope. I'll get it right someday!

Келесі