Self-referential structs (in Rust)

Ғылым және технология

Follow me on Mastodon: octodon.social/@fasterthanlime
Support me on Patreon or GitHub: fasterthanli.me/donate
In this video, we bump into some lifetime issues, try to understand why, and end up dangerously working around them.

Пікірлер: 143

  • @Kazyek
    @Kazyek Жыл бұрын

    Scoped threads were added in Rust 1.63.0, which could be used to cleanly solve your specific simple example without unsafe

  • @m.sierra5258
    @m.sierra5258 Жыл бұрын

    Just wanted to let you know that scoped threads now made it to Rust's Std library in 1.63.

  • @yardez5990

    @yardez5990

    Жыл бұрын

    1.63*

  • @dexio85

    @dexio85

    Жыл бұрын

    Still, another patchup on the misguided idea that compiler will "catch all problems".

  • @yardez5990

    @yardez5990

    Жыл бұрын

    ​@@dexio85 why is it misguided? every language's compiler tries to find as many problems in your code as it can, and if it doesnt devs try to make some additional tools for that

  • @m.sierra5258

    @m.sierra5258

    Жыл бұрын

    @@yardez5990 Exactly. And Rust does guarantee no undefined behavior without the unsafe keyword. This is not misguided, it is a guarantee. (and a compiler bug if not held)

  • @m.sierra5258

    @m.sierra5258

    Жыл бұрын

    @@dexio85 If I learned one thing during my years of working as a programmer, then it is that developers are absolutely never reliable, so every bit that the compiler enforces is valuable. Look at it as built-in sanity checks. Why is this misguided?

  • @gomesroney
    @gomesroney2 жыл бұрын

    Very nice video. It's hard to find examples online of memory corruption in Rust, so it was nice that you demonstrated how that could be possible. Also, as you iterated over the example I was getting high hopes that you would explore pinning but then the video ended, haha. I was ready for more 30 minutes just on Pin 🙂. I hope you can explore that in the future as your examples oriented approach makes everything much more comprehensible.

  • @m.sierra5258

    @m.sierra5258

    Жыл бұрын

    Pin then definitely no longer counts as "simple example" :P

  • @Iifesteal
    @Iifesteal Жыл бұрын

    Thanks for starting with a normal for loop and working your way up to idiomatic rust. Really appreciated by a starter like myself.

  • @flyingsquirrel3271
    @flyingsquirrel32713 жыл бұрын

    Awesome! Thanks for the video! Although I knew most of this before, watching videos like this feels like casually "strengthening" my inner understanding of rust which is always appreciated :)

  • @daniellambert6207
    @daniellambert62072 жыл бұрын

    14:30 the worker thread will outlive main() {} if you decided to add a conditional panic!() before the join. 16:00 aah, so crossbeam's scope will be dropped at that diabolical panic, ensuring the threads join

  • @neopalm2050

    @neopalm2050

    Жыл бұрын

    I guess this illustrates why it's important that things are dropped in the event of a panic!()

  • @jotch_7627

    @jotch_7627

    6 ай бұрын

    ​@@neopalm2050no, actually. rust does not allow memory unsafety just because a value wasnt dropped. thats why std::mem::forget is safe to call. the way it actually works is that the callback is run in a way that panics are caught and cleanup is performed before continuing the panic

  • @Verrisin
    @Verrisin Жыл бұрын

    The more I learn about Rust (and what it makes explicit about how hard it used to be in C++) .... The more I appreciate simplicity of programming with GC.

  • @linkernick5379
    @linkernick53793 жыл бұрын

    I love your blog, learned from it so much.

  • @sardobi
    @sardobi2 жыл бұрын

    Love your work! this video was pitched at the perfect level for me

  • @MetalManiacBoy
    @MetalManiacBoy Жыл бұрын

    Insanely well explained, fasterthanlime ! Keep up the good work :D

  • @naitikmundra8511
    @naitikmundra8511 Жыл бұрын

    The feeling when you finally understand ownership! Great video.

  • @wololo1657
    @wololo16572 жыл бұрын

    I did not understand *all* of the parts of this video, but I still managed to learn a lot by watching it. I will be coming back to this video, certainly.

  • @abhishekshah11
    @abhishekshah113 жыл бұрын

    Yeah this was really insightful into subtle workings of rust safe code. Would like to see more of it. I'll try to write a "safe" fn woops this afternoon hopefully, now that I'm curious to make it work :P

  • @kajacx
    @kajacx Жыл бұрын

    18:00 The reason that move doesn't work is because rust compiler doesn't understand the difference between shallow and deep borrow. Since the "good_lines" vec borrows (points to) the heap where "input" points, it's actually safe to move input while good_lines exist. If good_lines pointed to the input's length, for example, then you couldn't move the input, because then good_lines would point to invalid memory. But the Rust compiler has no way of knowing that. There are no type annotations for it. The same problem is with shallow/deep mutability. Sometimes you have to declare a variable binding as mut even when you don't need shallow mutability.

  • @antonionatilla9848
    @antonionatilla98482 жыл бұрын

    Very interesting demonstration and explanation! I dealt with the same situation a couple of weeks ago, but I didn't delve into the "unsafe territory" since I'm "still learning" Rust. Finding out about this video was a pleasant surprise, in the past days!

  • @m.sierra5258

    @m.sierra5258

    Жыл бұрын

    It's hard to believe to me that many people still strongly advocate C/C++ where unsafe ist just normal

  • @Hwyadylaw

    @Hwyadylaw

    Жыл бұрын

    I think it's quite nice that writing unsafe code is so explicit that it makes you think "maybe I shouldn't", whereas in other languages you might write rust-unsafe code without even realising the potential issues

  • @squelchedotter
    @squelchedotter3 жыл бұрын

    Where is Cool Bear though 🐻 😞

  • @timo7964
    @timo79643 жыл бұрын

    Amazing video! thank you!

  • @numtostr
    @numtostr3 жыл бұрын

    This was indeed fun. Thanks!

  • @JihChiLee
    @JihChiLee3 жыл бұрын

    Love you videos! keep it up!

  • @alexzander__6334
    @alexzander__63342 жыл бұрын

    loved that part with the diagram

  • @joaoalves1359
    @joaoalves13593 жыл бұрын

    Excellent video

  • @pepoviola
    @pepoviola3 жыл бұрын

    Awesome video!! thanks!! :)

  • @user-lp1is2zq9n
    @user-lp1is2zq9n2 жыл бұрын

    Great video. How about Pin and Unpin. They should help with these problems (almost) without unsafe. Are there a any good guides on how to use it in self-referential structs?

  • @1000percent1000
    @1000percent1000 Жыл бұрын

    i have been wondering for months why joining threads doesn't join lifetimes, you are a lifesaver for demonstrating crossbeam's scope. so useful, thank you so much :DDD

  • @d3line

    @d3line

    Жыл бұрын

    Scoped threads are now in standard library btw

  • @SophieJMore

    @SophieJMore

    10 ай бұрын

    Joining threads doesn't join lifetimes because join() can be called at any time (including never), or it can be called in some conditions and not others. Also, the compiler is unable to reason about any complex control flow, it just relies on the lifetime system, and the lifetime of the closure you pass when you spawn the thread is 'static, which was chosen for the abovementioned reason. That 'static is literally everything the compiler sees, and it will reject any attempt to give it a closure that reference anything non-static

  • @thirdreplicator
    @thirdreplicator Жыл бұрын

    Very nice!

  • @kh0kh0
    @kh0kh03 жыл бұрын

    More please!

  • @CottidaeSEA
    @CottidaeSEA Жыл бұрын

    Once you mentioned the second thread, all I could think was "you could just do the entire thing in the new thread".

  • @LeandroCoutinho
    @LeandroCoutinho2 жыл бұрын

    Amazing!!!

  • @kibar9155
    @kibar91552 жыл бұрын

    awesome! thanks

  • @nikandfor
    @nikandfor Жыл бұрын

    What a good simple language!

  • @syncpoint
    @syncpoint2 жыл бұрын

    Hopefully there are more in que regarding rust.

  • Жыл бұрын

    Hi bro, thanks for all of your hard work. I would like you know, which plug-in are using, to show the errors on the line in Vs Code

  • @tuesss

    @tuesss

    Жыл бұрын

    I have only used VSCode very briefly, but I think it's Error Lens.

  • @manawa3832
    @manawa3832 Жыл бұрын

    Great video also keep in mind aliasing could still screw things up.

  • @jx6773
    @jx6773 Жыл бұрын

    what vscodes extensions are you using???? those seems useful

  • @mattdavies6820
    @mattdavies68202 жыл бұрын

    Great video - what is the drawing tool you're using for doing the diagrams?

  • @samuelhurel6402

    @samuelhurel6402

    2 жыл бұрын

    Have you found out ? I'd like to know too

  • @cthutu

    @cthutu

    2 жыл бұрын

    @@samuelhurel6402 No, but it's the same tools that 3Blue1Brown uses in his Maths videos. Coincidentally, I literally asked the same question on one of his videos. It's blowing my mind that you should reply to my original ask from 6 months ago!

  • @walterkrieg5396

    @walterkrieg5396

    3 ай бұрын

    It’s a python extension called manim

  • @Laura-hl3hg
    @Laura-hl3hg Жыл бұрын

    Hey, I love this video, you're amazing at explaining! Could you tell me what software you're using to make those diagrams? I feel like it could be insanely helpful for sketching out the logic flow of programs.

  • @fasterthanlime

    @fasterthanlime

    Жыл бұрын

    I'm using draw.io, now named diagrams.net

  • @zokalyx
    @zokalyx Жыл бұрын

    So... what approach would you use if you were to avoid unsafe?

  • @baggern
    @baggern3 жыл бұрын

    "load bearing keyword" lol

  • @smidimir
    @smidimir10 ай бұрын

    I'm currently new to rust but, I know C++ very well. In rust I tried to solve exactly this problem for about 3 hours using "safe" rust without success. Having my background, I had some assumptions about how certain things work in rust. - Unlike C++, rust "moves" are just memcpy's. - Moving the container (String or Vec) shoud not mean moving its data from one location to another. It's just a handle. I even checked if String in rust has short string optimization. And it doesn't. So, what's the problem, rust? Why don't you let me do this clearly safe thing? In your video you asked exactly the questions I had in my head while trying to make it work. I also started to think, that "unsafe" rust is the solution to the problem. Thank you for the thought process and the solution. Great video!

  • @dj-maxus
    @dj-maxus Жыл бұрын

    Currently, `std::thread` also have `scope()` which is pretty nice

  • @asdfghyter
    @asdfghyter7 ай бұрын

    4:19 was there any specific reason to wrap the filter function in a lambda instead of just writing .filter(starts_with_capital_letter) directly?

  • @jeffg4686
    @jeffg4686 Жыл бұрын

    The comment about a move being a memcpy, is that only true in the context of moving across a thread boundary? I always think of it as more of "compile-time rules enforcement" (playing the game of the language so to speak) than an actual memory operation at runtime, but I don't really know ... Was just the way I had envisioned. Maybe it's a runtime thing for thread boundary, but compile time thing otherwise. Something to ponder. I would think for a thread boundary, it would have to be an actual runtime consideration (move to stack of other thread), but otherwise I envision compiler enforcement only, and no memory op. I guess the bigger point is that a move may or may not be a memory op (i think...)

  • @ConnorMcCormick
    @ConnorMcCormick Жыл бұрын

    i don't think anyone has mentioned it but scoped threads were added rust's std 1.63.0, wish someone would have commented this earlier

  • @SianaGearz

    @SianaGearz

    Жыл бұрын

    Is it deep sarcasm or genuine? I can't tell.

  • @ConnorMcCormick

    @ConnorMcCormick

    Жыл бұрын

    @@SianaGearz i saw it mentioned 3 times so wanted to make sure it got mentioned at least once

  • @kajacx
    @kajacx Жыл бұрын

    This is about deep references. Still interesting, but I was hoping it would be about shallow self-referencing. Much more fun, since you cannot even move the struct then. Does "Pin" help there?

  • @AbelShields
    @AbelShields Жыл бұрын

    Hmmm... I wonder if this problem could be solved with a little language help - can we assume it's fine to hold a reference to a heap value, such as a T in `Box` while the box itself is moved?

  • @d3line

    @d3line

    Жыл бұрын

    It's not really about the move, it's more about the drop. Box could be moved to a function that immediately drops it, so box is deallocated and references become invalid. So no.

  • @AbelShields

    @AbelShields

    Жыл бұрын

    @@d3line huh, that makes a lot of sense. Thanks :)

  • @thepuzzlemaker2159
    @thepuzzlemaker21592 жыл бұрын

    Nice video! Just curious, what's that program around 8:18?

  • @fasterthanlime

    @fasterthanlime

    2 жыл бұрын

    That's draw.io, now called diagrams.net

  • @user-in5vf3ll8h
    @user-in5vf3ll8hАй бұрын

    What font are you using in the video ?

  • @Hector-bj3ls
    @Hector-bj3ls Жыл бұрын

    Why did we get Pin instead of relative pointers? The issue is that references and pointers have absolute addresses. So &a returns the absolute address of a. If we had some way to do relative pointers/references we could have a pointer that is more of a "take my address and sub/add x to get to the thing I point to" instead of "I contain the address of the thing I point to".

  • @miniappletheapple
    @miniappletheapple Жыл бұрын

    Which application are you using?

  • @asdfghyter
    @asdfghyter7 ай бұрын

    I wonder if it would be possible for Rust to be aware of the distinct lifetimes of the String struct and the string data it points to? when moving something, you invalidate anything that points to it directly, but it shouldn't invalidate pointers to things it points to. it doesn't seem like a super difficult rule to get right and it should be a fairly common case: if i borrow a pointer from a struct, you can move the struct, but you can't drop it nor mutate the pointer part of the struct. the last part is similar to rules that already exist, since you can partially borrow a struct and then the rest of the struct is still available *edit:* there seems to be a crate that attempts to solve this, called owning_ref. unfortunately it (and the example in the video) seems to be unsound/UB for some technical reasons, which is proposed to be fixed by the "maybe_dangling" rfc. *edit2:* another crate called _self_cell_ claims to be sound, so maybe that's the way to go. it uses a macro to define the struct and the allowed operations

  • @zperk13
    @zperk13 Жыл бұрын

    7:00 How in the world did you get rust analyzer to highlight line 17 too?

  • @zperk13

    @zperk13

    Жыл бұрын

    Figured it out. You need to set Error Lens to also highlight hints

  • @lucasemanuelgenova9179
    @lucasemanuelgenova9179 Жыл бұрын

    Which drawing program is that?

  • @joshuahab
    @joshuahab2 жыл бұрын

    What if you created an ImmutableString type that can be built from String? In that case it seems like self-reference to the ImmutableString could be made safe.

  • @DeviRuto

    @DeviRuto

    Жыл бұрын

    You can always leak() the string. Makes anything 'static

  • @SophieJMore

    @SophieJMore

    10 ай бұрын

    I think the main problem here is that you can't move values that are being borrowed, even immutably. Generally, it's unsound to do so, because it might invalidate the data that's being pointed to. Here it's not unsound, since the data that's being pointed to is actually on the heap. However, the compiler can't really meaningfully differentiate between the two cases. To it, &String and &str both borrow from String, so it would prevent the String from being moved. Even if you wrap a String to make it immutable (psst, better use a Box instead in this case), you still don't be and to make a self referential struct with it (without unsafe). So, if by "could be made safe", you mean not using unsafe blocks, then I don't think it's possible. Otherwise, you can always make a safe API around unsafe code, just be careful.

  • @maxwellclarke1862
    @maxwellclarke1862 Жыл бұрын

    I think RcString is the best solution here

  • @ibrozdemir
    @ibrozdemir Жыл бұрын

    relatable intro xD even tho i never coded in rust, i understand the pain you are having.. why would anyone make another language as fast as c++ but puts tonns of restriction so no one can use it as free as they want

  • @TheMrKeksLp

    @TheMrKeksLp

    Жыл бұрын

    Because having the compiler check the code for you is better than you checking the code and failing

  • @ibrozdemir

    @ibrozdemir

    Жыл бұрын

    @@TheMrKeksLp thats not checking thats restricting.. and yes you are right, with c++ you will get more crashes and bugs, however thats (in time) fixable.. since %99.9 of programmers not writing codes for spaceships, its allright

  • @JimmyZeng
    @JimmyZeng Жыл бұрын

    I think there's some serious misunderstanding about "move" in rust, "move" here does not mean the memory address is _moved_, it's just the ownership of a value been moved.

  • @micycle8778

    @micycle8778

    Жыл бұрын

    but when a closure in rust has the move keyword, would it not copy stack allocated memory into that closure's stack, therefore changing the memory address?

  • @JimmyZeng

    @JimmyZeng

    Жыл бұрын

    @@micycle8778 ​ I think the point is: when we are talking about move, we are talking about ownership, not address.

  • @calisti9308

    @calisti9308

    7 ай бұрын

    @@JimmyZeng Ownership definitely moves, while the address *might* move, or be optimized away.

  • @newclarity4228
    @newclarity42283 ай бұрын

    What software do you use for your diagrams?

  • @fasterthanlime

    @fasterthanlime

    3 ай бұрын

    In this video, draw.io / diagrams.net

  • @fasterthanlime

    @fasterthanlime

    3 ай бұрын

    In this video, draw.io / diagrams.net

  • @calisti9308
    @calisti93087 ай бұрын

    23:03 Swapping the pointer, length and capacity of the two strings (input2 and s.input) does nothing to the references stored in S::good_lines. And I would imagine it also does nothing at all to each heap-stored string slice (pointed to by the pointer field of input2 and s.input - so the pointers on the stack are swapped, but the bytes in heap aren't.) But why does it then start failing when enclosed in a scope (curly braces)?!?

  • @gauravtatke4793
    @gauravtatke479311 ай бұрын

    Hi, At 22:00 to 22:15 time, why does return S not complain about returning a reference to local variable input_slice? good_lines hold a reference to input_slice which is a local variable. Can someone please explain?

  • @calisti9308

    @calisti9308

    7 ай бұрын

    input_slice is a reference to memory on the heap, not a binding of a local stack value. good_lines reference sub-slices of the same heap string slice that input_slice points to. The reference's lifetime is made 'static by the unsafe block, even though the heap value is not actually 'static - which is why this is unsafe. (If the heap value is dropped, then both input_slice and the vec entries of good_lines point at garbage memory.

  • @contactron
    @contactron2 жыл бұрын

    So you are saying std::thread::spawn never drops any resource moved into it? That seems like a major source of memory leaks. Good to know.

  • @fasterthanlime

    @fasterthanlime

    2 жыл бұрын

    It definitely drops resources at the end of the thread, but the compiler cannot statically tell *when* unless you use something like crossbeam's scoped threads (which is not always what you want either). But it's easy to check that it does in fact drop stuff when they fall out of scope: play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=2968a70cb054657ff7134dce01679c49

  • @contactron

    @contactron

    2 жыл бұрын

    @@fasterthanlime OK, good, that's what I would expect. I originally misunderstood what you were saying at 14:00, I thought you were asking why it wasn't safe to drop input, which seemed odd to me because you had moved input into the new thread. But really you were asking *why* you had to move it into the thread. Anyways nice video.

  • @mattdavies6820
    @mattdavies68202 жыл бұрын

    `scoped_threadpool` may be lighter weight than crossbeam

  • @gabrielnuetzi
    @gabrielnuetzi Жыл бұрын

    In 2023 is there an safe way of achieving self referencial structs? ❤??

  • @zerker2000

    @zerker2000

    Жыл бұрын

    OwningRef seems like it would be useful here, if generally limited.

  • @bozhidaratanasov7800
    @bozhidaratanasov7800 Жыл бұрын

    Can anyone point me to good material on the topic of building your own LinkedList/ Graph type structures in Rust? This is what I understand by "self-referential".

  • @bozhidaratanasov7800

    @bozhidaratanasov7800

    Жыл бұрын

    To answer my own question, there is a reading online called "Learning Rust with entirely too many linked lists". It's dence with useful Rust examples.

  • @MasterHigure
    @MasterHigure Жыл бұрын

    4:30 filter(|x| f(x)) is the same as just filter(f).

  • @qm3ster

    @qm3ster

    Жыл бұрын

    Nope. `f` is `Fn(&str)` `|x| f(x)` is `Fn(&&str)` There is a dereferencing happening. The closure is unavoidable.

  • @WolvericCatkin
    @WolvericCatkin Жыл бұрын

    _`starts_with_capital_letter`..._ 🤦‍♀️ Or just `.starts_with(|&x| x.is_uppercase())`...

  • @Dorumin

    @Dorumin

    Жыл бұрын

    Yeah that function was really awkwardly written, but it did the trick I guess. Is &x necessary? Shouldn't the dot operator auto deref?

  • @TheMrKeksLp

    @TheMrKeksLp

    Жыл бұрын

    Many roads lead to rome 🤷‍♂

  • @qaqkirby9781
    @qaqkirby97817 ай бұрын

    😂😂😂

  • @MykolaTheVaultDweller
    @MykolaTheVaultDweller5 ай бұрын

    good video. but where is self referential structs? looks like clickbait title

  • @theLowestPointInMyLife
    @theLowestPointInMyLife Жыл бұрын

    GUI in rust lol

  • @alekxcrafter
    @alekxcrafter Жыл бұрын

    rust is cringe

  • @happygofishing

    @happygofishing

    Жыл бұрын

    Language for transsexuals that failed to use makefiles and pointers

  • @jonathanmoore5619
    @jonathanmoore56192 жыл бұрын

    Contrived. "Main" is the program. Join the thread and end, else send the changed input to another file. You're creating an unnecessary problem.

  • @Iogoslavia
    @Iogoslavia Жыл бұрын

    Oh man, I thought there was going to be some neat trick to create self referential structures without the unsafe keyword 🥲

  • @buzdxkysguge2334
    @buzdxkysguge23343 жыл бұрын

    Good work! May I ask what's the software you used to draw those diagrams.

  • @meain_

    @meain_

    3 жыл бұрын

    Kinda looks like app.diagrams.net/

Келесі