How To Make Your Python Packages Really Fast With RUST

Ғылым және технология

Everyone knows that program speed isn't Python's strong point. That's why so many number-crunching data science libraries are written in C.
BUT C can cause headaches. Segfaults are common and memory leaks are something to be feared. What if there was another option? Well, there is. It's Rust. Rust is a blazingly fast, memory-efficient and memory-safe language that's an absolute joy to work with. And I'm going to teach you how to use it to write your Python packages.
Check out the code:
github.com/isaacharrisholt/yo...
Resources:
Maturin: www.maturin.rs/
PyO3: pyo3.rs/
Rustup: rustup.rs/
The Rust Book: doc.rust-lang.org/stable/book/
uuidt: doc.rust-lang.org/stable/book/
Polars: pola.rs
pydantic-core: github.com/pydantic/pydantic-...
__________________________________________
Check out my other socials!
🐦 Twitter ▶ / isaacharrisholt
🖥️ Portfolio ▶ ihh.dev
📝 Blog ▶ isaacharrisholt.com
__________________________________________
Timestamps:
00:00 - Introduction and explanation
00:46 - Prerequisites
00:58 - Script overview
01:19 - Maturin setup
02:09 - Writing the Rust
02:53 - Benchmarking our function
04:08 - Rust limitations
04:43 - Common gotchas
#python #softwareengineer #rust

Пікірлер: 91

  • @yadukrishnan8577
    @yadukrishnan8577 Жыл бұрын

    "C has enough foot guns to cripple a centipede..." 🙂

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    Thank you thank you. I was WAY too proud of that one.

  • @somethingelse401

    @somethingelse401

    9 ай бұрын

    That made me laugh way too hard

  • @colinmaharaj

    @colinmaharaj

    4 ай бұрын

    Im a C dev, and my foot hurts

  • @vncstudio

    @vncstudio

    27 күн бұрын

    LOL. Nim is also great to interface with Python and very fast.

  • @IsaacHarrisHolt
    @IsaacHarrisHolt9 ай бұрын

    ERRATA: 4:24 - You can use a u64 on a 32-bit machine! The compiler will do its magic and sort it all out for you.

  • @electric26

    @electric26

    9 ай бұрын

    Was wondering about that since you can use 128-bit integers on 64-bit machines. TIL, thanks :)

  • @AkilManivannan
    @AkilManivannan9 ай бұрын

    There are a lot of videos on this topic but this one earns a like for being focused and to the point. Thanks 🙏

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Glad I was able to help!

  • @LowRezCat
    @LowRezCat Жыл бұрын

    Why are you so underrated. You have genuinely good content, why are you still don't get any views? Keep going, good content!

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    That's very kind, thank you!

  • @johanngambolputty5351
    @johanngambolputty53519 ай бұрын

    Damn, after getting more comfortable with the idea of calling opencl code from either python or rust, or c code from python, wrangling c types in all cases, I've been meaning to get to calling rust from python, did not realise it can be this easy (especially for end users too), this might be exactly what I wanted, you've saved me some research :). I kinda like c, but I do find myself missing generics, traits and maybe sum types and pattern matching, not super fussed about automatically checked safety, you still end up having to think about what needs to be accessed from where (and apart from rust-gpu, we still don't really have this for GPGPU), but hey, automating some of that with the compiler can't be a bad thing. All in all, looking forward to using rust for more stuff, apart from when I would find a repl more convenient (data analysis and plotting). Knowing what functions can and can't fail and where you might have to think about nulls is also pretty cool.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I'm glad you found it useful! You'll still be able to call the Rust functions from your Python REPL, thankfully. You'll just have to import the correct lib :) Unfortunately Rust doesn't have a REPL, though I'm sure someone's probably made one!

  • @johanngambolputty5351

    @johanngambolputty5351

    9 ай бұрын

    @@IsaacHarrisHolt I think there are a couple, but for just quickly poking around some slices in a matrix, or plots that you want to edit on the fly, good ol' python will do

  • @ofeki4533
    @ofeki45339 ай бұрын

    So informative! thank u! I can really tell that this vid was made with love

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Thank you! I'm glad you enjoyed it :)

  • @mCoding
    @mCoding9 ай бұрын

    Great video! I wonder what packages I'm already using are actually written in Rust.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Hey James! Big fan 😅 and thank you! I've actually made another video which goes over that, though they're the obvious ones again. Polars, Pydantic, Ruff. There are more and more where authors are choosing to use Rust instead of C/C++ for the critical path. The PyO3 readme has some examples: github.com/PyO3/pyo3

  • @proudmoroccan8164
    @proudmoroccan81649 ай бұрын

    Great stuff, thank you for sharing. Greetings from Morocco.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Awesome! I love Morocco. I want to go back at some point :)

  • @ChrisHalden007
    @ChrisHalden0079 ай бұрын

    Great video. Thanks

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Glad you liked it!

  • @krestofelo8757
    @krestofelo8757 Жыл бұрын

    Hi! I liked the presentation u made and was wondering what software u used for making it.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    I use Obsidian with the Advanced Slides plugin using the Monokai theme :)

  • @anishbhanushali
    @anishbhanushali9 ай бұрын

    aaaand ... subscribed !!

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Thank you so much!

  • @ErikCampobadal
    @ErikCampobadal9 ай бұрын

    I'm curious by your comment on "u64 might not be supported on every machine". What about u128? I am sure this is more than enough for 99.9% of use-cases.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I was actually incorrect about this! I added a correction in the pinned comment, but essentially you can use u64 or u128 for this too. Of course, there will potentially be memory constraints, especially for larger Fibonacci numbers, but they can be easily overcome using a better algorithm

  • @hanshosel7431
    @hanshosel74316 ай бұрын

    Hey great tutoriel sadly the command maturin doesnt seem to work. My maschine cant find a cmdlet, function, script or executable Program named Maturin. Maturin is installed with pip just like you did in the tutoriel. Thanks for your time and help.❤

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    6 ай бұрын

    Hey! It's likely that your pip binaries aren't in your PATH. I'd have a quick Google to find out how to do that for your machine and OS.

  • @filipengstrom3639
    @filipengstrom36399 ай бұрын

    Amazing video. You deserve more subscribers than mr beast.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    That's very kind, thank you!

  • @quintencabo
    @quintencabo9 ай бұрын

    C can be fun. It is super simple once you get the hang of it.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    True! C is harder to get started with from a tooling perspective though. Which compiler do I use? What formatter? etc. etc. Rust doesn't give you a choice, and all the tooling is in a single CLI app, which makes it really easy to start. Yes, the language is a bit more difficult, but it's much easier to write safer code in Rust than C

  • @user-sb5vt8iy5q

    @user-sb5vt8iy5q

    9 ай бұрын

    @@IsaacHarrisHoltI think it's better to have a choice, all of rust currently depends on LLVM...

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    That's not inherently a bad thing. My point is that the barrier for entry to getting started writing Python extensions with Rust is lower than that of C, mostly thanks to the amazing tooling built by the Rust community.

  • @snesmocha

    @snesmocha

    9 ай бұрын

    ⁠@@IsaacHarrisHoltwhy does this need to be such a difficult choice. Windows you use mingw for multi comparability or msvc for performance, Mac you use clang, and Linux you use gcc. And do people really need intelisense so badly? I’ve personally never used intelisense and just use default vim lol

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I think you've just proven my point 😅 for Rust it's cargo on everything!

  • @FirstLast-vn4kl
    @FirstLast-vn4kl Жыл бұрын

    nice to see a fellow wsl user

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    Only for videos and work! I use Ubuntu for most of my personal projects, and then WSL on my work laptop and for recording

  • @osky._.
    @osky._.9 ай бұрын

    What is the name of the font used in vscode?

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I use JetBrains Mono!

  • @osky._.

    @osky._.

    4 ай бұрын

    @@IsaacHarrisHolt Im late haha thanks

  • @nevokrien95
    @nevokrien959 ай бұрын

    With code that needs to take data owned by a different process do some modifications to it and give back control (Like python packages cuda ml kernals etc) Just us8ng pointers seems so nice and simple

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I'm not entirely sure how this works in that regard. I haven't looked into whether PyO3 gets values by reference or by value. I'll have a look!

  • @KrzysztofDerecki
    @KrzysztofDerecki9 ай бұрын

    Why would you recommend Rust for someone with Python background and not Nim?

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Good question! Mostly because I'm not familiar with it, but also Rust and Python is a very popular combination at the moment and I couldn't find many good videos on the topic, so I decided to fill the gap.

  • @filipengstrom3639
    @filipengstrom36398 ай бұрын

    I tried to create and return a simple list containing 1 million integers in range with the following: #[pyfunction] fn create_integer_list(n: usize) -> PyResult { let mut result = Vec::new(); for i in 0..n { result.push(i); } Ok(result) } when I compile it in RustRover it runs in 8 ms. However, when I create it in maturin as per your instructions, it takes roughly 110 ms to run. Any idea what's causing it? I ran maturin develop --release

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    8 ай бұрын

    It's very possibly the interaction with Python. When you say you're compiling it in RustRover, are you simply compiling it as a Rust binary or as a Python library?

  • @filipengstrom3639

    @filipengstrom3639

    8 ай бұрын

    @@IsaacHarrisHolt Sorry I was unclear, I meant that I ran the same code as above in RustRover which worked great, as opposed to when I ran that code through maturin. I have narrowed it down to the PyResult return of my function. It seems like returning a vec and "retrieving" a list in python is really slow. Gonna see if there is a way to return a list of values in a better way

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    8 ай бұрын

    That makes sense. The communication with the Python layer will be what takes up most of the time in your case, I would imagine.

  • @bogaczew
    @bogaczew9 ай бұрын

    don't know about it. there are good, tested modules like numpy, to make your counts in python, they are fast.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    They're really fast! Because they're written in C 😉

  • @archniki_
    @archniki_9 ай бұрын

    I try learn Julia but there are only English resources

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    That's a shame! But it also means there's a gap in the market 😉

  • @CamaradaArdi
    @CamaradaArdi9 ай бұрын

    What are you talking about? u64 and u128 are supported in 32 bit machines lol, the compiler fixes that for you. Please add this correction as that is highly misleading. That being said, it's very rare to build for a 32 bit architecture in 2023 if you're not in embedded, and there you have a whole lot of other things to worry about.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I've added a pinned comment with this information. Thanks for calling me out!

  • @denizsincar29
    @denizsincar29 Жыл бұрын

    wow, i did it!!! With poetry!

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    You got this working with Poetry? Awesome! What was the trick?

  • @denizsincar29

    @denizsincar29

    Жыл бұрын

    @@IsaacHarrisHolt in pyproject toml specify build backend as maturin and something else a littlebit. can i have your contact? I have telegram, whatsapp and email. I could provide a bit more detail because now my laptop is closed and i forgot what i actually did.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    My email address should be available on my channel's About page

  • @luisalejandroacunalopez3662
    @luisalejandroacunalopez36629 ай бұрын

    I wish there was anything similar for doing it with c++ and pybind

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I'd not heard of pybind! It looks like a minimal version of Boost, but it would be nice if there was a build tool for it. Technically maturin and PyO3 are separate tools, so there wouldn't be anything preventing you from creating the equivalent for pybind11.

  • @soniakolasinska3850
    @soniakolasinska38503 ай бұрын

    Let's not fall into syntax similarity trap. Rust is wonderful language, and I'd use it for other reasons than similarity of syntax. In fact popular modern languages such as Rust, Typescript, Scala, and typed Python share syntax with good old Pascal, which would be the first programming language that you would learn in my days (except basic). Personally, I have rewritten PoCs in Python in C++, and I didn't find it any difficult, actually I could achieve most of those rewrites by copy & paste of Python code, and then run some VI macros to add types, compile and all works! I don't think it is that simple with Rust.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    3 ай бұрын

    You make very good points! I usually only mention the syntax similarity because it slightly reduces people's "fear" of Rust. Lots of folks hear that it's hard and then don't want to try it, so it's just another reason it might not be that scary. Naturally, it's not the most important thing when considering which language to use to extend Python. Honestly, I think the most important consideration is whether you even SHOULD be using a different language when there are so many things you can do to optimise Python code first.

  • @loek8638
    @loek8638 Жыл бұрын

    what ide is this?

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    It's PyCharm! I'm using the new UI, dark theme and the red-green colour blindness adjustment turned on, if you're curious about the theme :)

  • @joaopauloalbq
    @joaopauloalbq10 ай бұрын

    Mojo 👀

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Perhaps! I made a video on Mojo, but it seems to be a lot further along now. I'll need to take a closer look at some point

  • @pablodenapoli1667
    @pablodenapoli1667Ай бұрын

    Better use something more similar to Pyrhon and easier like cython or for instance D or nim..

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Ай бұрын

    I've not tried Cython, but Nim is good, I agree

  • @jordixboy
    @jordixboy9 ай бұрын

    C is fun tbh

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I'm not denying it! I just prefer Rust for Python extensions :)

  • @yash1152
    @yash11529 ай бұрын

    C++ has enough leg bombs to cripple a tank

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    This is why we Rust 👀

  • @yash1152

    @yash1152

    9 ай бұрын

    @@IsaacHarrisHolt ohkay, i think i will start with rust then. one question - does it have proper tooling - like the IDE documentations & linting etc?? editors which i have available: vscodium, kate, intellij.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Yes! Rust's tooling is amazing. It has an LSP called rust-analyzer which you'll be able to use in both VSCodium and Kate, and all the JetBrains IDEs have a Rust plugin available (JetBrains are also working on a Rust IDE). I'd recommend looking at the Rust book and watching a few KZread videos to get started :)

  • @amortalbeing
    @amortalbeing9 ай бұрын

    This is good for rust developers but not C or C++ for obvious reasons. anything that enriches python echo system is welcomed by me.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    I don't think it's particularly BAD for C/C++ devs. They can continue to write their language - it just gives people another option!

  • @PaulJaros
    @PaulJaros Жыл бұрын

    I'm both a python (on job) and a rust (newbie and hobby) programmer. And before I start my criticism: Yes, pure python code will be slower than rust code most of the time. My issue is the abundance of oversimplified benchmarks on the internet like this one. Please stop doing those. New programmers will pick up those examples and will think these are mostly okay. Please, use real-world problems whenever possible and solve it with your best effort on both sides. Don't trust the results and don't jump on conclusions. Verify the results as good as you can and be as transparent as possible when publishing the results.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    While I do typically agree with your approach, this video is aimed at people who can't really write any Rust, so I needed an example that would easily translate from Python to Rust without causing any confusion. For that reason, I avoided anything with any data structures as they're topics that require some understanding of the Rust borrow checker.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    Жыл бұрын

    Ah, that's a good suggestion. I'll keep that in mind for next time, thank you!

  • @seriouce4832

    @seriouce4832

    9 ай бұрын

    @@IsaacHarrisHolt I fully agree with your perspective here. I guess the video was about "how to run rust code in python", not "look how slow python is"...

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Yes! Absolutely. Python is slow compared to Rust for certain, but this was just meant to be an example of WHY you should consider rewriting parts in Rust. The idea was to show potential gains, not necessarily to benchmark the two languages.

  • @DJenriqez

    @DJenriqez

    9 ай бұрын

    Wtf you are talking about, the tutorial does what video name says,... What real world problems , what oversimplified benchmark ? Benchmark is good, shows simply comparison of those 2 languages. What is Real World problem ?, in real world, like almost a year, I move data from frontend to backend, to database or file storage, read and open and refactor files,....

  • @andreasericsson6989
    @andreasericsson69899 ай бұрын

    I feel like every language-plugger and C-hater in the world keeps forgetting that their favourite language runs on top of about 5 million lines of well-tested and functioning C code. It's no longer feasible to write any language at all without libbing out to C for most of its basic functionality (like opening files, accessing memory, listening to sockets, connecting to servers etc etc), simply because if you spend 2+ years getting the basics done, your language will get overtaken by something else.

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    That's probably true! But it doesn't mean that everyone has to write C. My argument in this video is that Rust is easier to get started with for Python extensions. That's what I've found from experience, even as someone who's done more C/C++ than Rust in the past.

  • @gustavojoaquin_arch
    @gustavojoaquin_arch9 ай бұрын

    zzzZzz Imagine using pythonzzzzz,. instead of pure Rust

  • @IsaacHarrisHolt

    @IsaacHarrisHolt

    9 ай бұрын

    Unfortunately I'm not genius enough to use pure Rust 😓

Келесі