FASTER Inference with Torch TensorRT Deep Learning for Beginners - CPU vs CUDA

Ғылым және технология

Hi everyone! 😀 In the last video we've seen how to accelerate the speed of our programs with Pytorch and CUDA - today we will take it another step further with Torch-TensorRT!
We will focus on a Machine Learning process called Inference (which is when the model is trained, perfected and ready to make a prediction).
For this we will load a state-of-the-art artificial neural network and we will use it to classify a picture of my cat! 🙀🙀🙀
Specifically - we will borrow ResNet50 for our little Pytorch experiment! 😉
We will also run a speed test comparing Pytorch models running on CPU, on CUDA and on Torch-TensorRT - which of these do you think is faster??
⏲️ TIMESTAMPS ⏲️
-----------------------------------
00:00 - intro
01:05 - clone Torch-TensorRT
01:40 - install and setup Docker
03:52 - install Nvidia Container Toolkit & Nvidia Docker 2
05:02 - Torch-TensorRT container (option #1)
07:22 - Torch-TensorRT Nvidia NGC container (option #2)
09:00 - import Pytorch
09:16 - load ResNet50
10:25 - load sample image
11:45 - sample image transforms
14:48 - batch size
16:19 - prediction with ResNet50
17:12 - softmax function
18:07 - ImageNet class number to name mapping
20:10 - predict top 5 classes of sample image (topk)
23:33 - speed test benchmark function
27:33 - CPU benchmarks
28:13 - CUDA benchmarks
30:09 - trace model
31:20 - convert traced model into a Torch-TensorRT model
33:02 - TensorRT benchmarks
34:32 - download Jupyter Notebook
34:50 - HOW DID I MISS THIS???
35:31 - thanks for watching!
🛑 REFERENCED TUTORIALS 🛑
----------------------------------------------------------------------
⭐ CUDA Parallel Computing for beginners:
• CUDA Simply Explained ...
⭐ Neural Networks for beginners:
• Neural Network Simply ...
⭐ Machine Learning Databases:
• Machine Learning Datab...
⭐ Gradient Descent:
• Gradient Descent - Sim...
⭐ INSTALLATION LINKS AND BASH COMMANDS ⭐
--------------------------------------------------------------------------------------
1. Clone Torch-TensorRT and change directory:
$ git clone github.com/NVIDIA/Torch-TensorRT
$ cd Torch-TensorRT
2. Docker installation guide:
docs.nvidia.com/datacenter/cl...
$ curl get.docker.com | sh \ && sudo systemctl --now enable docker
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker
$ docker run hello-world
3. Nvidia Docker 2 installation:
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ && curl -s -L nvidia.github.io/nvidia-docke... | sudo apt-key add - \ && curl -s -L nvidia.github.io/nvidia-docke... er.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
4. Official Torch TensorRT Container:
$ docker build -t torch_tensorrt -f ./docker/Dockerfile .
$ docker run --gpus=all --rm -it -v $PWD:/Torch-TensorRT --net=host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 torch_tensorrt:latest bash
$ cd /Torch-TensorRT/notebooks
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888
5. Nvidia NGC Container (alternative to 4):
catalog.ngc.nvidia.com/orgs/n...
$ docker pull nvcr.io/nvidia/pytorch:21.12-py3
$ docker run --net=host --gpus all -it --rm -v local_dir:/container_dir nvcr.io/nvidia/pytorch:21.12-py3
$ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888
6. My cats picture:
github.com/MariyaSha/Inferenc...
7. Transforms Normalize - Pytorch Documentation:
pytorch.org/vision/stable/tra...
8. ImageNet class to name mapping:
github.com/pytorch/hub/blob/m...
9. Complete Notebook on Github:
github.com/MariyaSha/Inferenc...
---------------------------------------------------------------------------
💗 THANK YOU SO MUCH FOR WATCHING! 💗
- Sound effect by: www.zapsplat.com
- Icons by: www.flaticon.com/
- Resnet50 image by: commons.wikimedia.org/wiki/Fi...

Пікірлер: 99

  • @lukajeliciclux3074
    @lukajeliciclux30742 жыл бұрын

    Great video as always. One advice: in the newer versions of Ubuntu apt-get is replaced with apt because apt-get is deprecated and apt has better performance. Everything stay the same while installing things but instead of apt-get you will use apt.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much for the tip and for the lovely comment Luka! 😀😀😀

  • @buithanhlam3726
    @buithanhlam3726 Жыл бұрын

    This approach is much more simple than development in Anaconda environments. Thanks!

  • @americovaldazo6373
    @americovaldazo63732 жыл бұрын

    I love Python Simplified. All the tutorials are very well explained and the content is super interesting. Excellent work. Thank you very much.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much for the beautiful comment! 😀😀😀 I'm super happy you enjoy my tutorials! Will keep em' coming! 😊

  • @abdallahessam4671
    @abdallahessam46712 жыл бұрын

    I have a graduation project in which we will use Pytorch and NLP, so I want your support because your videos are so amazing and simple, we want more

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much Abdallah! 😁 NLP is absolutely fascinating! I think I have a repository on Github that might help you with your project: github.com/MariyaSha/StoryTeller It's a Pytorch Word Embeddings model that was trained on children books/fairytales and it predicts an entire story once you select the first 5 words 😉 It's a fairly stupid network, I must admit hahahaha, but if you're having issues with some of the processes - you might find a solution there. I also believe that's the network I cover in this tutorial: kzread.info/dash/bejne/n66WrMZpfsu9YqQ.html (It's one of the first videos I've filmed so my apologies in advance hahaha 😅) Best of luck with your graduation project! let me know how it turned out! 😀😀😀

  • @abdallahessam4671

    @abdallahessam4671

    2 жыл бұрын

    @@PythonSimplifiedThank you very much, I really watched the whole playlist and benefited a lot and we want more of your wonderful videos and it is my pleasure to tell you about my project

  • @MrMolekula
    @MrMolekula Жыл бұрын

    Thank you so much! Your video helped me to manage with TensorRT!

  • @CoopmanGreg
    @CoopmanGreg Жыл бұрын

    Every time, I hear you say "Holy Smoke", I have a little laugh (the only one I ever heard say that expression was my wonderful father and he was born in 1924). Your videos are great! ... very comprehensive and entertaining. Thanks so much!

  • @trtlphnx
    @trtlphnx2 жыл бұрын

    Another Fantastic Preesentaion ~

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much dear! 😁

  • @clashclan7549
    @clashclan75492 жыл бұрын

    Thanks for providing best contents.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much! 😊

  • @Nurlan_Turganov
    @Nurlan_Turganov2 жыл бұрын

    Thanks!! Clear!

  • @return_1101
    @return_11012 жыл бұрын

    Love you very much! Like your content! Its great!

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much for the lovely comment Andrei! 😁

  • @DJDozyDoh
    @DJDozyDoh Жыл бұрын

    Very nice content well explained and perfect for beginners

  • @wikicoin
    @wikicoin2 жыл бұрын

    I'm in Love 😂❤️

  • @olzh6285
    @olzh628510 ай бұрын

    Perfect

  • @Joaogtv
    @Joaogtv2 жыл бұрын

    Excelente conteúdo. I love you

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much Jose! 😁

  • @khetimachinery1336
    @khetimachinery13362 жыл бұрын

    Thanks for making this video. I was looking for this information. Excellent presentation.. keep posting new videos. please make one video on jetson with torch-tensor RT. Do we need to follow different commands or simply this tutorial will work with jetson.

  • @user-jc9kf2ml5l
    @user-jc9kf2ml5l Жыл бұрын

    You have my subscribe!!

  • @hamoudimaghoul3407
    @hamoudimaghoul34072 жыл бұрын

    👍😉

  • @freezoulou
    @freezoulou2 жыл бұрын

    Nice Fantastic import video ..thanks again :) you have the best channel for learning

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much! I've put some extra work on the installation portion in case some folks run into Docker issues 😉 How did you find Docker compares to Anaconda? I'm so used to Anaconda that I rarely try new stuff hahahaha 😊

  • @freezoulou

    @freezoulou

    2 жыл бұрын

    @@PythonSimplified me too I much prefer anaconda ...sometime you have to try new things ....it could be better :)

  • @ardavaneghtedari
    @ardavaneghtedari2 ай бұрын

    Thank you so much for these videos. Is it possible for you to make another video on how to run the opener Whisper model in TensorRT for near real-time transcription of voice-to-text?

  • @udbhav3760
    @udbhav37602 жыл бұрын

    I don't use computer and laptop because I don't have xDDD But no longer it will happen I will purchase one! :) But yeah I'm liking it now coding from phone! Great video 💪🙌🤠

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much Udbhav! 😃 I did mention your supreme phone coding skills during the livechat! 😉 Good luck with your PC/laptop search, let me know if you need an advice with hardware 😊

  • @diwakar_tsn
    @diwakar_tsn2 жыл бұрын

    Lol! I missed yesterday 😅 🔥❤️❤️

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Hahaha no worries, attendence is only recommended 😉

  • @fadl999
    @fadl999 Жыл бұрын

    You nailed "Habibi"😂

  • @LegitKev.
    @LegitKev.2 жыл бұрын

    You so good at this. How long have you been coding for ?

  • @verdirossi21
    @verdirossi212 жыл бұрын

    Não entendo nada do que fala mas adoro ver você falando.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    hahaha obrigado! Eu gostaria de poder falar português... por enquanto tenho que confiar no Google Translate 😅

  • @verdirossi21

    @verdirossi21

    2 жыл бұрын

    @@PythonSimplified Parte do que disse foi brincadeira, apesar de não entender o que fala, eu sou programador (Python inclusive) e aproveito bem seus exemplos. Obrigado.

  • @collinslife7021
    @collinslife70212 жыл бұрын

    Very Helpful

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much Collins! Super happy to help! 😃

  • @collinslife7021

    @collinslife7021

    2 жыл бұрын

    @@PythonSimplified absolutely your lessons have saved me a lot of time and I have learned so much, very grateful

  • @sharkyigor
    @sharkyigor2 жыл бұрын

    @pythonSimplified, Thanks for your great videos! would it be possible to make a video about openvino? what it is, and what benefits it brings?

  • @siddharthagrawal6990
    @siddharthagrawal69909 ай бұрын

    One important thing is that simply tracing also improves inference speed by 2x often. So I am unsure if the TensorRT is actually adding much additional speed for how much more of a nuisance it is to install.

  • @sushrutikarthik2540
    @sushrutikarthik25402 жыл бұрын

    Thank you for great video. Really appreciate this. What is process you follow for shooting/editing videos? Also please suggest the gear you use for shooting videos.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Thank you so much Sushruti! 😀 If you're just starting with making videos - I recommend to only get a nice microphone and some lights. You can start with your phone camera and connect it to your computer with Android's IpWebcam (I actually used it in a tutorial a while back: kzread.info/dash/bejne/h5h3tcuTgpveibw.html ) Then I use OBS Studio to combine the screen capture with the IpWebcam stream. You don't really need a green screen at first - but it's definitely nice to have one somewhere along the way 😉 To use a green screen, just add a "chroma key" filter in OBS and set it to green. And of course as you move along you can invest in a fancy camera and go from there - but the key note is to start with the tools you have and keep expanding 😜 In terms of video editing, I use the Adobe Creative Suite (a combination of Premiere, Illustrator and Photoshop). And you can find open source alternatives as well (Kdenlive instead of Premiere, Inkscape instead of Illustrator and Gimp instead of Photoshop) which may be a better starting point. My first few videos were edited with Windows Movie Maker (which was unfortunately discontinued when Windows10 has arrived) Best of luck on your new video making journey, let me know how it turned out! 😁

  • @carljames9654
    @carljames9654 Жыл бұрын

    Hi there, gr8 video! can i ask what gpu server you are using please? because i have a project that grabs a frame from a video, then sends the image as base64 to a ubuntu server with tensorflow serving server using flask. The problem i have is that it takes 30 seconds to inference the image and send the data back. The server is just a CPU server, but looking at renting a GPU server seems to be way to expensive. Do you have any thoughts how I can speed this up?

  • @arturovalle5990
    @arturovalle5990 Жыл бұрын

    what a big monitor you have

  • @ardavaneghtedari
    @ardavaneghtedari2 ай бұрын

    Thank you for this excellent video. Do you have time to make another video on how to convert WhisperAI models to TensorRT for faster (i.e., near real-time) conversion of voice to text?

  • @yeswanthgowd6102
    @yeswanthgowd61022 жыл бұрын

    Can you share a video on inference of large transformer model from huggingface on Multi-GPU with TensorRT

  • @imai_official
    @imai_official2 жыл бұрын

    This first video about trt tutorial. Could you try convert yolov5 models, will be good

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    I haven't had a chance to work with yolov5 before, but if I see a few more requests on it - will definitely film a tutorial on the subject! 😉 Thank you so much for your suggestion! 😀

  • @Agronomistapolo
    @Agronomistapolo Жыл бұрын

    Could you do a tutorial using Jetson Nano?

  • @ajithkj5253
    @ajithkj5253 Жыл бұрын

    Hi. I tried doing second approch of installing tensorrt. Jupyer notebook is not launching though i changed from hostname to localhost. Its still saying this site cant be reached. Please can you guide me what to do here????

  • @ChristianRodriguez-ou2qy
    @ChristianRodriguez-ou2qy2 жыл бұрын

    Mariya thanks for everything, you are great!! please if you can do the same but with Windows I would appreciate it very much!!! :p

  • @codewithyouml8994

    @codewithyouml8994

    Жыл бұрын

    Better to get wsl on windows and do the same

  • @artyomkolas6850
    @artyomkolas68502 жыл бұрын

    3% - carton because your image has brown color subject like box and a lot of straight lines. it's 3%🤗

  • @dantec.dagandanan3732
    @dantec.dagandanan3732 Жыл бұрын

    very nice, but maria what version is your ubuntu?

  • @Kennethlumor
    @Kennethlumor2 жыл бұрын

    Please and please I request you make video on how to use multiple databases in flask

  • @learnwithmunir9291
    @learnwithmunir92912 жыл бұрын

    Make a tutorial on hand written Character recognition

  • @MustaqimAdha
    @MustaqimAdha3 ай бұрын

    Is it applicable for NVIDIA Jetson Nano? What machine and GPU are you using?

  • @seyeeet8063
    @seyeeet80632 жыл бұрын

    Do you think you can make a crash video on how to from scratch install and use docker on a deep learning simple project and then how to provide it for the user to use it? I never got it how to set and use and deliver docker....

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Sounds like a great idea! 😀 If I see a bit more requests on it - will definitely film a Docker dedicated video (similar to the Anaconda one 😉) Thank you so much for suggesting! 😊

  • @user-jc9kf2ml5l
    @user-jc9kf2ml5l Жыл бұрын

    Can you please make a tutorial on DOCKER? Thankyou

  • @renanmonteirobarbosa8129
    @renanmonteirobarbosa81292 жыл бұрын

    we need Transformers

  • @WebWise_Wallet
    @WebWise_Wallet2 жыл бұрын

    when will u work on django

  • @kesavguru415
    @kesavguru4152 жыл бұрын

    Can you make video on how to create Windows Application with Python?

  • @fosskingdom

    @fosskingdom

    2 жыл бұрын

    It is already out.

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    I already have a bunch of videos for that, Kesav 😊 1. A Simple Greeting App with Kivy: kzread.info/dash/bejne/i3ikmZWQmrGvg8Y.html 2. A trivia game with PyQT5: kzread.info/dash/bejne/a52Orqanh8Lcp7g.html 3. A simple SMS Spam filter with DearPyGUI: kzread.info/dash/bejne/ZIajxbqEgLPeZNI.html 4. A PDF Reader with Tkinter: kzread.info/dash/bejne/m6iGrrSfpa_FgLQ.html All these are Python Desktop Application tutorials, so you have lots of things to investigate now! 😉 Good luck and have fun! :)

  • @mohammadjavadebrahimi5895
    @mohammadjavadebrahimi58958 ай бұрын

    yeah Im just starting machine learning if your beauty let me to concentrate

  • @hrishikeshkrishnakumar1443
    @hrishikeshkrishnakumar14432 жыл бұрын

    Can you make a Rust tutorial? 🙏🏽

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    I haven't had a chance to work with Rust yet, so I'm probably not the best person for the task... 😅 But I've just stumbled upon a Rust tutorial recommendation on Discord, you might find it helpful: kzread.info/dash/bejne/rHpnlsaLmdmyf9o.html

  • @rustybolts8953
    @rustybolts8953 Жыл бұрын

    Why zero predictions for Mafia cat?

  • @mohamedlaminekartobi5732
    @mohamedlaminekartobi57322 жыл бұрын

    Hey ! Thank you for video ! I really appreciate your tutorials. I'm just thinking that maybe the title of the video doesn't sound totally correct in fact that TensorRT is based on CUDA and it runs only with NVIDIA GPU, so for comparaison you can say Torch CPU vs TensorRT GPU Both of PyTorch and TensorRT are deep learning framework, CUDA is a parallel programming language for NVIDIA GPU's and ofc CPU and GPU represents hardware for training or inference

  • @karann6010
    @karann6010 Жыл бұрын

    Facing cuda out of memory issue when I call the model which is loaded in GPU memory. The GPU doesn't free memory

  • @PythonSimplified

    @PythonSimplified

    Жыл бұрын

    Try restarting the kernel and then give it another go. Otherwise - are you trying to load the entire dataset at once? or are you loading it in batches? if batches - decrease the batch size. If not - load the data batch by batch instead! 🙂

  • @karann6010

    @karann6010

    Жыл бұрын

    @@PythonSimplified thanks for ur help. The issue was there was no context manager torch.no_grad() used when doing foward passes to the model. With this change it released memory.

  • @Yachid
    @Yachid2 жыл бұрын

    Mariya~, (?) RU caught up near thoze Trucker 'eventz'; &, (?) what R y'r thought'z on that... ShoutOut from @ Michigan

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    You know I can't talk about it on YT... 🙃

  • @Yachid

    @Yachid

    2 жыл бұрын

    @@PythonSimplified Ya, I get that... who'da thought 'in the West' we'd loose 'FREE Speech' & have to bight our tounge'z!

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Good always wins evil, even in the darkest of times! Stay positive, determined and don't let them drag you into hatred. We're all on the same side - it's just that not all of us realize it yet 😉

  • @Yachid

    @Yachid

    2 жыл бұрын

    @@PythonSimplified Deep,

  • @Yachid

    @Yachid

    2 жыл бұрын

    @@PythonSimplified unReleated, could U do a video show'ng how to covert between number bases, w/Lg numbers, & w/'fractional-number-parts'; in my prj I waz us'ng Excel { base() }, but it iz limit'd to integers & to a certain range of values { I think (2^53)-1 }, I was just think'ng perhaps Py may be a better choice; specifically, I want to go from base 10 to base 24. & of course, y'r ML video waz great, az alwayz

  • @nowgamertv4148
    @nowgamertv41482 жыл бұрын

    А что там в Канаде случилось ?_?

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Ludi hotiat svobodu - provitelstvo ho4et rabstvo 😢 Neskolko dnei nazat oni zaiavili voenoie polozeniie iza tovo 4to dalnoboishiki bostuiut v Ottawe... oni sei4as v turme. takovo ne kogda v kanadskoi istorii nebilo, voenoie polozenie eto tolko dla voini protiv drugih stran - ne protev nashih ludei... tak 4to vse v shoke! Interesnie vremina... eto to4no! 😅

  • @Mouazbrai
    @Mouazbrai2 жыл бұрын

    I'm late back, looks like you've been busy with the Ukraine crisis, I hope you're fine

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    There's actually an exponentially bigger crisis in Canada... 😅 (and I wouldn't trust the news for reliable information in any shape or form, ask people who you personally know and live there - you'll get a very different angle, trust me)

  • @Mouazbrai

    @Mouazbrai

    2 жыл бұрын

    @@PythonSimplified We always wish the two countries well, however welcome back

  • @nowgamertv4148
    @nowgamertv41482 жыл бұрын

    Привет у тебя Вк есть ?_? Ну в контакте ?_?💖

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    Nea... u menja bil kogdato dovno, no uze ne pomnu parol i "username" 😊

  • @faresbadr2624
    @faresbadr26242 жыл бұрын

    hi my sister. please I want to learn how to scrape full data from Facebook like email, phone number and convert code to user interface please............. please......... please............ thank you

  • @KatySei
    @KatySei20 күн бұрын

    Love this channel. I have a little question, why in kzread.info/dash/bejne/m3p1ptWLdK3OdLA.html the dim=0 and not dim=1?

  • @unknownfacts3716
    @unknownfacts37162 жыл бұрын

    Mam love your content, but I have one question. You are already so cute and beautiful then why do you put that extra makeup

  • @PythonSimplified

    @PythonSimplified

    2 жыл бұрын

    why not...? does it really matter?😜 You would very rarely see people who go in front of the camera without makeup... even guys! (especially when filming in 4k quality... every little flaw is captured by the lenses)

  • @unknownfacts3716

    @unknownfacts3716

    2 жыл бұрын

    @@PythonSimplified yeah that's true even we boys also put a lot of makeup but I must say that I have never seen a teacher as beautiful as you are, God bless you and may you accomplish all your goals 😇♥️

  • @angvuhai964
    @angvuhai9643 ай бұрын

    how the hell she could remember all the command :v

  • @mmhshazin6155
    @mmhshazin61552 жыл бұрын

    What is your name dear??

  • @adilkhalid6625
    @adilkhalid6625 Жыл бұрын

    The attendance program through facial recognition was working without any problem, but suddenly it started giving the following error how can i fix ? module' win32 com has no attribute 'CLSIDToClassMap AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'CLSIDToPackageMap

Келесі