How to Speed Up API Requests With Async Python

In this video, I will show you how to take a slow running script with many API calls and convert it to an async version that will run much faster. I use AIOHTTP for the request library.
Need one-on-one help with your project? I can help through my coaching program. Learn more here: prettyprinted.com/coaching
Get the code here: prettyprinted.com/l/vZJ
Twitter: / pretty_printed
Github: github.com/prettyprinted

Пікірлер: 57

  • @LeThorquato
    @LeThorquato3 жыл бұрын

    Now that's the kind of programming content I was looking for, real examples not examples with threaded print jobs

  • @Yalchin403
    @Yalchin4033 жыл бұрын

    I've seen a lot of asyn videos and this one is one of the best ones, example sync code really shows you both theoretically and experimentally how important async is. Thanx a lot

  • @djanthony6662
    @djanthony66623 жыл бұрын

    Happy New Year, dude. Thank you for your efforts sharing your knowledge all this year. Stay safe!

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Happy new year!

  • @amirnoorani5017
    @amirnoorani50172 жыл бұрын

    I really liked your way of explaining things. you are a good teacher

  • @ahmedw5
    @ahmedw52 жыл бұрын

    Woah, this is just what i needed! Perfect, thanks man!

  • @eduardogil1775
    @eduardogil17753 жыл бұрын

    Excellent tutorial, Thanks for sharing! 😎

  • @Andrew-ht7rf
    @Andrew-ht7rf3 жыл бұрын

    I didn't think it was possible to be aroused by code but this is definitely doing something

  • @samsallee5924
    @samsallee59243 жыл бұрын

    Well I've been looking for something like this for a long time .... And this is perfect. It might be black magic, but still perfect

  • @conconmc
    @conconmc3 жыл бұрын

    Such a good video brother thanks!

  • @mohdsalmanansari5992
    @mohdsalmanansari59923 жыл бұрын

    Awesome!! Happy new Year 💕

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Happy new year!!

  • @AneeshMelkot
    @AneeshMelkot3 жыл бұрын

    Awesome! I was trying to implement something similar and was breaking my head for the last 2 days. I had >1000 requests to perform. Thanks!

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Glad it helped!

  • @Ruf4eg
    @Ruf4eg3 жыл бұрын

    WOW, this is definitely something extremely useful!!!! thank you very much sir!

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Glad it was helpful!

  • @jpcalvo6699
    @jpcalvo66993 жыл бұрын

    Man, this is sick. just like javascript. Thanks man

  • @ViniciusRaphael
    @ViniciusRaphael Жыл бұрын

    Extremely useful content.

  • @jakubprogramming29
    @jakubprogramming292 жыл бұрын

    Thank you for this video!

  • @alexcano2642
    @alexcano26423 жыл бұрын

    Hey Anthony, Have you considered showing off the httpx library for this use case? It (mostly) has a requests based API, and I personally found the transition from requests -> httpx with async code much easier! Also, don't forget about having requests / httpx build the querystring for you instead of using f strings ;) Thanks and loved the video!

  • @konstantin9888
    @konstantin98883 жыл бұрын

    Thank for video and Happy New Year ! could you tell me is it possible to use async with proxy

  • @user-stevegjffuhdry
    @user-stevegjffuhdry2 жыл бұрын

    You're a legend

  • @p3wx4
    @p3wx43 жыл бұрын

    I watched this video in 2x and now my APIs run 2x as on top of the speed enhancements I got from using what I learnt in this video.

  • @karthikkarthik100
    @karthikkarthik1002 жыл бұрын

    Super, Subscribed :)

  • @samuels.4678
    @samuels.46783 жыл бұрын

    thank you so much !

  • @dustiwe
    @dustiwe3 жыл бұрын

    thank you!

  • @XDTuber
    @XDTuber3 жыл бұрын

    Where would you add request throttling code (different from time.sleep I am assuming)

  • @andredempsey2687
    @andredempsey26873 жыл бұрын

    Hi Anthony. Happy New Year! Thanks for this video. One question...where are you getting the value for ‘key’ variable on line 68 of your refactored code?

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    I imported it and the variable exist in the global scope for the file.

  • @andredempsey2687

    @andredempsey2687

    3 жыл бұрын

    @@prettyprinted Thanks. I should have noticed that.

  • @hammershark612
    @hammershark6123 жыл бұрын

    Hi, If the video_id endpoint had multiple pages insted of just one as shown in the video ... How can we get all the pages of one endpoint using asyncio

  • @dinokrivic5486
    @dinokrivic54863 жыл бұрын

    What if I want it to have the same order as sync code.For example if i want to have all the videos analized from oldest to newest and then display it on a website ...Can i use async code then?

  • @CoderBB
    @CoderBB3 жыл бұрын

    Is it work with post request as well? Bcoz i need to dump json data in facebook graph api for each and every user...

  • @shoazizbakhtiyorov2895
    @shoazizbakhtiyorov28952 жыл бұрын

    await async.gather(*tasks) when this will be executed? After performing all tasks done?

  • @tejastundulwar622
    @tejastundulwar622 Жыл бұрын

    What if you have another function that takes those view counts and does some processing on that. And you want to return results after processing every view count as they get appended in the view counts list

  • @renzhengyu
    @renzhengyu Жыл бұрын

    What you want to do is that you want to do what you want to do.

  • @KirillBezzubkine
    @KirillBezzubkine2 жыл бұрын

    Sadly this wont work with many APIs since there s a limitation to frequest requests. How do i process a big list of items in batches (say 20 in one spin)?

  • @devijv1441
    @devijv1441 Жыл бұрын

    How about post json data for fastest performance?

  • @rangabharath4253
    @rangabharath42533 жыл бұрын

    Awesome

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Thanks for watching!

  • @ericxls93
    @ericxls932 жыл бұрын

    How about doing the same with the same session.? (Rather than request AND aiohttp)

  • @abeeya13
    @abeeya132 жыл бұрын

    Hi , How to handle asyncio.timeout error

  • @jijojoseph8721
    @jijojoseph87213 жыл бұрын

    How is it different from multi threading, and which is faster for this use case?

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    It's a lot easier to understand than multithreading, I think. And async is great for when you need to do a lot of things that require you to wait like sending an API request.

  • @Peterstavrou

    @Peterstavrou

    3 жыл бұрын

    @@prettyprinted what if you were using a request session to login to a large list of different URL's. Do you think async would be faster/better or multithreading? From what I understand, async would probably be faster because threads are limit to hardware (20 max for example) whilst async could send much more than 20 requests and await their response. For example, if 20 threads visited 20 slow sites they would all wait for each site to respond, however, with async it will load all of the URL's in the list while waiting for their response. is that correct?

  • @gsb22

    @gsb22

    2 жыл бұрын

    Finally someone said it. The code show above is basically synchronous execution vs parallel execution. The MEAT of asyncio was not shows. You could have achieved same, of even less than 3 seconds time with threading or multiprocessing libs python provides. I have seen tons, I mean it, I have seen tons of videos and read blogs, and no one really says where asycn would work faster sync code. If you were to process one task at a time, like you did in sync for loop, you aiohttp calls would be slight slower than request.get. The async model comes into picture where you want to increase server efficiency.

  • @MD-vm7tv
    @MD-vm7tv3 жыл бұрын

    Do you have full course about fastapi?

  • @prettyprinted

    @prettyprinted

    3 жыл бұрын

    Not yet. I hope to make one this year.

  • @shahnawazshaikh9371
    @shahnawazshaikh9371 Жыл бұрын

    How can I adjust access token in url ?? I wanna make 1k api calls that has authorization key

  • @muledeer1766
    @muledeer17662 жыл бұрын

    How can I send a cookie defined as a variable with this? I've tried everything and looked everywhere and cannot get it to send the cookies I want...

  • @savantofillusions
    @savantofillusions2 жыл бұрын

    More Cherrypy sockets!

  • @vigneshreddy9262
    @vigneshreddy92623 жыл бұрын

    Bro , I was collecting data from *API* and stored in an *array* Using *Flask* I send the *array* via render_template('index.html',data=*array*) It works fine in Local Host prints the all data in array After I deployed into HEROKU the array was empty please help me bro ❤

  • @edmondw6689
    @edmondw66893 жыл бұрын

    Please consider to post video to Lbry.

  • @RoamingAdhocrat
    @RoamingAdhocrat2 жыл бұрын

    maybe I _like_ watching request results scroll down the terminal window while I draft the next script

  • @TSalama7
    @TSalama73 жыл бұрын

    Thanks man! Question - how useful is async with typical http + db requests for different endpoints? I have a FastAPI GraphQL (Graphene) backend built with serverless aws lambda and SQLAlchemy built out and working well - but I'm not really making use of async as I'm not sure how to tell which requests would benefit from it. Thanks!

  • @gsb22

    @gsb22

    2 жыл бұрын

    dont use it. dont use asyncio if you dont have thousand of concurrent requests.

  • @IgoR.R.
    @IgoR.R.3 жыл бұрын

    I have to pay your attention, that you can be banned by a webserver for sending a lot of requests in short period of time. It can be treated as a DDoS attack.