You do not need to explicitly create task on line 18

@dpm_07

4 ай бұрын

why so ?

@PixelThorn

2 ай бұрын

Gather already accepts coroutines, so any async function can be supplied directly

I can't believe how fast you got to the point. thank you for reading the room

Thanks for your video!! Tried to send multiple async request for some api but my previous code was unefficient and your example helped me!!

@JohnWatsonRooney

Жыл бұрын

Great stuff glad it was useful for you thanks!

Awesome man , I was working with these things Today , I'll try this one too ♥️♥️♥️🙏

John, your channel is absolutely fantastic! congratulations!

@JohnWatsonRooney

Ай бұрын

Thank you very kind

@fmanca100

Ай бұрын

@@JohnWatsonRooney I don't say this lightly, best educational content I have seen in years. Straight to the point, rich and very well explained! (ok, and I now I stop :-)

ThreadpoolExecutor is slower than Thread but saves memory. You don't need to use asyncio for waiting. You can use joins, barriers, wait groups and mutex locks with Thread to achieve the same. It comes down to preference, although asyncio is more streamlined, and thread is better suited for manual optimization that requires utmost speed and care.

@JohnWatsonRooney

Жыл бұрын

Thanks for the info. I still have a lot to learn about threads!

Thanks for the video, makes me appreciate go routines and their simplicity even more :)

@JohnWatsonRooney

Жыл бұрын

I couldn’t believe how easy it was when I started learning go

@eduardocasanova-personal3064

Жыл бұрын

@@JohnWatsonRooney preaching to the choir. Even with its rudimentary error handling

Thank you good sir, you are a master at this!! You have helped me land and keep my data scraping job!! Thank you so much, truly an inspiration :))

@JohnWatsonRooney

2 жыл бұрын

Thank you very kind!

Thanks, I'm gonna implement this in my "yfrake" package (PyPI).

While using executors, if the capacity of tomcat to process is set to 200 only. What happens after 200 request? Will we have to wait till 200 request are processed by the server? Or, it will pick up some request when call passes from server to db.

Very good vid!! I just finished making use of concurrent futures (based on your previous vid) and it speed up my code considerably! Looks like I have the potential to speed up further 😀. Will making a lot of requests at the same time slow down the source server, thus passing the waiting time to the server?

@JohnWatsonRooney

2 жыл бұрын

It should be fine, obviously you can overload it (ddos attacks work this way) but as we are just trying to maximise our efficiency we can stay within any rate limits or server limitations set. It’s much more useful for getting data from many places at the same time rather than just one

hey sir👋 as a self taught dev You are one of my inspiration 🙌🙌🙌 i started learning web scraping recently. i downloaded some videos about scrapy and BeautifulSoup tutorials of yours and i followed along them and found pretty comprehensive and clear i did learn better by your videos. i do hope more and more tutorial videos!!! thanks million times !🙏

@JohnWatsonRooney

2 жыл бұрын

Thank you that is very kind!

@RickeDz

7 ай бұрын

I am scraping a lot of things on web, this is just a pretty thing to do, I like this too much, it's so beautiful to see all the thing beeing "gathered" ahaha, are you still on this thing?

I was looking for a good explaination of the difference between async and multithreading...so threading is for doing things in parellel and async is for waiting for future tasks to complete but dosn't stall the current program?

Very well explained

Thank you John

hey thanks, good info. how to run 'subprocess' Asynchronously, while I dont want to use 'requests.content' to download subprocess.run(["yt-dlp", download_link, "-o", f"{output_dir}/{episode_name}.%(ext)s"], shell=True, stdout=PIPE) Also, What would you prefer Multi-Threading or Async in this case ?

hi john, or anyone who knows, I keep getting the error "UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28855-28856: invalid continuation byte" , do you have a work around for this? the ones that I found on google I can't seem to implement correctly.. much thanks!

Dear Ron, I am going to send unlimited fetch requests by javascript code from console to the server. In backend or frontend there is policy for each user to send request with 300 mili second time distance. If I want to send multiple requests in a second, I am blocked. Is there a solution for this issue from your point of view?

How can we apply same thing in PHP? Currently I am using CURL Multi but it boost up my server utilization. So is there any alternative way to do same in PHP?

Hi good sir thanks for the vid and most importantly, being actively engaging with people in the comment section Cheers

@JohnWatsonRooney

2 жыл бұрын

Thanks I’m glad you enjoyed the vid 👍

very instructive thank you John, I think that scrapy uses async requests, that's why some scraping jobs can be impressive quick with scrapy

@JohnWatsonRooney

2 жыл бұрын

Yes scrapy uses twisted, which is asynchronous

Thanks for your helpful videos. Please, have you any idea how to avoid dat*ad*ome protection ?

Why does the threaded version take so much longer than the async one, when it should be sending all the requests simultaneously as well? The synchronous one on the other hand waits for each response before it sends the next request. You could have a version that doesn't use any parallel programming facilities in Python but still works in parallel by having the async / parallel stuff handled by a Python package, or handled in an underlying C library. Eg. if you used the (much lower level) socket module and just repeatedly called send (), your requests would all go into the send buffer and the OS would send them while your Python program is waiting or doing something else.

u can speed up threading by using maxworkers = x with concurrent.futures.ThreadPoolExecutor(max_workers=10_000) as executor: executor.map(scraper_sub, links) max workers = 10_000 means 10k workers work at same time OR a single worker work 10k work at a time

Awesome vid bro!

@JohnWatsonRooney

2 жыл бұрын

Thanks!

Thank you for the video. What theme of VS Code are you using please let me know

@JohnWatsonRooney

2 жыл бұрын

Hey thanks, this is PyCharm but the theme is gruvbox which is available for vs code too

Nicely done, a Proxy or socks5 "which i believe works with requests" will do the trick bypassing the traffic limit. But how can i implement it in this scenario? Thanks John.

@erenc8377

Жыл бұрын

proxies=proxies :)

That's so great! ow to do this with dynamic payload in a Post request?

Can you please help on how the asyncio would work in AWS Lambda?

hello john can you make a video on scrapping world population data website please i tried but failed beacuse the span tag is constandly changing

is the normal request library a blocking library?

Hi John, I want to use Asynio and threading in combination. I want to use Asynio for making requests and threading for making calculations. What do you think?

@sebastiangudino9377

Жыл бұрын

You could make a thread that runs a main loop (Inside of which async stuff is happening) while other threads are doing your calculations, so yeah, sure, they can both work But why tho? If you are already making a bunch of threads, why not also use threads for your network request? You are adding complexity to your code base by using both techniques when it seems that your use case would be perfectly fine with just one of them. Making your code slightly easier to maintain So yeah, you can use both of them, but do think about what you are trying to achieve, and if this will actually achieve that

We use asyncio to scrap around 300+ web pages of a site. Because of res = await asyncio.gather(*tasks) the res gets heavier and script stops there. We changed the logic to process 50 pages at a time using asyncio Also, I see you have not used loop = asyncio.get_event_loop(), does this affect the performance?

On Yahoo finance, a company's income statement has table rows which need to be clicked on to expand and show the data. Do you know of a way to scrape such rows that doesn't involve using selenium to click on the button to expand the row?

@JohnWatsonRooney

2 жыл бұрын

I haven’t looked for a while but I believe there was a Python package for working with yahoo finiance, yfinance I think? That made it very easy to get all the data

Hey I really like your color scheme! Could you please share the theme and color shemes used? Thanks

@JohnWatsonRooney

Жыл бұрын

Sure this is gruvbox material and I’m using PyCharm community edition

your videos are as efficient as your code

can i do the same things with Flask?

In this example when we send 2500 async requests , does each request have the same session or each api call had its own session ?

@dpm_07

4 ай бұрын

no `aiohttp` maintains pool of connection !

Hey John!! Could you make a video about handling cookies with aiohttp? With the ClientSession I can send cookies but they are shared with all instances. I just dont get it how to pass a cookie which is only shared with one website and how to retrieve them. Cookie handling with requests is way much easier, but i really want to use aiohttp because it is SO FAST! :D

"how to build a ddos attacker..."

@dpm_07

4 ай бұрын

Right 😂😂

THIS IS FREAKING AWESOME, THANKS MAN, NOW I CAN DDOS FBI SERVERS WITH BILLIONS OF REQUESTS

Thanks!

@JohnWatsonRooney

11 ай бұрын

Thank you, that’s very kind

What about running async bots that in turn run threads? 🤔

Can you please make a video on how to run a webdriver with headers on google colab? or if you have any tips Thank you I love your vids they really help a lot

for me, thread pool executer is best to go. using it from last few years and always gives best results. easy to handle in python standalone scripts or even in python websites with celery.

What if you need to render the content? What would be the best approach in that case?

@JohnWatsonRooney

2 жыл бұрын

Requests-html I believe can do that but I’d have to check

@oparpax

2 жыл бұрын

@@JohnWatsonRooney well that would be a nice topic for a new vid wouldn't i? :D

@danielgarcia1428

Жыл бұрын

@@oparpax He already did a video on that topic! its called Slow Web Scraper? Try this with ASYNC and Requests-html

Hello I get this error Event loop is closed

This method puts a lot of pressure on the machine, and to my use case, I want to send more than 170 thousands requests, and when I use ThreadpoolExecutor, I can divide requests to lesser group using max workers' argument. Can that be done using Async & Await?

@JohnWatsonRooney

7 ай бұрын

It sounds like maybe Python isn’t going to be the right language, something like Go or rust might be better for you as they much better built in concurrency models

@ahmedelsayed3133

7 ай бұрын

Which is easier to learn?@@JohnWatsonRooney

Hi Ron! I’m a business engineering student from Belgium currently writing my own thesis. As a part of my research I would need to make my own dataset by scraping Zalando. Would you be open to assisting me with this 1-to-1, ofcourse when publishing my work I would properly reference you and your help. I have tried some things using your very helpful videos, but I’m still mostly very stuck. I look forward to your answer

@renancatan4788

Жыл бұрын

still trying to do? it's possible to scrape this website in a very easy way. I'll be glad to help =]

i wish this used a real networked example as i have a real networked API example i'm trying to speed up. i'm already using async and it's literally not any faster. so i don't know if aiohttp will be any better.

That's a very neat trick! However is it safe to send so many requests in such a short period of time? I'm learning web scraping and from what I've read your IP can be banned by the website if it detects that all those requests are scripted. That's why in my web scraper I use sleep function to wait from 0 to 2 seconds (randomly) between each request so it resembles more human behavior. But I guess it's not the best solution since it takes about 15 minutes to scrape roughly 1000 webpages.

@JohnWatsonRooney

2 жыл бұрын

Absolutely, if you are working on something that needs to spread out the requests the do so. This is about sending as many requests a possible- if you have 1000 different urls you wanted some data from then having to wait 1 by 1 would be painful, so we can use async to make them quicker. Also learning async technology is definitely worth it as you progress as a developer

@volin_d

2 жыл бұрын

@@JohnWatsonRooney For sure. While I might not use async in this particular case it's good to know about it for the future

Hey, wanted to point something out: I've often timed my own code too, and I find that print() consistently causes a large delay. So if you're analysing these delays, I would suggest not including a print statement there. Love your content though!! Helped me with a lot of stuff.

@JohnWatsonRooney

2 жыл бұрын

Great tip thank you!

Also, I'm curious whats your day job? Is it related to web scraping?

@JohnWatsonRooney

2 жыл бұрын

It’s not directly no but I use Python everyday for data extraction over our APIs

How to login in "jumia"

Every devops developers nightmare.

How to send million request?

async for loop would have been more readable…

More polite title: How to do 2500 handshakes in 2 seconds and disappear. Next video: How to make 2500 people shake hands with each other in 2 seconds, for fun! require 'popcorn' // btw

Here is a good tutorial on how to destroy someone's backend in seconds :)

work in parallel with python threads? nope, GIL.

First!!

@JohnWatsonRooney

2 жыл бұрын

Nice 👍

@randyjd3706

2 жыл бұрын

Really appreciate the videos! Cheers from Australia 🇦🇺

503 incoming

that's 2499 requests