Awesome videos John. I wish I had found you before I paid money to learn everything you're explaining here more succinctly and free 👏

Good to see it in Scrapy. Your channel need more Scrapy tutorials. 👍

@JohnWatsonRooney

3 жыл бұрын

More coming, I have a few planned!

i requested for your scrapy x api video and voila it's right here, thank you!

@JohnWatsonRooney

2 жыл бұрын

Ha yeah I hope you enjoy it!

Love your videos, really helped me get a project off the ground. Could you do a video on not overwhelming an API server with requests? What is the best way to slow the requests to the server down? I would like to hear your thoughts, process, etc. keep up the great work!

@JohnWatsonRooney

3 жыл бұрын

Great suggestion! I'll add it to my list of video ideas

@lfcatchall

3 жыл бұрын

@@JohnWatsonRooney thank you, and again really wonderful job you're doing with your videos. Be blessed.

@datascience7928

Жыл бұрын

There are several ways to slow down the request, the most commons are: 1) Sleep after each run of a for loop 2) Limiting the quantity of requests made, for example: each 10 requests, sleep more 30seconds Often I go with the first one that is the easiest to implement: from time import sleep for x in my_links.json()['data']: sleep(0.5) print(x['id']) each loop the code will sleep for 0.5 seconds, decreasing the flood of requests..

@mohamedbhasith90

8 ай бұрын

@@JohnWatsonRooney Hi, can make a video or share guide on how to do the same process as in this video with POST Method? cuz, the website i'm trying scrape has this data in POST request.. can you help me pls?

Tremendous help - thank you so much.

Could you make a tutorial where you deploy the scraper on a VPS? I've seen many options like using scrapyd or running a cron job. I'd be helpful to see examples.

As always, detailed & clear explanation. Threw me off when Pycharm was fired up though 😅

@JohnWatsonRooney

3 жыл бұрын

Haha snuck it in there!

Thanks allot 💞💞, amazing video 😍

I'm getting a 429 unkown error. What type of method should I use to slow down my scraper calls?

seems to me every website has its own custom API and blocks access to these type of request to even HTTP GET data

Hey john! Many guitars in the back. So any plans for a music youtuber soon?

congrats with 10K

@JohnWatsonRooney

3 жыл бұрын

Thank you!

Thanks for this video.

Do you have a video or can you create a video on how to schedule recurring scraping? Ie, say having the scrape run every hour?

@JohnWatsonRooney

3 жыл бұрын

I’ve covered it in my cronjobs video- and am going to cover it again soon

super cool very well explained thanks so much. subscribed :)

Hi John Can you please guide about instagram scraping Thanks

is this works for privat company network??coz i failed just like need to login..or something

Great Video. I am wondering if scrapy can get that long URL by itself instead of copy and paste by ourself?

Hello John, could you compare Python Requests and Python Scrapy? I just found out about scrapy but want to know the caveats between the two.

That's a nice video. Like always. Hi, John, Can you post a video to show how to scrapy home depot product reviews? Thank you.

You did with GET method. How to do this task if you have POST request/mthod?

Great video! The hidden api I'm trying to work around is a fetch type though. And it's response is really not as clean as this one. I don't know how to work with it really

@JohnWatsonRooney

Жыл бұрын

Try saving the response to a file and opening it up in a separate Python script and work out how to extract the data you need then reimplement it into your main project. Notebooks are good for this too

@zahrastb3869

Жыл бұрын

@@JohnWatsonRooney how about I just use selenium? This seems like a lot of work, the text is so jumbled

@zahrastb3869

Жыл бұрын

@@JohnWatsonRooney though I never worked with selenium before

Awesome 👍

Loved the video! Any plans for a web scraping course soon?

@JohnWatsonRooney

3 жыл бұрын

Yeah I am planning one, just not sure where or how to release it. Or just make it free on yt

@mandarraut9565

3 жыл бұрын

@@JohnWatsonRooney You can try to upload on Udemy. As i have checked there is not much content available on Web scraping. And thanks for making short tutorial on Yt like always

@JG-ms4rb

3 жыл бұрын

@@JohnWatsonRooney would be great to learn how to do price comparison from website to website and how to track that data / store it.

Awesome! But what if the website doesn't make any xhr requests? Is headless browser the only way (by clicking and pretending to be a user)?

@JohnWatsonRooney

2 жыл бұрын

That is a way yes, but more last resort - if we can render the page with the headless browser and grab the html to parse that way it’s a bit better

John I came up with your youtube channel and it is an amazing resource! Right now I am working in scrapper project but I have issues with ng class elements in the website I have done my research but without luck. Can you recommend something or a video in your channel? (I coding in Python) Regards!

Is there a good reason to use Scrapy for this instead of the requests library? Isn't it bringing a gun to a knife fight?

@JohnWatsonRooney

7 ай бұрын

I try to show lots of examples and I like the easy expanding use case of Scrapy but yes here it’s not needed as such

@JackyVSO

7 ай бұрын

@@JohnWatsonRooney Good to know, thanks! Your videos are very useful.

Thanks for video!!!!!!

I am trying to scrape foreclosure .com and response.body will print just fine but will throw errors when I try to load it as json.loads

Fine

Thank you man!

@JohnWatsonRooney

Жыл бұрын

thanks for watching!

Bravo

hey sir if the scrapy.Request(-) is not showing any thing what should i do

do you have example for POST request?

Can you do a video showing how to scrape wikidata info on people like famous programmers?

Hi John, I tried scraping a website using hidden API like this, I succeed in parsing the first page. When I tried to loop the next page, it returned 403 error. Now, when I tried going back to parsing one page only, it also returned 403 error I have tried changing the user agent in the settings.py, but still no luck I can open the API endpoint link just fine in browser. So I think it's not an IP ban Can you suggest something ?

@JohnWatsonRooney

2 жыл бұрын

I think you need to copy the cookie over from your browser and put it in the headers you are sending. Copy the request as curl and see what the headers are and try putting them into your code

@DittoRahmat

2 жыл бұрын

@@JohnWatsonRooney it turns out I was actually blocked by Perimeter X when I manually visit the website. So assume this is IP ban right ?

@JohnWatsonRooney

2 жыл бұрын

@@DittoRahmat Yes sounds like it, bot protection. Assuming you don't have a static IP from your ISP you can restart your router for a new IP, or just wait they don't usually last for that long

@brothermalcolm

2 жыл бұрын

@@JohnWatsonRooney how do you put the copied cookie (and other headers) into the scrapy spider code can you cover this please?

@brothermalcolm

2 жыл бұрын

@@JohnWatsonRooney because I'm having the same isssue where I'm able to get it to work following your earlier video using requests and insomnia but not in scrapy

God bless 😎👍🏻

I did this but unfortunately the API is salted 😢

Can you make a video about android app scraping?

I need help scraping data from a website that has a firewall. Will pay

hi, i follow your content, and your content awesome at all! but, can i get tutorial scrapy scraping graphql & follow the link after first request? thank you!😉

@JohnWatsonRooney

2 жыл бұрын

Hey! Thanks. I’ll be covering some of those topics coming up, I’ll see if I can drop that in too!

Hi John, My comment is unrelated to this video. I've been trying to scrap music data from streaming platforms like soundcloud for the data only not the actual music. to create a analytics platform for independent musicians. and these websites are loaded dynamically so its been giving me a problem. I tried everything from selenium to request_html but it is just not happening. Could you please do a video on it. So I can learn.

@brothermalcolm

2 жыл бұрын

what's the website and fields your trying to scrape?

@HoustonKhanyile

2 жыл бұрын

@@brothermalcolm spotify, amazon music, tidal & youtube music. the fields are name, song name, streams, date uploaded and so forth.

What is software where was GET editor?

This is really cool but also kinda illegal, I guess it depends on what data you are fetching

@JohnWatsonRooney

2 жыл бұрын

The legality is a bit grey but we are only getting data that is publicly available online, it’s not behind a login nor are we abusing the website with 1000s of requests. I think if you use the data for personal consumption ie don’t try to sell it it’s ok

@perticomanonalto

2 жыл бұрын

@@JohnWatsonRooney thank you for the response!

Hi John, I sent you an email. I'm having trouble with something I hope you can help me.

@JohnWatsonRooney

3 жыл бұрын

hey man, i will get to it as soon as i can

this are not hidden apis -.-