Thank you for this. Learnt so much. The try exception in the function helped a lot as well.

I would love to hear you present a real world web crawler design, complete with IP proxies, horizontal scaling, rotating user-agents, anti-bot detection...yadda yadda yadda. I have no dount this is your bread and butter, but hearing about complexity considerations and tradeoffs would be *very* informative to us all. Just a thought. Thanks for everything John!

@user-sl9fg8rl4j

7 ай бұрын

Just needed a tutorial like this

Hi John, May be you can create a playlist just like a course by sequentially collating video list, would be great to have that as it is easier to flow and does provide a rhythm in learning the basics and advanced stuff pretty fast

@SkySesshomaru

9 ай бұрын

That would be neat

My friend! Thank you for covering this topic in a such understandable and straight to the point manner, it was a pleasure to watch your video

I approached web scraping like 2 weeks ago, and u are the one from which i learn the most... I'm so excited for this series thank you man

@JohnWatsonRooney

9 ай бұрын

Great to hear! thanks!

Thanks for this, looking forward to the rest of the series!

@JohnWatsonRooney

9 ай бұрын

I'm going to release part 2 tomorrow! its ready to go

@Ghazanfierce

9 ай бұрын

@@JohnWatsonRooney stoked.. 🤟

Thank you for sharing this comprehensive tutorial on web scraping with Python! This video is a great starting point for beginners like me who are interested in learning about web scraping techniques and tools. I appreciate how you broke down the process step-by-step, covering everything from setting up the environment to extracting data from websites. The explanations were clear, and the examples provided valuable insights into various Python libraries and their functionalities. The practical demonstrations helped me understand how to apply the concepts learned in real-world scenarios. I particularly liked the section on handling different types of data structures and navigating through HTML elements efficiently. Overall, this video has equipped me with the knowledge and confidence to explore web scraping further. Looking forward to diving deeper into this fascinating topic with your guidance. Keep up the excellent work!

Perfect! Exactly what I was waiting for. 😃👍🏻

@JohnWatsonRooney

9 ай бұрын

Great i hope you like the rest of the mini series too. next one is tomorrow!

Hell i always want to make a bot which kinda can dive it's way through the web using webscrapping and requests to find hidden spots in web :D tutorial looks awsome

Exactly what I was looking for.. I will start tomorrow thank you.

Hi John. your tutorial is much better than every other video i saw. from you i learn the most!!! looking forward to the rest of the series. thanks a lot.

@JohnWatsonRooney

7 ай бұрын

Awesome, thank you!

I come for the lessons. I stay for the typing skills (and the lessons). Touch type coding using Vim. RESPECT.

Thank you John, for such helpful material

You are the GOAT. Thank you for this video

That was very informative. Thank you so much.

You are an absolute legend. I hope you enjoy the time you have before exploding into one of the top dogs of this niche on the internet, because you're def headed there

@JohnWatsonRooney

8 ай бұрын

Thanks that’s very kind

Great video ty!

thank you! upload as many tutorials as you can 🙏

Enjoyed the video, looking forward for more tutorials

@JohnWatsonRooney

8 ай бұрын

Thanks for watching glad you enjoyed it, more coming (next one today)

Great Video. I just love the way you describe step by step. Keep uploading, please. And If possible please make a playlist.

@JohnWatsonRooney

9 ай бұрын

Yes more parts coming and a playlist will be created!

I also like this series so much that you used a real website that ALSO has stuff that won't just work right away! I was just following your steps in the video and I ran into errors and tried to understand why before I resumed the video and realized you also faced the problems too.

Thanks for this video, I wish this video was a bit longer, and go more deeply to extract links of each product and get data from product details page. looking forward to the rest of the series!

@JohnWatsonRooney

9 ай бұрын

this is going to come in part 3!

Keep posting

brilliant

+1 abo great content simple explanation top teacher .

Your setup looks so organized and efficient. Do you have any tips for configuring a similar development environment ?

@JohnWatsonRooney

4 ай бұрын

keep it simple and in time you'll find what you like and don't like!

Just knowing about the new html parser. Thank you.

@JohnWatsonRooney

9 ай бұрын

As soon as i found it i never looked back

big hugs from brasil.

ty for video 8)

thank you for creating a series, I learn a lot of cool and new things with your videos but they mostly do not have a chronological order so as a beginner I have troubles understanding them due to not having prerequisite knowledge. edit: may I ask in this industry is there a career path of position for people who are advanced with webscraping/webautomation? I'm mainly learning because I find it useful but I don't know if there are jobs that would require this skill set.

@JohnWatsonRooney

9 ай бұрын

thank you! yes there will be 4 videos I think, all leading on from each other in a mini playlist to help out!

Thanks 👍

@JohnWatsonRooney

9 ай бұрын

thanks for watching!

@chandrasekaran2429

9 ай бұрын

@@JohnWatsonRooney I'm learning to your channel

John, thank you for the very informative videos. The products you scraped in this video came from one specific category of the store's website. How would one scrape all products without going into each category separately? Thanks again

....❤....

Hi, John, your video content is very awesome for everyone who learn scrapping. But one thing I think everyone face that is blocked by some websites due to bulk of sending request. In this video you mention to avoid blocking while scrapping data. can you share how to get unblocked from these types of websites.? It's very helpful for everyone. Thanks

Hi John, thanks for this course. Absolute life save. Let me know the solution if the element I see cannot be found in the html what would be the solution to scrape that

I love you John Watson Rooney

Thaaaank you so much it is very helpful. One question please , how can deploy and host it as an Api?

@JohnWatsonRooney

9 ай бұрын

using a python web framework like fastapi we can turn this into a simple API easily enough sure!

Thank you, I hope to make tutorial how we can dockerize scrapy with postgres

@JohnWatsonRooney

9 ай бұрын

more scrapy content is in the works, I could look at using docker and postgres too

@talaldardgn2550

9 ай бұрын

@@JohnWatsonRooney thank you ..

@samoylov1973

9 ай бұрын

@@JohnWatsonRooney, please do. Waiting for continuation of this series and docker + PostgreSQL also. THANK YOU!

Have a question for anybody or John. If the response for get(url) is 403, I have read it is because the page has block the access for users to scrape his information and you need to use other libraries like Selenium. Any comment is highly appreaciate it.

Hello, what if the web-page has a login? I do have the credentials, but how to I make it log in in this scenario?

greetings, im following your tutorial and when i print the products (line 13) and run it , it just gives off an empty list [ ]. what am i doing wrong?

what do you do if you get code 302

Hello there, I have a question. I want to scrape a website, but it gives me 403 error, when I want to connect to it. Is there any way to bypass it? I tried changing the user agent, but it did not work

I know how to scrape sites and I do it sometimes writing a Python script, but I get scared I will get IP banned or blocked. It's frustrating.

Can you web scrape email addresses of realtors?

First thanks for the tutorial, I'm starting learning about scripe and found your channel. I'm trying to execute this tutorial but I always got a timeout. Can you help me please?

I was able to get everything working, except it would only give me one product no matter what I did! It wouldn't give me the full list of products on the page - just the first one. any suggestions?

if only all ecommerce website offered an endpoint from where to pull all the data we need, instead of relying to scrape their website

@JohnWatsonRooney

9 ай бұрын

shopify actually does that.. go to any store and add "/products.json?limit=250" at the end of the URL

is this a new series you are starting or just one vedio?

@JohnWatsonRooney

9 ай бұрын

series, so far 4 parts, next one is tomorrow and there will be a playlist in order

@abdifatahabdi3939

9 ай бұрын

@@JohnWatsonRooney i would like you to create videos about deep scrapy..otherwise thank you so much

why do u use venv and not conda?

@JohnWatsonRooney

8 ай бұрын

conda has loads of extra stuff i dont need, its aimed towards data analysts really

can this scrape JavaScript site without Selenium?

@JohnWatsonRooney

9 ай бұрын

afraid not, to render javascript you need a browser, which is currently out of scope for this series - but I may add to it to include a selenium/playwright version

I tried to install httpx for a couple of hours but it didn't go okay, at all :(

@JohnWatsonRooney

8 ай бұрын

You can absolutely use requests for this too if you prefer. Httpx is just my preference

damn now the website in the video is giving 403 http status code (access is forbidden)... even with headers