Thank you John for all this wonderful journey of web scraping various techniques that you illustrated in a professional and easy way, Keep forward and we wish you the best

Thank you John. I started my scraping journey with scrapy, playwright, scrapy-playwright, proxies with scraperapi.

Thanks so much John!I learned so much in your contents, keep going, you are amazing !

my new fav channel on python :) since yesterday i cant stop watching

The only KZreadr I have my notifications turned on for!! Thanks JR

@JohnWatsonRooney

2 жыл бұрын

Thank you!!

This video is so good! Thank you

Nice Video! Could you tell us more about how you work with the scraped data afterwards? Do you put them in a Data Lake? (If so, which one do you use? Azure, AWS, G-Cloud, something selfhostet, ...) Or do you save the csv files only in an system like Apache Hadoop? So in general how looks the system around the scraping part? I´m very interestered to hear more about that from you! Best wished!

John I am really grateful for your new tutorial on web scraping. Regards from Croatia.

@JohnWatsonRooney

2 жыл бұрын

Thank you!

Excellent video. Your cool and easy to understand delivery is a real pleasure.

@JohnWatsonRooney

2 жыл бұрын

Thank you Rob

Nice video. I wish I would have had that when I started webscraping - basicly everything you need to know is included. I would appreciate if you would make a video about your relation to web scraping. Is it your hobby or do you need it for work, e.g. as a data scientist? At the moment it's a hobby for me, but i'm a mechanical engineer and i'm trying to evolve a bit away from classic product design towards using big data technologies.

Super helpful thank you!

@JohnWatsonRooney

2 жыл бұрын

Glad it was helpful!

Thanks John. I jumped in (having scrapped in past, but picking the right tool for the job is a big one. Plus also knowing how the tool works helps too. I been using Selenium to scrap Yahoo Finance and getting frustrated. I have been learning Python and coming up with projects to get better at it. Thanks for your videos.

@JohnWatsonRooney

11 ай бұрын

Hi Ron - thanks for watching I appreciate it and glad you are finding my videos helpful!

@RonWaller

11 ай бұрын

@@JohnWatsonRooney Yes I am. I just started trying Playwright and Selectorlax. It worked well. Still a lot to learn. Thanks again

This is great.

Thanks for all these enlighten videos. Among our videos, I haven't seen Facebook or any social media site scrape. It that possible? if so, which would be the best approach to achieve fast results?

Thanks for this guide. You are a brilliant teacher. I came from whiskey webscraping. And I learned a lot from the comment section as well. This is a great community. I will move on to the Patreon soon. Nice job.

@JohnWatsonRooney

2 жыл бұрын

Thank you for watching, I’m glad you have got some good value and learning from my videos!

Thank you for this video, John. Any courses you know that teaches web scraping the professional way? I asked because I watched tutorials where I can't follow because I'm being blocked even with headers(only learned that after searching and reading a lot).

Hi John, I've subscribed to your channel!

@JohnWatsonRooney

10 ай бұрын

Thank you and welcome!

Hi John, of the three main methods for scrapping, which would you recommend with the following use case: I need to access a URL to download four data files. The site requires user name and credentials to login and access the files. The files are updated on a weekly basis. The files use the same download link; it's the data that changes. I guess you can say it's a dynamic link to a data file. I am thinking this could work with Requests and BeautifulSoup (and OS to land the files on my local directory), but wondering if you would have a recommendations. Thanks

Thanks for the video! I have a question tho...Can you give me a direction to look for in how to scrape amazon as fast as possible? what method should I use . THank you

Hey John, I’ve never scraped data before but really want to scrape RightMove for sold house prices in specific areas. Any suggestions on the best approach for this?

I can say it proudly and out loud, "I have an amazing teacher now" . Thank you John Just one question, you've been talking a lot about premium proxies usage like oxyylabs and iproyal. Im confused and dont actually know how to use a proxy service in a code. Would you please be kind enough to guide us through process also? Thanks Again

Hi John I just have a doubt in html5 we can have data-attribute in 'a' tag how to scrape that data elements when i try to use it is giving error, for example: data-index could u helpl me...!

Hey John, all interesting and I guess now a curated list of videos (maybe fill in the gaps) which covers your curriculum, step-by-step please?

@JohnWatsonRooney

2 жыл бұрын

Yes good idea I will sort a playlist

@sampatankar1977

2 жыл бұрын

@@JohnWatsonRooney Thank you!! Have a lovely Christmas!

Hi John, Are you on any social media. I'm a beginner web scraper.. just want to clarify doubts on my planned personal projects on webscraping. Would me much helpful to get the insights from you. Thanks for the inspiring videos 🤗

Great video. Where've you been when I was staring in June :D Btw, just out of pure interest, how much time you would say a person need(approximate, if they're persistent and spend enough time) to go from start(knowing only basics of python) to the point when they can start to work as web scrapers? For me it took 3.5month to get my first contract, but seems that it can be done much faster).

@JohnWatsonRooney

2 жыл бұрын

I’d say still at least a few months. There’s a lot to learn if you want to do paid work as you’ll need to be comfortable with things I didn’t talk about in this video, like running scripts from a server and saving stuff to databases. But like everything keep practising and learning

@DittoRahmat

2 жыл бұрын

1,5 months for me. But I do have basic programming skills from other language (C# & Delphi) so not exactly starting from 0. Usually when coming from zero programming background, learning about algorithms & data structures can be a bit challenging

@dhankkhush1088

2 жыл бұрын

​@@JohnWatsonRooney A video on freelancing requirements would be awesome! What's your highest quality/proudest project you've done for a client? and what skills/methods did you need to complete it?

Thanks for the informative video. How to crawl all urls associated with a website using scrapy.

@JohnWatsonRooney

2 жыл бұрын

Check out my channel for my video on scrapys crawl spider, that is what you want to use

the link is not working

Hey John love your channel, I’ve been having this issue when I try and paginate with BS4 on some sites, where the page number will change (I loop over the numbers and change the url or click the button) but the items on the page will not change.

you say start here, but you're communicating as if we all started way before here. You're talking as if I'm already somewhat knowledgeable on web scraping.

Mate, you need a full fledged udemy course or something. Nobody else comes close to your tutorials. 😁

@JohnWatsonRooney

2 жыл бұрын

Thanks I’m glad you enjoy them!

Beautiful soup is such a nonsense name