I LOVE THIS! thank you John. I especially appreciate your teaching method: trial, explanation, tweak if error, run to completion. You've taught me process and rationale and I feel confident applying this to any other page on Yahoo Finance or elsewhere.

20 minutes to explain what many others take twice as long. Learned more from this video than any of the others. Explained very well, describing each step along the way. Well done!!

@JohnWatsonRooney

3 жыл бұрын

Thank you!

Had some trouble until we turned into a function where everything worked fine. My first go at scraping, I subbed, Thanks!

im trying to track prices of stocks I find and after watching 4 other videos, none of them worked except for this one, thank you 🙏

fantastic, straight to the point and learned a lot in this 20 mins video! thanks

The best channel ever for web scraping

Mate, you are the most underrated KZreadr on scraping! I learn something new every video. Thanks again.

@JohnWatsonRooney

3 жыл бұрын

Thank you!

I am getting my master's degree and Professor can't explain better than you. Thank you, my brother.

Job well done. Thank you for your effort and everything.

The video is really helpful man. You saved my day.

That's the third vid of yours i've watched this week. I'm getting great content. I subscribed.

@JohnWatsonRooney

2 жыл бұрын

Thanks, I’m glad your enjoying them

Simple, nice and clean👏

Perfect tutorial - you're a professional!!! Thanks

thank you so much! The first tutorial that actually helps srsly Thanks

Great tutorial. Thank you.

Holy shit, this helped me a lot! I'm relatively new to coding and I had absolutely no idea how to start so I just watched this. Although I'm trying to figure out how to have the user input a ticker symbol and use that and get the price and change of whatever stock I chose instead of changing the actual code. Thank you for making this video 😁

i wanted to scrape stock prices directly off the website(yahoo finance, market watch ect,ect) using your method. get 9:30, 9:45 . look at the data to find which stocks are going up. learned a thing or two about scraping data. this video was really useful. thanks

@JohnWatsonRooney

3 жыл бұрын

Thanks Robin I’m glad I could help!

Thanks@! Pretty cool how you used Python.

Very helpful video, thanks!

got a question. ive been trying to scrape the moving averages and vwap. but they are elusive. the vwap span doesnt appear in the span tags until you houver the mouse over the tag. ive tried soup.find the span tag itself. and even backed out a couple places and did the soup.find then the soup.find_all('span') to see if the vwap data would show up. kind of stuck at the moment.

legend, these videos help a lot!

you explain very well, you're amazing

@JohnWatsonRooney

3 жыл бұрын

Thank you!

Great vid. This approach looks like its polling each stock price ? Is there a way to save the data every time the stock price changes, rather than polling ?

Thank for the tutorial. What are some ways to remove the brackets and make it look cleaner?

Great video, i learned a lot. Thanks!!

@JohnWatsonRooney

3 жыл бұрын

Glad to hear it!

Great video!

Hi John. Big thanks for the effort you've put in on this. I learned a lot from this and am now a subscriber. Keep up the great work please! As I'm not an expert in all the technologies, I was surprised that you bothered to output via json. What is the benefit of this - it's not that it makes the output any easier to read...

@JohnWatsonRooney

3 жыл бұрын

Hi Bob - glad you enjoyed the video. I always usually export to CSV for the human readable aspect, but this time chose JSON because this is generally the format used with APIs, we could take this data and create or own api using it for example

Well done Sir.

This is great from korea😍

Does anyone know how to obtain real-time stock data for LSE-listed stocks? All the data sources I was able to find provides delayed data only.

Great explanation of the whole process, thanks man

Amazing

Hi, do you know why I have this error? ModuleNotFoundError: No module named 'requests'

Dude! this worked perfectly. However the site I'm trying to scrape has styles, and this method doesn't seem to work with it. Any advice on this?

Nice, however for me , it half of the time through 404 error , even though by the browser the page opens, any help ?

I think this is best channel on web scraping, what you guys say???

how do i automated the scripts like daily append in the csv file

I wish could like a video more than once

Can someone explain what the headers are for?

I am new to python. This is video is amazing to a beginner like me . I tried further to get the price change during premarket/aftermarket period but it was not successful to find the right tag. It will be appreciated if you or anyone can suggest the right div or span tag.

When I started webscraping, things went well until I ran into JS elements on sites. I’m aware that Selenium, requests-html etc can handle these things but how can I tell beforehand if I’m going to need to use a JS requests package or just requests and Soup? I know it’s View Source but what exactly am I looking for? Thanks!

@JohnWatsonRooney

3 жыл бұрын

There is a couple of ways- I tend to copy some specific text from the page and search the source for it, it you find it inside html tags in the source it’s not JS. Or you can disable JavaScript support on your browser temporarily and that will show you whats actually on the page

Hello sir, how do I fetch "day range" data , please let me know. Thanks in advance 👍

dude, If I'm going to heaven I'm carrying you on my back

Thank you for your tutorial, it helps a lot. May I ask if I want to save the result as an excel file, how to do it?

@anirudhramachandran2133

2 жыл бұрын

Using pandas

i keep getting this error "AttributeError: 'NoneType' object has no attribute 'find_all'"

Anyone know why the urls aren't working anymore?

Thanks. if we have multiple links on a csv/Excel, can we use python to run through the links on the CSV and scrape them?

@JohnWatsonRooney

3 жыл бұрын

Yes absolutely. Read in each line for the csv and scrape the data

@icedgodz428

3 жыл бұрын

@@JohnWatsonRooney Can you please make a video on this? Any links as proof would work if you choose to make a video. Thank you

Thanks, but if i want to scrape exactly time price. İ mean how can i get "BTC" price at 2021.03.06 13:00

Amazing video bro, loved it . Please share source code too.

great content ...amazing ....thanks a lot for this . Have a query , for stock price , change and change % is avaiable after price . I managed to locate the DIV and Span details , however both change and change % are within single text string . how can i break it to store change is one variable and change % in other ?

@JohnWatsonRooney

3 жыл бұрын

thanks! I'm glad you like it. I think you could split the string with Python - .split() will break text up into strings and put them into a list

Can I scrape indicator (e.g. RSI)?

Yahoo also has (downloadable) historical stock data. I would love a similar tutorial of creating a big database of stock prices

@JohnWatsonRooney

3 жыл бұрын

Great suggestion!

@cammac57

2 жыл бұрын

Precisely my use-case that I’m going to try build!

can you put the link to the actual code to this in your ghub, i cant find it in there any where

Great teaching method. But didn't work in my case. Couldn't extract the attribute from the website. Although I'm pretty sure I was dong something wrong but don't know what...

Hi John, I was trying to use requests-HTML for this but it seems like it has trouble with css selectors that contain parentheses. Have you encountered such a problem?

@JohnWatsonRooney

3 жыл бұрын

Hi - no I haven’t, do you have an example?

@im4485

3 жыл бұрын

@@JohnWatsonRooney I don't have the example code saved, but basically I believe that there nothing in requests-HTML documentation that shows how to select an element using its attributes. You can select an element using CSS selectors, but not with attributes. You can access the element's attributes after you already found the element itself though. In Beautiful soup you can do both: select the element using its attributes and using css selectors. I hope I am making sense here. Long story short - I was not able to select the price element in this video using request-HTML...

@im4485

3 жыл бұрын

@@JohnWatsonRooney Apparently, this is the solution to escape parentheses: from requests_html import HTMLSession url = 'finance.yahoo.com/quote/COMP' s = HTMLSession() r = s.get(url) name = r.html.find('h1.D\(ib\)', first=True).text print(name)

The span element now does not have any class, what is happening. Please help me

How would you do this in idle

why did you denote the the class attribute as a key using colon : and curly bracket { as opposed to writing it as class_= thanks

@JohnWatsonRooney

3 жыл бұрын

It works both ways for class, but I think doing it like this {‘class’} is neater and easier to read back

Hi, apologies for the beginner question! I have the most up to date version of Python and Pip on Mac and have successfully downloaded bs4 via the terminal but whenever I 'from bs4 import BeautifulSoup' it says it doesn't recognise the module bs4. I've searched all over google for an answer but I can't find one in plain english for a complete beginner? Does anyone have any advice?

@JohnWatsonRooney

2 жыл бұрын

Hey! It could be a few things. I suspect you have different versions of Python installed. Try running “python3” from your terminal and see if you can import bs4. If not try running “pip3” instead of just pip - install it if it suggest you need to

@mustseetop10s3

2 жыл бұрын

@@JohnWatsonRooney Thanks so much for your fast reply! That's solved my problem all up and running! a question for yourself or anyone who has time to help a newbie - I am at the print(soup.title.text) part and the output is Yahoo as opposed to your output - do you have any advice or suggestions? Thanks again!

I keep getting this ""pip" is not defined" message. Someone help me.

cool tutorial, can we scraping using Regular Expresion python?

@JohnWatsonRooney

3 жыл бұрын

Sure you can, if you are after something very specific it can work well

Hello, I started following your channel which is great to learn scraping. Actually, how do you handle cookies consent ? I can't use those script right now because of this. Just the r.status_code in your example give me a 404.

@juricadevic7337

2 жыл бұрын

I have same issue with following this tutorial. This solved my probkem. r = requests.get(url, headers = headers)

How come when u do print(soup.title.text) part I get an error?

@jamesscanlon607

2 жыл бұрын

I do this and it prints “Yahoo” instead

great Video. it was really helpful i scraped data from a site by writing following code find_all('div',class_='book-wrapper') This successfully scrap the data for first 7 page and after that the class name changed as 'book-wrapper-list' . How to handle such things

@JohnWatsonRooney

3 жыл бұрын

I would look at using try: and except: for those cases. If under try: it doesn’t find book wrapper, under except: run find book wrapper list. Or using if and else statements would also work

@ALANAMUL

3 жыл бұрын

@@JohnWatsonRooney thank u

can I get the yahoo link?

How to get intraday data for multiple tickers in python and send to SQL server make step by step video

genius

says "ModuleNotFoundError: No module named 'requests'

@JohnWatsonRooney

3 жыл бұрын

No problem, - pip install requests - in your terminal (or at the bottom in vs code)

@lonnieo4676

3 жыл бұрын

@@JohnWatsonRooney thx, had to install beautifulsoup4 too...

Hi I must be missing something what I cant see is the actual price just what its gone up by and the % change. But nice vid

@JohnWatsonRooney

2 жыл бұрын

that's odd i get the prices, i just checked it: "pricing_information": {"currentPrice": 130, "standard_price": 130, "standard_price_no_vat": 108}

Anyone know how to fix a 404 error?

I think Yahoo changed the site and the code is no longer working

@JohnWatsonRooney

3 жыл бұрын

Unfortunately that does happen - the best thing is to learn the common techniques so you can apply them to any site

Great video. But disappointing to see how unstable this web scraping technique is.

Is it possible to get some support regarding data scraping? ..may I know your email to send my query ..

things have changed. span has been replaced with fin-streamer . it will get you a price but not the current price.

@HighClasAsian

2 жыл бұрын

did you figure out how to fix it?

@rverm1000

2 жыл бұрын

@@HighClasAsian no

@rverm1000

2 жыл бұрын

yes i figured it out. going to make a video today.

@omersiddiqui5101

2 жыл бұрын

just write fin-streamer, where he typed span

@rverm1000

2 жыл бұрын

@@omersiddiqui5101 yes

please always try to provide source code it will useful for lazzyy coders XD anyways thanks for the content you provide us thanks a lot

This is good and elaborative, but why go through all this trouble when you can do same in a smartphone app?