What data do you (want to) scrape from what websites? Let me know in the comments!

@pozytywniezakrecony151

11 ай бұрын

Market data seems to be hard to get as "website" or "refreshable picture", I mean like finviz heat maps or sectors. I am trying other sites one by one but I am mostly interested in whole charts not price points.

@crabsodyinblue

9 ай бұрын

i've tried scraping the Total pages printed off my HP printer webinterface as it is not an entity in the IPP integration, but for 1. it uses a PIN login, couldn't get the webform publish to work it, 2nd, the css selector off the site doesn't get me results on the inspector console or HA (lots off css error on the page reported by firefox inspector). I'm working on a video on how i managed to fetch the number thru SNMP...

@StereoSatellite

8 ай бұрын

Instagram Followers. And I'm not making it... How can I extract it from

@SmartHomeJunkie

8 ай бұрын

@@StereoSatellite Ah yes, I tried that as well. I even tried this with the API, but couldn't get it to work either. 😢

@nicolasrousset2897

8 ай бұрын

Follow the tesla cars prices in the inventory. I search a lot of time but it don't work...😔

Great video! Made a scraper for swedish namedays. A tip for getting the correct selector in the inspection view (which I had some trouble with): right click the selector you want to scrape, pick Copy and then Copy selector. Then you get the selector correctly. Like so: body > div.header > div.text-vertical-center > h1

@moorescommercial4624

Жыл бұрын

This is where I failed - thank you.

@ArjenEeken

8 ай бұрын

That is a perfect tip, thank you so much!

@Doeff8

5 ай бұрын

This is an ESSENTIAL tip.

Hi Ed This is a wonderful tutorial! Very clear and step-by-step with good examples. Thank you so much!

@SmartHomeJunkie

Жыл бұрын

Thank you!

Enorm bedankt voor je video. Het heeft me geholpen de scrape sensor eindelijk werkende te krijgen en heeft me meer inzicht in deze sensor gegeven. Een dikke pluim en duim omhoog voor je goede werk :)

@SmartHomeJunkie

Жыл бұрын

Thx!!!

Thank you. You're an excellent teacher. I've learned a lot from your videos. 👍

@SmartHomeJunkie

7 ай бұрын

Thank you for your compliments! 🙏🏻

That worked better than expected. Added local sea temperature as a sensor, perfect for swimming season!

Whoahh ! Great job !

Thanks for the video. It's east to understand what you explain.

@SmartHomeJunkie

Жыл бұрын

Thank you for the compliment!

Really interesting, thanks

Hi Ed, as usual, very well explained. As you said in the video, I'll play with that a bit to get the hang of it, but your video is a good guide. Thanks for sharing this.

thanks Ed, i need to focus more to learn this, very interesting function to automat

@SmartHomeJunkie

Жыл бұрын

Yeah, there's a lot of information in this video to digest. ;)

holyyyyyyyy cowww!!! I have now so much ideas hahaha thank you sir

Please do bring more multi dimension of HA with different use cases like these

Thanks!

@anthonydeangelis4498

Жыл бұрын

Thanks for all your awesome and helpful videos. Just getting into this and your videos have been so helpful

Super Cool

thanks!

Oke, briljant tut again, now i have to find the Vattenfall prices haha, Oh and the soccermatches (like from flashscore) :)

@SmartHomeJunkie

Жыл бұрын

Ah, that sounds like a perfect use case indeed!

Danke!

Great video. I never thought of doing that. It sounds very useful. As a different exercise, can you get share prices from somewhere too? I guess this couldn't be done by scraping but I assume there is some way to do it.

@SmartHomeJunkie

Жыл бұрын

Thank you. If there's a website somewhere that shows these prices, then it might be possible. As long as the website contains HTML, it should be possible. Sometimes it's not possible if a website contains a lot of javascript objects that are created on the fly.

Thanks a lot! How can I use the value.split to select a value between 2 different characters?

Great...I have few of my scrappers running in node red...which gives more flexibility to write manual codes in funtion node.. I mostly scrap from websites which requires logins... Could be great if you do 2nd part of this video.. And include websites which require login authentication

@SmartHomeJunkie

Жыл бұрын

You can use logins too. See the manual of multiscrape for this. It's just adding two fields to your scraper.

@james416

Жыл бұрын

Most modern sites do not use basic or digest. For token-based or oauth, you'd better to login manually with a browser and copy the token you got, and use that in your scrappers.

I had already come up with such a plan and was very happy when I saw your video. Only my energy supplier (Greenchoice) does not offer a rate sheet via the public internet, I must first log in to the My Greenchoice environment to view the prices of my contract. I've tried it myself with the multiscraper but can't figure it out. Could you also make a video for such situations?

@SmartHomeJunkie

Жыл бұрын

Hi Robert, Sometimes it's not possible if you have to log in first. It really depends on how the site is built.

Nice, I have modified the value template to make sure the values are integers and that you can calculate with it. value_template: '{{ value.split("€")[1] | replace(",",".")}}'

@EdwardJonkers

6 ай бұрын

This was needed for me to make it useful in the Energy Dashboard. Works like a charm!

I wanted a calendar for USPS Holidays and so I'm scraping a list from a website and using that to flag when it's a USPS Holiday or not. I couldn't find an iCal file that didn't include Valentine's Day which is ordinarily a delivery day so I had to scrape.

now that I have a temperature sensor, brought it outside and realized the humidity values for my area are way off. so I'm scrapping it from a local weather station I just found that actually matches my readings and my temp sensor can get back indoors.

Thanks for the video, got me going on scraping. But where is the split function shown around 9:00 documented? I'm trying to better understand it so I can use it in other use cases. Thanks!

@SmartHomeJunkie

Жыл бұрын

Thanks. This might help you: www.middlewareinventory.com/blog/ansible-split-examples/

Hi, thats really cool!, how would you use this method to get past a login screen, I assume there's a way to enter your username and password first and then go to the source page after the login page?

@SmartHomeJunkie

4 ай бұрын

Yes, there is. See the manual of the integration for this.

Great video. I'm pretty new to HA, so bare over with my question. How can I turn on the logger to see what is wrong. Thanks

@SmartHomeJunkie

Жыл бұрын

Look at Settings > System > Logs

@ihavn

Жыл бұрын

@@SmartHomeJunkie Thanks for your quick response. But how do I enable debugging logs from the multi scraper. Where to put the logger command's in the configuration files?

ok, so i'm very new to HA. I understand most of this and getting ready to try it. Could you show how you would use/display said information scraped from a site? (or point me to the video that you've already done)

Thank you so much. Very interesting! I have a question because I’m trying to read some energy data but I have some problem. The website request a password to enter and then insert the data. I’m able to enter with my credential but when I read the data the value is 01/01/2015, very old! This is very strange because when I use chrome to copy selector I see the correct data! How can I insert the actual data before scraping the data from website? Thank you

@SmartHomeJunkie

Жыл бұрын

Scraping does not work for every site. Sometimes a webpage is build up by Javascript and in those cases it may not work.

Here in South Africa we have frequent power outages due to the power company unable to meet the demand, those outages are published on a website to help people plan around the power outages. Knowing when my area will not have power can help me manage the remaining power left on my solar system.

@SmartHomeJunkie

Жыл бұрын

That sounds like a very nice use case. 👍🏻👍🏻

it's fantastic , thanks I have scraped the bus time table real time.. but how to access to USERNAME and PASSWORD website? I tried so many time but no result

@SmartHomeJunkie

Жыл бұрын

That should be possible with this integration too. I will investigate this too!

Is this possible to use in local network? Trying to get data from my solarpanel inverter

I love your video, I am trying to preproduce it. 1) Why do you only start with "Chart content" ? 2) Why can you use "col-md-6" and not "col-xs-12" as you do further below with "media-body" ? I am so confused.

@topgundk

Жыл бұрын

Same :) VERY new to scraping, and trying to understand where you start from in a structure and why there is # and . in front of some of the tags in example two... I keep on searching for clues :)

@stevethackwray450

Жыл бұрын

I am struggling with exactly the same sort of thing. Also if you use the copy selector for the field (which several sites suggest is the easy way to parse it) you get: "#chart-content > div:nth-child(1) > div:nth-child(1) > div > a > span > h4" - so confused!!!

What does the scraper put into the entities when the web page changes? Assuming it doesn’t just find something weird, but doesn’t find anything any more? Does it just keep the existing stale data or more usefully some kind of error message?

@SmartHomeJunkie

Жыл бұрын

It depends if the structure of the webpage is still the same. In that case other "false" data will be shown. If the structure changed, then it will probably show either unknown or unavailable.

i tought to myselfe, using a scraping tool i shoud be abel to get my energie price info. looked up a random tutorial. turns out its about energy scraping. Super bedankt, DOE ZO VERDER !

Is it possible for me to connect the code with google home, for example ask google what the 3 top box office movies are ?

Nice tutorial! To use the prices in your Energy Dashboard the following should be changed/add per identity: unit_of_measurement: "€/kWh" value_template: "{{ value.replace(',','.') }}" I used this ☝for my Powerpeers prices. To get the scraper ('select'), simple got to the site and right click with you mouse on the price. Select 'Inspect' (in Chrome). Now the devtool opens. right click and select 'Copy' > 'Copy selector' and put it to you select: of your entity in the multiscrape.yaml file.

@paulus1412

Жыл бұрын

any idea how to solve this? It keeps giving this message when i want to use the gas price for example inside the energy dashboard: Entity has non-numeric state The following entities have a state that cannot be parsed as a number edit: Maybe i got it working now with: value_template: "{{ value.split('€')[1]|replace(',','.') }}"

the element I want to scrape is this one 32 °C the value I want to scrape is the degrees, once the sun is gone the degrees wil be - I have watched the video several times but cannot find out how to do the unit of measurement and value template as there is no letter or text to do the split as in your examples

With the latest release, April 2023, this setup, which worked WONDERFULLY for me, no longer does, as Scrape is now integrated into HA. Can you show me how to use the latest version? The UI is different, and I'm looking to re-create what I already had working perfectly from the HACS version you illustrate here. Basically, we need an update now. :) Thank you as always for your excellent work!

@SmartHomeJunkie

Жыл бұрын

Probably the HACS integration need an update. Give it a week or so so that the developer can catch up with the update of HA.

@JasonPhipps

Жыл бұрын

@@SmartHomeJunkie It's been on-warning that the HACS would be removed since December 2022. It's been working nonetheless, but with this update, it's finally dead.

@SmartHomeJunkie

Жыл бұрын

@@JasonPhipps Are you sure? I use this one in the video and it's still maintained: github.com/danieldotnl/ha-multiscrape

@JasonPhipps

Жыл бұрын

@@SmartHomeJunkie Welp, now that you mention it with a link, I'm not sure any more. LOL. I'll do some digging... Anyway, there's now a Scrape Integration under Settings >> Devices & Services >> Add Integration >> Scrape. No HACS needed for that one. Might be a future video idea for you. I appreciate it again. I'll doublecheck your github link and compare it to what I had. I'm now doubting it was the same thing. 😕

@SmartHomeJunkie

Жыл бұрын

@@JasonPhipps I know, but the HACS scraper is a multiscraper which has a lot of advantages over the standard scrape. But both work.

rightclick (most browser provide this) and copy xpath and paste in the yaml to make it easier?

@SmartHomeJunkie

Жыл бұрын

Yes, that will also work.

Heel mooi en bedankt. Vraag: Waarom staat het € teken achter de prijs?

@SmartHomeJunkie

Жыл бұрын

That is a layout thing in Home Assistant. You can change that I believe.

@daytona1212

Жыл бұрын

@@SmartHomeJunkie Okay, I will take a look a that. 👍

What would be a good way to detect when a scraper is failing so a notification can be shown in HA?

@SmartHomeJunkie

Жыл бұрын

I "think" you will only see that the entity is unavailable. So, you can create an automation that triggers on the state "unavailable" for that entity and send you a message maybe?

Hi, is it also possible to add the scarped prices to the energy dashboard?

@SmartHomeJunkie

Жыл бұрын

Yes, you can add them to a helper and use that helper in your energy dashboard.

@Masterrrrwilliam

Жыл бұрын

@@SmartHomeJunkie Nice! now i can set the high rate and low rate in the energy dashboard :)

Liked the video a lot, but what to do when the value is not in the first table on the webpage, but in a 5th table?

@SmartHomeJunkie

Жыл бұрын

The trick is to see what the correct css tag is and use that tag.

@cwh6091

Жыл бұрын

@@SmartHomeJunkie Thanks for your fast reply!! Within the first table I can scrape every value using your example, but since my knowledge of css/html is none I don't have a clue as how to point to another table/use the correct css tag on a webpage.

Hello Smart Home Junkie, In the new release of HA 2022/12 a webscraper integration has been added. How is this new integration working compared to HACS multiscraper?

@SmartHomeJunkie

Жыл бұрын

There was already a scrape function in Home Assistant, but it couldn’t scrape multiple values in one call. The HACS version can. I didn’t look into this new version yet. Might be that it supports that now too.

@WilcoKakkenberg

Жыл бұрын

@@SmartHomeJunkie as far as I can see it is possible to create multiple sensors with the 2012/12 scraper integration. I spend some time but was not able te create a working scrape sensor

@SmartHomeJunkie

Жыл бұрын

@@WilcoKakkenberg Not all sites can be scraped. It really depends on how they are built.

@SmartHomeJunkie

Жыл бұрын

As far as I see, it's not a new integration, but just the UI interface is added.

@WilcoKakkenberg

Жыл бұрын

@@SmartHomeJunkie managed to get my values through the scrape UI integration. Thnx for your replies.

This tutorial has helped me a lot to set up a scrape sensor for scraping the daily Belgian gas price from my energy provider. Unfortunately the have changed the website last Thursday and now I can not get it working again. I am looking into the issue already for 3 days, and cannot find a solution. Although I always had a lot of advice and support from the Home Assistant Facebook pages, scraping is something hardly anyone provides help with. Is there any possibility I could ask your assistance with setting up the sensor again? Thanks in advance. Kind regards.

@SmartHomeJunkie

Жыл бұрын

You can find more info about my services at www.smarthomejunkie.net/services/

@RobinDeBecker

Жыл бұрын

@@SmartHomeJunkie Ik zal erover nadenken, maar kan je me a u.b. een schatting van het tijdsbestek geven? 75 € exclusief btw is namelijk een enorm bedrag voor iets dat maximaal 10 minuten duurt en al voorbereid is

@SmartHomeJunkie

Жыл бұрын

@@RobinDeBecker Je kan contact met mij opnemen via het contactformulier op de site.

@RobinDeBecker

Жыл бұрын

@@SmartHomeJunkie ik heb via het formulier contact opgenomen met een situatieschets. Voorlopig enkel informatief voor dat ik een finale beslissing maak. Alvast bedankt voor de snelle reactie

@SmartHomeJunkie

Жыл бұрын

@@RobinDeBecker Hmmm. Ik heb geen mail binnengekregen. Dat is niet best. Ik kijk het morgen na.

very interesting to 'capture' online consumption but how to for belgian people about Engie or Fluvius ?

@SmartHomeJunkie

Жыл бұрын

I'd say, check those sites and scrape them using the info in this tutorial.

Hi, I need to scrape 4 data energy. One for example is the energy of solar panel that have this selector: #pSolareMon Now I see 600W inside website but I read 0.0kW using scraping. Very strange! For this reason I try to scrape other variable not necessary, for example the date and I noticed that was different! Inside website is present a box to choose the date and probably is necessary set the date before scrape. Boooo!

@SmartHomeJunkie

Жыл бұрын

I'm afraid so yes.

I want to scrape the temperatures of many lakes near me so I know when it's perfect to swim where 😉

I’d love to be able to scrape tweets from a specific account on Twitter (like my daughters school) that have specific text in them (like “all schools closed” or “paved pickup”).

@SmartHomeJunkie

Жыл бұрын

That could be a challenge...

@RyanDuffy

Жыл бұрын

@@SmartHomeJunkie LOL, you’re telling me. I thought this would be a fairly easy thing with the Twitter integration. But that’s more for posting tweets than compiling/scraping them.

@SmartHomeJunkie

Жыл бұрын

@@RyanDuffy Yes indeed. The Twitter integration is just for posting things on Twitter.

your sample code is not working anymore, something changed in HA?

@SmartHomeJunkie

4 ай бұрын

It works fine here, but I understood that some sites cannot be scraped that easily.

I would like to scrape the solar radiation reading off an Ambient Weather Station down the road

cheap flights... prices for the next month(s) between 2 airports

Can not get this to work at all, it always unavaiable. nothing works tried so many ways. Like copying the entire css selector path and all. While scraping with home automation integrator works flawlessly .

@SmartHomeJunkie

Жыл бұрын

That's unfortunate. Not all sites allow scraping.

@mikej2489

Жыл бұрын

@@SmartHomeJunkie But the thing is that it does work, with the build in scraping. It just doesn't work with multiscrape.

@SmartHomeJunkie

Жыл бұрын

@@mikej2489 That's strange. It should work with both.

@mikej2489

Жыл бұрын

@@SmartHomeJunkie Yeah i am sure i am doing something wrong just don't know what haha. Even used the full path with css selector.

Too bad my gas company puts it in a pdf...