Scrape Sensor in Home Assistant - Extract Info From Websites
I’ll walk you through extracting information from websites, such as key dates, or facts, and importing them into Home Assistant using a scrape sensor and css selectors.
Code: github.com/willbob8/willsurri...
CSS Selector Tester: chrome.google.com/webstore/de...
CSS Selectors: www.w3schools.com/cssref/css_...
As an Amazon Associate I earn from qualifying purchases. By buying through any of the links below I will earn commission at no extra cost to you.
The smart home tech I have is here: kit.co/WillSurridgeTech/smart...
▼ My HA setup
Raspberry Pi 3B+: amzn.to/3gF588j
64GB San-Disk: amzn.to/3iDUErN
▼ Amazon Products
Amazon Music Unlimited: www.amazon.co.uk/music/unlimi...
Amazon Prime - 30 day free trial: www.amazon.co.uk/tryprimefree?...
Audible Free trial: www.amazon.co.uk/Audible-Memb...
Kindle Unlimited free trial: www.amazon.co.uk/kindle-dbs/h...
Amazon Business: www.amazon.co.uk/tryab?tag=ws...
▼ Camera Gear
Canon M50: amzn.to/3iGsGeU
Manfrotto Element Tripod: amzn.to/3e7FYxF
Boya Mic: amzn.to/3fez8Ic
64GB San-Disk: amzn.to/2Z78vPC
▼ DJI Products
Buy a Mavic Air: click.dji.com/AO_0cxV8Qmj5tig...
Buy a Mavic Mini: click.dji.com/AMWoUtCGIPFYetJ...
Buy an Osmo Mobile: click.dji.com/AM9khwrxMoEKDKV...
▼ Social Media
Remember to subscribe: / @willsurridgetech
Instagram: / willsurridgetech
Facebook: www. WillSurridgeTech
Пікірлер: 43
Will, love what your doing and your videos always entertain me, you are really animated and upbeat whilst explaining things clearly. Your presentations always make me smile as you come across like a really excited puppy, full of excitement, wanting to share info and help us all along. Keep it up I love it, it’s so refreshing!
@WillSurridgeTech
4 жыл бұрын
haha, thank you very much!
Hi Will, now I found out a good few usages for HA. Thanks a lot!
@WillSurridgeTech
4 жыл бұрын
Happy to help!
Hi will, clear and well explained tutorial as always! Thank you for the great resources you are uploading about home assistant!!
@WillSurridgeTech
4 жыл бұрын
My pleasure! Glad it helps :)
Wow! Your channel is amazing. Thanks man! Definetly a follow worth
@WillSurridgeTech
3 жыл бұрын
Awesome, thank you!
I was just looking that!! You are the best!!!
@WillSurridgeTech
4 жыл бұрын
Glad I could help!
How do you determine the polling interval in HA?
Great content and very well explained!!! thanks again Will for an awesome job to HA community... I appreciate that!!!
@WillSurridgeTech
4 жыл бұрын
Thank you, glad you like it!
Great video. I’m a little stuck. If I was looking for the average review or number of reviews of a product on Amazon, what CSS would you be using? I’m trying but getting nowhere. I’m wondering if it’s because Amazon is blocking (even with the Mozilla 5.0 header) of a, getting the CSS selection wrong....
thnaks a lot
css selector tester seems to be gone now. any other suggestions?
For some reason my sensors for KZread via Socialblade still shows unknown after adding the User-Agent The log reads "2021-04-25 10:34:26 ERROR (MainThread) [homeassistant.components.scrape.sensor] Unable to extract data from HTML for Forschi KZread Views"
Will, yet another great video. I really appreciate you covering different subjects to other KZreadrs. Keep up the good work.
@WillSurridgeTech
4 жыл бұрын
Thank you! More to come!
thanks for this good vid. what is frequency sensor is fetching values?
@hamhumtube
3 жыл бұрын
scan_interval: 3600 www.home-assistant.io/integrations/scrape/#bom-weather
@WillSurridgeTech
3 жыл бұрын
Yeh the scan interval is how often (in seconds) it will scrape the website for an update
I wish it had two extra functionalities: - scrape multiple data points from a single pull - set how often the data is pulled. (e.g.: zero reason to pull info every 30s that's only refreshed every week) Also, you can just r-click in the inspect "elements" on whatever you want and go copy -> copy selector
@Videxthebest
Жыл бұрын
you can install multiscrub from hacs. Then you can selec freqency, and yes multiple data points.
CSS selector tester is no longer available: pluginsaddonsextensions.com/google-chrome/css-selector-tester-chrome-extension_23245
I'm trying to scrape the weather alerts from the NWS alert page. Alerts show up as an #id with an h3. If there are multiple alerts there are multiple h3. Is there a way to scrape multiple ids? resource: url select: "#content h3"
I have my data in cloud and by doing http get request I can get that sensor values is it possible to create a sensor with http get request if yes please share the tutorial this could help me lot waiting for your video or any response
Annoyingly my council's bin collection site uses the same ID for two types of collections on the same page and no way to differentiate :( ugh!
Thank you
@WillSurridgeTech
2 жыл бұрын
No worries
I tried to use this to make a credit karma credit score sensor, but couldn't figure it out. thanks for the tutorial. I don't understand selectors or maybe it is not possible with that website.
@WillSurridgeTech
3 жыл бұрын
Its probably not possible I'm afraid - scrape sensors are very basic
How to add attributes to your scrape-sensor? I have a scrape-sensor which gets the water-temperature from a lido. So I want the coordinates as attributes. like: latitude: 47.3519 longitude: 8.5360 If possible I would even like to have an attribute with the website-link where I scraped the information. Are there some informations about how I can create such attributes or how they have to be named ("link:", "source:", "web:", ...)?
How do you set the interval between scrapes ?
@nakiros24
Жыл бұрын
Nevermind. The answer can be: Use Multiscrape from HACS instead and set scan_interval. Also it lets you scrape multiple information in one go. Maybe a new/updated video?
Could you make video about Scrape sensor with login field ?
@WillSurridgeTech
3 жыл бұрын
I don't believe that's possible because of the limitations of the sensor
The Link for "CSS Selector Tester" is faulty.
Could have used the KZread API instead.
@WillSurridgeTech
4 жыл бұрын
This is true, but then I wouldn't have had that example!
The over-the-top speech is completely unnecessary.