A big use case for this (at least for me) is scholarship/grant websites. They tend to rarely be updated until they become available for the current scholarship round. For example, there is one scholarship I am looking at that still has the "2023 deadline passed" banner, and I am waiting for it to be updated for 2024. This will be incredibly useful. Thanks for the video!

@TechnoTim

Ай бұрын

Great idea!

@Damia-cz8og

20 күн бұрын

@@TechnoTim docker network npm+dns+traefik how guide please

I recently read a forum post from someone who was using a Docker tool like this one to track the availability of 3D printing filament, and I meant to ask them which tool they were using or do some research myself. But here you are, delivering exactly what I was looking for, as if you have a sixth sense for these things 😂 Thanks, Tim!

@TechnoTim

Ай бұрын

Sounds like this will work great!

I plan on using this to monitor the county website for people trying to steal my property with a forged deed so I can fight it and fix it quickly.

I've used this for years. It's an incredible tool and a real must-have.

This is a great tool and I have used it quite a bit in the past for specific purposes. As mentioned - be sure to avoid an overly aggressive scanning interval, especially if running this from home. Many sites will block suspected bot scanners - and that is a very bad thing when your home IP is what is being blocked! Running from a cloud instance or through a proxy should at least avoid the problem of getting blocked at home, but it is best to avoid the situation altogether.

@TechnoTim

Ай бұрын

Well said!

Dang, this is huge. I was looking for this for a while ago now. Thank!

Now this is a tool I can use great find ❤

Great video. Thanks Tim!!

Wat a great bit of software with so many uses. Thank you for great breakdown

I tried to configure Huginn about 2 years when I first started using Linux. Did not understand head or tail. Came across ChangeDetection a little over a month ago. Most practical piece of software I host.

@nezu_cc

Ай бұрын

Huginn is really good if you need something very specific or you need to do a bunch of post-processing on the data. Huginn helped me buy a Raspberry Pi during the supply shortage

Good ol XPATH queries. Id be curious how often the elgato recorded steps get broken by Elgato redeploying their site. But it is nice that they give you the option to customize it to be more resilient. Awesome awesome video! And a fantastic tool that I dont know what Ill use it for, but definitely going to add it to my arsenal.

@TechnoTim

Ай бұрын

Thanks! I thought my xpath days were over 😅

Level1Techs spotted 👀

@TechnoTim

Ай бұрын

If Wendell is Level1, I am more like Level 0.01

Great! Thanks very much!

Thanks for the video. Wonderful idea. I'm playing with it. It's quite the touchy little creature isn't it?! It's favourite thing to say is "Exception: Target page, context or browser has been closed". Still trying to track this down but it's an interesting journey.

@GitPush1

Ай бұрын

I'm having the same issue on an Unraid server. Still trying to track it down

@lloydsshednanigans

Ай бұрын

@@GitPush1 I've been playing further and it seemed to be much worse when I used Tim's github link that he showed for a test. Perhaps loads of people are using it in the example and Github is getting annoyed? Just a thought. After taking that test out, it seems to have settled down a bit now. Are you using the example too? It may not be related but it was just a thought.

Might have missed it but "Visual Filter Selector" is a great option too. Think it shows up after the 1st screenshot grab.

Great video. Thanks TT

Great video. Can I ask what terminal are you using in the video?

The timing of this video and me seeing a comment in an unrelated Discord server about this service.

Tim, this is fantastic! Great job on this video! 😍 I love self-hosting services and this is certainly one that's being added to the "stack". 🤓(See what I did there?) 😉

Really cool, thanks

Love this video. Anyway you can do more software installs for those that use Synology Nas systems?

@TechnoTim

Ай бұрын

Thank you! What are you looking for specifically for Synology? I don't have one but assume they just use a web form to fill out?

Can you plot the price detected on a graph or do you need to parse the data to Grafana or whatever? Also, I would have liked to see the chrome extension. I know I can check the doc but I choose to watch a video, so...

I'm curious to know why you don't use docker volumes for the datastore, since you don't have to access/ edit those files... to me it feels that, in this particular case, bind mounts are just a needless complication (and going with volumes will unlock gitops and remove the need for ssh, but I get that it's an advanced topic). That said, I will definitely try this service!

@TechnoTim

Ай бұрын

Thanks! Unfortunately this is also where your sites are stored as well as your dashboard config. Agreed you can use a Docker volume but I've found bind mounting it to the file system when using Docker (vs. k8s) is much easier to manage, back up, and not accidentally delete 😅

Great video, biggest issue I'm seeing is it takes forever to check. Running on a proxmox docker lxc container with 8gb of ram. Noticing this only after adding Playwright

@ch716

29 күн бұрын

Did you get this to work? I have the same issue (not running on Proxmox) ,and I don't think playwright is working correctly.

Is there an LLM integration? What we really want is a vision model like llava2 to navigate the steps, not xpath

Is there a way to monitor an account balance? I have to log into that page and distil isn't seeing any value for the field. I think it is just looking at the publicly accessible version of the page. I need to monitor my privately displayed account balance. Thank you.

@-Zarion

28 күн бұрын

Did you try "Browser Steps" he showed it in the video, you can type there too.

Thats really good software

Just ran this up myself, noticed you always cut your video when you do a fresh check - is that because you came across this error "Exception: Target page, context or browser has been closed" and pretty consistently needs to be run a couple of times to get a clean check. Or were your checks all clean and just needed a few moments to run?

@TechnoTim

Ай бұрын

I usually cut out long pauses of the video. I did run into that initially when using the web driver + selenium but not since moving to playwright and chrome. You may need to adjust some of the settings but if it only happens on the first initial runs it might be hard to track down unless you look at the logs when it happens.

I've been self hosting this for a few months now to monitor Vanguard 7 day SEC yield on VMFXX. Great piece of software.

@TechnoTim

Ай бұрын

Awesome use case!

Is there any ethical way to stop Amazon captchas? Even when setting checks to every 1 hour I am still hit with a captcha.

@TechnoTim

Ай бұрын

Amazon has the best bot detection on the planet... seriously

@JZL003

Ай бұрын

Keepa or camel camel camel?

I use puppeteer to get my water consumption from the council website every day as they have installed wireless water meters which send the consumption to the council. Similar to what you are doing.

@TechnoTim

Ай бұрын

I love puppeteer. Use it all the time in CI.

@kriffos

Ай бұрын

Playwright is basically a fork of puppeteer … the API is mostly the same.

Those explicit xpaths -shudders- Is it possible to customise the selectors so you don’t end up with horribly flaky browser steps?

@nezu_cc

Ай бұрын

yes, that's the reason why it waits for you to click apply so that you can customize the step

@DuckSport

Ай бұрын

Nice, thank you. Is on my kid to try out :)

what about sites that use MFA?

@TechnoTim

Ай бұрын

I think you might be out of luck with that unless you can find a way to automate getting that code and supplying it, unless you only need to supply it once. If that's the case, you can drive the browser to supply it once and then hopefully the cookies are preserved so that the next time it runs it uses those. If not see if there's a way to persist those cookies between runs.

@squalazzo

Ай бұрын

@@TechnoTim I was thinking the way alexa media player integration works in home assistant, it register as a 2fa client... But that means if someone breaks into your ha, has your full Amazon account

"Elagto"

@TechnoTim

Ай бұрын

“Lagtoe”

First

You should blur your webhook address

@dklima

28 күн бұрын

No worries, used only for the episode

No. I subscribe to you as I can follow your instructions. The VScode section with adding in the SSH extension was not digestible for my small brain like most of your videos are. You skipped so many necessary pieces there. Im sad.

@TechnoTim

27 күн бұрын

Hey! Sorry! If you're getting stuck, hop in our Discord!

13:20 hopefully you removed that webhook already from your channel otherwise you must be getting spammed :)