What AI agent use case do you think it will unlock with browser & computer access? Comment and let me know!

@metinEsturb

6 ай бұрын

It can now watch youtube shorts for me, so that I can focus my lifetime on watching productive videos like yours 😉

@minutecodingtips

6 ай бұрын

Can you provide github repo for it sir?

@ahsin.shabbir

6 ай бұрын

Find me all relevant documents in my inbox and my desktop so that I can prepare my tax filing for 2024.

@rab0309

6 ай бұрын

would be interesting to try it out with models that are open source and not something where you'd need for example an open ai api key

@martinchudomel7117

5 ай бұрын

maybe automated testing

reCAPTCHA: Are you a human? Web AI Agent: *click*

You have been making some of the best videos on this stuff since day one. Always super practical yet very valuable content Jason. Keep it up and glad to see the HubSpot sponser! Esketit

Thanks for featuring my project!

Great job mate! Love to see all the agent possibilities

This is a really smart and intutive process. Thanks Jason!

Congrats on your work, really useful content. Keep it up!

Really awesome job. Keep such videos coming. 🎉🎉

Amazing work! Thanks for putting in so much effort to bring quality content to the viewers.

I've been very excited about the last video when I first saw the paper of GPT-4v and the automation applications. I do RPA full time and even though GPT-4v is not there yet in terms of what it can do on its own for form data entry and complex automation processes, I can see the use cases where it can outshine RPA when it's enhanced.

This is really valuable work Jason, beautifully conceived and executed. Keep going, this is huge. 🙏👍

This is an amazing tutorial of how to integrate GPT into daily productivity tools. Thanks man!

Great video again Jason love to see it

Bro, you are such a gangster. You're the only person I follow actually putting out valuable work at this level. All these AI grifters are on their six hundredth basic ass autogen demo, just zero value videos for clicks.

@stephantual

4 ай бұрын

This exactly.

@hereusername

4 ай бұрын

same for me! it is really hard to find valuable info about AI beside just basics

great explanation, like this kind of tutorial on AI projects!

Mind blowing! Fantastic.

thanks jason for this valuable work. This toutrial are advance but very valuable spec in the market. You could easliy make big money from this method.

Great Jason... this is kind of innovation is mind blowing!

Great video! I would love to see if this can be done with an open source model like LLaVa.

Super interesting Jason!

This is so sick.. thanks man

Great Work Jason!! would love to see you explore creating a Autogen like framework with Open source Multimodal agents, seems like it can open a lot of possibilities without the huge costs associated with using openAI's APIs for intensive tasks. Keep up the great work.!

@EricWatyekele

6 ай бұрын

Awesome work Jason, I was thinking the same @paresh3915 - Open source Multimodal agents instead of OpenAI

Jason, wondering if you have some kinda plans for diving into details of finetuning open source models like llama2?

AutoGen now has a AgentBuilder class. Its exactly doing how it sound, building a swarm of agents.

This is Amazing !

Another great share, cheers! One request... please consider adding links in the description to the products mentioned in videos; perhaps as affiliate links for yourself. Makes for improved UX for viewers.

Thanks man!

Hey Jason, how would you implement the ability to type into textboxes? They usually do not have text in them, so the AI cannot identify them. This feature would likely be needed to add the ability to do more stuff like order a pizza.

Thanks!

You are awesome. You should have your own instructional (paid membership) website for all of this.

Wicked sick!

Nice, was wondering how to build a chrome plugin to control web like hyperwrite, this is handy 👍 Also the hubspot research is actually good

@carterjames199

6 ай бұрын

Same here trying to figure out how to connect the local repo to a browser plugin

Absolutely beautiful! Really appreciate you sharing this insightful video. It's always great to see such valuable content. Thanks for broadening our horizons!

Bro you are LOVE

Any idea about how to implement scrolling in the web agent?

Amazing

I really appreciate your video, I am wondering if you know a good resource or if you can do a video that walks through how to do this 100% from start to finish using windows and vscode

mind blowing

let's get it!

Nice work- thank you!! Do we really need to run node? Can’t we just use python browser libraries ?

@amandamate9117

6 ай бұрын

yo

Been looking for such approach for while now. Until i see this video , very exciting. But I also concern over the pricing of GPT-4 vision, I hope other alternative works as well like LlvAa 2 vision or CogVLM & CogAgent

Crazy !!!

how can we provide the assistant api key of gpt 4 which is not currently supporting the image input?

Hey Jason amazing video as usual! Can you do one using canva to setup and automate a youtube channel..... from creation of videos using canva to scheduling the posts... do you think this will be possible?

@wizaaeed

6 ай бұрын

It's not about automation, it's about being authentic. If you think posting unlimited auto generated videos to youtube will give you an advantage over something, you're far off the line.

@avi7278

6 ай бұрын

LOL, you'd end up paying $50-$100 for a single crappy AI video that nobody will watch. There's cheaper way to spam your AI crap videos if you want to do that.

how to build query bot ,it should query from the database like mangodb where the data are stored from the application

Did you have trouble getting your chrome profile working with Chrome Canary? That seems to be my only sticking point.

why are not all open source LLMs working on these autonomous web agents? Are there any open source, PC based, replacements of Rabbit R1?

🎯 Key Takeaways for quick navigation: 00:03 🌐 *Web AI agents with direct computer access are a trending topic, allowing AI models like GPT-4V to control computers and browsers for various tasks.* 01:28 🏢 *Robotic Process Automation (RPA) is one market category that self-operating computers can impact. RPA automates repetitive tasks in enterprises.* 03:32 🖥️ *Self-operating AI agents can handle complex and non-standardized tasks, potentially reducing setup costs compared to traditional RPA solutions.* 04:27 📊 *Understanding the end-to-end workflow of specific job functions is crucial for delivering useful AI worker solutions.* 05:36 🤖 *Two common implementations for AI agents to control web browsers involve sending simplified HTML or using screenshots with annotations for better instruction.* 08:09 🖥️ *Self-operating computer frameworks like the one discussed in the video use annotations on screenshots to instruct AI agents for interactions.* 10:02 🌐 *The tutorial demonstrates how to create a web scraper using GPT-4V for extracting data from web pages, providing cleaner results than traditional scraping.* 16:59 ⚙️ *A more advanced AI web agent is discussed, capable of interacting with websites, clicking links, and performing complex web tasks.* 21:19 🌐 *The web agent script can handle URLs provided by the agent, allowing it to navigate to specific web pages.* 22:29 🔄 *The web agent continuously clicks on links to navigate, find information, and scrape new websites.* 23:11 📊 *The web agent can interact with websites to answer questions or complete tasks, making it a valuable tool for various purposes.* 23:53 📷 *The web agent can perform complex tasks like conducting a Google search, visiting Instagram accounts, and extracting information, showcasing its potential for various applications.* Made with HARPA AI

Can we combine ai with something like ui.vision? Have the ai agent learn how to use ui.vision and then it would be able to control computer and browser

Hi, I do the same as you but my Chrome Canary still require login

Hi Jason, I've been trying to reach out by email for a professional consultation for a company. It looks like the email might have fallen through the cracks of a million others or under the spam folder. Is there any other way I can reach out to you?

can we use Gemini Pro Vision instead of OpenAI? Could you please provide the tutorials?

Hello! Thanks for the video, and yes, it’s already February 2024, and I would like to know what’s new, what crazy projects are being implemented using this technology? what business cases are already working today? And yes, can you help me set up this process for my task? You are super! I looked at it in translation, but I realized that you are very skilled! Bravo!

Is it possible to implement with Llama 2 ?

More of JavaScript and node please :)

Just throwing this out there: If using OpenCV to drive your annotations, you can annotate coords directly on the image without too much difficulty (to give some guidance for when GPT4V is trying to recommend a location to click). Haven't done this with puppeteer, but I'm sure it can be done ;) Anybody wanna collab on some experiments in this space? 🙏 Lolol

24:28 Having an AI agent spend tokens to check the weather when the same search in Google results in the same reminds me of that robot that is told its whole purpose is to pass the butter.

One question here , can the AI agent bypass cloudflare captcha check as a humain ?

Amazing as always! I'm still trying to build my browsing ai agent from the sxsw hackathon in Sydney lmao 😅

@KamilKaczmarekSolutions

6 ай бұрын

meaning clickolas cage? 😂😂 love that

@techfren

6 ай бұрын

@@KamilKaczmarekSolutions haha yup!

Isn’t reading the dom tree better than screen shots given how the latter needs to interpret images as opposed to structured data?

@AIJasonZ

4 ай бұрын

Yea DOM will be more effective, however the issue is each website DOM is very different and including lots of messy data; but if you figure out an universal way to clean up DOM then it will be effective Also with DOM, it lost ability to do some img related tasks

What if you have an AI agent for a consumer product that handles event management tasks normally performed by the event host? For example if there is a school doing a field trip, the AI agent can answer many of the parent's queries, send updates about the trip such as weather forecasts that may effect the trip, collect certain forms, and more. Same can be applied for events like conferences, weddings, birthdays, housewarming, etc.

@fullcrum2089

6 ай бұрын

i'm building this

@ahsin.shabbir

6 ай бұрын

@@fullcrum2089 how far along are you?

@amandamate9117

6 ай бұрын

build ngga build @@fullcrum2089

Might i know the cost per screenshot and api?

does this work with ollama?

Do you thing someone with zero coding ability can build what you've just shown? If not do you build this for people to use? Great video, found it very interesting and inspiring.

All I need to know is how I can use this to create a trading bot that can learn and update trading knowledge based on my trades, then it can trade going forward without me

Are there any of this sort of tutorials out there that don't use an open AI subscription and actually do the job ? Is there an actual value in this if you have to pay open AI for every new page chatgpt 4 has to process??

Hi the github link is missing alot of files can you update it please also with the requirement.txt files.

Can we use LlaVA?

Can you scrape websites without getting detected?

Can you share the repo link?

How much does each run cost on this?

is this less detectable than using puppeteer alone?

Do people recommend building an AI agent from scratch using Puppeteer and GPT or using an existing self-operating system like OthersideAI or MultiOn?

Don't sacrifice quality for views! You have an amazing KZread channel that could be useful and profitable for you and for your viewers. I think I am not the only one who thinks that you should open a patron account and create premium content for people looking for more developed material 😁

@stateportSound_wav

5 ай бұрын

That’s not a bad idea. Do you feel the content has been watered down for the sake of views?

@emmcode3414

5 ай бұрын

@@stateportSound_wav No I don't, and I don't want to. That's why I am trying to encourage the channel to open a Patreon account, to preventing ourselves for loosing quality.

@stateportSound_wav

5 ай бұрын

@@emmcode3414 10-4

@stateportSound_wav

5 ай бұрын

@@emmcode3414 I hope that his skills have been able to get him to a point of not relying on youtube for financial stability, but that’s his business. Jason mentioned previously working on startups, so it sounds like he’s no rookie to business, and could be well off to the point your comment could (I hope, for Jason and ourselves!) be irrelevant.

@hereusername

4 ай бұрын

@@stateportSound_wav Many channel about AI started just posting random news about AI or making clickbait videos just to get views which is really annoying. At this point I only check this channel

Can this be done in Python?

I dont know python. is there an alternative for Js, google didnt help me

@KamilKaczmarekSolutions

5 ай бұрын

just take this code and go with it to gpt-4 and ask for rewrite

Hi Jason, thanks for this practical video. I would like to have your thoughts on my podcast on Process Automation and mining using AI, would be interested ?

you are ... so fking smart man.

Interesting use case. Could you add the OpenAI API costs as reference?

@Humpty0Dumpty

6 ай бұрын

A simple 4-5 step task will cost you $0.24, try complex tasks and reply back the numbers, if possible

@moelkblau

6 ай бұрын

@@Humpty0Dumpty thanks!

@vladimirgetselevich4704

6 ай бұрын

@@Humpty0Dumpty Pretty expensive at the moment, to do anything valuable in quantities.

Has anyone here connected WebQL to a Crew agent? If yes, I would like to know how or a tutorial that can assist me. Thank you.

Market size unlimited

You sound exactly like a mature Jian Yang 😆

Twitter bot armies have never been easier to make. :)

Why would you jump back and forth between JavaScript and python? Just pic one or the other this seems super ineffective and inefficient

Cheat Layer Desktop has solved this problem at the desktop level using a new multi-modal model we're launching in the coming days, which enables a generalized agent that can detect UI elements directly: kzread.info/dash/bejne/e4WppKNwfdm7kdY.html

@avi7278

6 ай бұрын

I'm not convinced. The step by step setup cost is too high and it still relies on GPT4 so the cost savings aren't even there. Or am I mistaken?

@CheatLayer

6 ай бұрын

@@avi7278 our marketing agents drive traffic that would typically cost thousands more per month on google, and the sales agents get up to 11% reply rates--better than our human reps with another service.

im lost

This is insane!

why no one has built an online tool with API for this process? @aijason

Sky torrents

never trust this guy you will end up hacked

@amandamate9117

6 ай бұрын

i have the same feeling. last time he promoted some free openAI APis and also his codes are fishy. or? also his hair or face looks like is processed with some AI filter. i think he works for chinese government.

@mrfalvo6671

5 ай бұрын

lol@@amandamate9117

@KamilKaczmarekSolutions

5 ай бұрын

@@amandamate9117 he 100% certified chinese government agent, agreed. in fact ai agent * 💀

@hereusername

4 ай бұрын

@@amandamate9117 you can just check his codes yourself. It is not like secret program where you don't know what data is going off from you computer