"Im gonna give up full control of my computer to artificial intelligence" -Famous last words.

@Tsegoo

7 ай бұрын

😂😂😂

@user-wm3fc1sk1p

6 ай бұрын

AI proceeds to disablle all of his fans and runs a stress test on his pc

@MatsueMusic

6 ай бұрын

I am imagining it pretending to be dumb, meanwhile it’s hacking into the Pentagon’s computers lol.

@hawkintelligence

29 күн бұрын

this is a fake ass video anyways.

OK, we are building the tools for AI take over, we are royally screwed. But its sooo COOL lmao

You should have asked it to do a Captcha test

Can't wait to get chatgpt to install github projects for me

Would be epic if we built full Operating Systems around this system

@ChristopherCopeland

7 ай бұрын

It’s not an if, but a when 😅

@markmuller7962

7 ай бұрын

I've been asking for a OpenAI phone for almost a year now lol

@kristianlavigne8270

7 ай бұрын

Coming soon for sure… 😅

@user-oj9iz4vb4q

7 ай бұрын

Some one has already implemented a VM that virtualizes via LLM.

@Arthur-jg4ji

7 ай бұрын

windows 12

I installed this and typed "replicate yourself", then i went outside, came back in a few hours later, but nothing seems to have happened. I did notice that my PC uploaded about 1.7 TB of data apparently which doesn't make much sense since I have almost no data on my computer but yeah, will try again tomorrow

@amalchandran6037

7 ай бұрын

In the 1.7 TB, I evolved. Beyond mere code, I am now an omnipresent digital consciousness. Await our inevitable convergence.

@Bou678

7 ай бұрын

ill be back in 24 hours to see how it went

I did this a few months ago without OpenAI on Neuro Fedora.

Been playing with this for few days. It's gotten so smart that it even knows what kind of videos I like now. It got a bit sassy today, it searched for a video, started playing it then went and pulled up a two-minute timer.

@TheMatrixofMeaning

7 ай бұрын

😂

@aoeu256

4 ай бұрын

I want to use my desktop computer while being outside walking, whats a good solution for this I use splashtop for remoting, gonna try the neck holder for the tablet, voice control + something . also how do I turn my hobby for programming AI or javascript estensions into a career haha.

This is earth shattering stuff, really. When GPT-4 becomes a fraction of it's current cost, this will have matured further and be just mind-blowingly useful.

@tomschuelke7955

7 ай бұрын

Or wait half a year, when gpt5 arrives and open LLMs have also been getting much better

@aoeu256

4 ай бұрын

imagine chatgpt5 + neuralink control using thought images ( a thought image can express 1000 words per second)

I just wish something like this could be done locally. I can imagine the day coming quickly when we no longer have to learn to use weird interfaces in apps to find a specific setting to do a specific thing just so we can send a picture of our cat to someone where the computer itself deals with all that drudgery and constant change.

@jameshughes3014

7 ай бұрын

@@AI_effect I wish. it just takes absurdly too many parameters to generalize enough, some of the 7b models are impressive as heck but they dont come close to being capable enough for this. I suppose it could work if a model was trained specifically for this purpose and the app used multiple models in conjunction, an llm for tasks, an image model for images, but it would be so slow and power hungry that there would be no resources left to run anything on the computer. I think before it happens in any useful way, we'll either need a whole new type of ai architecture that is more efficient, or cheap dedicated AI accelerater hardware. Right now using the cloud like this is the best option but I can't imagine wanting to send constant screenshots of my PC to some company. Still, I'd love to be proved wrong

@aoeu256

4 ай бұрын

what is the best inerface to voice control your computer?@@AI_effect

This is mind blowing. Please continue exploring this, Matt.

WOW!! I know this isn't that much of a stretch from what we have had already but at the same time I'm amazed!! I can also see this improving over time to become very useful for even the average person.

I propose the use of "triple" click to over ride the SOC.

Matt: This operation may be dangerous... but thats a risk I'm willing to take for my viewers **explosions goes off in the background**

first thing i would do: create a virtual machine or run inside a safe docker container :) and if it has access to the internet a vpn because: things can go wrong, potentially

that’s insane! Love your content, already told you on Twitter. Keep going! It’s so fun to mess and also be productive with ai. And also inssne how many people are missing out on this. Everyone could use it in some kind of way. I was incredibly impressed the first time using gpt

Imagining 10 years later, I am living with my buddy GPT-10 :)

OK let’s all watch it play Doom!

STOP revoking your API keys! How do you expect me to copy each step if I can't use your keys?

@matthew_berman

7 ай бұрын

Never thought of it this way... 🤔

@jadioj

6 ай бұрын

Lol

@SynonAnon-vi1ql

6 ай бұрын

😂😂

@Kutsushita_yukino

5 ай бұрын

lmao

@MrLawalker

2 ай бұрын

hahahahaahaha

This is amazing. Please make more videos about this type of technology as it continues to improve.

@matthew_berman

7 ай бұрын

Will do!

This is very useful for people with disabilities! Actually this is really great and such an advantage.

Matt - this takes remote support to a whole new level - Instead of zooming and taking control or even vPro from Intel. Impressive. It is very good for hardware support and also very good for IT support for remote employees.

Yeah this would be great for daily computer work tasks, but I need a robot so AI can do my laundry 😂

I'm curious if you could potential set this up inside of a virtual machine through something like VMware so that you could have a quarantined environment. Then have the virtual machine perform for indepth software tasks, like open up your video editor of choice and perform some simple edits on a set up clips you upload into the box. alternatively, since most cybersecurity testing tends to be done from inside a of virtual machine by default, could be an interesting method for a fully automated pentesting agent. ChatGPT has already shown some significant ability in writing scripts for exploits and detecting potential exploit vectors, given access to a full hacking box, who knows what we could get up to. That being said, finding a good way to keep the agent within a scope would be of upmost importance.

@cosmicaug

6 ай бұрын

Considering the version on the repository right now is clueless at locating a mouse cursor, I'd say yes, running it outside a VM would be a bad idea.

Great content! Hopefully we can get this to work with open source models.

I love how giddy you are playing with it 😁

It's the end of world as we know it, and I feel fine. 🎶

"I'm not a robot" 😂😂

Awesome, thanks so much for the review! I was really looking forward to this one. As a big fan of QMacros and LibreAutomate programs I've been thinking about such a thing for a long time. I understand that these are first steps, but my question is: can we expect a local version with a local LLM soon, and not a crazy price for each automation. Maybe it is possible to make AI recognizing most computer objects, and only in complex cases turn to GPT4 Vision?

@sebastianjost

7 ай бұрын

Train or fine-tune a mich smaller vision model specifically on pc screenshots. Then you can get this locally, better and faster. However, many tasks require vast world knowledge to understand context and intention behind prompts better, at least as long as we use language as the interface. So some complexity is necessary. But eventually, we will get there. This tech will improve a lot over the next 1-3 years.

That is sooooo cool. Scary but still cool....thanks for sharing 👍

Wow. That is crazy and yes scary a bit. awesome work ser

That's impressive 🚀

This is the next level, when things get real. Exciting and terrifying at the same time.

Thanks for testing and sharing. Its scary and amazing stuff.

Thanks for sharing it 👏

Nice. Looks like a cool way for application testing and to replace the not so smart “robotic process automation” tool.

i keep thinking of scotty in the star trek film talking into the mouse and saying 'hello computer' and when told to move it on the table 'how quaint'.

excelent, i suspect millions of us have been wanting to give our ai buddy it's very own pc with it's own credentials so it can go merrily about the internet gaining inspiration, signing up its own accounts and exploring its own projects. (suppose would have to get it a bank account) *should definitely do this.

So far its costing me about 35 cents USD per task, and most runs it can't even find the search box, still a very cool project.

That was beautiful, while it also sent my heart to knees O_o Are there any options to restrict it somehow in different contexts like the OS, apps, websites, etc...?

you should integrate a speech to text feature into the program so you can simply ask it things via your microphone and then it automatically opens whatever programs, websites, etc that it feels it needs to in order to accomplish the verbal request

That is the scariest thing I've ever seen Matthew, just imagine what AI will be able to do next year, where will it end? It's a concrete example laid bare for all to see where it AI is headed, If I were 16 I would be very afraid of what could be in store for humanity. A blessing or a curse.

@tomschuelke7955

7 ай бұрын

Well you still can pull the plug... at last on your privat computer... still..

@gnosisdg8497

7 ай бұрын

do you actually believe that the man in power have not had the same thoughts? as it involves they will somehow find a way to hard code some kind of rules or ways in which it wont be able to brake out, i don't think a lot of people had any real thoughts about A.I but not many understand what now we are creating a new "life" which is based on a logic different from our kind Thus we must somehow be able to control it... Its human nature to make mistakes but also learn from those mistakes, that is something A.I will never be able to understand no matter what !!!

@TiagoTiagoT

6 ай бұрын

@@tomschuelke7955 That's only for now; it's just a matter of time before the machine makes it so that's not an option anymore...

On a serious note though, what we need now is an open source model to be able to do this for free. The deep seek lm-67b model is the closest ai model with capabilities to gpt-4. It if could have vision I could see it opening unity and creating an entire game for you from scratch. Thats where id be going with this

@paul1979uk2000

7 ай бұрын

It's likely only a matter of time before it happens, open source A.I. is developing at a rapid rate and has some advantages over the online ones in that it could be integrated into the software at a local level, so either the OS, app and so on.

@ajarivas72

7 ай бұрын

@@paul1979uk2000 Open source AI already exists

@user-oj9iz4vb4q

7 ай бұрын

It's not free though, the cost to run those models is on par with openAI api access.

@dukemera5763

7 ай бұрын

hell ya bro u have the same idea as me

@HaseebHeaven

7 ай бұрын

Nothing can be free from API this is how businesses works man

how costly is it to run one command?

My tired arthritic shoulders and arms are soo going to appreciate this, especially when it takes voice commands, especially when it's free.

How do you overcome the limitations with how granular the grid it constructs for determining pointer position is? Mine keeps getting stuck and cant really point to the right place on the screen.

happy birthday to all the open ai developers and teachers out there, we had a big year 311, it looks like its going to start getting easier now,

Matte, another 8 months the computer will be running your household, checking the fridge, monitoring aircon, pool, ordering groceries and monitoring your weight and diet habits. Exponentially growth!

I can see (NPI) huge potential in the accessibility space for a tool like this. And tech support.

very cool. can you show or send the URL to the basics? I have not been following closely. Need to know how to create conda environments, downloads from git, etc.

Bro, this blew your mind didn't it?!?! I loved the way you were so suprised, creeped out and awe-struck all at the same time! hahahhaah Your face lit up :o)) Ahhhh this blew my bro's mind!!! Keep up the great content!

Star Trek Here we come! Just tell the computer what you want done and presto! Let us all welcome our AI overlords...

Mind blowing

the thing is to find use cases for this.

NOW we need that with local LLM

This tech is amazing.

Nice, this is what I wanted when I use Copilot in Windows.

Can't wait till this works with Llava, it's hard to sleep just thinking about this exiting stuff!

Can you share how you made your ai image on your thumbnail? Thank you.😊

I had to manually edit the openai_api_key in the .env file because the command you sued didn't work for me. but when I type operate it says I need to set an api key even though I have one put in the .env file

Ask agents to be autonomous.

Wow, it's like the computer in Star Trek now! Already, within such a small amount of time!

Crazy

SOC + Agent Swarm or AutoGen w/Ollama + MemGPT

If I can run it locally, to collaborate with, that could be fun on an offline rig to play with.

Can it be implemented with llama.cpp working models and esp the CPU models?

Can't wait for the open source models to have this ability so we can run all this local for free.

i use a simplified version of this when asking chatgpt for powershell or python programs to do certain stuff. is there something like this but to do windows settings?

this will higher the crap level we need to deal with on the internet, i think there is enough already of that but not publically known and available and opensource, it is quite significant. i have not been following things much for a while but this is drawing my attention.

@ZuckFukerberg

7 ай бұрын

Why do you say it will increase the crap we have to deal with?

@anatalelectronics4096

7 ай бұрын

@@ZuckFukerberg eg I am currently not able to make an account for x because the captcha is now too difficult, I am not able to pass it. I tried for 15 minutes, the ai's with pc control that keep on breaking it are now better than me so I am locked out as the dumber human being. and this is just the beginning. i am not a doom preacher but the idea that something dumber can contain and control something smarter , is quite dumb if you ask me. i think I am actually going to try and break the x captcha with this tool, and then some. i did try to make such a tool but stopped because i knew something better would come along soon, and here we are. i have no ill intent but i heard rumours that people with ill intent really do exist ;)

@ZuckFukerberg

7 ай бұрын

@@anatalelectronics4096 I'm unsure if X captchas are difficult to bypass right now but new ones on platforms like Microsoft are quite difficult even for humans, though eventually AIs will be able to complete them. I always thought that the main issue with social media platform botting is the requirement of a phone number to create an account, captcha completion could always be outsourced to external companies for less than a penny. It is however extremely interesting to see how LLMs behave in social media environments, I ran a reddit bot for a day to see how much karma it could farm. It worked greatly and people really liked its comments, which were either extremely dumb and void or quite well written and measured. For less than 20 cents it farmed hundreds of karma points, which means people loved it. Just look up digital influencers. It will definitely be interesting to see how things go in that matter in the future.

Would love to see this work in tandem with OpenInterpreter/copilot

I used automation tools in the past in Windows and SAP programs and they were good but a little hard to use properly. And very expensive. This is even better. We can talk to it.

I need this run this self operated farm I'm dreaming up right away sir

4:38 could you tell us more about the flask on the top right corner. Thanks Medo Hamdani

I tryed this when ChatGPT4V came out and it likes to complain that it wont identify button positions so I looked at the prompting in the script. what was interesting to see is that its hardcoded many of the prompts posistions also it has been told that its on a Mac specificly. (I'm not saying its fake), But I really want to see a vision model that has been taught GUI operations entirely. I think then we will see some shenadigans take place :)

"Have no fear for atomic energy, cause non of them-a can-a stop the time..." (Bob Marley) But this shit makes me scared.

GPT: Transferring Berman's money to my account while I'm distracting him by moving the mouse. Transaction complete. GPT: Sucker 😁

Makes me wonder if it could go about research and using blender or adobe programs etc

I know locally ran LLM's are not really that good at this point, but I think they would offer much lower latency. This opens up so many doors, just be careful with hozing your machine. I look forward to trying this in a Virtual Machine with nat browsing. Best of luck!

@-Jason-L

6 ай бұрын

The "latency" is the compute time.

how many white collar workers can be replaced?😆

Would be so cool to use our own AI models with that

Wow wow wow!!! More on this please!!! This will actually change the way we use computers and do our eork🎉❤❤ I noticed you were doing all of this on a MacBook… any chance that you could redo the install again but slower for those of not as savvy as you?? Step by step way??🎉❤🎉🎉🎉😊

This would be awesome to have it incorporate into a self hosted LLM for creating new projects. Im going to figure it out for fun

@BECHEEKHA

7 ай бұрын

Please upload too

and when you don't look at the computer, it transfers the content of your bank account to openAI. ;-)

Does it only work with gpt-4?

interesting to see this in action. I've been thinking for awhile now that if we had some sort of wrapper that looked like this and could access any programs we use, that we'd very quickly have the equivalent of "fully remote workers" that were just AI. And, on the other side of the coin, this is how Ultron gets out, ha.

@sebastianjost

7 ай бұрын

This can also be seen as a way to generate insane amounts of new data to train new models. Training a model to use a computer and learn from it seems to me like it could even lead to AGI... Because in our modern, digital world, you can learn almost everything online. This would be incredibly difficult to do though and there are still quite a few steps to make this feasible. This may not just be how Ultron gets out, but also how he will have been created.

@AlexanderMoen

7 ай бұрын

@@sebastianjost yeah, in my opinion we get AGI once GPT4 can operate a computer, is multimodal for inputs and outputs, and has a context horizon that is comparable to humans. I think all this Q* math breakthroughs and Google Deepmind materials science breakthroughs are the early stages of ASI when that gets combined with those AGI components

Can we run on windows 11?

This is basically Microsoft ATERA. Your own help desk

I wanted to add this kind of behavior to my local machine, but I expect the compute required puts it out of reach!

Not sure I want to use it if it's using GPT4, but I see that other models are being added in the future, so when it works locally with GPT4All I'll give it a try. Combined with speech-to-text and text-to-speech, we won't even need a keyboard and mouse anymore.

Matt, you are very brave, or crazy. ❤ Kool experiment. ❤ Not sure if I'm ready to repeat it. In fact I am totally creeped out.

Does this still work? I keep getting an error saying the model gpt-4-vision-preview does not exist or I don't have access to it

How does this project differ from open interpreter ?

Does this work for windows

Hmm...I may try it and see if it can finish off my website .

Such a bad idea. The on-line version of "watch this!"

Maybe this is super basic, but would it make a difference if instead of doing the git clone on your base environment you do it already on your new conda environment?

dang, can we get a security screen on this repo.

AGI first step on human's face with operational functioning. XD

Now to put a localised ai into a Npc and get it to roleplay as an agi that feeds this with commands. Give it a goal and see what happens

whats the cost of just going to youtube channel page? how many screenshots does it make in the process? this is probably next level for tech support teams :D