My "AI Stack" is RouteLLM, MoA, and CrewAI. What about you?

@craiggriessel1872

20 күн бұрын

AISheldon 🤓

@shalinluitel1332

20 күн бұрын

It would be best to have alternatives to all these which are free and open source. Maybe later down the line.. The video is really cool tho! Thanks Matthew

@santiagomartinez3417

20 күн бұрын

Is MoA mixture of agents?

@AIGooroo

20 күн бұрын

Mathew, please do the full tutorial on how to set this up. thank you

@smokewulf

19 күн бұрын

RouteLLM, MoA, and Agency Swarm. Should do a video on Agency Swarm. I think it is the best agentic framework

Would love to see a tutorial on how to set this up.

@AlexBrumMachadoPLUS

19 күн бұрын

Me too ❤

@bamit1979

19 күн бұрын

I think some other AI enthusiast covered it a few days back. It was quite easy. Check KZread.

@ChristianNode

19 күн бұрын

get the agents to watch it and do it.

@sugaith

19 күн бұрын

On how to set this up IN THE CLOUD as well or preferebly

@averybrooks2099

19 күн бұрын

Me too but on a local machine instead of a third party service.

Please do a tutorial for local installation for this. Thanks

@camelCased

19 күн бұрын

What exactly? As I understand, RouteLLM is not an LLM itself but just a router. You can install local LLMs very easily using Backyard AI.

@m8hackr60

19 күн бұрын

Sign me up for the full tutorial!

@DihelsonMendonca

19 күн бұрын

​@@camelCased Or LM Studio

@bigglyguy8429

19 күн бұрын

@@camelCased But how to use the router with Backyard?

@camelCased

19 күн бұрын

@@bigglyguy8429 Why would you want to use the router at all, if running LLM models locally?

it'd be cool if you did a vid on setting it up and running it locally

@anubisai

19 күн бұрын

Olama or LLM studio?

Matthew - for those of us who develop line-of-business apps for SME businesses - local LLM deployment is a must. Would certainly like to see you demo RouteLLM with orchestration - Thanks!

I am a cyber security analyst who knows very little about coding so, between your videos and just straight asking ChatGPT or Claude, I am ham-fisting my way through getting AI to run locally. Please keep making tutorial videos - I am excited to see how to impliment RouteLLM!

@s2turbine

20 күн бұрын

I agree, I'm pretty much in the same boat as you. The problem is that my knowledge is outdated by the time I finally figure things out because there is so much advancement in so little time. I think we need a "checkpoint" how-to on how to do things now, as opposed to 3 months ago.

@DihelsonMendonca

19 күн бұрын

If you don't know much about anything, like me, but want to run LLMs locally, you just need to install LM Studio. No need to understand anything. On the software, it has even the option to download and install them, and run. That's what I use. Now that I learned a bit more, I will try to install Open WebUI, Ollama and Docker, these are way more complicated. 🎉❤

Yes! Please show us a comprehensive breakdown of this great tool! I’m also interested in your sponsor’s product, LangTrace. Can you possibly show us how to use it?

Absolutely do a detailed tutorial on how to get this up and running!

Great breakdown, much appreciated. I definitely foresee local LLMs becoming dominant for organisations as soon as next year. My advice during consults is for them not to invest a massive amount in high-end data secure cloud systems, but just to hang on a little, work with dummy data on current models to build up foundational knowledge, and then once local options exist they can start diving into more sensitive analytics.

Yes, please provide a tutorial on setting up the described language model.

Hey Matt, yes it would be great if you could show a demo of how to setup this model on Azure OpenAI or Azure Databrix and then use it in the application.

I feel like everyone is realising things at the same time. I started 2 projects, the first an LLM co-ordination system and a chain of thought processing on specific models

Thanks for this video. Very informative. Please make a full tutorial about the setup of route llm and what the recommendations of the local pc should be. Thank you in advance!

I also saw where they will have 20TB m.2 drives in a couple of years. Running this LLM locally will be really cool.

Just popping up to say thanks Matthew. You have become almost my only required source for AI news because your take is right up my street every time. Great work, keep it coming

There seems to be a hold up on the highest end models as the leading companies continually try to improve safety while watching their competition. Nobody seems to want to jump in and release a new/better model at risk of the potential "dangerous" label being applied to them. So a lot of the progress remains hidden in the lab, waiting for competition to finally engage.

@steveclark9934

19 күн бұрын

Improve safety really means neuter.

@davidk.8686

19 күн бұрын

So far with LLM's "data is code" ... it is inherently unsafe, unless something fundamentally changes

How would this effect mixture of agents? Could we have multiple route llms combined together since they use such lower compute?

can this be used to route between agents as opposed to model instances? for example routing to chain of thought agent vs simple q and a agent?

Yes! The tutorial. Great vid. Sharing with my crew...Just beginning an AI Consultant agency and cost is an existential threat!!!

It would be interesting to see how this will work on your AI benchmark. Please do a setup and test

Yes! please we need a full tutorial!

The only channel where i actually watch ads. Very interesting stuff

Yes I really like your idea of a complete install and running tutorial. Go for it. 🙏 Thanks 👍

So, how to run it. And on what hardware?

@threepe0

19 күн бұрын

Lmgtfy

Danke! As always great!🎯👍

You are so consistent bro, keep it up 🙌

I think you misrepresented the graph. The "ideal router" point on the graph is likely just that - the ideal. I don't think that's claiming actual results

Yes - please do a full tutorial on setup and use. Thanks.

The article used GPT 4, not GPT4-O, which is already 50% of GPT4 cost. Or am I missing something?

Would love a local setup tutorial! Thanks 😊

Yes please full set up!

Hey Matt, what are these routers exactly? (They are not LLM I understand) And how do they determine where to route to?

I'd be interested in what hardware is needed to run something like this locally. I was waiting until late fall or early next year to buy, but I might need to get an intern system to train up. I am big on local control except when needed to reach out.

This sounds interesting, but does it offer the same capability that the OpenAI API offers with customizable assistants, RAG, and function calling? I still have yet to find anything that compares. Would love to see something open source that can do this.

Please do a tutorial. And need to measure performance to validate the performance - cost graph.

I would love to catch a tutorial of you setting it up!

Would love to see a tutorial on setting up RouteLLM! 🙂

That's good stuff. I will take a look at the codebase. Thanks for sharing

Always yes for full tutorial

We need something locally for non difficult pourpouses. For example local home Assitant control.

Hi Mathew, I love your channel. I’m curious if you would be willing to explore Pi ai? It doesn’t compare to the others in the same way. Maybe it’s hard to test. But very interesting. It’s trained to be empathetic and you can actually have a conversation with voice that feels satisfying.

Yes do make a whole tutorial on it

First thing I did a year and a half ago was routing different LLMs via a zero-shot classifier. Looks like Route has done the same thing lol. I figured it was common sense.

Does your local model search the internet and summarize a few web pages? That's what chatgpt does for me, and that's all I need.

Ive seen you do techdev before and would love to see you do this one as well please.

You should consider including Semantic Kernel and GraphRAG in that ideal stack

Yes please... Full tutorial on setting this up to run locally. Also, I'd like to know how to setup multi-modal so I can show my images and casually talk to it (local).

For coding this would be insane. Mixed local and api endpoints

Is this an LLM that we can use in LM Studio?

19 күн бұрын

its just a proxy to send queries to two models, weak vs strong. It's not a new LLM.

I’d love to see a demo if possible.

Hey Matt, please do a tutorial on setting this up.

Yes, give a walkthrough video for RouteLLM

I think they are saying the brown dot is where an ideal LLM would be placed, I'm not sure that Route LLM is better than Claude 3 Opus. SO not sure where on that chart their router actually is. probably down with Llama 3 8b. Cause it's only job its to route.

After developing exclusively on GPT models, then joining an org with a ridiculous amount of free GCP credits and being pushed to use Gemini family instead- I can honestly say that while differences on benchmarks may seem small, they end up being really extreme in practice. I spent days smashing my head against a wall trying to get Gemini to provide quality responses, and after switching to 4o, I was literally ready to deploy. There still don't seem to be great benchmarks that represent performance of generative models well.

I would love to see a tutorial on how to get this running locally.

I’d love a guide to easily set this up for myself

Could you do a tutorial on RAG (retrieval augmented generation) ? I think I'll be pretty massive thing in agentic archetecure. Also I think RAG might soon be more than just text and PDFs 😂 in the not too distant future.

Yes please make a tutorial on the routerLLM.

It would be really cool to have a tutorial on how to implement this with ollama

Problem with this method is that there are some trade offs. While it maybe cheaper at answering a question directly; you sacrifice its social intelligence. Even though you get the right answer, the way the answer is phrased can be the difference between either a toddler or a graduate student. Personally I wauld want to talk with the graduate student.

Please do a demonstration

how??

Would also love to see the tutorial :)

It's time people admit that benchmarking off GPT4 is stupid. When GPT4 came out it was amazing. Now its no better than any other LLM. Ever since OpenAI introduced cheaper Turbo models, the quality has gone down hill. They sacrificed intelligence for speed to the point where they have plateaued in quality and its not getting better no matter how new models they release.

@orthodox_gentleman

19 күн бұрын

Thanks for being real bro. I absolutely agree with you. I barely even use ChatGPT anymore because it sucks.

@irql2

19 күн бұрын

"Now its no better than any other LLM" -- do you really believe this? Seems like you do. That's certainly a take.

@kyleabent

19 күн бұрын

I agree man I don't care about speed as much as I care about accuracy. I'll happily wait for a better response than rapidly go through 2-3 quick responses that need more time in the oven.

Claude 2.5 Sonnet missing from the chart.

This wasn’t just released. It had been around for a while. Now that GPT-4o and Claude 3.5 Sonnet exist things are much cheaper. I can understand using a local LLM with these two but overall the cost savings are not as big of a deal as before.

19 күн бұрын

API for claude and GPT is sitll expensive.

wonderfull know you can offer a low cost service and a primun service at diferent prices

Reading and watching anything about AI is like a live broadcast of the Manhattan Project in 1942. The current year is 1944?

If runs on local machine, why needs OPENAI_API_KEY ?

19 күн бұрын

because it still needs to query weak models (like mistral) and strong models like GPT

A tutorial would be great!

Promising, but a bit mess with naming. They are using GPT-4 to mean at least GPT-4 Turbo and GPT-4 Omni in various places. I am not even sure if on some place they don't really mean the older model GPT-4.

I suspect these features, plus dedicated non-GPU hardware will eventually reduce energy costs per "thought" to less than the human brain. Currently perplexity using Sonnet 3.5 thinks GPT4 uses 25x more.

Can you please look into alloy voice assistant

An implementation would be interesting

yes we need detailed video

Every single time I’ve seen a claim like this, the output in real usage never compares

When "data = code", how can you have security while having a actually useful / powerful AI?

Yes please, do the tutorial.

yes I vote for tutorial for set up please thank you

Full tutorial pls

Yessss! Go open source

Do you and Wes Roth collaborate? They uploaded an identically titled video 2 hours ago.

@matthew_berman

19 күн бұрын

No. He just copied me exactly...again

Let me launch this... Online models are saving themselves yet because of context.

It doesn’t change anything. LLMs are good at certain tasks (most of which aren’t as useful as we need, and most don’t help us earn money). AI has plateaued. They haven’t replaced software engineers.

95% of my queries, even GPT 4o or Sonnet 3.5 can't answer so I don't know what your queries are that local models usually handle fine.

Yes Please. a tutorial!

Please do a tutorial on setti g up

I really don't find this to be a big deal. I expect people select the model to use themselves on a per-task basis on what they believe is the most appropriate one for the task. For me the decision process is really simple: 1. is it code or requires complex problem-solving? -> Claude 3.5 Sonnet 2. Do I want to have a deep conversation with a creative partner -> Claude 3 Opus 3. Is it anything the other models would refuse? -> GPT-4o 4. Is it too private for any of the above? -> Local LLM I don't need a router for this and I wouldn't trust it to reliably choose the same way I would either.

Please do a tutorial !

Thank you Matt🖖🤖👍

Now, who has this in production?

If I could like and subscribe harder I would. Tutorial would be fantastic thanks 👍🏿

4:59 you lost me completely with langtrace! What does it do in why would I want it?

Can you make a comparison video with LangGraph X GraphRAG?

Yes, please.🙏

Tutorial please

While this looks promising, it is just a router that forwards simple queries to weak models while forwarding hard queries to strong models. This assumes that the queries can be divided between strong and weak models. If your work is truly intensive, I don't see much reduction here as it still requires querying strong models most of the time.

90% GPT4o Quality? More like 100% snake oil! Where do I sign up for this "RouteLLM" deal?

don't ask in the comments each time JUST DO IT !!!

do a tutorial on this please