the holy grail problem to prompt it with is one in which each contributor gives different advice, and the decider model produces output that's different from all the contributors and different than what the same model would have produced alone without MoM

@mwdcodeninja

24 күн бұрын

Give each actor its own motivation. Like a murder mystery. The king is the detective. Have it play Clue.

@techpiller2558

24 күн бұрын

Different personalities would be good.

@EinarPetersen

23 күн бұрын

I assume this is all on the GitHub repo to play with

@EinarPetersen

23 күн бұрын

Wrong thread

I loved the "crucial insights" into gravity 😂

@AllAboutAI

23 күн бұрын

😂😂

Good, but as noted in the comments it will be interesting to see the results for "plain" GPT4 and Claude for the same set of problems

@AllAboutAI

23 күн бұрын

yeah, kinda regret not inc a basetest

Kris, That’s an extraordinary project with surprising results. Thank you for sharing your talents.

Cool experiment! I'd like to see a sort of "deliberative democracy" architecture--where the models have a chance to discuss back-and-forth or to see one another's initial answers before providing a final answer and then voting

Perhaps it would be even more powerful to make a Chain of Models/Chain of Experts architecture that has a line of models and each model builds upon the previous models' responses. Making a Network of Models would be even better, where you have layers of multiple models and each next layer gets the previous layer's outputs and ultimately combines everything in the end to form an answer to your question.

@AllAboutAI

23 күн бұрын

very interesting, noted, tnx:)

@c.d.osajotiamaraca3382

22 күн бұрын

If just one agent is wrong the entire work product crumbles. Wouldn't you need a hallucination check, or some kind of redundancy at each stage?

@hypercoder-gaming

22 күн бұрын

@@c.d.osajotiamaraca3382 Yeah, I didn't think of that. Maybe in the system prompt, it could say there is a chance that the output of the previous model(s) is incorrect and to consider that before making a decision. Or between each layer it could have an internet search if applicable. Not sure how I would program that as I don't know how to do HTTP requests but it seems possible.

Excellent...It'd be great if the models could have been run in parallel. This way, less time would be needed.

@AllAboutAI

23 күн бұрын

great idea, will def check into this!

Very interesting you are not only thinking about such solutions but actually programming and testing them. Which is really great! What about this architecture: King architecture BUT with a twist that you will execute question batch with king being first GPT4, then Opus, then llama3 70b and you will then chose only answers with majority of votes ie if GPT, Opus agree.

@tubaguy0

24 күн бұрын

And if they don’t agree, loop back and resolve the differences via Socratic method until there’s consensus on either a solution or a recommended plan of action to improve the input for the models to try again. I wonder if the local models could reason out the problem super cheaply if given enough reflection and conversation iterations, or if the general models are just too effective to leave out. Probably use case dependent.

@AllAboutAI

23 күн бұрын

cool idea, noted :) tnx

@prolamer7

23 күн бұрын

@@AllAboutAI :-)

Excellent idea! I understand Opus and OpenAI as the overseer, but if local LLM could proof being the heroes here, would be supberb!

@AllAboutAI

23 күн бұрын

true!

You could combine tree of thought into this mixture of models Where instead of having one model pretend to be three different experts, you use a tree of thought process across these different architectures to generate a more accurate output across the entire task

@AllAboutAI

23 күн бұрын

great idea :) tnx

This was really interesting, I really like your thought process. One suggestion I have to validate your results is to run each of your tests 3-5 times and see how consistent each architecture is, since you’re going to get different outputs each time. I also didn’t see whether you were controlling model temperature in your setup, and I wonder if putting a temp 0 for at least one participating model as a sanity check would improve output (another opportunity to run a batch test battery). This cuts really close to what I’ve been imagining for agent framework cooperative problem solving, so thank you again for showing us how to put ideas into code.

@AllAboutAI

23 күн бұрын

yeah good idea, thnx :)

Damn this was a beautiful experiment! Really really can see this being used effectively on a grand scale. I mean even think of the little LLMs if we had those all deciding and the larger one is Llama3 or something we can run 100% locally. This can be really powerful for those with less GPU capabilities and low funding if they run them sequentially.

that's epic man! well done.

Really interesting! Like others mentioned, even if the added reliability of a MoM architecture, it may not be worth the computing power and additional context length, and GPT-4 might be strong enough to figure most of these out on its own. I think a good compromise would be using less LLMs in the hierarchy

@AllAboutAI

23 күн бұрын

yeah, or you could just use OS models with llama.cpp or ollama i guess

Looks like the Ai jury is about to be seated in a courtroom. and voilà!

The democratic arch should probably have a step where you have one strong model that decides which answers are identical and remove all but one of them before voting. Otherwise you will get a situation where there are maybe 7 correct answers and one wrong, then the wrong answer has an unfair advantage since the models will spread their votes on the 7 correct answers, but not the one wrong answer. Also, solutions like this is where Groq with 300 tokens/s really shines.

@AllAboutAI

23 күн бұрын

great idea, will try this. tnx

Amazing idea, and excellent result. Well done!!

@AllAboutAI

23 күн бұрын

thnx :)

Thanks for sharing your research! I had an idea about pairing a large model with a tiny model, where the tiny model is asked a question and the large model evaluates the response. The large model identifies any issues and iteratively adjusts the prompt to help the tiny model get the right answer. Not sure if it would work in practice, but if it did maybe a tiny (and cheap) model could be made effective to solve more problems than it otherwise could have?

Add a gamifying token layer to learn which models' votes should be heeded/ignored in which situations

@AllAboutAI

23 күн бұрын

yeah cool idea, noted

Great idea! Like your experiments. Keep up the great work!

@AllAboutAI

23 күн бұрын

thn mate :)

Seems like the second and third method are both kind of getting the best answer of each model, which makes it kind of average. The first architecture on the other side takes all answers into a consideration, adds all insights and creates the best answer from them all. So in a sense it gets only the good stuff and excels the most. This is the way.

@justtiredthings

23 күн бұрын

You could prob achieve that with a more deliberative democratic architecture

I love these experiments soo much.

@AllAboutAI

23 күн бұрын

thnx :)

This was a fantastic video thank you...brought me up to speed on the various models and structures of their hierachys was fascinating. Not dabbled in agents for a while so was nice to be surprised at progress in this area vs my early attempts that spent tokens and produced nonsense.... question....do you have any teams of agents setup doing actual real world useful stuff? (I'm sure I'll find out by watching more). Thanks for the video.

🤣😂 I just have to say it. Monarchy seems to be the solution for most problems. However jokes aside, this experiment turned out really great, and fascinating how the models behave in the different setups and interactions.

@someverycool4552

19 күн бұрын

Monarchy would be best only if the king is also by far above others in mental capacity, in all areas.

@jiyuhen

19 күн бұрын

@@someverycool4552 Agreed

@youtuberschannel12

14 күн бұрын

​​@@jiyuhenHis demo is not really a monarchy. In a real life a king is able to have his own ideas or solution, he has the freedom to completely ignore proposal put forth to him. But in this demo the king doesn't have any of those freedoms and powers. He has to judge and decide what others put forth to him. Let the king have his own ideas and the freedom I mentioned earlier and I bet the results will be different.

Next projekt is a Mo(MoM) model where we compare the result of all 3 MoM systems with these systems again. Think of recursively going down the rabbit hole.

@YT_Jx

24 күн бұрын

lol

Great content!

You should've add a baseline with just gpt-4 turbo and compare.

@AllAboutAI

23 күн бұрын

100%

wow, you have an innovative mind. 🤓

King is King!

I think that it would be better to estimate performance when you have some simple model at the output level. GPT-4 can operate with all of provided problems without previous models solutions

@AllAboutAI

23 күн бұрын

yeah true, will try diff configs for this setup

actually, the most likely place for the marble to be is on the floor, because it is no easy feat to turn a cup with a marble in it upside down and put it on a table without the marble falling out before.

@AllAboutAI

23 күн бұрын

that would be the big brain human answer yes hehe

Great experiment!! I wonder if adding in reasoning to their answer, if democracy would vote differently? Or if the answers could be discussed before voting. Having a bunch of AI/people voting on bad assumptions is never good. "before voting, discuss each answer for a) meeting all the requirements and b) being the most accurate".

@AllAboutAI

23 күн бұрын

interesting =) noted for future itteration

how exciting what an awesome video thank you

@AllAboutAI

23 күн бұрын

thnx :)

A note: the marble problem has actually been solved in Matthew Berman's testing by WizardLM-70B-V1.0-GPTQ, Mistral 7b OpenOrca, Mixtral, Mistral Next, Mistral Large, and LLaMA 3 70b Groq. That is however only 6 of the 32 models he's tested. I will note that in my own testing Reka Core passed the marble test while it failed when Matthew was testing it.

Makes sence we create theoretically using something between fast and slow thinking. Maybe something like q* needs to use Claude 3

I was following the explanation In the beginning about the different architectures very well, until he opened the code😂

@AllAboutAI

23 күн бұрын

haha, its not that bad

How much did each "tournament" cost in terms of tokens?

@AllAboutAI

23 күн бұрын

since 8 models was local on ollma, not much =)

So this is all in the GitHub repo to play with I assume?

Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.

22:38 it only got one vote because each model provided 10 unique answers. This shows that the democracy model isn’t a good architecture for all kinds of problems (specifically, the ones where the answer is determinate).

thats great can you please give us paying users access to the github

I would probably choose to do the tally locally using an optimized solution a simple python script bypassing any AI that way you eliminate the models sucking at math and use architecture specialized for solving the math problem of counting 😊

lovely

Waw this much be so expensive lo but great

The api call cost will go out of roof.

I came to the same conclusion that arbiter currently works best.

You feed the answer into the next model and ask it is true or false

means that autocratic systems are better compared to democratic - lol

@AllAboutAI

23 күн бұрын

haha lol

All seems rather pointless without comparing to the results by gpt4/opus on their own ...

@AllAboutAI

23 күн бұрын

yeah that was my regret, should have done a "basetest" to comp

are you sure chatgpt4 wouldn’t just solve all these problems without the other models

@MarkoTManninen

24 күн бұрын

I was thinking same. So the problems should be more adequate. But it was a proof of concept, very nice work. I planned aomething similar with websocket agents last summer, but it became too expensive to test for a hobby. Now it is right time because ollama!

@YT_Jx

24 күн бұрын

That’s a great question. In the past at least, ChatGPT4 failed the marble and the apple questions.

@justincase4812

24 күн бұрын

Do you cut wood with a hammer?

@jichaelmorgan3796

24 күн бұрын

Different models have slightly different training, context, and perspective, so this creates a broader perspective to approach the problem at hand. This will not be practical for everyday prompts.

@DarrenAllatt

24 күн бұрын

@@jichaelmorgan3796 I disagree on the practicality point. The way it is Setup in this video from a testing the methodology perspective yes it’s not practical.. But behind it is actually very practical. Because all of this stuff that is going on can be done in code so the user doesn’t see everything that goes on behind-the-scenes and just gets the final output

The marble problem proves nothing. Every model vendor cheats on some problems and sticks these problems into the training datasets. Same for other problems. You need to use new problems every time. Otherwise these reviews are b.s.

you need to go deeper into their reasoning because some of the wrong answers are actually right

your mom

@YT_Jx

24 күн бұрын

@william5931, your dad. :)

This video is a whole lot of fluff for a very simple concept: combining outputs from multiple LLMs to get better answers. The creator makes it seem revolutionary with fancy names like "King," "Duopoly," and "Democracy," but it's just basic ensemble methods. They feed the same problem to a bunch of LLMs (including weaker ones like Llama), then use those answers as context for a stronger LLM (GPT-4) to make the final decision. The "Duopoly" is just GPT-4 and Claude arguing about the best answer, and the "Democracy" lets all the LLMs "vote" on the best answer. He doesn't go into detail about how the voting works or what constitutes a good vote. The results are a mixed bag, sometimes better, sometimes worse than just using GPT-4 alone. The real takeaway is that ensemble methods can be useful, but this video overcomplicates it with theatrical names and presentation.

Why is nobody talking about "Where's Daddy?" or "Lavender" or "Gospel"? these are the most advanced AI systems and are already committing mass murder!

@tellesu

24 күн бұрын

Why do you bots make the same stupid fearmongering comment on every post?

@aaronpainting5643

24 күн бұрын

@@tellesu I'm not a bot, I just want to hear smart people like this guy break it down. more people need to be talking about this, it is AI development towards Armageddon

@illuminated2438

24 күн бұрын

@@tellesu that's not fear mongering it is simple fact. A certain tribal state is using advanced AI to engage in the mass murder of civilians without any human oversight. The where is daddy program literally watches targets leave so-called military locations and orders an attack only after they are at home with their family. It is deliberate mass murder of civilians, and it is powered by AI.