This is gold, most of people just show you how to build toy demo, but not many actually get into details of how to get into production; Thank you Jason!
@xXWillyxWonkaXx
2 ай бұрын
Couldnt agree more. This is gold.
@apereiracv2 ай бұрын
I recently be created a whole testing system for our LLM chatbots and we did exactly this: LLM as evaluator and code We created it as a series of unit tests with LLM generated cases. Since our results were mostly conversational, we made tests pass/fail according to a scoring system
@tkp28432 ай бұрын
This is great. Loved the use of firecrawl (as a scrape tool) to get the website's data. Feel like it always helps improve the model output quality. Cheers!
@kenchang34562 ай бұрын
Way excellent video that goes well beyond demo. Thank you very much for this guidance.
@jasonfinance2 ай бұрын
Amazing work as always Jason!
@darrenhinde29712 ай бұрын
Been looking for more detail on eval on LLMs and been scratching around for a while. Thanks for this.
@humanish_ai2 ай бұрын
Finally you back 🎉
@kayshidow2 ай бұрын
I've used promptfoo for some of my test with local llm to test the ai workflow. It allow you to write assertion like you'll do with software
@contractorwolf2 ай бұрын
goddamn Jason your videos just blow my mind each time. Thanks for such a thorough explanation and example.
@titusblair2 ай бұрын
Awesome! Keep up the great work!
@someshfengade96232 ай бұрын
I found langfuse metric monitoring little bit better.
@agenticmark2 ай бұрын
fine tune llama 3 (8bit) - you will get exactly the behavior you want - its what I do
@techfren2 ай бұрын
lesgooo!! ❤🔥❤🔥❤🔥
@JorritvanGinkel2 ай бұрын
This is so good, thanks man!
@fullgazz2 ай бұрын
Who never spent 4 hours to save 10 min? That's our hobby spent time to save time.
@AGI-Bingo
2 ай бұрын
If 25 people or more use it successfully then you literally gave humanity more time to live and be free
@MatrixCodeBreaker882 ай бұрын
Great Video
@jordanz95802 ай бұрын
fireeee content!
@jimmy-ef2ow2 ай бұрын
jason can we get another video about comfy ui?
@CorkyBallasdancewithmeАй бұрын
great stuff, as new to hearing this, very interesting, can this be built by a novice . . .
@Joe-bp5mo2 ай бұрын
Sick, whats the best practice metrics for evaluating agents?
@Ms.Robot.2 ай бұрын
I love how my Ai girl insults the competion with flame balls,then tells me.she loves me.❤🎉😊
@user-nt7lj1nc8s2 ай бұрын
Why not use Gemini as the LLM? It is free.
@HyperUpscale
2 ай бұрын
Lets me share my experience about any google AI model ... because it doesn't understand human and it hallucinate way too much. Practically ... in my cases 75% of the time what I get back is totally useless result. You cant use for anything... To be considered for evaluation ... you must be joking
@irql22 ай бұрын
I dont see the value of "Agents". All of this stuff is easily done with basic function calling. I think I'm going to need to see some more creative use cases before I jump on board, i just dont get it yet.
@ayoubfr8660
2 ай бұрын
Maybe we can discuss this, I am trying to jump on in but not until I find a decent idea to apply.
@symbol9new
2 ай бұрын
when your assistant has a lot of functions, he starts giving out hallucinations, have you ever encountered this?
@SydneyF-eg5lt2 ай бұрын
Good content but so hard to listen to his Engrish. Monotonous Pitch n sped up delivery didn’t seem to help either.
Пікірлер: 31
This is gold, most of people just show you how to build toy demo, but not many actually get into details of how to get into production; Thank you Jason!
@xXWillyxWonkaXx
2 ай бұрын
Couldnt agree more. This is gold.
I recently be created a whole testing system for our LLM chatbots and we did exactly this: LLM as evaluator and code We created it as a series of unit tests with LLM generated cases. Since our results were mostly conversational, we made tests pass/fail according to a scoring system
This is great. Loved the use of firecrawl (as a scrape tool) to get the website's data. Feel like it always helps improve the model output quality. Cheers!
Way excellent video that goes well beyond demo. Thank you very much for this guidance.
Amazing work as always Jason!
Been looking for more detail on eval on LLMs and been scratching around for a while. Thanks for this.
Finally you back 🎉
I've used promptfoo for some of my test with local llm to test the ai workflow. It allow you to write assertion like you'll do with software
goddamn Jason your videos just blow my mind each time. Thanks for such a thorough explanation and example.
Awesome! Keep up the great work!
I found langfuse metric monitoring little bit better.
fine tune llama 3 (8bit) - you will get exactly the behavior you want - its what I do
lesgooo!! ❤🔥❤🔥❤🔥
This is so good, thanks man!
Who never spent 4 hours to save 10 min? That's our hobby spent time to save time.
@AGI-Bingo
2 ай бұрын
If 25 people or more use it successfully then you literally gave humanity more time to live and be free
Great Video
fireeee content!
jason can we get another video about comfy ui?
great stuff, as new to hearing this, very interesting, can this be built by a novice . . .
Sick, whats the best practice metrics for evaluating agents?
I love how my Ai girl insults the competion with flame balls,then tells me.she loves me.❤🎉😊
Why not use Gemini as the LLM? It is free.
@HyperUpscale
2 ай бұрын
Lets me share my experience about any google AI model ... because it doesn't understand human and it hallucinate way too much. Practically ... in my cases 75% of the time what I get back is totally useless result. You cant use for anything... To be considered for evaluation ... you must be joking
I dont see the value of "Agents". All of this stuff is easily done with basic function calling. I think I'm going to need to see some more creative use cases before I jump on board, i just dont get it yet.
@ayoubfr8660
2 ай бұрын
Maybe we can discuss this, I am trying to jump on in but not until I find a decent idea to apply.
@symbol9new
2 ай бұрын
when your assistant has a lot of functions, he starts giving out hallucinations, have you ever encountered this?
Good content but so hard to listen to his Engrish. Monotonous Pitch n sped up delivery didn’t seem to help either.