If you are interested in learning more about how to build robust RAG applications, check out this course: prompt-s-site.thinkific.com/courses/rag

Thank you bro. Today itself i switched the LLM in RAG to Llama - 3 8B. It is performing really well.

Excellent presentation.

@engineerprompt

2 ай бұрын

thank you.

I found that the llama-3-70b from groq does not do as well on the test rag task I ran versus a local version, so they might have quantized it a lot on groq.

@engineerprompt

2 ай бұрын

I have seen people saying that. That might be the case.

Would love to see a reliable way to utilize function calling on completely local model. I saw a fine tuned model on HF designed for function calling, but users said that it had issues Anyone know if this has been done locally? relatively reliably?

You want learn RAG beyond basics? Make sure to sign up here: tally.so/r/3y9bb0

No module named 'packaging' - does not work in windows or wsl

Function calling without groq would be cool. we are looking to self host 70B with OpenAI compatible functions/tools. so far there is nothing promising except Trelis' models.

@_SimpleSam

2 ай бұрын

Use grammars and any model can do function calling. Just need to struggle to get the "bnf grammar" perfect. Try to hard code as many characters as possible, it improves the quality of the output. Quickly you'll realize how inefficient the OpenAI JSON style format is, and you'll go down the parsing rabbit hole, trying YAML TOML, etc. Good luck!

@engineerprompt

2 ай бұрын

I have seen a few finetunes for function calling. Will cover some of them soon.

Can I run llm on 4 GB ram and 16 bit processor laptop? Please tell me

@premusic242

Ай бұрын

U can run but using open source locally will not perform well . Instead of using llama 3 locally using ollama use groq API it would be insanely fast

fully decensored?