Deepgram is the most slept on AI player.

@DataIndependent

Ай бұрын

yeah they have a ton of good data to work with

Thank you for creating this tutorial it's exactly what I was looking for. Great content!

Thanks Greg you do alot for the community ,I have respect for you did in semantic chunking in langchain repo

@DataIndependent

Ай бұрын

Nice!

Looks cool. Thank you for showing this idea. Will definitely implement it in my project.

@DataIndependent

Ай бұрын

Nice! Good luck! What is the project you're building?

@souravbarua3991

Ай бұрын

@@DataIndependent I didn't started yet. But I will start it soon.

I built a voice bot with this exact set up steaming and all quite a few months ago and the biggest issue was the latency! So excited to see this is no longer a Problem!

I experimented with integrating Groq with Siri, and it was quite interesting. The response time was pretty impressive.

this is exactly i am looking for . thank you youtube. love you lots

@DataIndependent

Ай бұрын

Nice!

I purchased a ReSpeaker Mic Array v2.0 for this purpose. It captures speech with great clarity. Works out of the box on Linux, so should be able to make a standalone voice assistant with it.

@DataIndependent

Ай бұрын

Cool that sounds good, please share when it's out

Great heads up on new STT/LLM/TTS technology. I had been working on a Unreal Meta Human demo which got pretty close to RT using a Google STT/ChatGPT/TTS. One of the other things to think about if one want's to get into 2D/3D talking head chat apps is streaming back viseme data as well as TTS audio. Plus maybe emotion tokens of some kind. I can't wait for all of this to be offered on one platform/api service.

This is exactly what I wanted to do this weekend. Great timing.

@damien2198

Ай бұрын

Same, I want to try to do translation (language identification per speaker / switching whisper model could be tricky)

@DataIndependent

Ай бұрын

nice to both of you!

@avi7278

Ай бұрын

@@DataIndependent I ended up pulling down your project and ran into a few issues, fixed them and made a few improvements like response interruption. It's pretty good but groq being limites to llama 2 and Mistral is a shame. It is at least able to fill the time of latency of the more powerful models so that at least it still feels more natural even if the first 10 seconds are filler until openai or Claude 3 ingests input and starts streaming.

@DataIndependent

Ай бұрын

@@avi7278 That's awesome! If you were open to doing a PR to add interruptions I would definitely want to share it with the community. Did you add filler too?

@avi7278

Ай бұрын

@@DataIndependent it's on my list for this weekend, cheers, appreciate the starter.

Great stuff! Did you not cut in the video to reduce the waiting time for the LLM response? If you did not, the speed is impressive! Thank you so much!

Great video, Greg!

Awesome, subscribed on the strength of this video.

@DataIndependent

Ай бұрын

Ha thank you!

Super fast example - really highlights the power of this tech

@DataIndependent

Ай бұрын

Big time Cam!

Thank you! Amazing!

Great video, Greg! You gave me some interesting ideas :)

@DataIndependent

Ай бұрын

Awesome - excited to see what you build

Wow - incredible!

@DataIndependent

Ай бұрын

Nice!

Thanks for this! Testing it now ... the TTS audio isn't streaming and im struggling to fix it but ... THANK You for sharing your code! Super helpful and informative video!

@TheColdharbour

Ай бұрын

I’m in the same situation, got the listening and response working great but no TTS. Spent all day breaking and fixing it 😂 still confused why it won’t talk! Great guide (I needed python 3.11 to get I back working and had some issues with dotenv, ended up hard coding the APIs) - it’s a great piece of work! 👍👍

@DataIndependent

Ай бұрын

Try these updates github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381

Awesome Greg! Best TTS-STT demo yet. Do you have any ideas on how to modify your example for two people having a conversation, and the AI participating as a third person. For example, debate students are debating and want the AI to be the judge to help them improve their debate skills. I would love to hear your thoughts on this. Thanks for this tutorial. I've been looking for this solution since the 90's!

conversational endpointing is a great idea, but I'd like to see that combined with a small model agent that was only looking for breaks in the conversation and an appropriate time to interject. Maybe with a crude scale for the length of the response. So if the user has a break in the point they're trying to make - we don't want the user interrupted and the conversation moved on - what would be more appropriate would be a simple acknowledgement. But once the point is complete, we would then pass back that we want a longer response.

@frothyphilosophy7000

15 күн бұрын

This. I need something like this for a project, but I'm not very familiar with Groq or Deepgram yet; just starting to dig in. This thing starts responding with the first little pause, so it constantly cuts me off when I'm just pausing momentarily to think of how I want to phrase the rest of my sentence. If it wants to send data at every minor pause in order to understand context, predict the full query, and begin formulating a response, that's fine-- but it needs to wait until I've finished my entire input before verifying/sending its response. Out of the box, this is like a person who doesn't actually listen to what you're saying and is just waiting for their turn to speak. Is there an easy way to affect the response times and/or understanding of when the user has finished a full thought or do I need to develop logic/rules from scratch?

@HideousSlots

15 күн бұрын

@@frothyphilosophy7000 not that I’ve seen. And this would be a massive leap in improving conversation. It literally just needs a small model to parse the text at every pause and see if it’s an appropriate time to interject. Just the same as a polite human would do. The groq api should be able to do it. I’m really surprised we haven’t seen this effectively enabled anywhere yet.

@frothyphilosophy7000

15 күн бұрын

@@HideousSlots Gotcha. Yeah, guess I'll need to implement that, as it's unusable otherwise.

Holy hell that is incredibly fast

@DataIndependent

Ай бұрын

Right? Ya it was super cool

Rasa NLU would be good for intention detection like greeting, closure , and the domain or custom intentions

@DataIndependent

Ай бұрын

I'm not familiar with that - thanks for sharing I'll check it out

This is great, thanks for the awesome demo. I’m not a tech person, but can I clone my sales agent’s voice using ElevenLabs and integrate it to this process? I guess I can train the LLM to respond in his tone and sales style? Thanks

@DataIndependent

Ай бұрын

Yeah you could for sure. It would take a bit of work back and forth but it’s there

Thanks! How does it compare with assemblyai universal 1? do you know which speech-to-text support arabic with the best accuracy in real time?

FYI you dont need to use an API for TTS or Speech to Text both can be run locally using Faster Whisper for TTS and Coqui for Speech even if you dont have the worlds most expensive GPU because both of those only use a couple of GB of Video Ram. Going forward, on device will be the way to go for TTS and STT because they simply do not require that much processing power.

@DataIndependent

Ай бұрын

Nice - what kind of latency are you getting with those? Along with accuracy

@vaisakhkm783

Ай бұрын

There is also Piper for tts, it's open source and wirks realy well in my non-gpu laptop

@moodiali7324

15 күн бұрын

there is a caveat with ur solution becuase fatser whisper does not detect silence out of the box and hence, you wouldnt know if the user has finished talking or not, which Deepgram does.

This is exactly what I was looking for. Thanks. But the text to speech functions seems to be not working in my windows. I tried writing the audio in a wav file (for debugging purpose), but the file is also can not be played. I thought may be codec issue, so I tried to convert the file in online to mp3, but got an error message showing " Invalid data found when processing input". Any idea, how can I make the text to speech function working? Another note: it does not show any error message in terminal, it just does not play the audio.

@ocin3055

Ай бұрын

Same issue here. Hoping for an answer, too.

@DataIndependent

Ай бұрын

Try again with these changes github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381

Hi Greg, Rutvik here. We have created something similar to this but using GPT as LLM and Elevenlabs as TTS. We are facing issues with Silence detection with Deepgram. I know you did mentioned in your video at 3:53 that we need to make sure that we dont talk too slowly. And unfortunately Deepgram only has MAX value of 500 ms for endpointing (Silence detection). Can you confirm if we are using proper configurations with Deepgram? Following are my configurations. 'punctuate': True, 'interim_results': True, 'language': 'en-US', 'channels': 1, 'sample_rate': 16000, 'model': 'nova-2-conversationalai', 'endpointing': 500

very nice, many thanks. latency of the llm still way too high. maybe 6-12 months to bring it down to

@DataIndependent great video! Do you know how to get ChatGrok / Mixtral working with Langchain 'bind_tools'? I'd love to swap out my ChatOpenAI for ChatGrok if possible!

@DataIndependent

Ай бұрын

I haven't tried that out yet - sorry!

do you know if any of the open source models (STT and TTS) with Groq give the same latency?

@DataIndependent

Ай бұрын

My guess is no - oss usually isn't as fast as paid

ElevenLabs turbo with deepgram stt. As tested. Cant beat it.

Can this be added to a website?

Amazing video.. Just wish it had a walk though of instructions

on windows, having some problems with libraries. After fixing all dependencies, etc. I think I'm having problems with ffplay for the computer to send the audio to the speakers. Taking a look. Somehow I can't listen to ffplay.

@RememberingAI

Ай бұрын

Debug the response from TTS without streaming first. I had to change the URL (it was some beta URL), they have changed it now and the one in Github right now returns an internal error (non 200 response), so the audio won't stream.

@DataIndependent

Ай бұрын

Both good points, I'll update the URL at least

@ferencdalnoki-veress163

Ай бұрын

@@DataIndependent I tried it on Ubuntu Linux and (I think I) also have a problem with ffplay. However, somehow while it does convert my voice to text and LLM responds with text it does not convert text to sound. To test it I did different checks to verify that the connection with Deepgram is working. I used a test script where I used the Deepgram API with a text message and then I streamed the audio directly back to ffplay for playback and it worked. So that is why I am puzzled why the code is not working on the Linux side. Any help is appreciated. I truly enjoy your thoughtful tutorials and videos.

@ferencdalnoki-veress163

Ай бұрын

There was an issue with payload. When I commented out the "voice": self.MODEL_NAME it worked. I also changed to pygame from ffplay on Linux. But that may not have been the issue.

Wow that was fast… Grok literally went open source 😮

@DataIndependent

Ай бұрын

I was able to sneak early access for the video

I suppose Whisper/ tiny model would be faster than this Deepgram ? have you tried?

@DataIndependent

Ай бұрын

Nope not yet - let me know how it goes

Thank you very much for the vide, how can I integrate RAG into this pipeline?

@DataIndependent

Ай бұрын

Put it in the LLM step, set up a retriever and then call to it

@zahaby

Ай бұрын

@@DataIndependent Yes, thank you very much. vectordb = FAISS.from_documents(all_splits, embeddings) retriever = vectordb.as_retriever() BUT LLM doesn't accept retriver param, So I used ConversationalRetrievalChain

Am i the only one not able to hear the TTS back even if i installed the ffmpeg? What am i mising? i tried both from vscode and from normal terminal.. nothing plays out, no errors though. connections to deepgram is checked and successful. Any hints?

@TheColdharbour

Ай бұрын

I’d also love to know this. I’m in the same boat! It works .. just silently for me too

@DataIndependent

Ай бұрын

I updated the code, try again or make these changes yourself github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381

Getting error when firing off the "request" in "speak()" - "err_msg":"Failed to deserialize JSON payload. Please specify exactly one of `text` or `url` in the JSON body." It works though, if I take the "voice" attribute out of the the "payload" AND also change the model to the model in their docs, which is "aura-helios-en". Other than that, thank you so much for sharing! It's hard work!

@DataIndependent

Ай бұрын

Thanks for this - I found this same problem. I had it working in beta (before the video game out) so these were the changes needed for their prod version. Updated!

@Todorkotev

Ай бұрын

@@DataIndependent Thanks Greg! Awesome content! I appreciate you!

The original Groq demo had an impressive speech demo. How did that work? Interviewer interrupted Groq repeatedly.

@DataIndependent

Ай бұрын

Can you link it?

@ScottSummerill

Ай бұрын

kzread.info/dash/bejne/ooaJxsaEZtbedMo.htmlsi=Kg15nRUEEr1AHTGx

I miss your videos!

@DataIndependent

Ай бұрын

More coming!

This app won't find my API keys either Groq or Openai though they are there. Too bad. Any suggestions greg?

@urglik

Күн бұрын

API's being found either!

I have tthis error C:\Users\vnt>pip install deepgram ERROR: Could not find a version that satisfies the requirement deepgram (from versions: none) ERROR: No matching distribution found for deepgram

@Todorkotev

Ай бұрын

You trying to use their python-sdk? Maybe try "pip install deepgram-sdk"

@DataIndependent

Ай бұрын

Yeah - the suggestion here is the key github.com/deepgram/deepgram-python-sdk?tab=readme-ov-file#installation

Could you help me understand what's going on. I'm running this in Docker and keep getting an error when it gets to running the .py: "ModuleNotFoundError: No module named 'langchain_groq'"

thank you so much for this video, i have a problem when i run the scipt : i have this error message :" Could not open socket: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997) " any help please

@DataIndependent

Ай бұрын

I had this problem a while back too and it was super annoying. The solution was a lot of googling and I think I might have even used it as a push to upgrade my python

@loryo80

Ай бұрын

@@DataIndependent This doesn't look good for me then, hhhhhh, was the solution obvious or did it require a lot of manipulation?

How’s this compare against VAPI ?

@DataIndependent

Ай бұрын

Haven't tried that yet

@mmdls602

Ай бұрын

Vapi is extremely expensive. I think in a couple of months we should be close to Vapis performance with open sourced models and tooling.

how could we interrupt the voicebot can anyone help (pls)

Is there any alternative for Spanish? I cannot find it.

how can we interrupt the ai??? plsss helpp

how to add those filler words?

Facing WARNING: API key is missing Could not open socket: server rejected WebSocket connection: HTTP 401 on Stt model

@DataIndependent

Ай бұрын

Ya…you’ll need an API key which I can call out better in the read me

@BitBlendAi

Ай бұрын

I have my own api key with 300$ credit. Still it's showing this error Can you please send me a screenshot of your .env(hide your api)file​@@DataIndependent

@BitBlendAi

Ай бұрын

I have my api key with 300$ credit but still it's showing this error massage. So can you please send me your screenshot of.env( hide your API)​@@DataIndependent

i cloned the code but response time is like 12 seconds. 4 weeks past and i'm late to the party

Is there any developper who can help me to set up this?

Showcase it within a streamlet app doing Langchain shenanigans! Please!

@DataIndependent

Ай бұрын

Like what kind of shenanigans?

@RADKIT

Ай бұрын

@@DataIndependent Starter difficulty Steamlit app using Langchain and Deepgramdeepgram that allows you to upload a pdf / embed then when ready we can simply chat with it live , asking a question , retrieving information Advanced difficulty / Aspirational An agent using function calling and a set of tools like web search / calculators etc ... if we can have an asynchronous continues conversation with one supervisor agent who asynchronously can "ask" other agents to do time consuming tasks, like being able to talk to the supervisor agent in Langgraph schema

Would be great for training lawyers to take depositions and do other oral tasks.

@J3R3MI6

Ай бұрын

Yep it would… But unfortunately lawyers are probably done for… like 99% of them.

@DataIndependent

Ай бұрын

It'll help augment roles ha, not replace

Doesn't this suck? I've been subscribed to Deepgram for quite a while now. You'd think they would have some good competition. Alas, none that have stuck around, and open-source is infested of dragons.

I'm not a coder, but can someone please build one of these that I can pay for so I can basically have my own Jarvis to talk to on my computer?

@DataIndependent

Ай бұрын

Have you tried this one? www.hume.ai/

as a total beginner, I can't find a way to run this.

@mnagy0101

24 күн бұрын

Same here..

I can't drop the LLM below 3000ms :( maybe im fucking it up?

Nice. But in reality your demo is 3,500-4,000 ms from when you stop speaking to getting a response. It does not match the numbers you are printing...

No good can come of this.....lol

@DataIndependent

Ай бұрын

yeah - it'll be tricky to navigate but there will be use somewhere

Sir can you provide full tutorial of this plzzzz