More suggestions: add a "thought completed" detection layer that decides when the user has finished speaking based on the stt input so far (based upon context and natural pauses and such). It will auto-submit the text to the AI backend. Then have the app immediately begin listening to the microphone at the conclusion of playback of the AI's tts-converted response. Yes, sometimes the AI will interrupt the speaker if they hadn't entirely finished what they wanted to say, but that is how real human conversations work when one person perceives the other has finished their thought and chooses to respond. Also, if the user says "What?" or "(could you) Repeat that"" or "please repeat?" or "Say again?" Or "Sorry I missed that." the system should simply play the last WAV file again without going for another round trip to the AI inference server and doing another tts conversion of the text. Reserve the Control-C for stopping and starting this continuous auto-voice recording and response process instead. This will shave a many precious milliseconds of latency and make the conversation much more natural and less like using a walkie-talkie.

@SaiyD

3 ай бұрын

nice let me give one suggest to your suggestion. add a random choice with 50% chance to re play the audio or send your input to backend.

@ChrizzeeB

3 ай бұрын

so it'd be sending the STT input again and again with every new word detected? rather than just at the end of a sentence or message?

@deltaxcd

3 ай бұрын

I have better idea to feed it partial prompt without waiting user to finish and it starts generating response if there is a slightest pause if user continues talking more text is added to the prompt and output is regenerated. If user talks on top of speaking AI. Ai terminates its response and continues listening this will improve things 2 fold because moel will have a chance to process partial prompt and it will reduce time required to process the prompt later if we combine that to now wasting for full reply conversation will be completely natural there is no need for any of that say again because AI will do that by itself if asked

Some of the best work and fun that I have seen so far. Can't wait to try on my own. Keep up the great work!!

Awesome! Time to replace my slow speech to speech code using openAI. Also, added eleven labs for a bit of a comedic touch. Thanks for putting this together.

Just subscribed! can't wait to get my hands on it, looks super cool!

Hey Kris - that is awesome. I like it very much. Great that you do this open source stuff. Very cool 😎.

very impressive! I'd love to see them implement this in smartphones for real-time translation when visiting foreign countries / restaurants.

@optimyse

3 ай бұрын

S24 Ultra?

@deltaxcd

3 ай бұрын

there are models that so speech to speech translation

very cool, low latency voice, thanks for sharing, i watch all your videos, and i look forward to the next one,

Thats supah cool, I actually built something almost exactly like this yesterday. I get about the same performance. The hard part is needing to figure out threading/process pools/asyncio. To get that latency down. I used small instead of base. I think I get about the same response or better.

@user-rz6pp5my4t

3 ай бұрын

Hi ! Very impressive !! Do you have a github to share your code ?

@CognitiveComputations

2 ай бұрын

can we see your code please

@limebulls

2 ай бұрын

Im interested in it as well

I am still using Tortoise but Open Voice seems to be promising! 😊 Thanks for this video!! 🎉🎉🎉

I've been trying to figure out how to do this. Great job. I want to support your work and get this up and running for myself, but is KZread membership the only option?

I have tried open voice and bark, but VITS by far makes the most natural sounding voices.

In the US we have this concept, if you watch a football game which is notorious for having a shizload of commercials (ie latency), if you start watching the game 30 minutes late but from the beginning, you can skip most of the commercials. If you just shift the latency to the beginning, 15 seconds of "loading" would probably be sufficient enough for a 5-10 minute conversation between the two chatbots, and also avoid loops by having a third party observer who reviews the last 5 messages and determines if the conversation has gone "stale" and interjects a new idea into one of the interlocutors.

Just earned yourself a sub sir!

It will be very interesting to see this in a web application

I mean, this is dependent on your hardware... Are the specs anywhere for this "inference server"

Good job. Interesting video

I know about this for more than a year now and it still blows my mind. wtf

this was more comedy show than tech , lol. so hilarious responses from johnny.

This is great. But personally I think a speech recognition with push to talk or push to toggle talk is most useful.

wow very cool! Thanks

This was great 😃👍

Fascinating. Any chance you could upgrade it so that specific voices could be used and a recording made automatically, Could make for some interesting Biden v Trump debates

I wonder if you are (or can, if not) caching the processed .mp3 voice model after the speech engine processes it and turns it into partials. That would cut out a lot of latency if it didn't need to process those 20 seconds of recorded voice audio every time. Right now it's pretty fast but the latency still sounds more like they are using walkie talkies than speaking on a phone.

@levieux1137

4 ай бұрын

it could go way further by using the native libs and dropping all the python-based wrappers that pass data between stages using files and that copy, copy, copy and recopy data all the time. For example llama.cpp is clearly recognizable in the lower layers, all the tunable parameters match it. I don't know for openvoice for example however, but the state the presenter arrived at shows that we're pretty close to reaching a DIY conversational robot, which is pretty cool.

@JohnSmith762A11B

4 ай бұрын

@@levieux1137 By native libs, you mean the system tts speech on say Windows and macOS?

@levieux1137

4 ай бұрын

@@JohnSmith762A11B not necessarily that, but I'm speaking about the underlying components that are used here. In fact if you look, this is essentially python code built as wrapper on top of other parts that already run natively. The llama.cpp server for example is used here apparently. And once wrapped into layers and layers, you see that it becomes heavy to transport contents from one layer to another (particularly when passing via files, but even memcpy is expensive). It might even be possible that some elements are re-loaded from scratch and re-initialized after each sentence. The python script here appears to be mostly a wrapper around all such components,working like a shell script recording input from the microphone to a file then sending it to openvoice, then send that output to a file, then load another component with that file, etc... This is just like a shell script working with files and heavy initialization at every step. Dropping all that layer and directly using the native APIs of the various libs and components would be way more efficient. And it's very possible that past a point the author will discover that Python is not needed at all, which could suddenly offer more possibilities for lighter embedded processing.

Interesting, this is similar to a small project I made for myself. But instead of a chatbot conversation, the whisper output is fed into SAM (yes, the funny robot voice) and sent to an audio output. Basically makes SAM say whatever I say with a slight delay. I'm chopping up the speech into small segments so it can start transcribing while I speak for longer, but that introduces occasional weirdness, but I'm fine with that.

thanks, good project. Whisper can translate my spanish to english to spanish directly with little change in code? and tts i need change something also? thanks!

Could you get it to read information and context from a video game, like X4: Foundations, so that you could ask it like a personal assistant to help you manage your space empire?

I'm daunted by the idea of setting up these development systems just to use a model. Any chance people can bundle them into one big executable for Windows and iOS? I sure would love to just load-and-go.

I really like it! It this already on Github for members (could not find it)?

Hello, please can inform to us what is your cpu, gpu, ram etc?

I am trying to clone a voice from a reference audio file, but despite following the standard process, the output doesn't seem to change according to the reference. When I change the reference audio to a different file, there's no noticeable change in the voice characteristics of the output. The script successfully extracts the tone color embeddings, but the conversion process doesn't seem to reflect these in the final output. I'm using the demo reference audio provided by OpenVoice (male voice), but the output synthesized speech remains in a female voice, typical of the base speaker model. I've double-checked the script, model checkpoints, and audio file paths, but the issue persists. If anyone has encountered a similar problem or has suggestions on what might be going wrong, I would greatly appreciate your insights. Thank you in advance!

This gave me an interesting idea. Once could build streaming LLMs that at least partially build thoughts one word at a time (I mean the input, not the output). Basically precomputing most of the final embedding with an unfinished sentence, and if it has the full sentence and it's time to answer, it only has to go threw just a few very low latency, very cheap layers. Different but related idea: Similarly you could actually feed unfinished senteces into Mistral with a prompt that says "this is an unfinished sentence, say INTERRUPTION if you think it is an appropriate time to interrupt the speaker", to make the voice bot interrupt you. Like a normal person would. Would make it feel much more natural.

@deltaxcd

3 ай бұрын

Actually AI can do that you can feed it partial prompt let it process it then acd more and continue from where you left. thats huge speedup. but prompt processing is pretty fast anyway to make it respond faster you need to let it speak before it finishes "thinking"

I wonder if you could directly convert embeddings to speech to skip text inference

If you are fine venturing into c# or c++ then I know how you can improve the latency and create a single .exe that includes all of your different parts here, including using local models for the whisper voice recognition. I have done this myself using LLama sharp for runnign the GGUF file, and then embedding all external python into a batch process which it calls.

@matthewfuller9760

Ай бұрын

code on github?

@gabrielsandstedt

Ай бұрын

@@matthewfuller9760 i should put it there actually. I have been jumping between projects lately without sharing much. Will send a link when it is up

@matthewfuller9760

Ай бұрын

@@gabrielsandstedt cool

wow! I would love to have such an assistant to practice languages. The “python hub” code, do you plan to share it at some point?

Is it also possible to adjust to different languages?

Hey cool…anyway to run this self hosted for an online speech to speech setup? Want to drop this into a chatbot project…what level membership to access the code thanks

Hi, I´d like to know the computer specs required to run your speech to speech system, I m quite interested but I need to know first I my computer can handle it. thanks.

do you know of any offline/local way to do translations? i've been searching but haven't found a way to do local translations of video or audio using LargeLanguageModels

@deltaxcd

3 ай бұрын

there is a program "subtitle edit" which can do that

I run the oobabooga silero plus whisper, but those take forever to make voice from text, especially silero.

Nice! I was about to create something like this for myself but I still couldn't use OpenVoice because I keep failing to run it on my venv instead of conda.

@Zvezdan88

4 ай бұрын

How do you even install OpenVoice?

really interesting, lot of it i can't understand because I don't know coding but speech to speech could be a big thing within few years

Kinda feels like something the "rabbit R1" does with the whole fast speech to speech thing

how much RAM do you have to run mistral 7b locally? using gpu or cpu?

looks interesting but where is this community link you mentioned? :)

How can i do this with the Apple M1, this is soooo awesome i need to figure it out!

Can I configure it in Spanish? so that Mistral speaks Spanish and open voice in Spanish? I would like to confirm this to join as a member and access the github and try to make it work since my native language is Spanish, thank you for your work, it is incredible, you deserve many more followers, keep it up.

does openvoice perform better than whisper's TTS?

hi I dont have talk.py, but is there another way of running it im missing?

is there anyway that you can give the python 90% of system resources so it would be faster

this is awesome, but voice could use some fine tuning to sound more realistic

can the llm handle being told in a system prompt that it will be taking in the sentences in small chunks? say cut up into 2 second audio chunks per transcript. Can the mistral model do that? Anyway if so you might even be able to get it to 'butt in' to your prompt. now thats low latency!

@deltaxcd

3 ай бұрын

No it cant be told that but it is not necessary. just feed it the chunk and then if user speaks before it managed to reply more restart and feed more

Where can i find whole GitHub repo ?

Would love to see some realtime animations to go with the voice, could be a face, but also can be minimalistic (like the R1 rabbit).

@wurstelei1356

4 ай бұрын

You need a second GPU for this. Lets say you put on Stable Diffusion. Displaying a robot face with emotions would be nice.

@leucome

3 ай бұрын

Try Amica AI . It has VRM 3D/vtuber character and multiple option for the voice and the llm backed.

@fire17102

2 ай бұрын

​@@leucomedoes it work locally in real time?

@fire17102

2 ай бұрын

​@@wurstelei1356Again, I think a minimalistic animation would also do the trick , or prerendeing the images once, and using them in the appropriate sequence in realtime.

@leucome

2 ай бұрын

​@@fire17102 Yes it can work in real-time locally as long as the GPU is fast and has enough vram to run the AI+Voice. It can also connect to online service if required. I uploaded a video where I play Minecraft and talk to the AI at same time with all the component running on a single GPU.

I think at even 1/3rd the speed with my rtx titan it would run just fine to learn a new language. Waiting 3 seconds is perfectly acceptable as a novice language learner.

Do you have a video for showing how to do this step by step? I was going to be paid member but I couldn't see how to video in your paid channel?

no api = pure genius

How I can try this in my computer?? I couldnt find the talk.py in github code??

would be funny if you had this in games - like the people on the streets of gta having convos fueled by somthing like this. maybe it's already happening tho, i'm not in the know. awesomesauce!

Great tutorial but I wish you could share gists or share your code

So if you disconnect your computer from the Internet, will it still work?

@jephbennett

2 ай бұрын

Yes, this code package is not pulling APIs (which is why the latency is low), so it doesn't need internet connection. Downside is, it cannot access info outside of it's core dataset, so no current events or anything like that.

the predefined answer timing is what makes it not real conversation. It should spit answer questions at random timings like any human can think of something and only then answer. Randomizing timings would create more realistic conversations

Why LM Studio over OogaBooga? What are the pros/cons of them? I have been using Ooga, but wondering why one might switch.

Running on 4080 🤣 makes sense the damn thing is very fast

Surely the response time is a function of what rig you are doing this on - an RTX 4080 as you have is no doubt a major contributor here, and I would guess you have a beast of a CPU and high speed memory on a newer motherboard.

I have the uncensored model the same one and when I ask it to list curse words it says it can't do that. ???

@jungen1093

3 ай бұрын

Lmao that’s annoying

What GPU are you running?

@AllAboutAI

3 ай бұрын

4080 RTX!

I think to decrease latency more you need to make it speak before AI finishes its sentence unfortunately there is no obvious way to feed it partial prompt but waiting until it will finish generating reply takes asy too long

Thanks . Is whisper api free ?

@m0nxt3r

11 күн бұрын

it's open source

I look forward to actual speech to speech LLM, not any speech to text translation layers, pure speech in and speech out, it would be revolutionary imo

❤❤❤🎉 nice

It would be interesting to make a real-time translator. I think it could be very useful. The language barrier would end.

@deltaxcd

3 ай бұрын

meta didi it already they created speech to speech translation model

I swear on my mother’s grave lol… this AI is hilarious! 😂😂😂

BRUTAL! hahaha

Anybody done with this and have a code or link can share please

I would like to see how a chat room full of different models would problem solve... ChatGPT + Claude + * 7B + Grok + Bard... all in a room, trying to decide what you should have for lunch

How can I install this on my PC? I am a member of the channel

@AllAboutAI

Ай бұрын

did you get the gh invite?

@jerryqueen6755

Ай бұрын

@@AllAboutAI yes, thanks

@miaohf

Ай бұрын

@@AllAboutAI I am a member of the channel too, how to get gh invite?

Just to paraphrase your models: “Dude ! Are you actually grabbing the gorram scrollbars to scroll down an effing window !? What is this? 1996 ? Ever heard of a mouse wheel? You know it’s even emulated by double drag on track pads, right?” 🤘

we can make youtube content with those conversation between bots😂❤

you need to get out more man lol. that was toooo much!

I dont see Dan.mp3

can someone please give me that source code

Just wao

This made me laugh so hard!

Jumanji Alan

How is a system that goes out to openAI, "local" ????????

@seRko123

13 күн бұрын

Open air whisper is locally

Try fast-whisper. Open source and faster

AI: "We got some rich investors on board dude, and their willing to back us up!" I think this script just announced the games commencing in the 2024 US Election... [not in the US so reaches for popcorn]

the voices are Mehh...cool project tho. You always have some fire content. You could train a LLM just off your content and be set haha.

haha AI conversations could have their own chanels

Dude just made a JARVIS embryo.

So where's the source code mate? Or is this just a hook for your newsletter marketing and crap website?

@Skystunt123

2 күн бұрын

Just a hook, the code is not shared.

Could make for some interesting Biden v Trump debates

Sadly this video has fewer hits than it should have. I am looking forward for a more automated version of this. Hopefully the low amount of views wont hinder it.

now add a metahuman and auidio2face :)

Thought this was really cool, love open source. But this really isnt open source if youre hiding it behind a pay wall... smh

Too bad you take open source and make it closed.

@mblend27

3 ай бұрын

Explain?

@NoLimitYou

3 ай бұрын

@@mblend27 You take code openly available, and ask people to become a member, to receive the code of what you demo using the open source code. The whole idea of open source is that everyone contributes without putting it behind walls

@Ms.Robot.

3 ай бұрын

You can in several ways.

@NoLimitYou

3 ай бұрын

You take open source and make something with that and put it behind a wall.

@TheGrobe

3 ай бұрын

@@mblend27 You make someone pay to access something on github you comprised of open source components.

its great but OpenVoice doesn't allow for commercial use. Would be nice to do this with a model that can be used for commercial use

Do not make AI lie on your face, man. Thankfully this is local.

It is funny.

WORST OPSEC for a hacker. :D