📌 Code: github.com/aurelio-labs/semantic-router/blob/main/docs/encoders/openai-embed-3.ipynb MRL Paper (how OpenAI packed this perf into 256-d embeddings): arxiv.org/abs/2205.13147

This is actually perfect we were debatting what embedding model we should use at work yesterday, thanks for sharing!! ❤

@jamesbriggs

4 ай бұрын

also look into cohere, honestly, pretty good too - often better depending on use case :)

Love your content! Quick tip: Audio quality seems a bit off... I have to turn the volume way up and still have trouble hearing clearly.

*Summary* *Introduction to Semantic Routing and Embedding Models* - 00:00 Introducing the topic of semantic routing for AI agents. - 00:05 Discussing the use of OpenAI's third-generation embedding models. - 00:10 Examining the effectiveness of a reduced embedding size of 256 dimensions from the default 3072. - 00:27 Questioning if performance can still be better than GPT-3's Ada model with smaller embedding size. *Code Setup and Initial Testing* - 01:08 Using OpenAI's documentation to set up encoders. - 01:21 Describing the dimension feature introduction in encoder version 0.19 and OpenAI API version 1.10. - 01:34 Specifying the need for specific versions to use all features. - 01:46 Setting up protective routing to avoid undesirable responses from AI agents. - 02:27 Explaining that routing is based on semantic similarity, not just fixed phrases. - 02:51 Setting the OpenAI's text-embedding-3-large model to 256 dimensions to test its effectiveness. - 03:00 Mentioning that an OpenAI API key is needed to proceed. - 03:14 Defining a route layer with an encoder and predefined routes. - 03:36 Confirming the creation of 256-dimensional vectors with the route layer. - 03:49 Testing the model with example questions and verifying the correct routing. *Optimizing the Model* - 04:35 Suggesting that performance could be further improved by model optimization. - 04:41 Discussing threshold optimization using a larger dataset. - 04:57 Running threshold optimization and commenting on the different threshold values for the new model. - 07:28 Noting that optimal threshold values are significantly lower for third-generation models. - 07:59 Achieving an accuracy of 88.57% after optimization. *Comparing Embed 3 and Ada 002* - 08:45 Remembering the performance of the optimized Embed 3 model before comparing with Ada 002. - 08:48 Testing Ada 002 for performance comparison. - 09:00 Noting that it is just one test and not definitive of overall model quality. - 09:20 Starting both Embed 3 and Ada 002 from the same base point for fairness. - 09:57 Observing accuracy improvement possibilities. - 10:31 Reviewing updated threshold values and re-running the optimization. - 11:03 Comparing final accuracy of Ada 002 (87.4%) to Embed 3. *Conclusion: Embed 3 with 256-d Outperforms Ada 002* - 11:20 Concluding that in this test, the 256-dimensional Embed 3 model outperformed Ada 002. - 11:34 Emphasizing that this is based on one test and more extensive use is needed to form a solid opinion. - 11:53 Expressing surprise at the performance given the reduced embedding size. *Closing Remarks* - 12:02 Wrapping up the test and hoping the demonstration was informative and useful. - 12:09 Announcing the end of the video and promising to return in the next one. - 12:27 Closing with a goodbye. Disclaimer: I used gpt4-1106 to summarize the video transcript. This method may make mistakes in recognizing words

@jamesbriggs

4 ай бұрын

surprisingly good, thanks!

Great video, thanks for sharing ! Might be interesting to compare text-embedding-3-small 512 vs text-embedding-3-large 1024 ?

Thanks for the video James! Curious if you’d make a video on the latest langchain updates? Seems like they’ve made some significant changes for example to how streaming works.

@jamesbriggs

4 ай бұрын

yeah I know the team I work with has recently been swapping out the streaming stuff, planning to jump back into langchain stuff soon

Great video as always, James. Will the semantic router come to TS?

@jamesbriggs

4 ай бұрын

if it comes to other languages, TS will be the first

@dawid_dahl

4 ай бұрын

@@jamesbriggs Awesome! 😃🙏🏻

Do we also need to take care of completion also under this semantic routing tunnel ?

👍

the Americanisation of British English is truly sad.

@jamesbriggs

4 ай бұрын

you can't say I sound american 😂

@prashank

4 ай бұрын

Welcome to language!

@davidw8668

4 ай бұрын

Its the shirt 😂​@jamesbriggs

@jamesbriggs

4 ай бұрын

@davidw8668 maybe - I'll make sure to wear a tweed jacket and flatcap in future videos

@ArseniyPotapov

4 ай бұрын

Americanization 😅