Well there is also the consideration of integrating both, then depending on the task mix and match them or specific use of semantic search vs traditional keywords. Many vector stores allow pre-filtering the data based on the metadata before the semantic search is performed, this is an example of using traditional KV + Similarities searching. With it being not that much of a hassle implementing it, it's worthwhile to keep the tool in mind, large search engines like Google have been using it for years, they just never used it for AI augmented search until now, along with other tech giants. I'd say the few places where vector database shouldn't be used is when exact data is expected, but as soon as context is a variable intent in search, KV is not good enough.

@DataIndependent

10 ай бұрын

Totally agree - good call

absoulutely agree, some use cases, traditional data base query will be sufficient.

Hi Greg, another well structured and insightful video thanks. I have been experimenting with practical large custom KB types apps and would 100% agree. What I have settled on now is a combined, multi-layered approach incorporating standard relational DB lookups and vectorstores, which seems promising. Also, by extracting keyword and other metadata on ingestion and adding these to the vector embeds it is possible to do relational DB queries as a “first pass” then use these results to filter the vector search retrieval through metatdata, therefore providing a more fine tuned context for the semantic search before passing all this to the LLM. I have also recently developed a technique to parse PDFs by semantic structure (i.e. headers -> sections) and using this to split docs for ingestion, as opposed to arbitrary size-based chunking. This further improves relevance and semantic coherence within the semantic search and retrieval and ultimately the LLM context, also leveraging the new 16k token window. I will hopefully be able to share the basics of the technique soon, if you are interested?

@DataIndependent

10 ай бұрын

First off, thanks for the awesome comment and sharing! Second off, yes! I think both would be interesting to read about, but I'm a big fan of the 2nd (chunking) because I see this problem all over.

@sethhavens1574

10 ай бұрын

ok dude no probs - what is the best way to contact you to share the repo when ready?

@MarxOrx

7 ай бұрын

@@sethhavens1574I’m interested too and I need this for a project a non profit project. Please do share!

Great explaination.

@DataIndependent

10 ай бұрын

Awesome thanks Matthew

Great video - the technique that you described works perfectly for me.

@DataIndependent

10 ай бұрын

Nice!

Agreed, it's important to look at the problem you're solvinf before commiting to any tech stack. Many problems simply require relational, precise data. Vector stores are a great fit for quickly building something that needs a fuzzy semantic retrieval. (though BM25 is surprisingly effective and like you say, cheap!) You can go a long way by just "stuffing" top-k results into the prompt but the limitations will soon become obvious, mainly the shallow data representation and lack of control. This could be addressed somewhat by adding metadata + filters on top of the vector index but that feels like a band-aid to me. All of this makes me excited for the future of this space!

@DataIndependent

10 ай бұрын

Nice! I totally agree. Retrieval as a whole is on my top list of interesting areas in AI right now. IMO our bottleneck right now isn’t gpt4 or reasoning, but rather the super structures around it. Of which good retrieval is 75% of the limiting factor.

Short video that packs an important point. For context, I have built a production feature for our product using OpenAI APIs. When scoping the build, it was confusing for a minute to see how vector stores were everywhere in resources. It's a powerful tool but it has been oversold.

Definitely agree with this - I fell into this trap while developing and at one point I was like “Wait, why do I need this to be stored in the first place…”

Agree. Retrieval must be a mix of tech approaches: semantic/embeddings, lexical search, structured SQL type queries, etc...

@DataIndependent

10 ай бұрын

Totally! Start with the problem first, then solution. Not the other way around

Well it’s clear enough that there are a lot of hypes in tech, we’ve all experienced them. This video would add in value if it dove deeper into *why* vector stores aren’t a silver bullet - the pros, the cons, the use cases. Would be good to understand when exactly to use them, when not, what the alternatives are etc.

@DataIndependent

8 ай бұрын

Nice! Thanks for the feedback on it

Fully agree with that 💯, I am implementing my own vector store over a SQLite database works amazing with my small data set application (thousands of records so far)

@DataIndependent

10 ай бұрын

Nice! Thanks for sharing

Totally agree. There is no substitute to understanding first a customer's needs. Otherwise you end up with a solution looking for a problem to solve. But perversely, it does solve the problem of company valuation.

@DataIndependent

10 ай бұрын

Big time - I want to make a whole video reinforcing this point. The same tenants of product development still apply.

Totally Agreed bruh...

My boss insisted I store CSV/Data Frames in vectors 😂 He claimed he was trying to be “visionary”.

@DataIndependent

8 ай бұрын

Ha! Yes - exactly

Hi Greg, I’m new to LLM, and found your videos super helpful! Recently I’m working on a usecase that is to summarize top X frequent topics from a long text file (each line is a topic). The approach I took is to put it into vector store and use retrievalQA from langchain. But it responded based on limited number of source documents (4 by default), which is not a good summary from the whole file. I wonder if there’s any better approach for this. Thanks!

@DataIndependent

10 ай бұрын

Hm, that sounds like a fun project. I haven't seen a great way to do that yet. I would need to check out the data to give a better recommendation

I get your point and I agree. But we are talking about the usage of LLMs. Of course if the data is a database which is ready to be searched by the LLM, then no need for a second vector storage. But keep in mind, the LLM itself relies on a huge vectorstorage. It can only analyze your structured Database because it knows so much "embedded language". So in some way embeddings and vector storage are the foundation of LLMs. The hype is bullshit, I agree. But large quantity of data in form of text/PDFs etc. (which LLMs are good at) can only be parsed as embeddings in vectorstores. Why is this important? Because the new use cases for data analysis arises exactly from this (mostly) new possibility to investigate large pieces of text documents, like contracts. Sure you can also use LLM to analyze well structured data and it will be more easy, but those are not are not the main use cases.

@DataIndependent

10 ай бұрын

First off thank you for the in depth comment, this is the fun part - talking to people in the industry. > can only be parsed as embeddings in vectorstores I would slightly disagree and say it depends on the project at hand. Totally agree that there is a new world of data unlocked w/ embeddings. What I was trying to argue is that it is not the only way and one should start w/ the problem they're trying to solve first.

Greg, how has the quality of retrieval been among vector databases? (At least among the ones you’ve used). Is there a clear difference in quality? Or are they all fairly close? Also, great video as always. 😊

@DataIndependent

10 ай бұрын

It's been all the same for me, no clear winners yet. But there will be differentiation soon. The limiting factor of retrieval has been my ability to code up a more robust solution rather than the DB itself

Could you do an example where vectorestores are misplaced? For any company related llm question I immediately think about vectorestores. thx and br

@DataIndependent

10 ай бұрын

Depends - what kinds of company related questions? Something like "what is apple's revenue in 2022?" doesn't need a vectorstore

Agree Greg. I think we got a lot of new people into the data space thanks to AI (LLMs). As such, many of them just don't know the whole data search and retrieval environment. They just know the vector stores and ChatGPT

@DataIndependent

10 ай бұрын

What’s the best way to get the word out? More projects that highlight other types of search?

Please add some tutorials about combining Knowledge Graphs with LLMs. Thanks in advance!

I am finishing up a small events app project and my experience started out with 'AI, vector, yada...' yet ended up with a more traditional implementation. I did use AI effectively, just not as first envisioned.

@DataIndependent

10 ай бұрын

Nice! That's a solid case study, thanks for sharing

I was wondering if there are approaches where the context is not only retrieved based on semantic (embedding wise) similarity to the search query but from a kind of knowledge graph? I can think of a lot of use cases where the relevant pieces of information might not be retrievable by simple semantic embedding distance approach.

@DataIndependent

10 ай бұрын

Llama index has advanced structures like that gpt-index.readthedocs.io/en/latest/reference/indices/kg.html

@TerenceChill286

10 ай бұрын

@@DataIndependent Awesome thanks a lot

With the right workload, vector stores have improved my latency, upload and indexing time, etc.. Otherwise what are your thoughts about the langchain sql agent potential? But dear god keep LLM out of INSERT operations. :-) Can you imagine?

@dualityninja

10 ай бұрын

Would be cool for discovery and natural sql db interaction... From a read-only perspective 😅

@DataIndependent

10 ай бұрын

I like the potential of them, but totally depends on the problem at hand. I haven’t experimented with them too much

@collectivelogic

10 ай бұрын

@@DataIndependent a good use case is replacement to old school ways of building data dashboard and data insight reports. Natural language to query and analyze existing data for additional insight, better and more relevant correlation is possible too if the right models are trained and integrated. Build in diagrams and chart generation into chat, and have a good time. Amazing for fraud and accounting analysis, contextual cross referencing (useful in larger multifaceted organizations). All this allows you to create different dimensions to existing data...worse case, it's great for training with existing data. Customer service bots and direct integration for self help. HR related matters, without outputting PII data to other sources. Use temporary memory vectors for conversation. Etc

Another challenge for enterprise apps is usage and ownership of data. Pinecone specifically states you give them the right to "access, use, reproduce, modify and display the Customer Data", and goes on to say that they own your aggregated data "Customer further agrees that Pinecone shall OWN such Aggregated and Anonymous Data and may retain, use and disclose such data for any lawful business purpose, including to improve its products and services.". If you are using a vector database for large amounts of corporate documentation, how exactly is your data being made anonymous when the entire body contains proprietary and identifiable data. Kudos to Pinecone for being very transparent about this, but it is a challenge for serious business apps to use vector dbs.

@DataIndependent

10 ай бұрын

Wow I didn’t know that - I feel like that isn’t well known. Thanks for raising that

@insigh01

10 ай бұрын

Can you share the pinecone link where it says that?

@TimeWasted8675309

10 ай бұрын

@@insigh01 Hmm, tried 3 times to post the link but it keeps disappearing.. It is pinecone DOT io SLASH user-agreement. Maybe that will work.

Are vectore stores suitable for NER? I tried using them when extracting book titles from a transcript, but It didn't work well.

@DataIndependent

10 ай бұрын

If you want to extract book titles from transcripts check out my "topic modeling" video. That method may help. I wouldn't use vectors for that task

@ahmadalis1517

10 ай бұрын

@@DataIndependent Of course! I've learned and am still learning a lot from your tutorials, especially on topic modeling, summarization, and deploying to a Streamlit app.

I am waiting a new video from you about fine-tuning gpt 3.5. Thanks in advance!

@DataIndependent

8 ай бұрын

Working on it now - starting my exploration to match tone twitter.com/GregKamradt/status/1695123517766078534

I'm interested in the concept of llama-index's tree index as an alternative to vector store, but haven't seen much content on it. If you're looking for content- would be good to go over pros/cons of different alternatives like tree index.

@DataIndependent

10 ай бұрын

Nice - thanks for the ideas. I'll check it out

I don't think these startups are "evil" or "lying", but if the only tool you sell is a hammer, you'll see every problem as a nail. And most content creators kinda latched on because "it worked for someone before" and "it works in this use case". In my case, I want to make a chatbot that takes a wiki that no one has time to properly tag for semantic search, a daily newsletter and a lot of PDF manuals and gives people summaries and points to the right page when someone needs. So far only a vector DB managed to take this chaotic mix and return useful information with a minimum success. Normal search engines get stuck on keywords and fail to rate relevance automatically and a QA index needs someone actively maintaining it. But if there's an easier or cheaper automated way of doing it, please do a video about it. You already have my view and like when you do

@DataIndependent

10 ай бұрын

Nice! For that use case I totally agree vector is a good place to get the mvp going.

@DataIndependent

10 ай бұрын

Also, thank you very much for the follow and thoughtful comment - this is the fun part of talking to people in the industry

I didn’t hear any argument against vectoring. I heard bad implementation is an issue, but that can be said for any tech. No real reasons given against them. They are indeed a novel way to group near/like data easily.

@DataIndependent

10 ай бұрын

Totally - the main point I was trying to emphasize is that market would lead you to believe it’s the only solution for AI apps. I love vectors, but it’s only a single tool in the toolbox

Hey i completely agree with not using vector store on database but what above if we have 300 pdf file each of 300 pages. Should we go with openAI embedding or offline embedding ?

@DataIndependent

10 ай бұрын

Depends on the problem you’re trying to solve. If you want all paragraphs that contain the word “dog” then no need for vector search. If you wanted all paragraphs connected to the idea of animals then vectors come into play

@prashantpandya5397

10 ай бұрын

@@DataIndependent what if you have some financial relevant and bond data and yes definitely paragraph

@cuttheace6407

10 ай бұрын

Openai vs offline embedding is not the right question. The question is whether embedding is the correct approach. What are you trying to accomplish?

@prashantpandya5397

10 ай бұрын

@@cuttheace6407 a chatbot that can go through all the pdf text data and if I ask questions about how much revenue is making each stream it can go in depth from those pdf file

@cuttheace6407

10 ай бұрын

You should find good results with data chunking and embedding. You have to figure out: Which embeddings tool or API will you use? Openai embeddings are good IMO and are not expensive for calculating embeddings for a few documents here and there. Then you have to figure out where to store the embeddings. Pinecone, weaviate or a local solution like chroma. If you are just running this for your own benefit and not trying to put it on a server for public use, just use a local vector store.

Nothing wrong with vector stores, imho, for unstructured data or multi modal (semantic) data combining text and images. For pure structured data it is overkill. I prefer a hybrid approach combining semantic (embedding vectors), lexical (e.g keywords BM25) and structured (relational) data. Either as add on to already existing relational databases or solutions that incorporate these features by design.

@DataIndependent

10 ай бұрын

I'm with you on the hybrid approach

Seems to me vector stores are only useful if you want them to do the similarity search for you, and you have enough vectors that brute force search against every vector is not fast enough. If not, store your vectors locally (csv? Pickle?) and iterate with cosine similarity, done.

@DataIndependent

10 ай бұрын

Ya, that's not a bad approach! Totally depends on the type of app you're trying to build

@willsmithorg

10 ай бұрын

@@DataIndependent thanks for the quick reply. Although using chroma in langchain is a few line no brainer, easier to code than pickle/csv.

Thank you for calling this out. It's just blatant marketing.

@DataIndependent

10 ай бұрын

Thanks! It'll pass eventually

We went through all this when MongoDB and NoSQL were the shiny new object that would solve all scalability problems.

@DataIndependent

10 ай бұрын

;) How’d the story end?

The audacity of VC's is kinda baffling.🤦🏾‍♂

@DataIndependent

10 ай бұрын

They have an incentive to call out vectorDBs as it’s own category. The feedback loop works for them.

Audio and video are not syncing, it is like a robot

@DataIndependent

10 ай бұрын

ooo whoops!

I’ve found vectors retrieval to be very hit and miss. Sometimes it gets the right answers sometimes it doesn’t. Way to unreliable

@DataIndependent

10 ай бұрын

Ya - totally depends on the answer or target chunk you're trying to look up

Brother you spent the entire video bashing people using vector stores for retrieval, acting all smug af, and didn't even explain why, nor did you propose an alternative solutions for retrieval. Are you okay? What was the point of hitting record on that camera exactly? To act smug?

@DataIndependent

10 ай бұрын

Yo! Thanks for the feedback! What are you building? How can I help?

@kongweiying3892

10 ай бұрын

Did you suggest alternatives to vector stores though, genuinely was curious to your title claim.

@DataIndependent

10 ай бұрын

@@kongweiying3892 I would push back and claim that the question is "what other types of retrieval do you like?" vs "how to do you replace vector stores" Totally agree that vector stores have a place in our heart, especially when doing anything with semantic reasoning. But a bunch of other retrieval mechanisms out there will do fine for non-semantic searches