This would be brilliant for the AI community. We would not need these gigantic language models for many tasks. But instead we could get away with small locally hosted models connected to knowledge graphs. And then there is the potential for fact checking existing information.

Жыл бұрын

Yes, that's exactly what I would hope for!

@sujith3914

Жыл бұрын

@ Similar concept is what I was envisioning in 2017. The thought came to me when I thought about how we ask for directions from people to navigate to a new place when we aren't using maps. But I didn't know how exactly to place that hypothesis in the realm of AI and I am just an amateur enthusiast, thanks for this, I feel validated and would love to learn more to get it forward.

This is timeless, its about principles probably every generation needs to reinvent the wheel

Great talk @DennyVrandecic! Thanks for recording and sharing it.

Жыл бұрын

Thank you!

I like that you placed knowledge graphs in the context of tool use. For an old rules-based AI guy who became disillusioned with general purpose rules while working on the CYC project, LLMs just what I was waiting for because they provide the "soft" reasoning and facts while (1) controlled by "hard" rules and deterministic programming and (2) informed by "hard" knowledge in databases and knowledge graphs. I believe that assembling a rule-based AGI with a human-understandable representation scheme is too hard for humans. If there was a rules-based AGI built by someone much smarter than us, it would be impossible for a human to understand. But now the possibility for building hybrid agents based on tool use plus top-level orchestration opens amazing opportunities and I'm now diverting most of my resources to that approach. Recent papers in which LLMs dynamically generate tools makes me think of possibilities for "rules-based AGI" that run faster and cheaper. If they generate enough tools and the tools become larger (like making and updating expert systems on the fly), that is a step in that direction. They may be able to write more complex rule sets than any human, rules that use nuanced knowledge representation schemes appropriate to a given context. I'm going to try a little of that with an intern this summer, seeing if I can mine genomic data by using GPT-4 to write custom parsers for genomic data, rather than trying to hammer data with discordant schemas into a single "harmonized" schema. Access can then be fast and then use the results to fuel "soft" reasoning. Wow, I really do monologue when I comment. A character flaw that I should work on.

Жыл бұрын

I think this is a super interesting idea, and if I were doing active research these days, I might totally try to do that too. I hope to read about your results and insights! The output would be somewhere between the brittleness of rule based expert systems and the intransparency of neural networks and their weights, and I wonder about the interesting sweet spot there. Very cool!

@manslaughterinc.9135

11 ай бұрын

We're integrating KGs with our autonomous agent system, AgentForge. Our approach is combining KGs with VectorDBs to enrich prompts. Essentially, when the prompt comes in, you do a semantic search in the VDB, but also send the prompt to a categorization agent. You use that category as a query to the KG, and pass the results of the KG to the VDB for additional context on a separate search. The full results from the VDB becomes the context for the LLM prompt.

@sfilkin

3 ай бұрын

Have you had a success with genomic data mining using LLM?

I had the same notion that LLMs don't replace knowledge graphs (though could help to use in tandem), but this is very thorough on all the reasons for that. Thanks!

Жыл бұрын

Thank you!

Wow, nice job Denny. Sums up so much in just a half hour. Lot's of great sound bites as well. I'll be returning to this over and over.

Жыл бұрын

Thank you for these kind words!

Thankful for going over your video show giving key bits of knowledge on the qualities and shortcoming of LLM and the significance in the realm of artificial intelligence.

Жыл бұрын

thanks!

Astonishing how captivating the talk was! You hit the nails on the heads!

Жыл бұрын

Thank you for your kind words!

Excellent presentation, Denny! It talks about so many things I was thinking about in the past few months. Moreover, it provides excellent ideas and suggestions for possible solutions to the problems that LLMs currently have, and you put all that together in a clear and interesting presentation. I appreciate it a lot, and I will definitely come back to this presentation, and share it with others, over and over again. Thank you!

Жыл бұрын

Thank you for these kind words!

This was so amazing to watch! I am going to share and rewatch again to focus on some of the things I might have missed. I am going to learn more about what you mentioned about LLMs being a tool for knowledge extraction. Thank you again!

Жыл бұрын

Thank you for your kind words!

Great work Denny . Everyone is feeling the FOMO now, when experts who have been working on Knowledge graphs talk about things like this. I am sure it makes things a lot clearer. Personally I am going to wait for 2-3 months and see how this space evolves.

I am quite sure that structured, verified data, with notion of sources, is the next step. This was golden.

Жыл бұрын

Wikidata offers that with an open license.

Hello Denny, Ziko here. Thank you for the clear and understandable essay. This could be indeed important for the future relevance of the whole Wikimedia movement - provide knowledge in a different and distinct way compared to ChatGPT and potential other competitors. I am afraid that for many people convenience will be the most important factor, and maybe the Wikimedia movement is not always very competitive on that field.

Жыл бұрын

Agreed. I am rather worried about Wikipedia, less about Wikidata. It's going to be interesting.

Grateful for coming across your video presentation providing key insights on the strengths and weakness of LLM and the importance of KG in the world of AI.

Жыл бұрын

Thank you!

Great presentation!

Жыл бұрын

Thank you!

Any additional reading materials for using LLMs to build the knowledge graph? I’m also interested in how this architecture changes the training process for llm - Would labels now change into queries and query types?

Жыл бұрын

Not my expertise, but I hope someone else will have links. There must be work on Arxiv, or conferences such as EKAW, ISWC, etc.

Excellent presentation, Denny! Thank you for posting it.

Жыл бұрын

Thank you!

My take away is that the future structure of succesful LLMs are going to be modular, each module requiring specific chip designs optimized for it's task.

I love this. I have been feeling much the same. This is in many ways the biggest thing that could have happened to Wikidata.

Жыл бұрын

Thanks!

Great story, thanks for these insights. Spot on in current discussions.

Жыл бұрын

Thank you!

Great presentation!!

5 ай бұрын

Thank you!

Hi Denny great talk, very clear and engaging. Your talk focuses however on the limitations and issues of LLMs but skips over the limitations of KGs, e. g. the enormous cost of manually buildng, updating, verifying, etc. the large or small KGs. Yes you mention the use of LLMs for that but still these costs are huge. Not in terms of machine time but in human time terms

Жыл бұрын

That's true! KGs aren't free. Amusingly, I think that LLMs can help us reduce the costs of building and maintaining LLMs. But also let's remember that training a model such as GPT4 cost more than $100M. I don't know many KGs that saw similar investments, and none in such a short timeframe.

Humans are awesome... I does not matter how complex a machine or artifact is, we still can talk and teach a lot about it in simple words! AMAZING!!!! -. Thank you so much for this video. You've got a subscriber. Cheers from Mexico City!

7 ай бұрын

Cheers to beautiful Mexico City! I'll never forget my visit there

Question about your point at 26 minutes: Instead of "It's complicated", why not just add a certainty score to statements? Or an attribution to who said the fact? Like, if the RDF says: > ex:source . > ex:source . ex:religion etc. etc. Then when you ask the language model "who wrote the bible" and since it knows that the user is Catholic then it would answer "The author is the Holy Spirit" but if the last line said then the language model would respond with "The author is a collection of unknown authors".

Жыл бұрын

We already have attribution in Wikidata, every claim can have references attached to it. We don't have confidence scores, because these numbers are super difficult to find agreement on and are rarely available. "It's complicated" is meant for things that are ... more complicated than competing statements. E.g. "collection of unknown authors inspired by the Holy Spirit" is something we would have trouble representing right now.

Pretty good in the age of Reddit being shutdown. You could do a video on an example AI tool built by new emerging github trending Ai but connected to econometrics. Now we know how much it will cost if we want to converge 5 small AIs into one. Or even an AI who knows how to expertly distinguish the wheat from the chaff and make the new AI economic but not diminished in sophistication (aka faulty or dangerous). No one is doing a video on idea efficacy related to out-of-pocket costs.

Excellent presentation of this important topic! thanks for clarifying.

Жыл бұрын

Thank you!

Excellent presentation, @Denny...learned new perspectives. It would be great if you could share those slides specifically the comparison part wrt sparkql. Thank you!

Жыл бұрын

Slides are here: docs.google.com/presentation/d/1_fUGHjPj8C18bI-hAPdI1HuhoN2ef1n2mzO_nn3ACKA/edit#slide=id.g23790ae4808_0_0

Very Insightful . This will be the Future LLM for reasoning and actions and KG for Knowledge

10 ай бұрын

Thank you!

this was a pretty good insight! appreciate it!

10 ай бұрын

Thank you!

Thank you for presenting on this and suggesting a better path for both knowledge and AI.

Жыл бұрын

Thank you!

Thank you for this great explanation! I couldn't agree more with most of what you said. Language models are better used to verbalize the results fetched from a knowledge graph than to store the actual data.

11 ай бұрын

thank you!

Thanks very good presentation, I also would like to be able to mark things as need more care i.e. its a more a todo list but you dont have time/skills to fix it now but think it will be possible...

Жыл бұрын

Oh, that's a good idea!

Great presentation and thank you for sharing!

Жыл бұрын

Thank you

Brilliant talk. I was thinking about exactly this the other day with some colleagues. ❤ Externalizing knowledge is a good optimization strategy for LLMs among other strategies. I know some have found a way to make them efficiently run in CPUs instead of GPUs. Exciting times ahead.

Жыл бұрын

Thank you!

The guessing aspect of LLMs is, of course, by design. They are, after all, probablistic word predictors with a large context to refine the probabilities. There is a good chance that part of the way we think is similar, as our retrieval of memories of events seems to follow a similar algorithm. When humans experience an event, gaps in the actual experience can be filled in with hallucinations, and we can't tell the difference afterwards. There are ways to minimize the hallucinations in LLMs. The simplest is to use a large shaping prompt (e,g, "system", etc) to shape the context in a way that penalizes uncertain answers, but the best is likely to use chained LLMs and external tools, such as your knowledge graph. A great recent example is the "Tree of thought" approach, which on reflection, seems to better emulate how we think. But of course, that could just be my hallucination.

Жыл бұрын

Yes, that all sounds right to me. I do hope that humans are a bit better at telling hallucinations from knowing apart than these systems are. Also, I think it is much rarer (but not unheard of, as studies in bilingual people have shown) to store knowledge in a language dependent way.

Awesome presentation! Amazing!

Жыл бұрын

Thank you!

Super presentation!

Жыл бұрын

Thank you!

Brilliant I think we this.

Жыл бұрын

thank you!

Great insights

11 ай бұрын

Thank you!

I told my coworkers that GPT4 probably is already doing this on the backend because brute forcing all that info into neurons was not smart; and OpenAI has specifically said they don't care about model size if they can find better alternatives. But this is definitely the future of how AI will be set up, and it's honestly a bit frustrating that it's not a bigger focus of LLMs. Heuristic lookups still serve a VERY important purpose for efficiency.

Жыл бұрын

I'd be really curious to know. My guess is that GPT-4 really was betting on a huge number of parameters. But no one knows.

Great useful information.

6 ай бұрын

Thank you!

Great video. And thanks for sharing. You do present great insights about the size of the model, the cost of the training, hallucinations, making stuff up on the fly, how to updated the data or how to add new data. The part about training the model to make stuff up on the fly is "funny", but it's true. As someone that worked with Prolog in the past, I have always be interested in the knowledge representation and reasoning. I always felt there could be a way to combine symbolic ai/reasoning with the connectionist methods. I discussed with some professors and they believe this is a viable approach and that deep learning techniques will have limitations, some of which you mentioned already in this video.

10 ай бұрын

Yes, I agree. I think there is a fertile space to investigate neuro-symbolic hybrid methods.

@jeromeeusebius

10 ай бұрын

@Thank you for your reply. One think that I kept coming back to in your video, is why do we need to run through all the parameters of the model for something that is a "fact", e.g., where the actress you cited was born. Once we know the actual place, we hardcode and just retrieve it from the source at all times. This will definitely be more cost effective. Also, the fact the LLM knowledge may be "language" (spoken language) dependent presents another problem. I am not expert, but wondered how knowledge can be represented in a compact form, especially for planning and reasoning problems. I will follow some of the other works in this space to get some ideas.

10 ай бұрын

@@jeromeeusebius I'm a big fan of using structured knowledge representations such as knowledge graphs. They can't do everything, but they get a lot of things done.

Excellent presentation!

Жыл бұрын

Thank you!

Very insightful talks, thanks Sir. 👏 I agree that LLMs will help building more quickly the bridge between symbolic and sub-symbolic AI to make them complement each other in the right way ✌️

Жыл бұрын

Thank you!

Wow this is truly amazing!

Жыл бұрын

Thank you

I was really looking forward to the video, especially as I am a big fan of your calm and level-headed way of lecturing on the one hand and on the other hand because I have asked myself exactly the same question in my daily way. After LLMs, is there actually still a need for classic databases, storing information in graphs or using the usual REST/Http interfaces between systems, where everything is now language and A.I.-based? Even if the comparison of training costs is too short-sighted for me, especially since an LLM can do many things that are expensive to buy but that a knowledge graph cannot, I basically agree with you. But is an LLM without "knowledge" even conceivable? Or is it then a kind of language model with knowledge of a general ontology but without concrete knowledge of expressions? Say: a president is a person and the USA is a country, but in order to find out who was president of the USA and when, I have to build a plugin for a lookup in an EKG? I don't think that's enough in the lecture. Where does the LLM start, where does the Knowledge Graph end, or vice versa, and is it best to integrate the two? SPARQL and Q-IDs may be suitable for Wikidata, but not for the self-built Knowledge Graph, and yet there is a lot of knowledge in companies that is hidden internally behind the firewalls and cannot be found in Wikidata. Is the solution Embeddings or Functions or LangChain or Cognitive Search? How do you bring the worlds together? I think the lecture lacks some details in practical doing. But in any case, it is an exciting topic that you have presented well, I would like to hear more about it. Thanks for sharing with us.

Жыл бұрын

I totally agree that I certainly didn't give details on these important questions, mostly because I don't have these answers. I only sketched the corners of map, but it will be to others to actually chart the territory and fill it out. I think you are right that an LLM without any knowledge will not work. Even for simple things like "wait, it has to be 'the Netherlands', not 'Netherlands' when I use it in a sentence" would require some knowledge about the existence of the Netherlands. But the population or the area of the Netherlands would not need to be encoded in the LLM. So we know the extremes, but not where the border lies, how much it overlaps, and how to effectively build a system that combines those. There is a lot that we don't know yet, but now is a great time to figure it out if you are a researcher, or a pioneering practitioner. Thanks for sharing your thoughts and your friendly words!

Brilliant!

Жыл бұрын

Thank you!

Excellent talk - shows that Green AI is not just a buzzword :-)

Жыл бұрын

Thanks!

Was not able to reproduce Zagbreb for Ena - gpt4 gave me Split for english and crotatian question.

Thank you for sharing this, Denny. Very interesting! I was wondering, do you see the Wolfram Alpha plugin that's currently available for ChatGPT as a similar approach to what you are suggesting here?

Жыл бұрын

Yes!

What do you think of LLMs + RAG system? Where you don't store information on a knowledge graph but as raw text data do you think in that case knowledge graphs could become obsolete? BTW great video

8 ай бұрын

I think that is a great question! Let's put it this way: I think it is much, much more likely that a RAG+plain text will be be able to answer questions with a high precision than an LLM alone. A RAG just using plain text can probably compete with a RAG+KG+plain text in terms of precision. But I would still expect that a RAG+plain text would be considerably more expensive than a RAG+KG+plain text, simply because the LLM would need to read and interpret the plain text every time, instead of just doing a look-up. I can imagine that KGs become a "behind the scenes" technology, where we compile plain texts to, basically a better search index of the plain text, but it would not become obsolete, just as an inverted index won't become obsolete in a RAG. My current bet would be on a RAG+KG+function library+plain text+services for the most effective architecture for an assistant-like system. All of this would still require us to get a solid grip on the hallucination problem, but I am confident about that being solvable in the short or medium term. Thank you for your kind words.

Excellent presentation

Жыл бұрын

Thank you!

Thanks. Validity check is important. KG would be internally verifiable. But picking up the LLM hallucinations are not easy

I recently realized I needed to use a knowledge graph rather than a language model to explore a large data set. Do tools to help create a new knowledge graph using a mixture of rules and language model tools (eg entity training and extraction extraction) exist already, or tools that could be modified?

Жыл бұрын

That's a great question, and I am no expert on what is currently available in term of tools, sorry. I hope someone else would have an answer.

Interesting talk, thank you. In the short term it might be useful to hitch graphs to LLMs - it should be possible to conduct an empirical trail of doing so, using real world interactions rather than fact oriented queries. In the longer term it doesn't feel workable to maintain graphs and LLMs, as the effort of enriching graphs as you suggest, as well as maintaining, and auditing them, is huge. But more importantly, graphs don't adequately represent reality. The analogy I'd use is that graphs are like early Wittgenstein while LLMs are like later Wittgenstein.

Жыл бұрын

I disagree. The main cost factor of a large knowledge graph is due to size, but they are, even in maintenance, a relatively cheap operation compared to LLMs. And in the end it makes sense and is similar to humans: we don't need to remember the population of every city in the world, we just look it up when needed.

Great video! Would it ever make sense to store the LLM parameters in a knowledge graph?

Жыл бұрын

Depends what you exactly mean. Like a vector to be an embedding for a certain fact or entity? That might possibly be of use. But certainly not all parameters, but strategically selected embeddings I would guess.

This week OpenAI released ChatGPT release functions. Seems like they were listening.

Жыл бұрын

Yes, I am pretty excited about that! I need to look deeper into how they work.

The decentralized knowledge graph project Origin Trail is at quite an advanced stage, you might find it interesting.

9 ай бұрын

Thanks to pointing me there. I read the white papers and watched a KZread video or two, and I can maybe see how for certain use cases such an architecture might make sense, but for most use cases it seems that simply dropping signed RDF files on a webserver seems sufficient, no? For others, such as public knowledge - what Wikidata covers - this seems to make things more complicated. Would like to chat constructively talk about this.

hi denny, a little late to the party but great insights on a world with KG-enabled LLMs. just wondering if you've any thoughts on vector dbs as well? i see them as cheaper alternative to KGs (to setup) but perhaps less reliable? or do you think that they arent alternatives to each other and can both be used concurrently with LLMs to ground them?

10 ай бұрын

Vector databases are great to store embeddings and work with them, but I wouldn't see them as a replacement for a Knowledge Graph. They seem great to help connect text to nodes in the graph - but in the end, the knowledge would be stored in the graph, no?

@zactamzhermin1434

10 ай бұрын

@ oh i was meaning more of just relying on vector dbs for "knowledge" and context instead of KGs to ground the LLMs. my kind of weak understanding is that they are both trying to hold sources of truth, would this be a wrong way of thinking?

But in your experience , do knowledge graphs tend to perform better , precision-wise , time-wise (latency in orders of single digits microseconds)and memory-wise for those queries which grabs data which are not fixed by nature , so variable's names stay the same but their values differ as a function of changes of other variables 's values and as a result relationships and properties too change on a constant basis? your input is highly appreciated

Жыл бұрын

I'm not sure I understand, but it sounds like functions would be better at capturing that than a knowledge graph, as far as I understand it. Can you give me concrete examples?

Kenya, yw

Interesting presentation! Of course, GPT can generate SPARQL queries, but if it makes even a small mistake, will a person without knowledge of SPARQL be able to correct it? Learning SPARQL, SQL or XQUERY is quite daunting for humanists. Creating a universal and user-friendly graphical environment (UI) for searching various knowledge sources is not easy, while natural language is such an "environment" and LLMs give us this opportunity. One may wonder how effective the recently popular Q&A tools using LLM, large text corpora, and vector databases with embeddings are. Is that all we need? Effective methods for finding matching texts in a vector database and an LLM that prepares an answer from the information found? Don't KG become redundant in this situation? The problem with wikidata/wikibase is the huge amount of work required to complete them. I am currently testing LLM as a tool for biography processing and automatic knowledge extraction for a Wikibase instance. These are interesting times, no doubt!

Жыл бұрын

No, I don't think that someone without SPARQL knowledge can fix an LLM created query. I'm even more worried about the system creating an answer that runs and looks plausible, but is subtly wrong. There are researches such as Basil Ell who have been working on verbalizing queries. I think a combination of these technologies could be interesting to create a robust system. Plus fine-tuning. But it's not out of the box. I'm just saying it's super promising.

How is knowledge graphs different from vector data base?

man i really loud at ITS COMPLICATED PART... when you said that :)

9 ай бұрын

Thank you!

@tartaria936

9 ай бұрын

one more thing. what you are doing and saying is real treasure. its obvious that you are not the guy who just trashing around to be interesting. you have real value to our community 💘@

I see this as well. This is really obvious to me. And the ability for LLMs to make knowledge graphs and be customizeable to different knowledge graph building (like the kind of representations you want to make from some data) and the generation of the queries that traverse them. Makes them pretty limitless in terms of knowing and operating within some logical bounds

Жыл бұрын

Thank you!

So I have been looking for a way to represent a vast body of words stretched across 900 files, all pertaining to one project, as one singular visual Knowledge Graph. Akin to what Chris Harrison did for the Bible and all its impressive cross references, I also have a heavy themed narrative that I wish to distill from that original body, down to 10 million words. That is how I came across this video, is there anyone who knows how I could possibly achieve this context output desire?

I got more or less the same xperience as you did but testing different questions and asking for scripting procedures , while the overlook looks legit , when it comes to details , it is far off !

But you don’t think knowledge graph is limiting, given the fact that it needs to fit within the confines of linguistics

2 ай бұрын

I totally think knowledge graphs are limited, and in fact much more limited than just by linguistics. I'm not saying use knowledge graphs instead of LLMs for everything. That would be wrong. I say both should be used for their strengths.

QLoRA has brought down LLM costs by orders of magnitude

Жыл бұрын

QLoRA doesn't bring down inference cost, but fine-tuning cost. Which is great. Inference cost still remains an order of magnitude higher than lookup in a knowledge base.

Stopped watching the video when i saw the main argument was retrieval time when comparing a natural language query vs a structured query. Most of the complexity relies in that translation.

i think we will have better algorithms designed by llms to train llms and fix llms 😂😂. purpose based multiple llms will be the future. no one needs one model solution lime gpt for all the problem. independent expert teams can tune their models to perfection. and all these group of models could be built to communicate with each other like a hive of models to create agi.

Жыл бұрын

I am sure that there is a lot of space for improvement and that we will cover some of that space, but I wonder if we can reach "perfection" with reasonable costs. I guess we will see.

@voicevy3210

Жыл бұрын

@ sir any optimizations or new theories/ways being proposed to replace vector databases to support llms better. do you feel an completely new data modeling and Asics will help achieve this? all our thought process and hardware seem linear and very naive. going multidimensional with data and data processing and data relation processing (graph ) with our current technology and thought process seems to be in no sync to achieve leaps in performance. i feel we are trying to push the same technology forward rather than going to drawing board and redesigning from ground up. probably things happening in cutting edge teams .

Жыл бұрын

@@voicevy3210 that is not what I see. Chips such as GPUs and TPUs certainly allow more than just linear processing, which is one big reason why neural models have made so many incredible leaps in the last decade or so (in addition to more available data, novel algorithms, and a lot of attention). The same chips have so far, in my opinion, been underexploited for symbolic approaches. I think vector databases are an excellent tool, but their advantages and disadvantages compared to say good old knowledge bases should be taken into account, too.

There is no value to ask a question like "Why our solution is better then LLM". We should all ask ourselves "How does our knowledge/expertise/solution can be useful for LLM to build greater technology and life?" Because if it is not, then we have no future.

Жыл бұрын

I agree. I don't care which technology it is that helps us build a better future for everyone. And I certainly don't say that LLMs are better or worse than KGs, only that there are certain use cases that have different characteristics when trying to solve them with one or the other.

Why are you whispering?

Sorry but maybe when you can remember the sequence of GPT and not GTP, I’ll put some trust in what you say. At this point LLMs are in their preliminary phase, while knowledge graphs existed for a long time, so most of your arguments will fade away soon

@generichuman_

Жыл бұрын

What an incredibly petty thing to point out... but I guess when you aren't capable of countering the strongest form of an argument, correcting the pronunciation of an acronym is some nice low hanging fruit... Ironically, this comment demonstrated why no one should put their trust in anything you have to say...

Жыл бұрын

I'll try to remember the order of the letters. So many acronyms going and passing over the years :)