What tests should I add to the new coding-specific rubric?

@linuxtechrusgaming3479

6 ай бұрын

Can you add some regex generation, explanation and regex fixing? Make sure that the questions are unique!

@bertilhatt

6 ай бұрын

Build an e-commerce store with one item, many, items with features (size, color, material), subscription business; secure back-end.

@stickmanland

6 ай бұрын

CLI based todo app.

@jameslafritz2867

6 ай бұрын

C# code tests, C++ code tests, see if it can create the snake game in Unity Engine or Unreal Engine. I tried this with GPT 4 and it failed, it was able to create the code and tell me step by step how to set things up Unity, but it failed in being able to fix an error in the movement logic. It understood the problem that I described but was unable to come up with a solution.

@Moyano__

6 ай бұрын

React app, Regex, some coding challenge from a page, etc.

VS Code tutorial would be great!

@jaysonp9426

6 ай бұрын

There's a million of those

@ryanfranz6715

6 ай бұрын

Let’s make it a million and one. I for one don’t find it obvious how to run a local (or in my case cloud) open source coding LLM and plug that into VS code copilot. This would be much better that the existing copilot since you can train on your own code base (and it’s already amazing)

@jaysonp9426

6 ай бұрын

@@ryanfranz6715 use pieces

@milindgaharwar827

6 ай бұрын

VSCode tutorial please

@liadp3472

6 ай бұрын

Yes please!

Yes, please. A tutorial for LLM in VS Code would be great. How do we deploy them, local Vs cloud and how would a basic workflow look like? Thank you 😇

@antixdevelopment1416

6 ай бұрын

Totally would love a tutorial on setting it up with VSC.. the least amount of money I have to give to microsoft the better.

@wrOngplan3t

6 ай бұрын

100th agree :)

Please make the tutorial, this looks like a ton of fun!

@MrHaomich

6 ай бұрын

And if possible not only for VSCode but also for neovim. At least if there are neovim plugins for this open source models, please mention it. Cause I use a proprietary plugin for coding and would love to switch to the open source one.

I love it!! Snake works perfectly! The 7b model is also impressive! Reaching also pretty close to GPT4 for a 7b and its also capable of creating Snake Locally!

@matthew_berman

6 ай бұрын

Crazy, right?

I appreciate these overviews, and this is great to see! I teach undergrad CS and before teaching full-time my work was in creative text generation, creative agents, and using AI to support creativity (glad I defended before GPT-2 dropped). I've been hoping to find coding models we could run on our local servers for (at least a subset of) students to eventually use, but don't have the time to keep up this the wild progress without videos like these. +1 for sure on VS Code usage, anything on deployment and practical workflow integration is fantastic. Great vid as always! Oh, and in terms of some niche use cases for different kinds of code benchmarks, beyond generation, code reviews, checking if requirements are met or violated, and checking test case coverage have all been almost fully hallucinations in the limited tests I've done, so these are always appreciated (assuming the model is meant to support instruct)

Really appreciate everything you do. Thank you!! Tip delivered!

For formatting the !=, i think the term is ligature and different fonts can include them. Ligatures are special characters in a font that combine two (or more) troublesome characters into one. For instance, in serifed text faces, the lowercase f often collides with the lowercase i and l. To fix this, the fi and fl are often combined into a single shape (what pros would call a glyph).

@matthew_berman

6 ай бұрын

Awesome, thanks for letting me know.

@marcfruchtman9473

6 ай бұрын

Ah -- a ligature. Thanks for the info.

@Microphunktv-jb3kj

6 ай бұрын

"troublesome characters" lool.. is that answer from AI haha.... funny af

You are amazing, one of the few channels that I have with all the notifications enabled. Of course, I would love that you make a video with the installation of local VSCode programming assistant. Thank you very much.

This was one of the better code testing processes I've seen you do on model's. I don't expect most of them to get things on the first try but after some encouragement. It would be great to see creative and effective ways to use open source llm models to build useful applications.

I love the direction this is going 🙂 Thanks for sharing 🙂

Insane. What parameter model did you use for the test, you didn't say I think? EDIT: If it was the 33B then I think it would be worth the hassle to test the other models or at least the 3B or 7B because that's I think what pretty consumer level hardware can run pretty easily

My favorite moment by the end of the week when I discover new impressive project with your video 😊. Lovit

a local copilot would be wild if you can get that to work!

Thanks for this deepseek video. I have tested it and it really blows my mind. A copilot video will be highly appreciated.

Finally... this is awesome. Thank you for reviewing this model.

Thank you for doing that multi-turn error testing, that's what all coding LLMs testing should go into!

Go for it Love t see the Visual Studio off line integration, thank you Matthew

I love your code model testing!!

Awesome video, please make a video about using an open source llm as a copilot on visual studio code. That sounds very interesting.

Amazing. Super duper keen to see how to integrate this as a copilot substitute in VS code!

Hey Matthew, Really enjoy your videos. Just a question on the code challenge section. If you change the engineer prompt to "Please find the issues with this code and explain in detail: ". You will find the outcome to be more inline with expectations.

@matthew_berman

6 ай бұрын

Awesome, thanks for sharing this.

Let's get that tutorial, save us some time haha. Love your work Matthew, thanks for your contributions to the AI space.

YES!!!!! VS Tutorial for the win!!!!!

Thank you for another interesting video. Please do make a video that implements a local copilot. I use Olama, Litellm, Conda, and VScode on my M1 Mac Mini. I've also installed Autogen and Langchain with this setup. So far, my only problem has been using Docker to run code from Autogen, but it runs fine on the CPU. IMO, you are working on the most important projects for helping the community march towards AGI.

Really happy that you changed the way you test the coding models, it really hurt me to see you only ask one very basic coding question that every model was getting right and then a super complex question that no model was getting right. With this new format the improvements are more visible. Also would love to see a way to use this as an AI code completion like copilot. (Maybe find a way to use it in Visual Studio as well as Visual Studio code, I've found some projects for VSC code but none for VS :( )

Wild! Loved it!

Love this!

Very good, Sir. Thanks for the video.

The ≠ formatting is a rendering option. VS code supports it. Under the hood it's still the same text.

Super cool! +1 for a tutorial on the vscode integration, is possible to use a local LLM for coding without the cut and paste in the files?

@matthew_berman

6 ай бұрын

I'm sure it's possible, especially with the VS code integration.

@alexbabich2698

6 ай бұрын

@@matthew_berman If you could show us how to have a coding model understand an entire relatively large code base at the same time it would be the biggest leap in AI since chatGPT

@itlackey1920

6 ай бұрын

I'm going to test this model with the continue vscode plugin. It should work and prevent some copy and paste. I'd love to see other options though. So +1 for a video

The score-results vs GPT-3.5 and GTP4: I'm guessing all the scores are for un-quantized models? It would be interesting to see how the scores were for all different quantized versions of the open models too.

Thanks for this powerful information

You mentioned Phind and I wanna make sure you know that while Phind did release an open model Phind/Phind-CodeLlama-34B-v2 but later iterations have all been proprietary and behind a commercial cloud offering. So when people say "phind is the best" they generally aren't talking about Phind/Phind-CodeLlama-34B-v2 rather they are taking about the proprietary commercial offering.

@matthew_berman

6 ай бұрын

That's really good to know, thanks for sharing. Is there a significant difference between v2 and the subsequent closed-source models?

@bobbyboe

6 ай бұрын

I asked Phind who it is... and I was told that there is GPT 3.5 under the hood, modified / optimized by the company for coding

@CognitiveComputations

6 ай бұрын

@@matthew_berman it's known to be significantly better, but we don't know how they built it

Good stuff as per usual. This one hits all my needs... So let's get this.into autogen and see what kind of damage we can do.

Yes, please. A tutorial for LLM in VS Code would be great.

Very cool :) Can you make a few videos about AI code assistants for C++ ?

Vscode tutorial of course - but would like to see the code suggestion/completion feature as well in it

this is AWESOME!

I have a suspicion that the snake game was in their training dataset, that's why it always nails it.

@JohnChristosMolura

6 ай бұрын

very likely.... most of the code gen llms so far have been disappointing... feels like code pairing with a junior fresh out of the uni... id like to see it actually build a proper lambda script that pulls something ... transforms something and serves it... in an optimal manner...

@JohnChristosMolura

6 ай бұрын

or even better... a snake game but over telnet.... 14.4k style

@marcfruchtman9473

6 ай бұрын

It might very well be in the training dataset... but it passed almost every test thrown at it... even the "what does this code do" test... which I found to be very impressive. As these models get better and better we need to expand the testing to see if it can handle variations. But... this is so impressive. I don't recall any of the other models that Matthew has tested as doing this well before (even if we take snake out of the equation).

I would be really interested in seeing a video on how to set up this model in VSCode!

This is the ONLY model aside from OAI 3.5 and 4 that will actually write good code, python, js, bash, or whatever I have tried. Great Video!

So what's next on the game test front? Breakout? Space Invaders?

@matthew_berman

6 ай бұрын

🤔

Yes please to the VS code open source code pilot. Folks working behind a corporate firewall would be rescued by that!!! 🙏

When running the aider benchmark tests it also scored very well

Definitely want to see how you set this up. I wonder if you can ask it to increase the size of the tiles in Snake game without breaking? Aider couldn't do that.

Awesome! I want this as an API.

A tutorial for LLMs in VS Code would be super awesome!!

Very impressive !!!

Would love copilot tutorial

As a programmer with 20+ years of experience I can say that now I'm starting to become impressed. The code (in this case quick sort) identification element is extremely valuable on its own. Finally an local AI model has as much smarts as a high schooler.

@ArianeQube

6 ай бұрын

Running this now on an RTX4090. It works but damn it's slow. So "local" may not be the right word yet.

@foobars3816

6 ай бұрын

@@ArianeQube Do you know what the bottleneck is? Are you sure it is your gpu?

@foobars3816

6 ай бұрын

As a programmer with 16+ years of experience I can say that you sound like you have been sleeping if you think it wasn't interesting before this.

@horikatanifuji5038

6 ай бұрын

@@foobars3816 As an anime watcher with 0 years of coding experience, I believe he was referring to the code, not artificial intelligence. It's true that Machine Learning is impressive and it's been achieving amazing feats, but the code it was generating by itself meant nothing, at least not until now that it's making some code that can actually be used.

@LaravicCode

6 ай бұрын

As a programmer with less than 1 year of experience i can say that this is not yet interesting for me.

hi, pls let us know how to setup the LLM in visual studio as a replacement to copilot. thanks.

I test this model to beat leetcode. It works like a beast, almost every coding challange is solved even for the hard problems. I guess we have a practical coding model already. Even better, it's open source.

Do that vs studio guide. It sounds great, I will definitely watch it.

9:38 - it's just a 'font ligature' feature. working with a special font and modern editors

Yes please! Can we please have a tutorial for DS integration in VS or any other IDE. Thanks!

Definately the best coder!

Did the gguf model get fixed or are you running aws or something?

Bro how you said that the background is ugly lol😅😂😂 which is true. And that’s the thing I like about you is that you are true to the core, brutally honest. Who can agree

What software do yu use for video editing?

Thanks!

@matthew_berman

5 ай бұрын

Thank you!!

Interesting that it used the unittest framework which is pretty much secondary to pytest these days. I would ask it specifically to use pytest. I had this problem all the time when generating python code with tests. It’s also very interesting to generate test fixtures as well with parameters to make really good unit tests. Pytest is awesome.

Yes I need to know how to run this model in vscode on local

Hi Matt: I find your videos very informative. Is it possible that DeepSeek was pre-trained to ace the Snake game? I am scientist not an experienced coder. I used DeepSeek locally and was able to get the Snake game to work. However, It failed miserably when I prompted it to create another application (e.g app to analyze stock performance). GPT4 gave me excellent code for the same and other tasks. You may need to refresh your rubric; what do you think?

Yes please do the local coder tutorial!

Yeah, a tutorial for how to install an LLM as a copilot alternative would be fantastic.

please help us how to run in ML Studio this model with what preset? its just not working with any preset and gives garbage.

Please I would LOVE to see this tutorial in how to do It to run it on local. Thanks a lot for your great work and aim to share knowledge with everybody.

First open source LLM into awesome level

This is really handy.

Hi Matthew, fantastic stuff, please do make a tutorial for this DeepSeek Coder AI in VS Code.

It would be great if you could briefly discuss the licenses these adopt and the type of hardware you’d need depending on the parameter sizes.

@ojikutu

6 ай бұрын

I run the 6.7B on 3090 using ollama and it's super fast. It's performance is almost indistinguishable from gpt3.5.

We need a 70b coding model!!!! It would crush GPT-4

Tutorial +Pointer App, thank you

@matthew_berman

6 ай бұрын

What's +pointer?

@quebono100

6 ай бұрын

@@matthew_berman your highlighted mouse pointer

How much ram+vram if possible for local inference on the 33B model?

Could you test the models on strongly types languages instead of dynamically typed languages like Python and JavaScript ?

This year: Snake Next year: Optimizing Crysis to run on a raspberry pi

When are you gonna release the llm finetune vid?

I think it'd be super interesting to see if we can get this to work with aider

Can this be set up on Linux so I can access it within my network at home from a browser (or API tool like Postman)

Hey Matthew... is there a model working for 3js yet?

Using it as a VsCode editor would be awesome!!!

Isn't it great to be astonished almost on a daily basis by the progression of AI ? I will have to get this going sometime this weekend to see if it can help me finish up my Inventory management app. One question: Say I wanted to use an LLM to make a local AI for use by my company and website. Can you give an overview of what is required as far as training etc..?

@sheriffusman508

6 ай бұрын

I would love to see this... ❤

Vs Code Tutorial would be really great

how does gpt4 perform on the coding-specific tests?

pretty cool

That'd be great! A tutorial on using it with VS Code

Did you test the base model or the "instruct" version?

autogen expert when?!!!

@matthew_berman

6 ай бұрын

Lol...working on it!!

it’s a slept on model. Competes very well against the GPT4 coders for sure

While it is impressive, it might not mean much. LLM KZreadrs have been using the same tests to evaluate models for the past six months. If I were a model creator, to get some hype, I'd make sure that the model I'm training passes those "tests."

I'd really like to see a video on how to combine an embedding model with something like this. For some reason, (using llama_index) whatever model I use to generate the index/embeddings, it never works together with the model that I want to use to retrieve from the vectorindex (sometimes I get shape/dimension errors, sometime other errors), it's really frustrating!

Your own free local copilot? Hell yeah, that would be interesting!

Dude, i can't believe you have to ask if we want to know how to hook this up to VS code. This would be next level

+1 for visual code intergration, would also like to know how it performs on C#, and if I can add the libraries and documentation and preferences what I normally use to this model like embed it. Similar like what you can do in a custom GPT

Deep seek inf loops so much!

The first model to create the Snake Game !

A tutorial will be highly appreciated, thanks

Maybe it's good on basic, widely known code like snake. But it fails on something more complex which wasn't included in dataset I guess.

I am totally new to this but, is there a way to get this to work with GPT4All?