Using this concept I started building tdd-buddy. It is a cli tool which is capable of executing tests on your machine. If any unit tests fail, the error message, together with the failing code (or not implemented code), will be sent to OpenAI and GPT-4 will respond with a "corrected" version. The returned code is saved, and tests are rerun; if the previously failing test passes, then the program goes to the next failing test. If the provided code is still failing, it keeps sending an error message to GPT-4, and the conversation continues until the test passes (or you reach a timeout limit). The idea behind this is that technically you can ask chat got to write functioning code in any programming language as long as you are providing tests, which are (as you mentioned) the explicit specifications of what you want your code to do. If you can engineer an automatic feedback loop which can provide meaningful errors on why code is failing, I don't see why we can't engineer a program that automatically writes a reliable code. At the end of the day, we, as software engineers, design and implement tests. If LLMs are capable of making those tests pass, it is all that matters.

@branvandermeer

4 ай бұрын

The act of writing the test IS designing the bigger ideas of the code (separation of concerns, abstraction, coupling), LLM's can't do that yet. See the second part of the video.

@janiscakstins2846

4 ай бұрын

@@branvandermeer yes I agree, that is what I am actually implying in my comment. It is driven by the developer by writing tests first which are intended to capture the desired behaviour of the code (not implementation details). Once you have written the behaviour, you can ask LLM to write a code to make the test pass. We are talking about the same thing LLMs can help with the first part of Red->Green->Cycle. Designing of the code (aka bigger ideas) are in developers hand. Please see my comment again

Really great tips

It sounds to me like its a bad idea to ask gpt-4 for documentation of what the code does (im talking about the Unicode bits), considering it still likes to dream about not existing things... Using llms with a TDD approach is probably the most usable way I have seen, because you can immediately test the output, but I'm still not sure if its really useful for larger projects. The thing is the llm can only write code that uses stuff it already knows about (from its training data) and i don't think it can consider enough of the code i already have to be useful for anything but simple examples and it will probably stay like that for a long long time. It's good that you talked about the shortcomings of this approach, i have seen to many people just jumping on the hype-train and set way, way to high of expectations of what is possible right now.

Thank you for the video, I found it really interesting, but it seems not to be a pure TDD approach, since instead of following write failing test/write the code/refactor cycle, a bunch of tests were written upfront

@branvandermeer

4 ай бұрын

You're right, I've simplified it to show the power of ChatGPT while keeping the video short. But it works either way, give it a try! The one-test-per-prompt just requires a bit more copy-pasting and a bit more time.

This is some good content! I think the comment about not accepting the first code output is super important. I am always tempted to copy paste but iterating on that same code can produce the best results, definitely better than I could come up. While, I agree that SWE is much more than "just making it work", I think that most code is "good enough to make it work". Hopefully using AI will help elevate the standard for code, do you think it will?

@branvandermeer

4 ай бұрын

Thanks! As for your question: I think there is a very real danger of code quality lowering because humans often take the easy route. The weakness of my proposed sorution (never accept the first output) is that it relies on dicipline.

Rich conclusion I must say, with facts and examples. Bran! Please remove the rainbow object from background, it kinda distracting the focus.

Very transformational ideas -- well done! Thanks for your efforts, Bran. QQ: What resources would be good primers/guides and/or references that drive the particular TDD process you are using? In other words, I can see that you are following a very specific style of defining a TDD framework, and because I have pretty much zero experience in TDD, it would be good to get pointed to a learning path to rapidly spin up on the approach you are using (e.g. "describe(...) { it(...does this..) } ) I don't care whether your TDD kung-fu is "Southern Style Preying Mantis" or "Mountain Style Monkey". I'm just looking for good spin up resources (for both Python and JavaScript) in a path used in the real world by a "grizzled grey-beard" who's seen a lot of... "schtuffs".

@branvandermeer

4 ай бұрын

I've got a few more video's on Unit Testing and TDD, I recommend those. Remember that Unit Testing is different from TDD. As for the describe/it naming, that's just part of the Vitest and Jest testing libraries for JS, but it had a history in BDD.

@DanielTheCunningham

4 ай бұрын

@@branvandermeer Thanks, Bran! I also noticed, when I checked out your personal website, you had a list of recommended reading, where you listed: Test Driven Development by example, by Kent Beck. Would you say that that book defines the seemingly VERY well-defined process you were showing your obvious (and admirable) experience with? Or is there another guide that more clearly lays out your preferred TDD process?

@branvandermeer

4 ай бұрын

@@DanielTheCunningham Yes, that's the recommended book, it defines the process. I also recommend Kent's last book: Tidy First?

@DanielTheCunningham

4 ай бұрын

@@branvandermeer Thank you, sir! I will look into that one as well.

'Bánh mì' and 'Phở bò'? They sound interesting, haha! xD

@branvandermeer

4 ай бұрын

Haha I love vietnamese food, and they make for complex diacritic examples :)

@antoni2nguyen

4 ай бұрын

@@branvandermeer + subscribed because you love my country food, haha

I've never seen gpt3 or 4 write working code over 100s of times I asked it to

@branvandermeer

4 ай бұрын

In that case I recommend to learn about Prompt Engineering, because that's a solvable problem :)

@francisco444

4 ай бұрын

In all fairness, prompting for code is not as hard as coding manually but it's still hard because of the complexity. It will take many tries to get proper code. it's basically learning to code using natural language, which I find to be more enjoyable

Its faster to write a function you mentioned in video by yourself then waiting for gpt to generate answer. Nothing more complicated gpt can produce yet. Maybe for simple projects its valuable, but the code gpt can produce you can write without help from gpt. For anything remotely complex the amount of time you spending to convert code to human language and back is just stuguring.Welcome on hype train =)

@branvandermeer

4 ай бұрын

I'm not sure it's faster to write it yourself, depends on the function, and the prgrammers exporience/seniority. I think you are missing the point of the video: it's applicable to any size project when you work tdd, because chatgpt doesn't need more context than the tests. And I'm not converting human language into code, I'm giving it code: the tests. Btw, it's not a hype anymore, it was a hype (for a few months) over a year ago. It's here to stay, one way or another.