Such powerful tool! appreciated!

@IsaacHarrisHolt

10 ай бұрын

You're very welcome!

Ever since I heard about Haskell's QuickCheck, I've been fascinated with Property Bases Testing. So cool

@IsaacHarrisHolt

10 ай бұрын

It's great! It can really help cover a lot of bases you wouldn't otherwise

This seems very inefficient compared to a proper fuzzer, which uses the code under test to efficiently find test cases that take new branches in your code. For example, an email address starting with "postmaster@" is special and can't be the name of a real mailbox in email, and therefore someone might add a branch to this code that throws an error if the username is "postmaster". A test suite written with hypothesis would have a 1 / 26^10 = 0.0000000000007% of finding this particular username and hence is unlikely to ever observe this exception, whereas a fuzzer would see that the string comparison always returns false and try to find an input where that is not true. Hypothesis generates a lot of test cases, but they're all the obvious ones (ones you can find a strategy for). Coming up with strategies that correctly exercise all of the corner cases in your code doesn't really seem that much easier or harder than coming up with the corner cases yourself, so I don't think it's worth making everyone on the team understand a new library for this.

@IsaacHarrisHolt

10 ай бұрын

I totally understand your point, and yes, your example makes sense. However, one of the benefits of Hypothesis and property based testing is that it can quickly test a wide range of inputs, and it's really quick to set up. There's not a huge amount to learn, and suddenly your test cases are a fair bit more robust. I wouldn't suggest ONLY using tests like this - you'll want to do proper integration and unit tests with known inputs and outputs - but it's a great starting point for expanding test coverage quickly.

3:42 task successed failfully

Fascinating idea. Can’t find anything for swift to do this tho…

@IsaacHarrisHolt

3 ай бұрын

Sounds like there's a gap in the market for you to fill 😉

Python tests are slow, if it will "generate hundreads of lists" just for this. It's a waste of cpu time and time needed for tests to finish will grow rapidly

@IsaacHarrisHolt

10 ай бұрын

This is true! But personally, I'd rather wait the extra second or two and have software I KNOW meets product requirements and is unlikely to break due to bad input. If you're at the scale where the extra time matters, you should probably be looking into test caching systems like Pantsbuild.

@clumsyjester459

10 ай бұрын

You don't have to run all the tests every time. You can split the regular tests and the property-based tests into two different pipeline jobs. Then only run the property-based tests once every night and before merging into the main branch. And if you only run them this rarely, you can even crank up the number of test cases they generate. Also, whenever they actually fail, you found an edge case that you should turn into a regular test.

@IsaacHarrisHolt

10 ай бұрын

This is a great suggestion!