The Problem with Benchmarks

Ғылым және технология

Benchmarks are great, but they have limitations... and choosing to buy a computer because it has great test scores, is not always the best plan.
In this video, I show how common benchmarks, like Geekbench 6, arrive at a total score, and explain some of the limitations and edge cases.
Join this channel to get access to perks:
kzread.info...
PLEASE SUPPORT THE CHANNEL:
As an Amazon Influencer I earn from qualifying purchases
Apple Store on Amazon
USA Store: amzn.to/3rInBt9
UK Store: amzn.to/3gFyUw4
Gaming Laptops on Amazon
USA Store: amzn.to/48yn9jZ
UK Store: amzn.to/3uZwPp6
#benchmark #apple #pc

Пікірлер: 25

  • @ConstantGeekery
    @ConstantGeekery6 ай бұрын

    RE-UPLOADED DUE TO AUDIO GLITCHES - apologies to anyone who left a comment on the broken version, and sorry for not checking my work properly!

  • @deavo74

    @deavo74

    6 ай бұрын

    Quite right too! lol. Only messing. Watching again so you get the view 🫡

  • @Bucking_Fastard

    @Bucking_Fastard

    6 ай бұрын

    Video so nice you have to upload it twice.

  • @deavo74
    @deavo746 ай бұрын

    Great video, good to see someone breaking down the different elements.

  • @garanceadrosehn9691
    @garanceadrosehn96916 ай бұрын

    Thanks for fixing the audio!

  • @POVwithRC
    @POVwithRC6 ай бұрын

    Fair comment. It feels like the benchmarks need to assign individual weighting to each test based on how it would generally affect the average user. Perhaps a little less weight on Clang or Navigation when a final score is calculated.

  • @sueside
    @sueside6 ай бұрын

    Just wanted to drop that you are very informed, unbiased and smart in your videos. A joy to listen to them.

  • @ConstantGeekery

    @ConstantGeekery

    6 ай бұрын

    Thank you 😊

  • @glenmeredith8027

    @glenmeredith8027

    2 ай бұрын

    Agreed!

  • @AlanW
    @AlanW6 ай бұрын

    Great rundown, also, the set is looking clean and cozy!

  • @doctorscoot
    @doctorscoot6 ай бұрын

    Great video! Reminds me of the way people would rely on micro-benchmarks to determine which programming language was ‘faster’. Which completely ignores the context of the programming - for example modern Java JREs have ‘just in time’ optimization, which means that as your program executes in real time the JRE starts optimizing the code path based on the realworld performance. Compared to static analysis based optimization typically found at compile time in C programs. Or if you were comparing python - for example if the python only executes once you have to factor the compile time to a .pyc but if that same .pyc is compiled once and just executed over and over millions of times then the compile time is irrelevant. Similarly, you can have a fast and efficient compiled program but it has to call an old database over the internet pretty much making your choice of language a nonfactor in application performance. Etc.

  • @MeinDeutschkurs
    @MeinDeutschkurs6 ай бұрын

    The only question I'm interested in is whether the LLM fits into the VRAM. 😁 - But yes, we’d best use case related benchmarks. Are there any?

  • @ufopsi
    @ufopsi6 ай бұрын

    It’s hard to justify yearly upgrades. An Apple Silicon machine is such a good value that, unless you really need a new feature for your workflow, a new purchase won’t be necessary. That said, the AV1 and ray tracing features are a nice addition.

  • @bryans8656
    @bryans86566 ай бұрын

    I appreciate that your explanations are always so clear, thanks.

  • @Technocrat.
    @Technocrat.6 ай бұрын

    I've been watching reviews on Intel Arc video cars, and found out that they are killer for video and photo editing. They have 4 or 5 various decoders built in, kind of like M1/2/3, plus, if you have a newer intel 12th,13th gen cpu, it combines the apu acceleraors with the Arc gpu and multiplies the performance, theoretically by 2x to 4x.

  • @masterphoenixpraha
    @masterphoenixpraha6 ай бұрын

    People were always fully focused on numbers.. back in the days i remember hearing... oh, the CPU only has XXX GHz (well, I remember even MHz), or the scanner only has XXXX DPI... Maybe because I was in the HW selling business, I was always thinking bit differently about this... In 2012 I bought my first MacBook Pro, the very first Retina model. I certainly didn't need the performance, I simply liked the computer. Got me one of the higher end CPUs, more RAM, bigger storage. For my needs it was bloody expensive and simply too much of a power... But I found myself using it sill after 10 years, just had to replace the battery, it didn't get the latest OS updates... But still was fast enough for 'normal' work. Bit slower and louder for intensive work... So when I decided I wanted a new computer, went for the future-proof concept again... So got me M1Max MBP and will see the experience after 10 years... And BTW my original MBP Retina went further, a friend of mine got it from me for his web browsing, movies watching and easy office work... And he is very happy with it 🙂

  • @basbas63
    @basbas636 ай бұрын

    Interesting stuff! Ended up going for a single CCD Ryzen CPU after having used a multi CCD CPU that gave me issues in some tasks. Same as that putting a GeForce GT710 in a system with a Haswell i5 improved general computing speeds (browsing the web and such). I think, and I cannot confirm this as I do not have the knowledge nor tools to do this, that having dedicated VRAM offloads RAM bandwidth use next to being slightly faster in general. The card was put in because the HDMI port on the board was bad though I did not expect this side effect.

  • @Misteribel
    @Misteribel6 ай бұрын

    "Always do your research". Whole-heartedly agreed! I've been waiting for the Threadripper Pro 79xxWX to finally come, and I'm aiming at 7975 for now, paired with dual A2000 NVidia cards for AI and LLM dev. Another niche, i know. And man, that stuff is expensive! My last system (dual processor Xeon, 12 cores each, 48GB mem) lasted over 10 years. After some minor upgrades (faster SSD, mem and graphics) it's time for something new. Been researching for weeks now, but hey, you only buy these beasts once a decade. My prev system is still faster than my high end 2023 laptop lol. Not in benchmarks, but in actual every day workload.

  • @steveseidel9967
    @steveseidel99676 ай бұрын

    Nice video. I agree with the points you make. However, the average user is unlikely to be able to fully understand what a given benchmark represents. That point is self evident based on some of the comments posted to this video already. So, given this lack of understanding, general benchmarks are still safer bet than the results of any one specific application.

  • @brentsmithline3423
    @brentsmithline34236 ай бұрын

    Great view of this issue. Thing is for normals the move from Intel to M1 even normals could feel the difference. Thing is for normals the M1 still does everything they need in a host computer system. These benchmark tests, and real world testing for the most part only effect the professional part of the Apple grid. Maybe five or more years down the road when most normals will be looking to purchase a new host system will they notice the difference in technology. When inside and application like viewing content I do not notice between my now OpenCore Sonoma 14.1.2 Apple iMac (Retina 5K, 27-inch, Late 2014) with 2TB SSD storage and my MacBook Air M1 chip with 8‑core CPU, 7‑core GPU, 16‑core Neural Engine, 16GB unified memory, 2TB SSD storage. It is when I am moving around, and opening applications that I noticed the difference. Had my Apple iMac Retina 5K, 27-inch for seven years before replacing it as my daily driver. Will have my Apple M1 MacBook Air a least until spring 2027, and more than likely longer than that before I replace it. I love these video clips about the latest technology, and the soon to be technology. Thing is the difference always comes down to "Show me the money ! ! ! ". When time is money and it makes sense to purchase the latest and greatest you need to do so. Thing is most normals I hate to say are not watching these KZread clips as they are no pointed at them, and what they need to know.

  • @zumbate68
    @zumbate686 ай бұрын

    Hello. Great video. For developer who considers it best to buy M1 Max 14" 32 Gb or M2 Pro 14" 16gb. Thank you for your professional opinion

  • @ConstantGeekery

    @ConstantGeekery

    6 ай бұрын

    Thanks 😊 The M2 Pro has a more powerful CPU, whereas M1 Max has more graphics performance and twice as many video encoders. For development, CPU is probably more important… but the RAM may be a factor too, depending on what sort of dev work you’re doing. Not having enough RAM would probably have more impact on your workflow than the difference in CPU performance.

  • @daveh6356
    @daveh63566 ай бұрын

    [REPOSTED] Well done. Yes, benchmarks can be highly misleading by drawing our attention to irrelevant/away from relevant considerations. It's still crazy that we're obsessed with ray-tracing benchmarks when it's an edge-case using dedicated silicon blocks (RT cores & SIMD) which almost no other task uses - the very definition of misleading. BTW. Geekbench 6 differs from GB5 in that it no longer runs multiples instances of the tests concurrently on the available CPU threads instead it adopts a similar multithreading architecture to real applications ditching the previous artificial scalability (much to the disdain of some users). Cinebench has also been updated as R23 tested the SIMD with an Intel-controlled library (Embree) which disabled Apple Silicon 2nd NEON SIMD. Also, Metal currently limits GPU access to 75% of total RAM but it's way more efficient than physically capped VRAM.

  • 6 ай бұрын

    Yeah, Lots of them are just flat out irrelevant for real world now TBH. Looking at 3d, They’ll use Blender or C4D CPU renderer which no one uses any more, cos we all use a GPU renderer outside of very niche cases Or they’ll use a benchmark that maxes all the cores, but your software just doesn’t work that way. Also Adobe software is grossly inefficient in many places and just doesn’t take advantage of what it’s offered for a whole raft of reasons. Looking at so called “blender and C4D” benchemarks would make you think you need a threadripper, whereas actually an RTX 4090 and a CPU with a high boost speed and fewer cores would be better for you

  • @user-rt5iw8hi1r
    @user-rt5iw8hi1r7 күн бұрын

    Men lie women lie but numbers don't

Келесі