Your PyTourch was great. I recommend it to anyone learning ML, really can help you understand how GPT is built.

@mrdbourke

5 ай бұрын

Thank you Bill! Glad you enjoyed!

Yo, it’s been a while I saw my teacher, nice to see again and good video by the way. More blessings bro.

Good to see you again you made machine learning and ai fun

Nice! Missed you buddy!

@_MrCode

5 ай бұрын

I'm sure this video touched your heart.

@anthonypenaflor

4 ай бұрын

I actually thought this was your video when it popped in my feed!

@AZisk

4 ай бұрын

@@anthonypenaflorI don’t have such a beautiful desk.

is your test is using the Mx GPU : are TensorFlow and Pytorch optimized for Apple GPU silicon ?

So cool that you used 10 shades of green in your graphs. It's very convenient to distinguish

@mrdbourke

4 ай бұрын

You’re right, I made a mistake here - I only really noticed this reviewing the video, I guess since I made it I could tell the difference. Next time the graphs will be easier to distinguish!

@hyposlasher

4 ай бұрын

⁠besides that, the video is awesome and very informative

Glad to see you back.

Surprised, that you did not include RAM bandwidth in the beginning. Whenever you do non-batched inference, the memory-bandwidth becomes your main constraint, instead of your GPU-performance. As shown in your M1 Pro to M3 Pro comparison. llama-cpp's M-series benchmarking shows really nicely, why the M3 Pro with it's 150GB/s instead of 200GB/s memory is a problem, not its (faster) GPUs. If one just does inference and has large models, requiring lots of RAM, the M2 Ultra really shines with its loads of 800GB/s RAM. Totally agree, that with learning and batching, it's different and NVIDIA's new GPU performance blows away Apple silicon.

@imnutrak130

5 ай бұрын

that's KZread quality education, good enough but most of the times they are missing crucial details and due to that mistake twisting the truth especially for performance. Although this person has studied and gets paid BIG salary to know such details ..... weird but I boil it down to maybe a simple human mistake. Still a good video!

@mrdbourke

5 ай бұрын

Woah, I didn't know about the lower memory bandwidths between the M1/M3. Thank you for the information. I just wanted to try raw out-of-the-box testing. Fantastic insight and thank you again.

Hi Daniel, will you be teaching something more than image classification? You are the best programming teacher I have ever followed. Looking forward to your new deep learning course on ZTM.

this video is exactly what i was searching for, thank you so much for proving such clear and usefull information.

Thanks a lot for the valuable information. You saved me a tonne of time to come to a conclusion. cheers mate

OH FINALLY, waiting for that. U are king bro

Thanks, Daniels, for the video and for sharing the materials' links. You're a legend. Got an M3 Pro 14" (11-core CPU, 14-core GPU, 18GB) last month and have been wondering it was an optimal move.

Excellent comparison, thanks 😊

Very good one. Thank you so much.

Thank you for the Knowledge it really gave me an insight.

Hi Daniel love your video, can you please suggest which laptop is good for deep learning mac or windows or linux

Great video, could you please update us if the new mlx change the result or your conclusion at all? Would love to know if the m series chip is as good as what the others are saying .

This is interesting! It seems between the m3 pro 16GB (150GB/s) and m3 max 32GB (400GB/s), and considering the m1 pro 32gb (200GB/s), would you suggest that RAM is a much important factor to these ML tasks than memory bandwidth? Or other? Would be keen to see a test between an m3 pro 32gb vs your m1 pro 32gb to see if memory bandwidth of 50GB/s difference has any real world result differences. (also one less GPU core but faster boost in M3 pro)

These machines are great as laptops, for desktops, Intel 14th Gen i9 plus Nvidia GPU smoke them away.

If portability isn't a requirement, then the Mac Studio Ultra should be considered with its 60 GPU cores and 800GB/s memory bandwidth.

Hey Daniel, Consider trying their MLX versions as some of the models enjoy performance gain as high as 4x compared to their torch counterparts

@siavoshzarrasvand

5 ай бұрын

Does MLX work with Llama 2?

@user-qj2xm6fl2u

5 ай бұрын

@@siavoshzarrasvand yup and much much faster than llama.cpp

I believe you can also target pytorch to run on Apple silicon's NPU rather than the GPU. And I am sure it will perform better. Though not sure about how much memory the NPU has access to. It will be great if you can explore this and do a video on it.

Can you also make comparison with Neural Engine of M processors?

Please redo using MLX as that's what the developers using this laptop will probably be using.

@andikunar7183

5 ай бұрын

Especially since this week Apple released MLX with quantization support and other stuff.

@mrdbourke

5 ай бұрын

Fantastic idea! I started with TensorFlow/PyTorch since they're most established. But MLX looks to be updating fast.

@miguelangel-nj8cq

4 ай бұрын

not even that much, it doesn't even come close to those who really use Tensorflow and Pytorch, besides that if you have your production environment in the cloud, those 2 libraries are better integrated than MLX, in addition to the fact that for quick deployments you already have the containers preconfigured and optimized with those libraries and CUDA since the cloud servers are dominated by NVIDIA and not Apple's "Neural Engine".

Hi sir! Can you suggest the proper/recommended way to install tensorflow in macbook?

I am planning to buy M3 pro. Which one should i go for 30 core GPU or 40cire gpu. My use will be around running some prototype models in LLMs.

Question: how did you get pytorch / tensorflow on the m3 max chip? There is no current support?

Would you be able to benchmark maxed out Mac Studio with M2 Ultra, 192 GB ram and 76 GPU cores against the nVidias?

Finally somebody explains this shit properly not like all the other youtubers that only use it to create videos

Thanks for this, really useful and confirms my initial thoughts on just getting an M1 Pro 16GB over M3 8GB (M1 Pro is slightly cheaper). My M1 Pro is similar to yours 10 cpu + 16 gpu but just 16GB and has been slightly faster on both pytorch benchmarks. I then was curious to see how it compares to a quad RTX 1070. I modified your code (I will make a PR) to use all four GPU for CIFAR100. In general it is faster than the M1 Pro, what is interesting is how it compares to single card vs quad cards. CIFAR100 on small batch it was really bad, however by 512 batch size it was faster than a single card (34 secs on 1024 batch). It keeps on improving until 3072 with 16 secs, then gets worse at 4096 back to 19 secs similar to 2048. Also by 4096 batch size the GPU VRAM is almost full and close to 8GB.

Very helpful thanks Daniel. I was going to race out and buy an M3 to do my ML work, but I will hold off for now. I suspect Apple will do something to help boost performance considerably on the software side, but who knows.

Sir I follow all of your blogs, vedios etc I want to be a ML Engineer so i enrolled in your 'Complete ML and Data Science course on ZTM'. What a marvellous way of teaching ❤❤

One thing is clear even as a PC person Mac had a steep advantage with M3's dynamic ram to vram conversion and mow power. Sure they don't have the hardware or software of nVidia but for some Ai users, the entry price for the VRam is a winner.

Hi Daniel! What a great PyTorch tutorial you have made. Thanks for that! Also thanks for that speed comparing video. Can you record the video that comparing the speed of different Colab versions? I mean free, 10$ and 50$. Also here can be added M3 max and your Titan (which you already have done). Maybe one of your friends has 50$ account and he can do that tests for you [for all of us :)]

I'd love to see you test the M3 Ultra with 64 GB RAM when it comes out, I am using the M2 Studio Ultra at present and wonder if it will be worth upgrading. Running batches, it gets warm, but I've never heard its fan yet.

Question: I bought the M1 max with 64 GB ram, and 32 cores GPU. Like you, I am now extremely satisfied with my purchase two years later. Question: I like your set up using the Apple machine in conjunction with a box with that RTX4090 installed. Would that set up run in parallel with my GPU course? And similarly, if I added equivalent ram to that box, would it work together with my installed 64 GB?

In the process of learning ML/Ai related tasks. Based on your experience would you prefer a 13” MBP M2 24GB RAM ($1,299 new) or a 14” MBP M3 Pro 18GB RAM ($1,651 used)?

@mrdbourke

2 ай бұрын

The 24GB of RAM would allow you to load larger models. But it also depends on how many GPU cores the two laptops have. Either way, both are great machines to start learning on

Would be interesting to see how the 128GB version of M3 Max performs compared to the RTX cards on very large datasets, since 75% ~ 96GB could be used as vram in that Apple Silicon.

Your tests just prove how bullcrap synthetic benchmarks are. Love your work.

Appreciate hardwork. But please consider using better color scheme for bars. They all look the same.

Happy Christmas Sir❤❤

@mrdbourke

5 ай бұрын

Happy Christmas legend!

I would like to see if you have utilized intels gpus.

You missed memory bandwidth, the M1 pro has higher bandwidth than the non Max m3 macbooks.

@mrdbourke

5 ай бұрын

Thank you! I didn't know this. Very strange to me that a 2 year old chip has higher bandwidth than a brand new chip.

Hey daniel, just wondering can i fine tune my llama 13b param model on m3 pro with 14 core gpu 11 core cpu 18gb ram

Do apple silicon chips handle the workload on neural cores themselves or do they need to be specifically invoked via an sdk from the code? what was the workload on those during each test? I wonder if they were invoked at all. if they were, it sounds like they do not matter compared to GPU, however it's claimed they can do something like 17 tops which outperforms any google coral. Moreover, apple claims neural cores are 60% faster on m3 compared to m1. confused now.

can you test snapdragon elite x when it comes out with a laptop vs the apple m3 please. they say it has better npu than m1,m2 and m3. coz im planning to buy a deep learning laptop next year.

Great job! Would be great to include some popular Windows laptops as well in the comparison :)

M series doesn't allow for external GPUs so how do you hook a 4090? This would make a good video.

Great video

Outstanding

At one point you say the bottleneck is memory copies from CPU to GPU and back, but the M-series doesn't have to do memory copies because it's all shared memory. In fact, one of the first optimizations for code on Apple Silicon is removing all the memory copying code because it's an easy gain. Have you accounted for this in either your code or the library code you're using, or both?

what are the parameters that you used for powermetrics? I liked the monitoring you had in terminal.

@mrdbourke

5 ай бұрын

I used the asitop (github.com/tlkh/asitop) library for monitoring in terminal

can ypu try llma 2 70b with 128gb ? m3 max

from my experience, tensorflow optimization is a little better than pytorch for convolutional models.

Are they faster than Intel Ultra Core 155 or 185 ?

I think it would be interesting if you standardized your measures by memory and number of cores.

The comparison between the M1 Pro and M3 Pro is not ideal. The M3 pro you are testing is the binned version with only 14 cores however your comparing it too the full M1 Pro. To get an accurate performance measurements its best to measure both the full chips rather than the binned version that way we can truly see if the memory bandwidth has any difference when it comes to Machine learning

Has anyone tried the same stuff on mlx? I am wondering if it makes it faster, I had insanely fast inference using it on q4 mistral.

tell us the difference with m1 max 44 GPU 32 gb ram with the M2 M3 max please

although it's nice to see vision models most people wanted to see inference w/transformer LLM's then fine tuning LORA, SFT. llama2 q40 is hardly a test even an 8gb mac metal can run that. would like to see different quants at 33b and 70b with different loaders, AWQ, GPTQ, exllama etc.

This is really a great video. The problem I have is all my development is on a laptop and I think this is wrong. The conundrum is simple, I will present my work, that's a given, so how do I dev on a much more powerful desktop and still have the ability to present my work? I hate powerpoints of screenshots, I want to really show what I'm doing.

@eddavros264

4 ай бұрын

How about connecting with ssh to your desktop from your laptop?

There is a CoreML optimization for PyTorch on Apple Silicon. Was this used?

@greenpointstock6345

5 ай бұрын

Do you have more details on this? I've looked for something like this before and all I can find is something that seems to let you convert PyTorch to CoreML, or info on Pytorch using the. GPUs but not ANE. But I probably am missing something!

Do you have a masters/phd degree on ML ? Does your job require data science degree?

how come m3 max is slower than m3 regular, and m3 pro in PyTorch test?

I'm really struggling to see the difference in some of the darker shades of green, which makes it hard to know which bar is which

@mrdbourke

4 ай бұрын

Yes you're right, I see it now too. In future videos I will use a different colour palette :D

7B parameters / 25 ( 25 and delete 7 zeroes or divide by 250 000 000) = 28GB which is close enough for a simple maths for GB Memory for Molde Parameters.

IMHO macbooks are only inference machines, not training. It's great to run locally 7B, 13B, 30B LLMs (depends of your # of RAM), run quick stundents training on something like MNIST. I personally write code for training and run experiments with small batch size on my M1 pro, than copy the code on my 3090 PC and run long training with bigger batch and fp16. While PC is busy, I run next experiments in paralle on laptop. If you load with big training your main laptop, you will have uncomfortable experience if you want browsing, gaming, etc in parallel with training.

It's a good idea going for a new M3 MacBook Air model with 16GB for starting to learn ML?

@mrdbourke

2 ай бұрын

Yes that would be a perfect laptop to start learning ML. You can get quite far with that machine. Just beware that you might want to upgrade the memory (RAM) so you can use larger models.

Thank you, I was hoping someone would look into how these machines perform on ML, not only video processing. The results are quite disappointing.

You posted the single-core CPU scores for the M3 Macs, that's why they are all the same pretty much.

@tybaltmercutio

4 ай бұрын

Could you elaborate on that? Are you referring to the ML timings?

Thanks!

@mrdbourke

2 ай бұрын

You’re welcome!

Why didn't you compare M1 max?

The M3 Pro being slower/not much faster in some tests is probably because of the slower ram. I'd be interested to see how 30 and 40 series cards stack up but considering the cost of the laptops already this is quite the effort so no complaints.

@kborak

5 ай бұрын

my 6750xt will beat these things lol. You macboys are so lost in the woods.

@nadtz

5 ай бұрын

@@kborak I'm not a mac user, I wouldn't buy Apple hardware for love or money. But the chips are still pretty good so it's interesting to see how they stack up to a better GPU for this kind of workload.

your M1 Pro RAM is about twice that of your m3 pro, so maybe that is why it performs better than the latter.

@mrdbourke

5 ай бұрын

Yeah you're right, I also just found out that M1 has a higher memory bandwidth than the M3 (150gb/s vs 200gb/s) thanks to another comment. That likely adds to the performance improvement on the M1. Strange to me that a 2-year-old chip can comfortably outperform a newer chip.

@JunYamog

5 ай бұрын

I have only a 16 GB M1 Pro, on the first 2 benchmark I get similar or slightly faster speeds. I will try to run them on the other benchmarks, I got side tracked modifying the 1st benchmark to run on a quad RTX 1070 setup.

Finally a useful video. Too many “reviews” focus solely on content creators. Now I know I can do light ML on my Mac. And do the heavy lifting with my 30 series RTX card.

Bro can you please tell me which latop i need to go for machine learning Windows or M3?

@Outplayedqt

6 күн бұрын

Windows - specifically, the MSI Titan.

Which one would you prefer buying now between M3 (8gb ram and 512ssd) VS M1 (64gb ram and 512 ssd) M1 = 2,286 usd M3 = 2045 usd

While this is a nice buying guide for my next laptop, this is just a shining endorsement for Google Colab. What an insane value for new-comers looking to learn while not being hobbled by old equipment.

~$3000 for that deep-learning PC seems super cheap. It will cost double the price where I live...

The Titan is five years old. Would have been nice to include a current GPU. like the 4090. It can be 2.5× faster than the 3090, which is newer than the Titan.

💁🏽‍♂You should try MLX instead of PyTorch w/ mps backend for Macs and then compare results with CUDA ones.

.... and what about M2 pro mac ?

I wish you would make videos covering AI news your probably more qualified to talk about new developments in this space then 80% of these “AI channels”

Also worth noting that the GPU on a macbook only has access to 75% of the unified memory.

On my m1 max 64GB... I'm getting 8208 on Core ML Neural Engine... My Core ML Gpu falls more in line at 6442... All this while powering 3 screens. Watching youtube and a twitch stream. Not that I expect those things to add much load... But it is nice to have a machine that can basically do everything at once with near zero penalty.

14" thermal throttles btw

please do a mlx coding video

🔥

I don't think the choice of the graph colors was good

Your tests are probably gated by RAM so aren’t showing processor and GPU deltas. While most tests ran, there was probably a lot of VM swapping. :-)

Would be very interesting to normalise the results based on cost of hardware, after all it always comes down to spend!

@jks234

4 ай бұрын

Hm, in my opinion, a strange metric because "effectiveness per dollar" doesn't really tell you much. My bike costs $300 and my car cost $10000. My bike averages around 20 mph and my car 75 mph. That comes out to 30x the price for 4x the speed. Did this tell you anything? In my opinion, no. What is a far more useful metric is the options the purchase makes available to you. If I have a car, traveling 10 miles for food is a very easy decision to make. If I only have a bike, traveling 10 miles is a major decision. With the right hardware, you unlock options like "iterative experimentation" whereas before, you had to carefully choose your workloads. And as he mentions, certain configurations simply lock you out of certain desired avenues. (8 GB of RAM is too little for many projects.) So yeah... spend is not a very useful metric, in my opinion. Choosing the bike over the car is a pretty pricey choice for reasons beyond money.

@aparkeruk

4 ай бұрын

Interesting analogy, but with the car many other features (in the warm, carries 4 people....). When buying compute power for AI then yes you could also consider laptop might be better than desktop for convenience, but not really like the car example. If you were comparing mainframe to laptop to desktop then might be nearer this analogy. Guess will not matter soon, as cheapest will be cloud purely by volume!@@jks234

Sir please make vedio about career in ML in 2024

Both the M3 Pro and the M3 Max you tested have lower bandwidth than the previous M1/M2 Pro / M1/M2 Max and since bandwidth is hugely important that was reflected in your results. The M1/M2 Pro have a 200 GB/s whereas the M3 Pro only has a 150 GB/s. The M1/M2 Max have a 400 GB/s bandwidth but the M3 Max model you chose only has a 300 GB/s bandwidth (there are also M3 Max models with 400 GB/s).

@PMX

3 ай бұрын

As an example, I get 30% faster inference speed on my M2 Max (400 GB/s memory bandwidth) as you got with the base M3 Max (300GB/s bandwidth).

@mrdbourke

3 ай бұрын

Wow! I didn’t even know this… excellent info. So what makes the bandwidth increase from the base models? Is it RAM upgrades or storage? Or something else?

@mrdbourke

3 ай бұрын

@@PMX makes sense!

@PMX

3 ай бұрын

@@mrdbourke The M3 Max with a 30 core GPU has a 300 GB/s bandwidth and the one with a 40 core GPU has a 400 GB/s bandwidth

@mrdbourke

3 ай бұрын

@@PMX woah so the 40 cores is worth the upgrade. Is this information on Apple’s website? I must’ve missed it

hi lenovo loq i5 12450h 8gb 4060 80k vs ideapad ryzen 7 5800h 6gb 3060 71k purpose machine learning college purpose

Rtx 4090 has better Tensor cores, so that's hard to compete even with a M3 Max.

@andikunar7183

5 ай бұрын

not for pure non-batched inference, where the memory-bandwith as well as memory-size is the main constraint. There the M2 Ultra's 800GB/s vs. 4090 1080GB/s is not so bad. The higher GPU-power of the 4090 really shines with batched processing.

Seems M3 is crippled on most tests due to low memory to be a real M version vs, and more a “how low ram can hurt you”. I would have loved all models at the same ram, or all being base or maxed out models. That said, interesting insights on the effect of ram and how nVidia performs when we’re talking strict GPU.

@mrdbourke

4 ай бұрын

All M3 models are the base variant in their category. Only upgraded model was the M1 Pro (can’t buy anymore). But yes you’re right would be cool to see them all on the same RAM!

Was about to sleep 💀

Where's the Windows PC?

If you really want to show the Apple silicons advantage just wait till the M3 ultra comes out with 256GB Memory and then use a model that needs that much memory. Then the only comparison would be ~3 A100s. With apples new MLX and flash is all you need we might even get better results

@OmarDaily

5 ай бұрын

Can’t wait to pick one up, I was planning on a M2 Ultra but, I’m expecting to keep this machine for a good while as part of my server rack.. So M3 ultra it is!

@SloanMosley

5 ай бұрын

@@OmarDaily I’m very jealous 😛 I have the 64GB M1Max and if if a new MOE model comes out that rivals gpt4 it might just be worth it