Coming out of warp speed when those drives spin down. What a fantastic sound.

@MrFaaaaaaaaaaaaaaaaa

2 жыл бұрын

It's unreal! so cool

@PsychEngel

2 жыл бұрын

It's also the absoutly worst, what could happen. If all drives spin down at the same time, it causes vibrations so hard, it can damage the disks so they die...

I feel like you are helping the community by testing the niche cases and thus help the main cases along the way.

@JeffGeerling

2 жыл бұрын

That's the goal! If someone wants to build a large(ish) storage array with a Pi, at least with the CM4... they can avoid doing things that are definitely dead ends :)

@meqativ

Жыл бұрын

yesss!

Yeah using old versions of broadcom/LSI tools to flash firmware images because they don't check signatures or card type is a somewhat well-known thing in the crossflashing community (turning old crappy RAID cards in HBAs). Good to see that this time-honored tradition is still alive and well with more modern cards

@amp888

2 жыл бұрын

Yeah, also useful for crossflashing cards (e.g. to turn HP-branded models back to the vanilla LSI variant, which sometimes removes restrictions). The HP H220/H221/H222 variants, for example, are often significantly cheaper than the original LSI version. You can't crossflash the HP variants using the P20 sas2flash tool, but the P14 version works perfectly well instead.

@kyoko703

2 жыл бұрын

So what is supposed to be the real way to update the firmware then....? Or are these older cards not supported by the new flashing utility?

@leonkernan

2 жыл бұрын

Having just converted an old HP DL380 G7 into a TrueNAS box, this is all too familiar to me.

@marcogenovesi8570

2 жыл бұрын

@@kyoko703 Jeff in the video is flashing the right firmware for his cards so the tool should NOT fail like that. It is probably a bug. Crossflashing is another thing, where you flash a different firmware to convert a crappy RAID card to HBA (because of cost reasons, there is plenty of old low end SAS RAID cards in the used market and their price is much lower than the price of HBAs)

@PsychEngel

2 жыл бұрын

@@marcogenovesi8570 The problem was, that the firmware was way to old and the LSI tool wasn't able to verify the firmware. LSI verifies every version, so you won't brick the card. Older version of LSIcli probably doesn't do this. Sometimes you've to do a upgrade way from version 1.6 to 2.4 and from 2.4 to 3.9 and so on, till you're at the latest version 5.0.4. Thats, cause some firmware versions have major changes, which are only able to flash fine in a special way and the vendor doesn't guarantee a working card, if you don't flash in this particular steps.

Sounds like a turbine shutdown. 😬

@sayaaep

2 жыл бұрын

No. like Supra

@gus473

2 жыл бұрын

On a MH-65 Dolphin helicopter 🚁

@ejrandler

2 жыл бұрын

@@gus473 Or an S-3 Viking

@n0madfernan257

2 жыл бұрын

'All passengers, please remain at your seat until the seatbelt fasten sign disappears'

@PsychEngel

2 жыл бұрын

And death drives 😬

0:24. The Seagate Sound....On steroids!

Right now there's a bunch of sound engineers at Industrial Light & Magic facepalming that they could have gotten the best space ship power down sound just by turning off a bunch of hard drives 🤣

@Santor-

Жыл бұрын

Thats funny. The other day, a tram drove by us, and a kid commented "that sounded just like the X-fighter zooming by in starwars", and it totally did. So much so, I wonder if that perhaps is how those sounds were made.

My mother-in-law went to the pet store to get what's essentially a dremel for dog/cat nails. It was called pedi-paws or something like that. Instead she asked the worker for a pedi-file. Good call on the name you picked.

@JeffGeerling

2 жыл бұрын

hahaha

What does it sound like when you turn off 60 disks? The THX noise, apparently.

@MrJef06

2 жыл бұрын

Exactly what I thought. "THX, the audience is listening"...

Oh the joy of juggling different storcli versions and notes on which one works with what for unknown reasons - I know that game.

@JeffGeerling

2 жыл бұрын

lol the scars of a sysadmin

@RhodderzX

2 жыл бұрын

*twitches*, i still have a usb drive somewhere with a mass load of storcli and megacli versions on it (along with bunch other tools for dell and hp variations cause screw us right)

The fact that this is possible at all blows my mind

sadly at this time it's easier to buy 1.2 petabytes than it is 1 pi

@flethacker

2 жыл бұрын

there are lots of pi$$$ on ebay

@bknesheim

2 жыл бұрын

@@flethacker I do not like to be robbed. :-)

@JFat5158

2 жыл бұрын

I think I got lucky, bought an RPI4 on a whim a couple of weeks ago and it just happened to be in stock I guess. Good little nas/server and hardly uses any electricity.

Although the process of debugging and ground breaking is stressful, just how much fun was this. It looks like, to me, that this project was a blast.

Very nice! I'd like to see the test with the SAS extender and only one HBA: the system PCI should be more stable (no PCI switch, Gen2 speed) so better local speed (for array rebuild...)

Firmware flashing HBA's is the gift that keeps giving. The gift is frustration.

@PsychEngel

2 жыл бұрын

Or you read the manuals of the vendor (at least Broadcom has for the most cards a well documentation) and follow the instructions of the way to flash the cards, especially from which version to which version the upgrade is supported and use the "easiest" way the suggest, for example, flashing directly the firmware from the bios of the cards, from the EFI shell, from the EFI/UEFI itself (this depends on the EFI/UEFI you has and which support the PCIe cards has themselves for EFI/UEFI interaction) or use a flashingtool from the OS itself. Some ways are easier, some are harder, some are brutal^^

What you really want is multiple Pi hosts with smaller groups of drives running as a Ceph cluster. (Oh man, as soon as I started typing that, I got to the part where you say that you're doing exactly that. Good job!)

Amazing Jeff. I love your endeavors. Thank you for all your hard work

0:25 that sounds a little bit like the Prague metro C slowing down

Jeff and Wendel collab, nice :D

I ran a 60 SCSI disk Linux software RAID array in 2005. I was up to /dev/sdbc or something. 🤣 I had to backport kernel SW RAID code because the older kernel only had 4KB of memory to store the dev names. When I’d start the array the kernel would panic because the RAID info overwrote the buffer in memory to keep track of the array info. Fun times. Great video.

Yay! You showed my comment (7:00). In other news, I was surprised at how poorly the power consumption scaled when you spun down some of the drives. 200 watts for 15 drives seems a lot worse than 500 watts for 60 drives. I'm curious where all the draw is coming from.

@JeffGeerling

2 жыл бұрын

The fans alone on this thing probably eat up 30-40W. They're pretty massive, and spin at high RPM (each one is like 1 or 2A at 12v)

@MaxDad7

2 жыл бұрын

@@JeffGeerling If you did a smaller 15 drive rack, could one use less airflow? Or is it needed to push past the rows of drives?

@vk3fbab

2 жыл бұрын

I've got a PSU that draws 60W with no load. So you might find a lot of bare minimum kind of things in that 200W. Also PSU efficiency should increase as load goes up and then probably dives at some point before the maximum power.

Congrats Jeff! I knew you could do it. OMG, the hoops you've jumped through to get there is mind boggling. Is 'Winner Jeff" another one of your characters? The "thrill of victory' is a pleasure to behold. Even better to feel, especially after those "agonies of defeat", eh Jeff? Just goes to prove that it's all relative. We rarely appreciate the good times, until we've had to struggle for it. 😎 Great job, Jeff. Thanks for sharing.

Always wholesome to see Jeff shouting out smaller KZread Channels like LinusTechTips.

Thanks Jeff for the update of this wild experiment which is weird and pretty much fun, and I do wanna know more about ceph cluster with several Pis.

I love watching Wendall when we're talking storage, RAID, ZFS, Linux, etc. good call

Jeff , I’m so jealous of this! Awesome work and thanks for letting us in on the magic

Love the editing, keep it up!

The Rockchip RK3588 looks so sweet. Been keeping my eye on those. Thay would make great servers and high performance routers/switches.

It's really neat seeing you push these tiny SBCs to such extremes!

Awesome project! I'm very surprised how well you got this to run! Can't wait for Wendell guest episode.

That is fricking amazing Jeff !!

Thank you so much for taking the hits for us :)

Impressive feat! Thanks for sharing your expirence with all of us 😀

That sound of the drives spinning down! Jeez thats cool!!

Really cool video. I admire your perseverance 🙂!

I love your ambitions projects!

ngl - shutting down sounds reminds me of my old vacuum cleaner i used to have. Noise it made was straight from hell

Great video dude 👍

That's insane I'm so happy to have heard this 60drives spinning down

So want that sound bite of the spin down!

7:20 I’ve been waiting for this Ceph video for so long. I’m pretty sure it’s gonna be awesome 🥳😛😍

I wish we paid .13 the kilowatt where Im at.... its almost at .28 here. Thats around $130 to run this a month

@JeffGeerling

2 жыл бұрын

Ouch. If you would like to deploy a petabyte at my house, I will offer you the low low rate of 0.23 and skim the profit :D Of course I only have 35 Mbps up, so if you ever need to access data from it, that'll be a couple megabytes per second :D

Watching on my flight back to STL. Keep it up and I’ll wave while landing at Lambert!

lovvvve this ideas with pis :) thanks dear jeff!

Thank you for all the hard work for our entertainment and learning. My question is what about multiple Pi's as a Cluster for controller?

Wooo CEPH Cluster FTW! Really looking forward to seeing how you did that and how it compares to mine.

That shut down noise was exactly what I wanted

Thankfully the one 8TB 7,200RPM HD my Pi NAS has is way more than enough for my use case. Is fantastic to see it can be done if I ever needed it. Or had the cash, for that matter.

One word... Amazing!

No ECC memory was the first deal breaker to come to mind. Cool and determined though, so much so that I watched it entirely!

0:21 Love it!

Thx 4 overclocking ! 10% Note throughput is quite nice. You could Test against an older Desktop Low end CPU ( less Power Draw For comparison.)

Awesome video as usual! 500w of constant power draw.... wooo yeah. I have a pair of QNAP TR-004 and those are loud enough. Only have them running when I'm writing or reading.

@KameraShy

2 жыл бұрын

500 watts is about half the draw of a typical small window air conditioner.

Thanks Jeff I need a new Apple keyboard now 🤣 Peta File made me spill my tea 💖🤣

Pretty impressive!

I really wonder how nice and warm that is gonna be for winter and if it'll keep Red Shirt Jeff warm. Honestly tho, mega project and world needs more people willing to do projects/challenges like this - just to see if it can be done 🤘🤘🤘😉

Fascinating!!

Oh god yes, a Jeff Geerling video, finally something to watch!

Yay for getting Wendell into this :D

@Jeff Geerling you can use pi cluster supercomputer as parallel computing so that cluster can handle many operation in raid 0

I’d be interested in how a cluster of pi’s would handle the 60 drives. It’s probably a massive task and you’ve already mentioned your working towards that. Anyway, thanks for making these videos!

bruh .... the amount of testing and effort .... bruh, we appreciate you!

0:21 sounds like a reverse audio of the THX intro.

Really looking forward to the Ceph cluster. It would be interesting to see if you can increase performance incrementally with one Pi at a time.

How good is that sound at 00:26 🥰

Love the Wendel shirt.

Is the issue of using 3 HBAs due to limitations with the interrupt controller on the Pi SoC? I used to have to flash firmware with older builds of the tools and then apply the updates sequentially, but that was way back in the day (like 15 years ago) so I was shocked to see that was still a thing! Thanks for taking the time to jump through all of the hoops! Nevermind, I made it to 07:45

Hey Jeff, it might be that pice switch that is having the problem with having 3 of the cards at the same time. If I’m remembering correctly that card was made mostly for mining. So it would be one video card a time. I would look there.

good to storage school work

Thanks!

I don't envy you on getting those HBAs flashed. I had one where I had to take it from Linux, to Windows, to FreeBSD to UEFI multiple times before I finally got it flashed and working.

I don't know why you didn't choose Peta-FIle...seems like a great name!

@MaxUgly

2 жыл бұрын

When he said that I heard AvE's soice say "aaaannnnd DEMONITIZED"

Those spinning down drives sound like a Jubilee line tube train coming into a station

The power down sounds like a Transformer laying down for a nap.

Astonishing!

Flash the HBAs to IT mode and compile in ZFS to the kernel if you want decent throughput and reciliency.

Jeff, you're an inspiration! I find it fascinating how you get all those niche projects working. It got me wondering if I could do something similar. I'm wondering if an M1 mac mini running asahi linux could become a decent NAS + self hosted application box, similar to how some NAS's allow you to spin up some dockerized apps on your home network besides simply storing and serving your files. I have no clue how to make it all work optimally yet, though. Similar to how you said adding 10GB ethernet on a raspberry pi doesn't really add any speed benefits, I am unsure if things like 10GB ethernet, thunderbolt 3/4 external harddrive enclosure, hardware / software raid, etc. would be worth spending money on or if the performance gains are minimal. Having HDDs for storage and an SSD for caching also sounds good, but not sure how I would implement that. I do think the energy efficiency and performance of m1 will make for a great base for this project, just not sure how to do the cost benefit analysis of the other hardware and software decisions I would have to make. I've got a lot to think about. What's your first impression of the idea? Do you think it's viable? Any tips? Have you considered doing something like this before? Thank you for your videos and always being so passionate in explaining your projects.

Hi Jeff I'm currently working on my own Ceph video / installation on raspberry pi. I'm trying to get version 16 to run on Raspbios but sadly the only packages I found was version 14 for bullseye and the packages had missing builds in the ceph repository. So my last resort was creating an emulated pi and build my own Ceph packages. Have you found another reliable way to install Ceph? Best regards Daniel

Ceph on Pis, omg 😂😂 .... can't freaking wait for it! XD Cephadm is great btw ;) But I'm not sure if they have aarch64 images. If not you should build them and use that, as every other deployment method is much more cumbersome

This thing would cost ~1350€/y in electricity bills alone here in Europe... Still peanuts in comparison to the initial investment cost, though.

@marcogenovesi8570

2 жыл бұрын

If you actually need one Petabyte of storage, you can probably afford the power bill for this

@MPnoir

2 жыл бұрын

Yeah as a German (one of the highest energy prices in the world) i wasn't sure whether to laugh or cry at those US$ 0.1296/kWh

@maxw1567

Жыл бұрын

In my area it would be around 2100€ a year.

5:42 High peak power consumption during boot-up (the experts among us call this "inrush current") once made the electricity supply during a trade show I attended collapse. I had a 15-bay server case with 600 W redundant PSU with me and wanted to show it to potential customers. Instead, they only saw pitch black darkness. :(

@JeffGeerling

2 жыл бұрын

Oopsie!

Awesome!

I wonder if there was some IRQ sharing issues when you get to 4 HBA devices. I know it's a weird one, but I've had this happen with PRI T1 cards in the past and had to make sure they weren't sharing IRQs on the PCI-Express bus

How's the MTBF on those drives when all placed in one chassis and sharing all that vibration? That's a real issue in case and rack design for storage arrays.

So you immediately ruled out "Peta-file", but *not* "The Big PP" ? I want to talk to HR.

@tylercgarrison

Жыл бұрын

😂😂😂😂

PetaPi is a good name. Though Petaberry Pi, Pi Byte, or Raspeta could also work.

Hey Jeff! I was actually wondering if you have any other sbc's that might have more pcie lane's to test projects like this against the pi 4? something like a jetson nano/tx2/xavier module

I genuinely think Pi-tabyte sounds pretty good for this :)

Sounds familiar. I was testing a Thunderbolt Fibre Channel HBA on a Monterey (intel) Mac today. The card seemed to work, but the config tool reported IO errors when attempting to check the firmware version. Thankfully, three year old drivers worked fine on Monterey to detect and update the firmware and now I can run up to date drivers and access our giant tape library from an M1 Mac Mini. So maybe that should be your next video? Talk someone into lending you a 3573 (Dell TL4000/IBM TS3200) tape library with a couple of LTO8 drives and a FC HBA? ;) "This time, I only have half a petabyte, but peak power consumption is only 150W!"

So i had bought old SAS server with only 6 drives, and it is consuming, according to idrac 175W, never gave it a thought, that drives can accualy take a big part in this. Also in Poland, electricity costs are bigger that u have so i pay for that almost 20 USD monthly... But i like the fact that i can run on it surveillance server, game servers (Valhaim for example), nextcloud ect. It is still cool project, i just like that i can just dump many things at my server, and it costed me 336 USD by todays course.

Sound effects engineers would pay for those sounds.

that is completely bananas 😀

Are you going to run a more typical threadripper type setup? And are you planning to do a software or hardware raid. For faster drives i know that a hardware raid card is much faster than doing a software raid especially even a threadripper gets bogged down with not too much through put with the software raid.

This is a very interesting experiment. Have you consider to run a cluster of Pi's to increase the throughput?

Hey Jeff, it would be interesting to see how you’d setup raid with a few SSDs for retropie. I was following another one of your blog posts hoping to combine the two concepts of creating a simple RAID1 Array and booting from the SSD. Definitely been a struggle on my end. I’ve been getting a “wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error” Long story short: I just wanted to know if you had any more information or tips on solving this (BTW I’m a cmdline noob 🙃)

Yeah I've seen things like this now and then. Firmware updaters are a pain. I have done similar for fiberchannel cards. I used the dos based updaters.

I feel you

YES the next Pi5 must have the Power like the Rockchip RK3588

I was rooting for the name PitaByte, but I was kinda hungry at the time, and that sounded kinda good to me then. Actually, still sounds good.

Ceph on Raspberry Pi? Sounds like fun!

When Wendell gets involved, you know you're in for a good time.