My favorite IT-tech channel :) I use almost the same systems, but after i discover Your channel, I became convinced that these are not my strange ideas, but common sense in spending money using open source in local governments sector. Best regards from Poland.

Love ZFS and Servers! Thanks Tom!

im a unraid guy but really enjoy your FreeNAS vids

We have PCIe NVMe disk cards in those raisers, that's good for cache/ZIL I guess. The vflash via iDRAC is as confusing as you found, it can only provision filesystems or ISO but no boot device. Booting from SD is a different option (I think ESX comes with IDSM)

pretty cool, thanks for the video

Tom rocking that free as shirt

Thanks for the vedio .. but i would you to add one thing that to install more ethernet network interfaces on this demo server and trunk it as LACP for more a data transfer throughput Just a suggestion

If I have a mix of 24 different brands/models of 2.5" 1tb drives out of used laptops could I put them in a server like this and use them as a large storage pool with redundancy?

just burn the iso to the pen with rufus then boot into it through the idrac or boot mode lifecycle control then config the ips there are vids on the tube to do it :)

Problem with this test methodology? Memory cache. You will get amazing read speeds because all the writes you just did (to create the file you are going to read) are still in memory. I have an encrypted array of 6+2 6TB WD-reds (not SMR :-) and I got a read performance of 10259Mb. Watching the disks via gstat, and they weren't being touched. The cach hits (zfs-mon -a) was > 99.5% (the machine was still running other stuff). I do have plenty of memory. Now if I set primarycache=metadata, then my read performance drops to 210Mb. 50 times worse. Admittedly the array has been running for years, is 80% full and 20% fragmented so there may be some extra seeks involved, but still, that seems more realistic for what I assume are random reads. I believe that the 24 drives would perform much better than this. Writing with no sync was up at 2680Mb. With sync 624Mb. I do have a zil. Its a little 64gb optane. its only 2 channel pci nvme (the bigger ones are faster) but it was cheap. I run all disks attached to VMs with sync. The optane has no on board cache, so it doesn't need any battery backup. Consumer SSDs can lie about when an item is actually synced to disk and have no built in battery backup to ensure that their cache is flushed to permanent storage (unlike many high end SSDs). This is the reason why only certain SSDs are actually "safe" when being used as a zil. Another issue I have noticed is that if the zil is encrypted, performance suffers. The extra latency makes a difference. The above is just my current state of knowledge about my experiments and reading on zil/zfs. If any of my current opinions seem to be in error, I am more than happy to be corrected.

@johnmadsen37

4 жыл бұрын

Right. He should test the writes. Never with the same file. Then reboot. The rest all reads. Never with the same file. With cache on and off. Direct io, write back / through, disk cache on off .... many many options But let’s be real. These guys are not professionals. They are youtube people making entertainment videos. It’s all opinion type content. They really have zero clue about the technology and software.

And just three years later I got one on eBay for $650 with 2 Intel e5-2680 v4 CPUs, 256GB of RAM, and 10 1.2TB SAS drives 😆

FYI: vFlash is where the SD card is used to store things like configuration data, previous firmware files (to allow rollback after an update), etc. It's not bootable. What you're thinking of is the IDSDM: Integrated Dual SD Module. They're configured as an active/passive mirror, and can be installed to and booted from. The confusing bit might be that on certain servers, both the vFlash SD card, and the IDSDM cards are on the same physical PCB (they are on 14G, can't remember if 13G is the same or not)

@CriticoolHit

4 жыл бұрын

I have a 730xd and unfortunately the vFlash slot is the ONLY flash slot on the device. At best you have access to 1 internal usb 3.0 header just behind the flex bays. That said. With Remote file share built into the idrac as well as the simplicity of pxe booting they really don't want us to be putting operating systems 'on' the metal anymore. Uniform metal just slid into a rack and given an ID. It feeds off the pxe and takes it's place in the chain gang with literally 'no' effort (a la enterprise). Not terribly conducive to a homelab but hey.

@R1PPA-C

Жыл бұрын

I've got the dual SD cards in my r720, it has exfi on there which I can't get access to due to unknown credentials, now... Is it wise to put the OS on these SD cards? I'm reading a lot of conflicting opinions on it, but with this being a mirrored setup I'm leaning towards the idea that this is what it was designed for? Would the SD cards hold up over time, the server is literally only being used for one program (it's a render node) and running server 2016, all files are saved to an external nas

Very Helpful..😘

there's vflash, and then there's a dual sd add-in module which is what you would put an os like esxi on, they're not the same. vflash is literally a small proprietary provisioned sd card from dell that you'd transfer iso's to to load up & mount while in idrac.

I had freenas on my 730xd but I found it a waste. Cpu sat at 4% so i moved it to unraid to run vm's. 4x6tb disk pool plus 4 dell 12Gb sas ssd in raid 10 for cache... cache write speed is 1440MB/s sustained... only down fall is lack of pci support. Every non-dell card I add forces fans to speed up making it too loud for small office or home use.

Hey Tom, thanks for another interesting video! Your comments about using a USB stick to boot the R730XD led me to ask you about your general opinion on the subject of using USB sticks to boot on production servers that operate remotely. I have a Dell R610 8xSFF with a PERC H200 flashed to HBA IT mode (LSI9211-8i) that for the life of me I can't force to boot from the ZFS pool once I install ProxMox on top of it. I need to move this server to a data center sooner than later as I will be using it for hosting live websites. This server has a pool of 4x Samsung 860 EVO 500 GB drives in a RAID-Z1 (~1.4TB usable space), which puts me in a somewhat silly situation. If I use another slot to install a small bootable SSD, I will be happy with reliability. Heck, I can even put two mirrored drives to be safer still. But if it turns out I run out of disk storage, I will not be able to put another set of four SSD and create a second pool. If I use a quality USB stick (e.g. SanDisk Ultra Flair 32GB USB 3.0) I can keep my four SFF slots in front free for a future storage upgrade. But am I not risking too much? What I fear is that if the USB stick fails, I will not know it until I reboot next time. And once it goes off line, I will have to rush to the data center (10km away). There are solutions for this (e.g. have two servers at the data center connected in a ProxMox cluster with active replication) but the datacenter cost is not yet justifiable (hosting two servers is the same as buying two 256GB SSDs every month....) So I'd be interested in your take on this.

@jj-icejoe6642

4 жыл бұрын

You can mirrored usb. That’s what I have on my r720 Unraid server

@julianhamann4925

4 жыл бұрын

It is possible that your RAID Controller doesn't make the drives available fast enough. I have that problem with 5 PowerEdge R620 and PERC 310 Mini IT Mode. The solution is to wait ~15-20 seconds before trying to import the zfs pool. This is described here: forum.proxmox.com/threads/proxmox-6-efi-boot-zfs-rpool-race-to-import-fail.56265/

@Darkk6969

4 жыл бұрын

@@julianhamann4925 RAID controllers never recommended for ZFS setups. Use regular HBA. ZFS have it's own software RAID so going through a hardware RAID controller just adds more overhead and headache. Dell servers are easy to swap out the controllers for HBA.

@julianhamann4925

4 жыл бұрын

@@Darkk6969 That is why you flash them to IT mode. From what I understand, it basically is an HBA then.

Does anyone know if the R730xd has any issues with Power Disable drives?

Hey Tom, a little off topic, it looks like nice universal rack "rails" you have there. Can you point me where to get them? i need to get some for myself. Thanks

@MajklKaplan

4 жыл бұрын

Avery's Tech thanks mate :)

Where is the Perc battery? I dont see it on the Perc RAID Controller...

What flash drives do you use for FreeNAS installs? I've had problems with the 16GB Sandisks I've been using. They don't seem to last very long before needing to be replaced.

@arigornstrider

4 жыл бұрын

Just ordered a stack of 16GB 3.0 Sandisks (silver body with black plastic loop around the back end). What model are you having issues with?

@Darkk6969

4 жыл бұрын

Been using SanDisk Ultra Flair 64GB USB 3.0 Flash Drive since Jan 2017 without issues. Yes I know 64 gig is overkill but thinking about endurance for number of writes.

Nice video. The same on unraid :) I have 2 R720

Did the perc 330 get flashed to it mode?

Hey Tom, double-check on the R730XD, but on the R720XD, the onboard SATA ports are disabled when anything is installed in the onboard RAID controller daughter card slot. I have an H310 in IT mode in my FreeNAS lab server, and cannot use the onboard SATA.

@arigornstrider

4 жыл бұрын

For clarification, this is in the Dell R720 manual if you want to confirm. The onboard SATA works normally in the R720, only the R720XD has this limitation for that generation.

@LAWRENCESYSTEMS

4 жыл бұрын

Thanks

Usb for boot. Ssd for cache. 3.5 for storage. Yummy.

@victorcoss2600

3 жыл бұрын

Mirrored M.2 for boot, RAM for cache, 2.5" SAS SSD for storage. Yummy.

Would this be a good system for a business?

Awesome partnership! I'm a new buyer/MS grad and entrepreneur in the market for a 730XD for modeling and ML. I'm looking to hook up an Nvidia Tesla GPU. Any recommendations? Thanks! I'll be checking out your link unless we can connect and find a better deal.

Can you make a video that covers all the different aspects of TrueNAS? I've watched a bunch of your videos, but I don't recall any that talked about what slog is for, what I need a cache drive for, is that not what RAM is for? What is metadata in the context of TrueNAS and how do I know if I need a separate drive for it?

@LAWRENCESYSTEMS

Жыл бұрын

I have that here kzread.info/dash/bejne/f2h4rqWhgruwn8o.html

What's your preferred HA or hyperconverged storage solution?

@LAWRENCESYSTEMS

4 жыл бұрын

TrueNAS kzread.info/dash/bejne/k4eqsLytf7aWqdY.html

Are you using the PERC H330, H730, H730P or the HBA330 ???

@victorcoss2600

3 жыл бұрын

HBA330, it's very important it's the HBA one, the others are RAID.

Would you recommend using a RAID card or replacing it with HBA card?

@LAWRENCESYSTEMS

3 жыл бұрын

for TrueNAS always use an HBA

Who else loves Servers and ZFS? Aren't we all nerds?

@kristopherleslie8343

4 жыл бұрын

David G gotta play with ZFS one day 😌

@ramhee98

4 жыл бұрын

And I thought everyone has a Dell R640 with 96 Threads in their basement lol

@jj-icejoe6642

4 жыл бұрын

ramhee No just a 720xd with 40 threads and 96 gigs

@ramhee98

4 жыл бұрын

@@jj-icejoe6642 Oh yea thats normal. 😂 Also got a R720 but with 48Threads and 384GB ram😋

@jj-icejoe6642

4 жыл бұрын

ramhee Nice little toy

How much does it cost to ship these monsters vis UPS? If you would'nt mind.

Interesting ... How many NVMe drives can the R730xd run simultaneously .. ?

@LAWRENCESYSTEMS

3 жыл бұрын

I think 4

Just be aware that many add in cards will cause the fans to never spin down below 50% as the server can't properly identify the card and will 'protect' it from overheating. Choose your cards wisely. I went through a few Intel pro VT and PT 1000s before finding one that wouldn't cause a fan spin up. That sucked. Otherwise at idle this server should be damned silent. 70db? what?

@victorcoss2600

2 жыл бұрын

I thought that was more specific to HP servers, not Dell. This is a Dell server he is reviewing here.

@xACExSHARKx

Жыл бұрын

@@victorcoss2600 No, unfortunately its also the same with Dell Servers

What is the part number of the card supporting the hard drives in the back of the blade?

@LAWRENCESYSTEMS

3 жыл бұрын

not sure, I don't have that system any more

Has anyone ever tried cramming 3 half-height Nvidia P4s into this server? If so, any luck running them? Performance?

I think R730XD is a bit too expensive for the normal homelab user, maybe in a few years.

@brandonhoover2120

2 жыл бұрын

I’d say. 2,200 as a starting point is a crazy for a normal home person. Really gotta be a major network tech enthusiast to pull that trigger.

How deep do you need the rack to be to rack mount it?

@johntice2007

4 жыл бұрын

standard 30" deep rack, I have several r620, r720, 720XD and R730

Hey Tom, shouldn't it be PowerEdge instead of PoweEdge?

@LAWRENCESYSTEMS

4 жыл бұрын

yup, thanks!

Nice but this is just a step up for the 720XD ( I love that server chassis) Good job on the video...

My head says that having two more spinning disks as cache isn’t worth it. If they were SSD then yes, worth it.

we have the 730R

I don't rerally like 2.5" drives for a storage server, but there are pros and cons.

@SirHackaL0t.

4 жыл бұрын

You can back a lot of drives in when 2.5”. I have 12 1Tb SSD drives in my 730XD at work.

730's support a dual sd boot using a IDSM card that work great with freenas..

@motonack

4 жыл бұрын

The problem with the SD adapter is that when one card fails you have to bring the whole system down to replace and rebuild the array. If you're gonna go with cheap flash, USB drives are way more convenient and you can hot swap them.

@kristopherleslie8343

4 жыл бұрын

Even with the raid 1 sd array?

@motonack

4 жыл бұрын

@@kristopherleslie8343 Yes the dual SD adapter is lodged inbetweem PCI-e risers and cannot be accessed without powering down the system. The rebuild process takes place in the BIOS and will not boot back into your operating system environment until the rebuild finishes. Using 32GB SD cards i've seen this take well over an hour. It's not a very good system.

@berndeckenfels

4 жыл бұрын

Isn't that ridiculous expensive?

@robthomas7523

4 жыл бұрын

@@berndeckenfels 60$

hey Tom ... using a single thumb drive as boot drive always seems a crazy decision to me, as being a single point of failure and, in general, thumb drives not being the most "enterprise" made devices. I know they will be write/read very few times, but they will keep powered for quite some time. Doesn't that worries you, in that sense, a single point of failure on a (usually) not enterprise-grade hardware?

@davidg4512

4 жыл бұрын

I usually go with high quality nand based flash drives for boot devices.

@mwilliamsr1

4 жыл бұрын

i have used USB drives in three (3) FreeNAS unit (Production) and they have been up for 3+ years with only one (1) USB failure, which did not down the system, as the system/syslog are moved to a small zPool (two disk mirror) for this purpose i.e. scripts, logging etc... During our maintenance window, replacing the USB, restore config/keys from backup took only 15 minutes.

Can I do this my r410 I have just sitting around

@diavuno3835

4 жыл бұрын

Yes

what is free Nas? what samethings in the market

@funnyarc

3 жыл бұрын

It's a filesystem based on FreeBSD, it's essentially a open source enterprise-grade network storage system with a nice interface to manage your pools and shares from. You will mainly make use of ZFS, which is a filesystem with data corruption protection (btw, it's recommended to use ECC RAM), RAID with automatic repair and load of features, snapshots, etc. You might wonder why servers with ZFS are force fed with RAM, it's not some BDSM fetish thing, it's just ZFS loves RAM! The look of it! The smell of it! The taste of it! The texture! It just loves RAM so much...♡ The more RAM ZFS can access, the more it will cache stuff from the pools, which is data you can Read or Write Really damm fast. If you give it 1TB, it will literally cache 1TB of data, if you have 1TB of data. But otherwise, you 32GB or 16GB (a little low though) is completely fine, but ZFS should be given at minimum of 8GB of RAM to work well for meta data sake. A pool is basically a collection of drives in a specific configuration, could be 8 4TB drives in RAID-Z2 (basically RAID6) or 2 drives in RAID-1 (is a mirror). Maybe a pool has a cache or a log/zil. Basically it's nice stuff for nerds. Welcome to FreeNAS. Make space for servers and surrender your data. You will add your hentais, pics and roms to the pools. Your culture will adapt to service ZFS. PS: Please keep making back-ups, k, even if it is very resilient, it could fail too one day, so please always keep back-ups on 3 seperate storage mediums in different locations (one outside your house in case of burglary or nuclear bombs).

Does it make sense to get the NVMe drives with passthrough to FreeNAS (on VM) where the proper software raid can be created and then present the created Raid drive back to VMware to be able to create datastore to share with other VMs? Has anybody tested it? Would there be a lot of performance overhead with this solution that will basically waste the whole performance benefit of NVMe drives? All this within one physical machine.

Thats way expensive for what you got. Are all the parts brand new?

@hiddenidentity8291

3 жыл бұрын

If it's new, it's cheap; if it's refurb/used then that's not a great price

@-Xaverius

3 жыл бұрын

The difference is commercial support.

@KnightRiderOfVoid

3 жыл бұрын

Those sas drives are *NOT cheap, trust me, I'm in charge of putting together systems like this for my company, and enterprise-grade SAS drives even if used are going to get your budget wrecked no matter how old or low capacity they are, specially those 12Gb/s ones. If he had gone with non enterprise sata ssds he would've gotten more storage (capacity) for better price and better latency. Of course, not the same reliability and parallel access, but hey, you get what you pay for. Enterprise grade stuff is expensive as heck, specially storage related or software licenses.

oodles of options.

You can remove background noise with the new nVidia tool.

raid 10 or raid 5 its safer on those two :)

@johnmadsen37

4 жыл бұрын

Raid 6 is fine too. Especially with 1-2 hot spares.

Where can I get a link to that static.ixsystems website you referenced?

@LAWRENCESYSTEMS

4 жыл бұрын

Follow the link to the forum post which has all the other links

@nothinglasts

4 жыл бұрын

Lawrence Systems / PC Pickup thank you, I’ll check it out

vfalsh SD card need to be DELL own SD card, bocks are diffent or sometihing...

@IM_A_BEAR_LOL

4 жыл бұрын

Additionally you can't use it as mass storage, you can load small images to the vflash and boot from those, but can't use as writable OS drive.

730 generation is not worth the cost of electricity it consumes. Dell epyc platforms are where the best bang for the buck is right now.

Easy for someone to start in a homelab? These are NOT cheap; it's not for everyone :)

very is GPU

Pretty but pretty expensive 😔

You ever look at you comments section? You ever answer any of them???

@LAWRENCESYSTEMS

2 жыл бұрын

Yes

Don't install OS (FreeNAS / TrueNAS) on thumb drive !

So, I'm gonna be a bit of a downer here and give some (hopefully) useful criticism: Whats the audience here you want to target? I'm a system engineer and if i were interested in buying new systems and seeing some benchmarks, this would have been mostly useless for me. The benchmarks were almost all sequential which is probably the least common traffic pattern in enterprise workloads. In an enterprise, when i'm looking at io-benchmarks i don't care about anything but IOPs and small, random workloads. You showed them briefly, but it wasnt really a focus. Also 64K Block size is far from real world as well. These nice, high numbers for sequential read look nice here on YT but thats it...

3 sets with z2 seems excessive, thats 6 disks redundant.

@drkskwlkr

4 жыл бұрын

You are not supposed to add the disks like that when assessing resiliency. Each of the 3 individual pools having a 2-drive redundancy *does not* equal protection against a 6-disk loss across the whole pool. Everything is a tradeoff: keeping the combined pool fragmented like this greatly shortens the time required to rebuild a pool underneath that has degraded.

Dell PowerEdge D730XDXDXDXDXD

I'm a database developer and this stuff is way overkill and I can only imagine how big the electrical bill is. I run a Dell Precision desktop with a Xeon E5 1650 V2 and 64GB with two 3TB drives. 300 bux all in.

@davidj231

3 жыл бұрын

And... nobody cares.

@KnightRiderOfVoid

3 жыл бұрын

Overkill for you, not for a small business like the one i work for. I mean you are a developer, not a system administrator after all, you should be fine with way less than a common business would. Probably a good desktop or laptop with a little nas would do for most cases, but again. Your job is to develop, not to load the thing up like admins have to on their jobs. It is overkill for most home labs, but those use cases are hot what these monsters are designed for.

Only 300G spinning drive ? This Is a waste of storage space and not cost effective

Remember, ALL of these servers, and always the rack mount servers are as loud as a leaf blower. You can not have these in a normal house. Maybe in the basement. And they put off a ton of heat. Yes, even the new hardware is stupid loud. Get a tower setup for your playing. Once you are doing this as a professional you don’t need all this shit at home. And you realize it’s Unnessecary. So loud. So hot.

@EyebrowsMahoney

4 жыл бұрын

My DL360 G5, I would agree with you. The Dell R700 series, I also would agree with you. But my DL360 G6, gets pretty quiet once it's up and running. When you put load on it... that's a different thing. I store mine in an air conditioned closet.