To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CraftComputing/ - Plus the first 200 of you will get 20% off Brilliant’s annual premium subscription!

@rushunt2131

Жыл бұрын

Your shirt is brilliant!

@amateurwizard

Жыл бұрын

You pronounced it correctly (not that anyone minds if you didn't, just nice to see)

@Nobe_Oddy

Жыл бұрын

why does your intro have the sound of water pouring into a glass??? you would think you'd have at least used a carbonated beverage pouring sound for it... right??? You have the MIC... you have the BEVERAGE.... now fix it!!! (and lemme have the beverage when you're done plz?? ... that's the whole reason I wrote this... I wuntz bierz plz :P) :)

@zachsoanes6417

Жыл бұрын

Forgot to put the command in the video description - as you said at 15:45 ;)

those mad lads at Mikrotik just sent not just one but a couple of their flagship 100gb switches plus accessories, no questions asked. Jeff truly wields the power of the gods

@BenCos2018

Жыл бұрын

Facts haha

i Still cannot justify the 2x 2.5 giabit switches i need in my basement and office, this guy is doing 100 gig! Good for you great content! Keep it comming!!!

@ewenchan1239

Жыл бұрын

I have a 36-port 100 Gbps Infiniband switch in my basement.

@bojinglebells

Жыл бұрын

yeah, really is frustrating that consumer networking has been dragging ass for the better part of TWO decades now. I swear they've introduced 2.5Gbps and 5Gbps just to nickle and dime us instead of going to 10 where they can keep overcharging businesses

@wiziek

Жыл бұрын

@@ewenchan1239 ah stop trolling

@wiziek

Жыл бұрын

@@bojinglebells it hasn't, you have no idea what are you talking about. most consumers don't even use 1gbs looking at how widespread is wifi. 2,5 and 5gbs was introduced for entrerprise, to reuse cat5e cables. anyway there are already 400gb devices available or even 800g, of course not for consumers or average enterprise...

@bojinglebells

Жыл бұрын

​@@wiziek they don't put 2.5G on consumer products so that enterprise can use them. But sure, keep being a know-it-all ass.

iperf3 tip: it natively supports multiple parallel streams with the -P flag, no need for multiple instances ;) great upgrade though, it gave me the itch to dust off my own 100Gbps cards I have lying around

@akmattb

Жыл бұрын

THIS

Your shirt... That little saying is grilled deep into my head. After many years of making thousands tens of thousands hundreds of thousands of patch cables over the last 20+ years.

This is the level of absurdity that I love from the channel. I was finally able to upgrade my NAS and main pc to 10 gig. Because I needed to upgrade my NAS and didn't want to deal with migrating 30TB the data over a 1g connection.

I’d like to see RDMA in Linux and Windows through SMB Direct, as well as iSCSI and NFS. RDMA should remove all CPU bottlenecks since the transfers will not longer use traditional file stacks. Make sure when doing any tests with iSCSI to turn off synchronous writes with TrueNAS, it will allow better performance for tests-although it shouldn’t matter once you get NVMe.

@ivanmaglica264

Жыл бұрын

Well, not quite correct... RDMA itself removes the TCP stack and goes RAM to RAM via Infiniband, like DMA does from let's say sound card to RAM, but between machines. File-systems are still involved, especially in userland program space, where fopen, read, write and fclose operations are used.

@AegisHyperon

Жыл бұрын

It's super driver dependent and the drivers suck ass. Tried to implement it for a fileserver project a while back but couldn't get it to not drop packets.

@rayl6599

11 ай бұрын

NVMe ROCEv2 requires converged enhanced ethernet support to avoid flow control issues else reliability issues may result - I do not think these are CEE capable.

Having a personal, local fiber network in you home just sounds so cool to me lol

Great video. With 100gb you could pretty much run all the VM storage over iscsi. Would make for an interesting project.

@llortaton2834

Жыл бұрын

That's what i did but with 40Gbe on my proxmox

@marcogenovesi8570

Жыл бұрын

most of the clients I work for have 10Gbit (either fiber channel or ethernet) for VM storage.

@ewenchan1239

Жыл бұрын

"over iscsi" or NFS

@RowanHawkins

Жыл бұрын

The way you are installing that is making my DC senses tingle. MicroTic designed this to be a back of rack device. I bet the fans are blowing the wrong way for your configuration. That's why the power and QSFP are on the same side. Edge switches have power on one side and ports on the opposite side because you generally link them to patch ports for your end devices off the front. Also MultiMode (OM4) is fine and spec'd up to 100G over 125m(400ft). The problem as you saw is the QSFP28's at that configuration want 8 or more likely 12 strand MPO and they are not cheap. I really wish you had been clearer that the switch to SingleMode infrastructure wasn't a limitations of the OM4 itself but a budget limitation of buying the hardware to allow OM4 to operate at 100Gb speeds. OM5 is the same issue, however old Datacenter infrastructure that I supported ran either 24, 48 or 96 strands to each cabinet using 1x12 MPO so even with the cost saving of SM which is a relative recent thing, it was still cheaper to buy the MM compatible 100GBSR QSFP28 modules than to replace the cable plant would be.

I bought a 2u wall mount rack and bolted it under my desk for my PDU. Works perfectly and keeps everything out of the way. Also installed some under desk pockets to hold power bricks and the like.

It's great to see a good use of 100gig. I upgraded to 10gig with the crs326 and crs317 switches a few years back and think I am good for the foreseeable future.

On systems with only one X16 slot, I would recommend running your 100G NIC in the X16 and your GPU in the X8 or X4 slot, the GPU might only lose a few per cent of performance, but your 100G NIC will thank you.

Awesome video, Jeff. LOVE Mikrotik!

It took a long time for me to finally try out some MikroTik hardware. It was their Wireless Wire kit, which is ironic given my long-standing distaste for infrastructure wireless, but genuinely the best choice. Other than some configuration bobbles - they're the link in the middle of a double-NAT setup - it's been _very_ nice. Incredibly reliable and simple to install. I know double-NAT is bad, but I haven't been able to successfully argue for taking over the ISP-provided external router yet.

I appreciate the shirt, after doing thousands of cable ends in my life thus far.

Great Scott's. That is some seriously impressive gear, thanks for sharing.

The fact that all of this is as (relatively) inexpensive as it is is freaking crazy to me

Love to see the Rack Studs, they're so great.

It's crazy how far transfer speeds have gone. It was only like 15 years ago we were splitting DS0's off of DS1's to backhaul voice on cell sites.

never clicked so fast on a video

I have been waiting for this video so much!! And it's 30 minutes! fun! Keep creating amazing content!

For file transfers and real applications (such us streaming) you would need to setup NFS over RDMA or Samba over RDMA. I don't think the bottleneck is in the SSD raid. Standard ethernet works fine for synthetic benchmarks using specialized applications (like iperf), but for getting similar speeds with network filesystems and non specialized apps you really need RDMA.

@jyvben1520

Жыл бұрын

Remote direct memory access

Awesome Shirt in the 2nd half. Cerritos Strong!

My goodness. I have a hard enough time using all of my 10GBE setup, even between servers doing large VM backups! hahahaha. Good video, looking forward to more videos on it.

Good job wearing green! This comes from a Professional Brewer in Ireland who was raised in The PACNW! Happy St.Patricks day. Let me Know if you ever want to Homebrew. Super fast switch I just ordered the 570/80 you just presented.

Nice network, good job!

@CraftComputing

Жыл бұрын

vkc.sh/product-tag/t568b-cheat-sheet/

@nticompass

Жыл бұрын

Looks to be from the KZread channel "Veronica Explains" 🙂

Wow. Overkill - Love it!

Looking forward to the Truenas testing with Cache and all that 🙂

When those lights first came on, Jeff was grinning like a kid who just had the best Xmas of his life! 😆

@CraftComputing

Жыл бұрын

It was like getting my Super Nintendo all over again 😁

@gearboxworks

Жыл бұрын

@@CraftComputing - There is an unmistakable look of pure joy, that can’t be faked. You had that look just then. 😆

I also had the same issue with Mellanox ConnectX2 and 4 Cards on some HPE Workstations.

Love this! Give me hope for my future ;-)

Jeff, In this video, you were just like a kid at Christmas, so Merry Christmas! enjoy your massive bandwidth.

One thing I'd really like to see is inter-vlan routing speeds. Hardware offloading etc. There seems to be very little information around correctly setting this up without just killing your switches/routers cpu. Usually if you're geeky enough to have 10 or 100gb networking then you're going to have vlans. :-) It would help me get on the 10/100gb train!

@stephenp4440

Жыл бұрын

I was going to ask how he setup inter-vlan routing with that Ubiquiti UDM-Pro. I went to 25 Gbps and I had to collapse my VLANs to put all of the 25 Gbps clients in a single VLAN if I still wanted to use the UDM-Pro. An alternative might be to use the Mirotik as the inter-vlan router but I don't know what the speeds are. The UDM-Pro just couldn't cut it for me.

Damn --- I have not been this jealous in a while lol. That's sick tho Youre setup gives me something to aim towards... maybe one day

I get about the same, ~ 35 Gbps per stream with iperf3, and about 16 Gbps with plain SMB. Using RDMA (SMB-Direct) I can achieve 40 Gbps file copies, but only in one direction. (Uploading from W11Pro4WS to Svr2022) Tried enabling PFC+DCB on my equipment, speed went down. More troubleshooting needed.

I grew up having a BBS on 9600 baud (9.6 kbps) and the move to 19,200 baud was a giant leap. That was still all over a circuit switched telephone network, i.e. not IP networks yet. The speeds you reach in this video are like 10 million times higher... Talk about progress...

That is fast. I know less than Linus about the linux universe but it is always fun to see the hardware. Also glad you mentioned the tree grinder across the street. For most of the video i was sure that you had a noisy fan near a microphone. Cheers

Oh, that t-shirt. I want one!

@Nealio6s

Жыл бұрын

I was thinking the same!

@CraftComputing

Жыл бұрын

vkc.sh/product-tag/t568b-cheat-sheet/

Love the shirt!!

I've very happy with my MikroTik gear.

is really amazing... readly proud for latvian guys... )) mikrotik well done )))

Beautiful, absolutely beautiful 😘

cheers!!! thanks!

Nice!

I do not have the kind of speed requirement, files, workloads, etc., that said it is definitely drool worthy. I agree on the ' typical ' workload usage and could do with that kind of speed and storage for my Steam Library and wonder what kind of load times you could get using a net-box on that network.

Oh my gosh, love the t-shirt!!! When I first noticed it I just laughed and laughed. Mostly only network geeks will get it. Well done!

@eivinha

Жыл бұрын

Quite outdated for *this* video though 😂

Awesome video, I have a question regarding safety though: Could you please be more clear on the dangers of leaving SFP/QSFP lasers open? It can permanently blind someone, and proper precaution should be taken when handling lasers.

I think the elephant in the room is how the hell you got those optics for 5$ each

The one thing I like about Mikrotik is I don't have to fight with GBIC and SFP compatibility! Intel, 10Gtek, Nokia (FTTH SFP). It doesn't care and it just works.

Took me a minute to realize what that shirt means. I'm drinking and eating as I'm watching this and thinking about that shirt. I like it.

I really want to see how you maximize the network. I've been meaning to pick up the same router, but I'm not sure how I should be setting up NVME drives to maximize. Should I be using a 16x pcie 4.0 slot with a 4x4x4x4x PCIE bifurcation card? Striped + Mirrored? What about sata ssds, how fast can those be made in ZFS. What about tiered storage with an HDD, SSD, and NVME pool. With the stuff you're actively working on being moved to the NVME pool? (

Damn 100gig... I would only need that between my two servers... Despite I was only cnsidering a 25g upgrade... Thanks alot Jeff... My bank account is getting drained

If you want to push the network, storage, and CPU a bit, database ETL (extract, transform and load) will be a good general stress test. Not only are you capable of saturating the network link, but for how long, and for how much data that has to be process, structured, and stored in a database that can then be queried quickly. I'm not sure what's out there for "canned" large footprint ETL benchmarks though.

That is one snazzy shirt you got, Jeff!

You made me an alcoholic. Always drinking your episodes with beer :)!

I had trouble using the connectx-4 on a board as well. It wouldn't boot with my U.2 16x bifurcated card installed. They can definitely be finicky.

So envious :)

Was waiting for someone to review this switch in the wild. Interesting regarding the cpu bottleneck. Assuming there’s a hardware DMA workaround that doesn’t involve CPU. We’re getting close to RAM speeds let alone storage.

I got a Celestica dx010 and can't even get to 25. Still love the overkill. I went with bcachefs for testing my storage in a tiered configuration, I don't have all that many nvme drives to put my data just there.

I would love to see a iSCSI netwoork boot comparisson between a sata ssd and nvme. In addition to that mount a GameDrive for steam and see what kind of cpu usage is generated through the streamed block storage with that much bandwidth

To avoid the scheduler on FreeBSD bouncing around on diff cores, use the command cpuset with iperf to lock down the process to a specific core. I was able to achieve much better rates when benchmarking like this.

100Gb very cool for most businesses let alone home, but that mains lead about to pull out of the wall, can you get 90° NEMA plugs so the strain is downwards?

I'm surprised how long it took me to understand that shirt... I always think OrangeWhite-Orange-GreenWhite-Blue.... in my head

Jeff if possible make a video on how to flash the bios of the mellanox cards from infiniband to ethernet in windows 10 thanks.

I would like to see the peak of cloud gaming

FINALLY!!!!!!

This comes up on my KZread feed the day two new 32x200Gbps switches where delivered at work 😂

Which sfp+ and sfp28 nics work with both truenas Core and scale?

Nice Video! Would it be possiable to see how long it takes to do a Snapshot of a VM?

does it work for improving self hosted Multipath TCP (MPTCP) -& MPudp- throughput on the lan. -does it have working openWRT drivers-

Can you use melanox connect x2 (10 gigabit) sfp+ pcie with proxmox on a hp proliant dl380p g8? I would really want to know, I want to possibly get a card with 2 sfp+ in a bond.

I like the Ethernet colour code T-shirt.

Looks like it is Christmas at Crafting Computing!

please add the command (and maybe a link to the documentation) to change the switching mode of the network cards :)

Recent convo with my self: I'm thinking of upgrading the house to a 10gb network network infrastructure. Me: cool your gonna need a new switch and nic. Also me: yeah I can get a 2x10, 4x2.5 switch and a 10gb rj45 nic, plus cable and supplies to do the house runs for like 500$ Me: wow cool do you work an IT or Tech job or something? Also me: I'm a welder Me:........ Also me: I have a problem.

Assuming since those pci express 100gb/s nic's support proxmox out of the box. Is there guarenteement that those same nic's could work with an actual Ubuntu LTS that is not in a virtual environment? I'm looking to build a datacenter from ground up within next year, speed is priority. Already have the switches in mind that I'm needing to get.

Our mikrotic gear is cheap but unfortunately buggy. Their OSPF implementation has a memory leak. CCR2004 has a few problems that severely limit bandwidth. Also ROS7 still has some blocker bugs for us.

I'm wondering if you tried using a ramdisk to increase the speed of transfer. I'm wondering if your ssd is the bottleneck

Curious what distance those single mode optics are rated for? Did you put attenuators on the (preferably) receive ends of the links? You are going to shorten the life on your optics if you are blasting 10km optic power levels over 3meter cables.

Hi Where can i find the link for the commands you used to update the Mellanox card firmware Thx

Hm, can’t you break out multiple virtual devices to spread out the interrupt load and send queues?

What, no 400Gbps?

@CraftComputing

Жыл бұрын

Baby steps.

Sweet shirt :)

When you said 100 gigabit full duplex it gave me a full duplex. I've been considering a network upgrade for a bit and I'd very much like to get 10g links in place, but 100g is almost an incomprehensible number.

@CraftComputing

Жыл бұрын

If your full duplex lasts more than 4 hours, make sure to call your doctor.

Why did you try the bandwidth utility built into Mikrotik to run switch to swtich to see what it could do?

i like that EIA/TIA 568B shirt :)

I'm a sucker for fast networking ^^ Awesome video, Jeff. Why would RAIDZ-1 be preferrable to a striped mirror (RAID10) configuration in your use case? Other than the the fact that you have more storage space. Wouldn't RAID10 be better for write speeds and potential rebuilding of the pool in a worst case scenario?

Great video! At this point, do you really even need to copy back the footage to edit it, or can you do that directly on the remote drives (using copy-on-write so you don't overwrite the originals or have to create a copy. Deduplication might net you some interesting savings there, too). I have a funny feeling you might run into issues and bottlenecks with SMB, the File Explorer equivalent or similar. But still, that would be a use case I'd like to see. Maybe you could also try playing some disk-intensive games directly from remote storage and seeing if they remain playable? Just a "totally overkill hardware for gaming" kind of idea. And thanks for the idea of running single-mode over multi-mode if you want "cheap" future upgradability. I'd like to run some fiber lines around the house for the connection between the desktop and the NAS. I'm looking at 10Gbps right now, but it would suck to be stuck there in the future.

oh an AE-5, Classic!

With 100G and DPU you could boot off network as it was local nVME drive. That would be interesting project, albeit expensive one.

Question, is there any place I could find a NR1200 storage server? Your link for it is down so I was wondering if you can still buy them for cheap. Thanks!

So it's 5 months later, and I have only now picked my jaw back up off the floor after hearing you say you got those 16 Intel 100gig transceivers for 5 bucks each.

@CraftComputing

11 ай бұрын

If it makes you feel any better, I still can't believe it myself.

I don't know if I've seen Jeff this giddy before!

@CraftComputing

Жыл бұрын

I'm a simple man. I see 100Gb lights, I smile.

Nice cable shirt!! Where did you pick that up at: I want one!

Ok, this is probably dumb. But I'm curious if there is any practical use for a ram drive on your server shared over the 100 Gbps network. I'm also curious how a 100 Gbps connection compares to 10 Gbps in latency. Also, regarding your future plans to use an NVME pool, I've heard that ZFS actively hinders NVME performance (according to a presentation by Allan Jude from last year), so a different file system for comparison may be interesting.

I NEED to know where you got that shirt, it would be an absolute hit at work.

Well I guess I'm done waiting for the 40gig Brocade icx-6610 part deux!

Is the mallanox card compatible with pfsense

I couldn't help but spit my coffee out when you mentioned how much you paid for those optics....

Is 100Gb finally enough to do gaming over the (local) network without noticing?