Reminds my several 'Beowulf' clusters for scientific computation that I built 10+ years ago by stacking miniITX small for factor I5-3450s (4core 16Gb ram each) and hooked them to a gigabit switch. Node0 was running Ubuntu with storage pools, management tools and the other nodes were booting by tftp with all the files hosted node0. DRBL linux package was handling the whole boot, os config of the nodes, network mounts magically. This cluster has served us well for dynamic fluid/structure simulation using MPI for more than 7 years and was totally unbeatable in perf/dollar. Maybe I could do that with those mini pcs. With the current CPU perf / watt and core count this ciuld be great..

@ktfjulien

10 ай бұрын

@Wendell TRY THIS ^^ !!

@DigitalJedi

10 ай бұрын

I used to have a single big box with a 6800K and dual Titan Xps for simulations, but I remember the computer science lab in my university having a stack of 1L office PCs with i5 3450Ts. IIRC there were 3 stacks of 6 each for 18 nodes or 72 CPU cores and a combined 144GB of RAM. It was really funny hearing them all spool up their fans. You could tell how hard somebody was hitting them by how many of them spun up.

@autohmae

10 ай бұрын

At some point HP released Open Single System Image (openSSI) Linux Cluster Project, which was also kind of interesting, but sadly didn't seem to go anywhere.

I'd love to see some more stuff specifically on hyperconvergence itself, what software supports it (XCP, Proxmox, eventually TrueNAS etc), how to use it, when to use it and maybe even using it with mixed hardware setups for the homelabber with a need for more PCIe lanes than these mini PCs can provide.

I personally would love to see more Thunderbolt (or USB4 for that matter) adoption in desktop motherboards.

@tames307

10 ай бұрын

Thunderbolt adoption was slowed by Intel's exclusive-use deal with Apple when it was first released. I'm surprised that it hasn't taken off since then.

@StaySic4Ever

10 ай бұрын

Yeah lack of USB4 even on expensive boards is simply horrible, to add just regular type-C adoption is also horrible, everyone slacking.

@Decenium

10 ай бұрын

ehhh idk, we are using docks these days at work for employees with their usb C connection points and honestly... I thought usb C was suppose to last long but it kinda sucks, they become so weak, never had that with usb A or B.

@potatoes5829

10 ай бұрын

@@Decenium agree, had a couple usbc ports bend in ways usba never would

I had no idea thunderbolt could be used for networking like this. This is genius!

@autohmae

10 ай бұрын

USB also supports networking as does Bluetooth.

@physx_yt1062

10 ай бұрын

@@autohmae How would you use USB?

@autohmae

10 ай бұрын

@@physx_yt1062 for this application ? I wouldn't maybe USB4 is good enough. Just saying that all of them have IP support.

@lunalucadou877

9 ай бұрын

It feels like the logical endpoint of Thunderbolt. Thunderbolt is PCIe, which is basically a packet-switched network, so why not use it as the carrier for another packet-switched network (IP)?

I bought my mother one of the MinisForum "game" PC's to replace the MicroITX computer I built her a couple years back. For the money, I couldn't build something better and certainly not as small. I was really impressed with the little thing.

@prashanthb6521

10 ай бұрын

When it breaks you cant even run around to get replacement parts. While a desktop computer can be repaired easily and cheaply.

So happy to see another happy outcome! Your hard work is making a difference!

I’m in the process of building a new lab and I have been struggling on what to get. This has helped me a bit. Thank Level1techs!

Wendel looking good Sir! Great content from the team. 2.5 G in a NUC is a win.

Useful info amongst the rambling

Yay finally the NUC cluster, I have been looking forward to this since you mentioned in another video

Yeah, the video I've been waiting for ever since NUCs popped with two TB3 interfaces 😊🎉 Real sad, that the latency is that high though. That makes it totally unusable for replicated storage...

Cattle not pets! Build availability into the solution, not into each device! Love it!

I have a Genoa 9124 with an air cooler. After boot is is actually very quiet. The cpu has a max power of 200 watts, but the chip area is enormous, so it has no trouble getting rid of the heat.

That's pretty cool, no added power consumption and heat from extra networking cards and for a cluster like this, the limit on the cable length is not an issue. I would be interested in the possibility of using a thunderbolt 4 hub to take all of this to the next level, but that would add cost and complexity and I suspect that a hub wouldn't work with multiple hosts in the first place, as that's not what it was designed for.

That in inconsistent ping is um... concerning

Love this stuff, I looked into it a while ago but with IP over USB3 which is a very accessible connector (a lot of homelabbers don’t have modern thunderbolt systems but old laptops / mini pcs). Would love to see if that’s possible (haven’t got round to trying it myself).

Love this!

In the days of old, I ran a NUC Now I'll need at least 3 those when I muck Cause I do a ping-NUC 1, NUC 2 Then I save a NUC for when there's nothin' else to do

This is incredibly cool

You're just great!

FYI old lenovo micro pcs have PCI so you could add a network card :D

not been here for...a year, maybe two... What happened you look good man!

@DarkShadowReign

10 ай бұрын

He got bit by a tick and got sick and had to change his diet.

Great thing for a K3OS or similar Kubernetes cluster on bare metal OS...

I wonder, and I know it sounds crazy, if introducing a TB to PCIe chassis with a proper NIC would work better or just increase the latency and overhead. It could show if it's the interface or entirely the driver.

I have often wondered why there is no daisy chain cable? For nucs with only 1 thunderbolt port this would be nice.

Nice idea for 5 node lenovo m90q mini cluste, but using 10gb and no switch. This could save a lot of money. I think this could be done on plain ethernet NICs.

If you only end up getting about 10Gb anyway you could use that GoWin R86 that has two 10 gig SFP cages on it already.

Now do it with firewire

Oh I'm not sure. It sounds like a setup for a complete Clusternuc to me.

Also is the ping introducing significantl latency that manifests when using the cluster? I need to do more home labbing....

"Soon" showed up!

In my opinion/experience, thunderbolt was a promise never even close to being realised. I'd avoid the headache personally. Maybe more mini pcs will start being designed with SFP ports which are slower but without the bs.

@RyTrapp0

10 ай бұрын

Like so many Intel developments over the last decade or so

I''m surprised this wasn't on the Linux channel

@CataclysmZA

10 ай бұрын

Level1Linux probably does not have the algorithmic reach that the main channel does, so this will do better here.

The Minis Forum Quorum 🥳

Would this work with USB3.2 (gen2x2,or whatever the 20Gbit version is called)? Even if it’s not explicitly called Thunderbolt? I’m assuming it would definitely work with USB4.

@rpm10k.

9 ай бұрын

Thunderbolt is not USB.

or go retro, use firewire (400/800) was faster than (in theory) than 100Mb, but also only point to point.

400,000K SUBSCRIBERS WELL DONE TEAM @ LEVEL 1

How about bridging the thunderbolt interfaces on each nuc, and running spanning tree. Now you have a flat br-thunderbolt network with just one ip per nuc.

@0LoneTech

10 ай бұрын

Spanning tree protocol essentially works by logically cutting links until the network has no loops. You'd end up with one node handling all the traffic between the others, even though they have a direct link going unused. That also adds the full latency to all the forwarded traffic. Running one IP per NUC is fine, as long as the routing is correct (multipath for redundancy).

The Jetson nanos can be clustered together to build a mini gpu cluster. Beastly for inference and more than capable for training small NNs. I've toyed with the idea of building a luggable/cyberdeck with 5 nodes and an ebike battery in a briefcase.

Is there an active occulink m.2 card that would work with this kind of stuff?

Silly question - how do you install a PCIe 3.0 x16 card into any one of these mini PCs???

How does the ngenius system compare to the Linksys omada?

1tb of cache? would that help an os #3d cache clockfriquency issues(handler prioraty)?

What I am hearing instead is don't bother with TB networking unless you have nothing above 2.5Gbps

ask eta about the gamecube and think makita toolbox plz and Thank you

I'd leave a positive comment but I'm all outta sorts without my morning new... Links with friend sharing...... I agree with guy who said we need Krista to fill in with Wandall and Ryu puppets....

While running this in cluster with 2.5gbps, doesnt it get slow?

So basically iSCSI, but with extra caveats? :D

Those latencies (ping) is unacceptable for using Ceph. That's worse than a WAN connection. Sequential transfers should be fine, but for clustering, that's a no go :(

Now that intel is discontinuing NUC... do we now call them PUC?

Why no level 1 news.. I mean level 1 sharing links with friends??

Can this work with USB4?

Bisides solar bitflips cancer is the main catalyst

I think I may have missed it, but what's with those ridiculous pings (on a local network??). Is this just a poorly written driver?

@AzureFlash

10 ай бұрын

15:57

ClusterNUC.

what the NUC?!?

what is death? a blesing

software networking, I mean it can work but its gonna suck down half your cpu cycles and run half the speed

We still have a situation where a low cost nvme will saturate ethernet socket becuase those mini pc's insist on not using 10gbe :/

I would actually like to put 8 orange pi 5s in my system that has a 7950x, and I'm wondering how easy this setup is gonna be with the way people talk about it. This seems somewhat daunting, and I may also opt to use the m.2 slots for interfacing instead of type c or on board ethernet

@stuffinfinland

9 күн бұрын

Unable to compute.

3d vcache vgpio? pathrought? think capturecard but quantom arm meets x86 #northbrydge (threadripper ipad pro brydge keys?

Anyone actually run iperf or netperf over a thunderbolt-net interface like this? In order to do highspeed ethernet a lot of driver/sw tricks and hw offloads are needed to run ethernet efficiently and im not sure if just raw thunderbolt is enough but maybe it is

can you do 5 node on that setup? please?

@CataclysmZA

10 ай бұрын

Three node is a practical limitation because there aren't enough Thunderbolt ports available.

@0LoneTech

10 ай бұрын

​@@CataclysmZA Only for full interconnection. You can still run a ring topology with two paths between any nodes, though forwarding through networks with this poor performance isn't going to be fun.

DeeeZ NUCz

Couldn't you bridge the interfaces and assign an IP so it's the same IP space. Simpler config with the same design.

@iodehk5205

10 ай бұрын

P2P is better. STP bad

What's my lowest-wattage and cheapest cost to having a home machine with: ECC RAM, 10GbE, Intel Quick Sync, and at least six cores/12 threads?

@marcogenovesi8570

10 ай бұрын

still pretty high, you are looking at Xeon D boards

@FrenziedManbeast

10 ай бұрын

The Xeon chips aren't APUs too, right? So I'd need an Intel GPU w/ Quick Sync support as well? Seems like a lot of PCIe connectivity to give up just to gain Quick Sync, that's a shame.

@bosstowndynamics5488

10 ай бұрын

​@@FrenziedManbeastIntel does make some consumer chips with ECC but you'll probably not get 6 hyperthreaded cores (they only do this on low end chips where there's no competing Xeon), so you'll need to decide if 4 high performance cores will do compared to say 6 older generation, more power hungry cores. Do you need QuickSync specifically though, or just hardware accelerated encoding? Because AMD's 7000 series will run really efficiently when underclocked slightly, is readily available in 6 core configurations for *relatively* cheap, they all support ECC and have onboard GPUs so they *should* be able to do hardware encode (do double check this first but as long as the encoders are there AMD GPUs are generally well supported on Linux).

@FrenziedManbeast

10 ай бұрын

@@bosstowndynamics5488 I should have specified that I'm looking for fully Registered ECC. I'm still curious as to what the lowest Watt + lowest cost solution would be. I understand that many consumer CPUs support ECC on the data sheet, but finding motherboards with support for Registered ECC for those chips seems...difficult? Impossible?

@MarkRose1337

10 ай бұрын

Many i3s support ECC RAM.

First, I finally got some internet points!

@cairath

10 ай бұрын

Cheers ;)

@marcogenovesi8570

10 ай бұрын

all the points, give them to him/her/they/it

@V1N_574

10 ай бұрын

I just imagine youtube sending firsts certificate

Pcie turning pi 2

This video title is far cleverer than it has any right to be.

So out of the box we get 1Gb and 100ms ping? Or we could use the 2 2.5Gb interfaces for 5Gb aggregate, and 0.1ms ping?

@snowwsquire

10 ай бұрын

2x10gigabit for a little setup when you first get the nucs vs 2x2.5gigabit oob is an easy choice for me, that 10gigabit is gonna be much better for me

@marcogenovesi8570

10 ай бұрын

If you are not using the thunderbolt it's not worth the 100-150$ markup over similar devices without thundetbolt. Also if you use both ethernet ports to do this then what network interface is used to talk to the outside world?

@tcnymex

10 ай бұрын

​@@marcogenovesi8570 perhaps talk to the outside world vua a wifi card in a slot where a 2nd ssd would normally go

Any dude in I.T. named Wendell has to be the man! LOL

risc-v cores can run an os at 1mhz zo can i volt it up to like 120dc and have the chassis run on 24v #capasotor pack+ discharge tool arcing prev.

You don't want with low security thunderbolt an hacker to use it and overheat until it catch fire😁

1tb virus or "Ai" xD #uranium forge eeprom updater

Jajajaja thunderbolt en linux y con AMD or nvidia hahahahade, please don't. Yet. Early adopted back & no Bueno

Escape from tarkov cheatercard fixes vigpo?

I wish this guy would learn, that the background music is very annoing... Otherwise, I love his videos. I'm not interested in music, when watching computer-videos, and when I want music, I prefer to choose it myself