Really enjoying these In-Depth XCP-NG video tutorials! Thanks for taking the time to make them. I am becoming such a fan of XCP-NG!

Thank you for making this video.

@LAWRENCESYSTEMS

2 ай бұрын

My pleasure!

Thank you so much for making this lovely video!!

I assume this requires the host motherboard and bios supports IOMMU and that it is enabled? Also, why the world is not just not an option in the UI? Same with Proxmox. Though, in Proxmox once you go through the steps to enable PCIe passthrough, you can just do everything else from the GUI. But still, the hypervisor should just detect if the hardware is capable of sharing PCIe devices and then let you set everything from the GUI.

@marcogenovesi8570

2 ай бұрын

The reason is that nobody cares enough (of the businness customers that actually pay the bills). Same for sr-iov. Afaik the only project that has full GUI for PCIe passthrough setup is Unraid, from version 6.9 onward.

@mt_kegan512

2 ай бұрын

ESXI *cough Broadcom has a really nice implementation. I tend to think the OP is more right than wrong tho. I do agree that this is more of a "home-lab" request... But we always want more. I absolutely love XCP-ng & Proxmox, but the full GUI implementation in ESXI was really nice. Just give it to us. It would FACTUALLY make their project better. Additionally, thank you Tom for putting this out. Your content is always great. Love the amount of videos you've been cranking out too. The community really appreciates you and all your hard work!

@marcogenovesi8570

2 ай бұрын

@@mt_kegan512 xpc-ng and proxmox (and Unraid for that matter) have a plugin API to let the community make GUI plugins for things like these that don't warrant the attention from the "main" dev team. That functionality in Unraid was originally a third party plugin as well.

@haxwithaxe

2 ай бұрын

Yup hardware support is required. Iommu support by intel is a mess. I hear AMD is much more sane.

@xandrios

2 ай бұрын

@@mt_kegan512 ESXi does this much better indeed. Combined with SR-IOV it is very powerful and we do rely on this for our business. In telecom there are still plenty of reasons why you'd want a PCI card, or port, passthrough in a production environment.

Just did this two weeks ago from your older video lol passed an hba card. Super simple

One aspect that was missed on the PCI Device Passthrough, is how to check if the machine you are using even supports it. (xm dmesg)

Thank you, great video. Would the same steps also work for GPU-Passthrough?

@LAWRENCESYSTEMS

2 ай бұрын

It should

Awesome video, Tom. One question: Is it not possible to run modprobe -r to remove the kernel module and avoid a reboot?

@LAWRENCESYSTEMS

5 күн бұрын

Not sure, also worth noting they have added pass through options to the web interface

I have also asked in the forums but for newer hardware to be supported they need update their kernel. Come on 4.19 back porting all everything because their storage solution depends on it

Two things: 1) I noticed that in the VM, under the Advanced tab, that there is an option for GPUs. Is this the process that we would follow to ennumerate said GPUs so that it will be made available to the VMs? 2) I am guessing that by passing through the PCIe device (e.g. GPU) that you won't be able to SHARE that resource between VMs, correct? (Unlike Proxmox where you can share a GPU across multiple LXC containers.) Thank you.

@tomklein6540

2 ай бұрын

This is voor vgpu only from what ive seen. If you run xcp-center you can pass through the gpu from there without having to go commandline for this. Although results may vary as this doesn’t work for the nuc 11 and 13 it seems. Devices are detected by the vm but you can’t use them. Did a post in the forum trying to get this resolved.

Great video 👌🏼 Ive been moving stuff off vmware to xcp ng last week and all seem to work great except this part. Been passing through the igpu for years in vmware but for some reason this doesn’t work in xcp ng 8.3. The device gets detected in linux it even shows hardware transcoding but just hangs from there. This happens on a NUC11 and 13. You or anyone else have hint on what could be the problem? I could try 8.2 to see if it is a bug in 8.3…

Hey Tom, watching this while trying to pass the SATA controllers and one of the M.2 drives through to a TrueNAS VM, and I have a feeling from something I've read that there may be issues in doing so for the AsRock Rack X570D4U-2L2T; is that a situation you've tested on your end? I know you use a couple of those (or at least similar), so figured I may as well ask. Will get a HBA if needed, I'd just love not to need to XD

@LAWRENCESYSTEMS

Ай бұрын

Not something I have tested.

@joshuawaterhousify

Ай бұрын

Okay, never mind, got the drives passed through. Now it's just hanging on boot instead...but they're showing in the boot sequence, so maybe I can somehow work through that? May end up as a sticking point, but I'll keep at it for now to see if I can work my way through it.

@joshuawaterhousify

Ай бұрын

@@LAWRENCESYSTEMS Didn't see this when I put up my other comment, whoops ':D Thanks for getting back to me though :) I seem to have got that sorted, but yeah, TrueNAS Scale's now hanging at "Begin: Running /scripts/local-premount ... done.", or occasionally working through at least some of mounting sda and hanging there instead. I did seem to get further with that than with Core, but I've yet to figure out just what's hanging. Having the issue both with 23.10.2 and 24.04-RC1, with Core 13.0-U6.1 not even getting through install. (Scale also didn't like rebooting after installation with drives connected, and went through install just fine before the SATA controller was passed through...might see how it goes with it passed through with no drives for testing purposes.)

@joshuawaterhousify

Ай бұрын

And I got it sorted out, as far as I need to for now at least; half the SATA ports passed through and that's all I need at the moment. Issues passing the other controller through, but I can pick up a dedicated controller down the track to resolve that I guess. Either way, it's been a great thing to work through, and your video was a massive assist on it, both for this and many other things I've been working on for moving my setup over to XCP-ng.

In a past video you said that passing a pcie card connected to a bunch of HDD's might cause some troubles and it was better to connect the disks directly to the VM (it was a truenas example). Would you still recommend passing the disks individually or the PCI card?

@LAWRENCESYSTEMS

2 ай бұрын

When virtualizing something that uses ZFS I recommend passing the controller card through.

Is having to reboot when attaching a usb device a limitation in Xen? My laptop with kvm does handle usb redirection without rebooting the VM

Thanks for the video and explanation. Looking at moving away from VMWare, and relying on passthrough quite heavily, this is pretty much a no-go for XCP-NG. Doing this kind of config from the command line makes it basically unusable for anything other than a hobby environment. Thats a real pity. Also, an important aspect with this is SR-IOV support, allowing one PCI device to be shared over multiple VMs. For instance, a 4 port adapter shared over 4 VM's each getting their own port.

@LAWRENCESYSTEMS

2 ай бұрын

XCP-ng does have SR-IOV support and passthrough will probably be added to the web UI in the future but for now it's command line. We don't see too much passthrough in the enterprise work because it bind the VM to that host.

@xandrios

2 ай бұрын

@@LAWRENCESYSTEMS It does not necessarily *have* to bind the VM to the host, though. If the same type of hardware device is available at multiple hosts in the cluster, nothing is stopping the software from allowing configuring devices on a cluster level (i.e. define a cluster-wide PCI device and assign a physical device to that resource from every host). The VM could then use the cluster-defined PCI device and run on any of the hosts that offer the PCI resource within the cluster. Of course things like live migration won't work, but HA for example could.

@LAWRENCESYSTEMS

2 ай бұрын

That is use case for SR-IOV

Do pass through USB devices survive a host restart? Testing earlier in the year I found that the USB device would disconnect, workaround being pass through the USB pci controller instead of the USB device

@LAWRENCESYSTEMS

2 ай бұрын

Rebooting the host does not remove the USB.

Not directly on video topic, but has Vates been moving features that used to be in the free version of XOA up into licensed levels? I spun up XOA about 3 years ago and I could do file level restore - now it looks like you have to have a Premium license - is that correct?

@LAWRENCESYSTEMS

2 ай бұрын

You can have all the features in the self compiled version but to my knowledge the file level restore has always been a higher license.

Thank you for the wonderful video. It shocks me that this simple thing cant be done in the gui

@LAWRENCESYSTEMS

2 ай бұрын

Maybe one day, but it takes time to write that into the UI. It's open source and if you are a dev feel free to help contribute the code.

The inability to hot-plug USB, as well as the lack of snapshots while they're attached seems like a pretty severe limitation... I know QEMU can do both at least, but is this maybe just a UI limitation for XCP-ng (as it is for proxmox), or does the underlying tech stack have the same limitation as well?

Do you have to have the XO from sources in order to see the USB's from the gui? I have the same version of XCP-NG running and can't see an USB passthrough using XOA.

@LAWRENCESYSTEMS

2 ай бұрын

Make sure you are using the latest version as this is a brand new feature. You can check by going to XOA then Updates in the menu and choose "Release channels Latest" to be on the most current one.

@RCrosswell

2 ай бұрын

@@LAWRENCESYSTEMS Cheers Dude. Sorted 🤘🏼

Thx for the video Lawrence! I tried GPU passthrough to a Windows VM and it works. However, when I tried GPU (RTX 3090) passthrough on an Ubuntu 22.04 VM, the OS would hang on startup. Is anyone else experiencing this? Does anyone know a solution to this?

Maybe I missed this, I understand that with USB passthrough you can't do snapshots and the VM has to be shutdown for backups, what about when passing a NIC or passing an HB, does one still have the restrictions on snapshots and backups, or is it only for USB devices?

@LAWRENCESYSTEMS

20 күн бұрын

Only USB causes the issue

But the backups wouldd still work for my Home Assistant (I don't care about snapshots)? I also just learned about the 2TB max disk size limitation :(

@LAWRENCESYSTEMS

2 ай бұрын

Stuffing large volumes of data into virtualized disks is not good storage design. kzread.info/dash/bejne/Y6lnw8ueZNqZYso.html

@AlexDiamantopulo

2 ай бұрын

@@LAWRENCESYSTEMS What would you recommend for 3-4TB MSSQL DB servers? Thank you so much for making these videos, we need to get rid of vmware...

@LAWRENCESYSTEMS

2 ай бұрын

@@AlexDiamantopulo if they are running in Windows than you can do an iSCSI attached drive

Damn, you literally did a video about the same thing I asked in the Q&A video!

Well... I just installed "patches" on my xcp-ng host with Home Assistant aaaaaaaaaand USB passthrough stopped working... All my Zigbee, BT, etc devices stopped working. Why? Because XCP-NG overwrites /etc/xensource/usb-policy.conf and who knows what else... Time to figure out what to do :./

@Zeric1

20 күн бұрын

I didn't experience this when installing the latest patches, I also have HA virtualized and use USB passthrough on a zigbee dongle. Everything started up and worked normally. There must be some other configuration difference that caused your usb-policy.conf to be overwritten, my usb-policy.conf hasn't changed in six months.

Well, that's not a fun dance to attach PCI-devices. You'd think you could do that from XO, given how you can do the same from quite literally every other hypervisor's GUI utilities....!

2:20 Having to reboot a VM to enable a USB device doesn’t look good. Imagine trying to do that to a server OS. That would imply downtime unless you schedule it

@LAWRENCESYSTEMS

2 ай бұрын

USB passthough is not a very common server use case.

Ugh. Wake me up when you don't need copy, paste into the command line! No snapshots possible when pass-through enabled can be a showstopper, but then, when, why would you need external USBs? Encryption? Passwords? Data? Can be done without the complexity (smb, rdp, even usb over ip). NIC/HBA pass-through to a VM is a real possibility and real scenario, but those solutions are usually "embedded", stateless or reproducible with 1 configuration file, so, not terrible, but they can surely improve or workaround the technical issues.

A showcase of why people are just going to pay Broadcom.