I think we need more videos about this card. Id like to better understand usecases and how it works.

@MustaMT

Жыл бұрын

Mikrotik, yes please..

@hexatested

Жыл бұрын

This card very useful in data center or network exchange environment where you pay per U. Rather than installing separate router and server that needed 2U spaces, with this card you only need 1U space.

@RB01-lite

Жыл бұрын

​@@hexatested Brilliant! You know of any server models that can accommodate this card? And do you have any idea what that could mean in terms of savings? Like, if all you needed was 1 server and 1 router, then you would cut costs in two? I imagine flexible rates - paying for bandwidth and electricity are common practice.

Another good thing to try if you want to maximize throughput to a single VM is to directly assign either individual interfaces or the whole PCIe card to a VM. This lets you skip the linux kernel bridge as a possible bottleneck.

@cldpt

Жыл бұрын

and QEMU/KVM. There's at least 3% perf loss on CPU/RAM from those but potentially more on i/o, maybe also IOMMU groups related stuff. Likely the issue here is the load on the CPU i/o controller handling the NVMe disk on the same group. Another theory is PCIe bottleneck/overhead: this card appears to be x8 electrically, likely PCIe 3.0. That's almost exactly 8GB/s _bidirectional_ which is what we seem to be getting. Even though traffic generated shows beyond 8GB/s on the external router, only really

@majordragon995

Жыл бұрын

​@@cldptpcie 3.0 8x is yes 8 Gigabytes/s but networking is Gigabits/s, 8GBps translate to 40Gbps. The pcie 3.0 8x is able to transmit both sfp28 to the host no problem, as it would be illogical to have chosen this interface other wise.

@cldpt

Жыл бұрын

@@majordragon995 omg I could swear I saw the traffic generator window show GB/s but it's really kbps and Gbps. My mistake

MT should fulfill it´s promises and support BSD for the CCR2004-PCIe. It would go great with a PFSense or Opnsense Firewall!

@radoslavmatusevicius6143

5 ай бұрын

Mikrotik router inside BSD router.., interesting

@DawidKellerman

2 ай бұрын

Did this happen?

Thank you for this video!

Very-very good, troubleshooting-style video! I’m not familiar with Proxmox, but it was interesting for me. Thanks.

Ideally you have SR-IOV support and poke that into the VM directly rather than use a virtual Ethernet card in KVM. I think otherwise you won't get the full capability of PCIe because the KVM guest has to jump through the kernel in both directions. One thing I have been curious with these cards - is it possible to make the card work basically separately, and then communicate back to the host system via one of the PCIe interfaces? Think like a security appliance where normal packets just come in one interface and go out the other and don't touch the host system's CPU, but packets you want to inspect make a trip through the host system.

@csvarasdy

Жыл бұрын

using SR-IOV you run into IRQ issues due multiqueue - at least with Intel X520 series I don't know, if this card event supports SR-IOV

another great monitoring app for CLI is glances, also give you good overview

Through many different evolutions of traffic generators we finally found that TREX was the most cost effective way to test devices at our ISP. A Dell R610 can generate about 10Gbps in ASTM mode using Intel Optical cards. TREX has been tested up into multiple tens of gigs and there are even anecdotes of it being used at 100Gbps but I cannot verify this.

@Wingnut353

3 ай бұрын

The issue is none of these cheap cards have any offloading so you can never hit anywhere near wire speed anyway because your bottleneck will always end up being in the CPU shuffling bits around for no reason.

Give us EVPN in these cards and you'll see stock go out next day. What an easy enabler of full L3 underlay, especially considering the price.

WONDERFUL!!!

I would love to try this is in a one of my Lenovo / IBM servers

An update video would be greatly appreciated, this is a good card for mikrotik as well as for other open source router OSs. Example, would something like this work with my lenovo tiny p330, could i use this and a switch to have the ultimate router + proxmox + whatever else ?

It would be great to have similar video with the VMWare ESX and Hyper-V :)

About Performance: Try to use openv-switch and set the cpu type of the vm to host.

@DawidKellerman

2 ай бұрын

Would be cool to hear the outcome of the suggestion

Can the switching acceleration hardware on the board used to make a high-performance firewall? Can multiple boards be used on a signle system with the acceleration hardware on the boards to make a larger fabric across the boards?

I ordered 3 of this 7 month ago to do exactly that. Haven't received a single card 😢

I just need to be able to get hold of the damn thing... been on pre-order for nearly a year :(

So I was just wondering if you have tried tweaking the MTU size to fit 25Gb speed ??? I know for 10Gb the MTU can be shaped to 9000 but in my experience leaving it default in production environment is easy troubleshooting.

@RB01-lite

Жыл бұрын

When it comes to the PCIe card itself, to attain the maximum possible throughput jumbo frames are required, but it should be possible to improve the throughput without resulting to that.

9:18 - THX for real values.

@RB01-lite

Жыл бұрын

That's the router running traffic generator and not the card.

@Miesiu

Жыл бұрын

@@RB01-lite Doesn't *cpu-used-by-cpu=100%* limit performance / transfer ?

@RB01-lite

Жыл бұрын

@@Miesiu It limits how much traffic can get generated in the first place, here the issue was that the throughput was lower than what gets generated.

@Miesiu

Жыл бұрын

@@RB01-lite Now understand. THX!

Thanks for the video. Since I saw this NIC announcement, I thought the idea was to run CHR directly on the nic, and not so much use it as a passthrough to other VMs. Is that possible?

@brwainer

Жыл бұрын

This is in the style of other “SmartNIC” or “DPU” cards - having your network card do some amount of helper offload for you, although in Mikrotik’s case its just RouterOS and not a system designed to do trivial data manipulation on the fly or similar. I wish you could run containers on the card (for use in other systems, not with a hypervisor) but it only has 128MB internal storage and no USB.

Product definitely looked interesting. However, the fact it simply stops working whenever it's rebooted kinda kills all use cases. Also, I experienced some kernel panics while running it. I suppose if they can fix the PCI-E initialisation issues (e.g. allow it to re-initialise after the host system has booted), it becomes a much more interesting product. Currently having two of these cards but not deploying them as it simply wasn't stable.

@RB01-lite

Жыл бұрын

Try upgrading to the latest RouterOS version. I rebooted my 2004 a couple of times while leaving the host running and it seemed to work fine.

@RB01-lite

Жыл бұрын

It might depend on the motherboard though.

@JorritPouw

Жыл бұрын

@@RB01-lite Thanks for your reply. So there's still hope. I'll give it a go again soon. 🙂

Isnt virtio limited to 10gps in the driver. The only solution is to pass through the hardware using iommu and making a dedicated VM driving the NIC.

Any use case?

The limit could also be the number of pcie lanes available.

Where can I get it?

Synthetic load is a not VM friendly by any means, maybe try passthrough the whole pci-e slot to the VM, or at least with IOMMU try to individually pass one of the cages

Product seems really interesting, but a bit hard to imagine solid use case for. As it lacks some features that other DPUs have. Albite this is more affordable. Make a version with more RAM and Storage, comparable to nVidia Bluefield and add NVMe-over-TCP support. Or show how it can be used to offload traffic encryption or firewalling. Make a video of more use cases for CCR2004-PCIe.

@RB01-lite

Жыл бұрын

NVMe over TCP already possible with the ROSE package ;)

@andiszile

Жыл бұрын

@@RB01-lite That's nice, but it's only a part of it. DPUs can interpose as a regular NVMe device to Host machine. This is the missing link, or it already can do it ?

@RB01-lite

Жыл бұрын

@@andiszile If I understood you correctly a DPU could just have the host load an NVME drive on bootup, that is physically elsewhere. In the current ROSE implementation you can only access the NVMe-over-TCP drive after the boot process.

@andiszile

Жыл бұрын

@@RB01-lite Ok. Looking into it. Maybe even other DPUs can't really be used as boot devices (unless UEFI can wait for drive to bootup :D ) but they can be used as storage device. Gain is that Host CPU doesn't need to process nvme-tcp protocol by itself.

@andiszile

Жыл бұрын

@@RB01-lite But probably i am too focused on this one particular use case. That's why i would like to see showcase of more use cases that utilize this as more that just a NIC to broaden my view.

Would be amazing as an integrated SAN controller of sort. Well, Network controlled SAN.

ordered it 06/2022 - still waiting. not available like many other products. i am certified for your stuff and need them for customerprojects, but cannot buy them anywhere. i am really pissed

Am I right in thinking that this thing does not support sr-iov? And more importantly - since main selling point is that it's a router - what kind of speed one can expect when this thing is being used as a router?

@RB01-lite

Жыл бұрын

Don't know much about sr-iov, but routing performance depends largely on setup. However it is safe to say that routing with the 25G interfaces will not deliver anything near the wire-speed that is possible in pass-through mode.

@csvarasdy

Жыл бұрын

@@RB01-lite 1) the card is a router itself, it runs ROS as on it's own. 2) seen in the specs, max of 12G routing in best scenario

Can this card work with ESXi 8?

Is the problem on performance solved? Could you consider using SR-IOV please?

@drunkbear889

Ай бұрын

If @mikrotik made a follow up video in 24-25 with SR-IOV backed pass-through to the Proxmox VM. The CPU probably could not coke eggs anymore. ᕕ(⌐■_■)ᕗ ♪♬

do we need license for this?

Card is similar to Radeon RX 6400 !

I wish that it we can just stick to any windows pc then run winbox just like any other mikrotik router :P

when we talk about performance, some words pop in my head, SR-IOV, multi-queue, OVS, DPDK as proxmox is a .... I mean compare to proxmox, vmware vsphere is a (more) enterprise ready platform, should perform best result out of the box (don't know if DirectPath I/O nic helps. but we seen vmxnet3 in vmware gives better performance compare to x520sriov, cause x520sriov driver only support 1 queue.) how ccr2004 pcie card running on that?

@tutugreen_dayo

Жыл бұрын

if you can emulate atl1c, how about emulate more pcie interface for sriov use, will be useful.

@masterTigress96

6 ай бұрын

If I am not mistaken, Linux is the only platform this is supported on. This card requires extra drivers that are not available on VMware. If I am wrong, then please someone let me know, but I recall reading this on the Mikrotik website.

@tutugreen_dayo

6 ай бұрын

​@@masterTigress96 Yes, one year later, this card is still not supported by enterprise virtualization platforms, It's a software emulated card without any hardware offload supported. If they can improve driver, it will be very promising, and we have seen the benefits brought by Bluefield and Amazon Nitro.

Despite my best efforts I cannot find one. I have checked several distributors and they are all telling me they have not had had one for close to a year

Does this card work with XCP-NG?

@jblow530

Жыл бұрын

In PCIe passthrough it should, but I have yet to get my hands on one to test. Looks interesting.

@masterTigress96

6 ай бұрын

What @jblow530 said, so no not for XCP-NG itself if you want to use it to e.g. migrate VM's to another host in a speedy fashion. Linux is as far as I know the only OS this is supported on. XCP-NG and VMware also run a modified version of Linux, but you need something like Proxmox (which is a more standard, full fat Linux distro) to get the drivers. Maybe a custom kernel for XCP-NG of VMware could get it to work, but I haven't tried it.

Uztaisiet, lūdzu, video, kā dabūt AWS site-to-site VPN'a pieslēgumā ar Mikrotik aktīvus abus tuneļus.

I never had great luck with proxmox XCPNG I've had much better network stability

This video is confusing. No agenda and clear scenarios....waste of time of the creator...