PVE 搞 PCIe 直通, IOMMU Group 过大怎么办? - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
sky96111
V2EX    服务器

PVE 搞 PCIe 直通, IOMMU Group 过大怎么办?

  •  
  •   sky96111 2023-10-23 10:22:24 +08:00 1884 次点击
    这是一个创建于 798 天前的主题,其中的信息可能已经有所发展或是发生改变。

    主板是 ASUS prime b350-plus ,系统是 PVE8

    已经试过加内核参数pcie_acs_override=downstream

    root@pve ~# uname -a Linux pve 6.2.16-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-15 (2023-09-28T13:53Z) x86_64 GNU/Linux root@pve ~# cat /etc/default/grub GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream" GRUB_CMDLINE_LINUX="" 

    PCIe 插了 w2100 ,LSI SAS2308 。现想将 lsi sas 卡直通到 NAS 虚拟机,然而我的 iommu 将 USB 控制器、SATA 控制器、有线网卡、和 PCIe SAS 卡全分在了同一组:

    root@pve ~# iommu.sh ... Group 13: [144d:a808] [R] 01:00.0 Non-Volatile memory controller NVMe SSD Controller SM981/PM981/PM983 Group 14: [1022:43bb] [R] 02:00.0 USB controller 300 Series Chipset USB 3.1 xHCI Controller USB: [1d6b:0002] Bus 001 Device 001 Linux Foundation 2.0 root hub USB: [1d6b:0003] Bus 002 Device 001 Linux Foundation 3.0 root hub [1022:43b7] 02:00.1 SATA controller 300 Series Chipset SATA Controller [1022:43b2] 02:00.2 PCI bridge Device 43b2 [1022:43b4] [R] 03:00.0 PCI bridge 300 Series Chipset PCIe Port [1022:43b4] 03:01.0 PCI bridge 300 Series Chipset PCIe Port [1022:43b4] 03:04.0 PCI bridge 300 Series Chipset PCIe Port [10ec:8168] [R] 04:00.0 Ethernet controller RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1b21:1080] [R] 05:00.0 PCI bridge ASM1083/1085 PCIe to PCI Bridge [1000:0087] [R] 07:00.0 Serial Attached SCSI controller SAS2308 PCI-Express Fusion-MPT SAS-2 Group 15: [1002:6608] [R] 08:00.0 VGA compatible controller Oland GL [FirePro W2100] [1002:aab0] [R] 08:00.1 Audio device Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series] ... 

    意味着我只能将 group 14 的设备全部直通到 nas ,这明显是不现实的。还有什么能将 SAS 控制器直通的方案吗?

    3 条回复    2023-10-23 22:58:08 +08:00
    sky96111
        1
    sky96111  
    OP
       2023-10-23 10:26:15 +08:00
    20015jjw
        2
    20015jjw  
       2023-10-23 20:26:32 +08:00 via iPhone
    这主板真的配吗..
    sky96111
        3
    sky96111  
    OP
       2023-10-23 22:58:08 +08:00
    @20015jjw 淘汰旧设备废物利用嘛。
    最后解决了,是主板的问题。只有显卡那条 PCIe 是走 CPU 的,其他都是走的南桥所以都在同一个 iommu 组。
    更换了一下插槽顺序,现在 SAS 控制器是单独分组了
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     5228 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 30ms UTC 09:14 PVG 17:14 LAX 01:14 JFK 04:14
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86