FreeNAS 11.2-U5 and 10Gbe

This is just a note as I see a lot of posts out there saying use Chelsio cards for FreeNAS as they are well supported. Well I tried this and yes the card was recognized but the DAC cable was not.

Aug  9 18:45:00 ale cxgbc0: <Chelsio T310, 1 port> mem 0xfb881000-0xfb881fff,0xfb000000-0xfb7fffff,0xfb880000-0xfb880fff irq
 42 at device 0.0 on pci10
Aug  9 18:45:00 ale kernel: PHY 0 i2c read of dev.addr a0.0 timed out
Aug  9 18:45:00 ale kernel: PHY 0 i2c read of dev.addr a0.0 timed out
Aug  9 18:45:00 ale cxgbc0: using MSI-X interrupts (9 vectors)
Aug  9 18:45:00 ale kernel: found old FW minor version(7.10), driver compiled for version 7.11
Aug  9 18:45:00 ale kernel: found old FW minor version(7.10), driver compiled for version 7.11
Aug  9 18:45:00 ale cxgbc0: firmware needs to be updated to version 7.11.0
Aug  9 18:45:00 ale cxgb0: <Port 0 10GBASE-R> on cxgbc0
Aug  9 18:45:00 ale cxgb0: Using defaults for TSO: 65518/35/2048
Aug  9 18:45:00 ale cxgb0: Ethernet address: 00:07:43:05:fe:ae
Aug  9 18:45:00 ale cxgbc0: Firmware Version 7.10.0

Firmware update - I don't think this was necessary as the driver auto updates we just need a reboot

cxgbtool cxgb0 loadfw t3fw-7.11.0.bin
Reboot

Next time around the firmware has been updated

Aug  9 18:55:44 ale cxgbc0: using MSI-X interrupts (9 vectors)
Aug  9 18:55:44 ale cxgb0: <Port 0 10GBASE-R> on cxgbc0
Aug  9 18:55:44 ale cxgb0: Using defaults for TSO: 65518/35/2048
Aug  9 18:55:44 ale cxgb0: Ethernet address: 00:07:43:05:fe:ae
Aug  9 18:55:44 ale cxgbc0: Firmware Version 7.11.0

The drive is plumbed into the kernel but the link won't come up to ACTIVE.

I was using SPF+ CISCO Passive Twinax cable part 74752-9519 connected to Mikrotik CRS305 switch but the link would not activate. I switched to a different 10Gbe card and presto.

MNPA19-XTR | Mellanox EN ConnectX-2 1-Port 10GbE Network Adapter

Aug 22 19:52:55 ale mlx4_core0: <mlx4_core> mem 0xfaa00000-0xfaafffff,0xf5000000-0xf57fffff irq 42 at device 0.0 on pci10
Aug 22 19:52:55 ale kernel: mlx4_core: Mellanox ConnectX core driver v3.4.1 (October 2017)
Aug 22 19:52:55 ale kernel: mlx4_core: Initializing mlx4_core
Aug 22 19:52:55 ale mlx4_core0: Unable to determine PCI device chain minimum BW
Aug 22 19:52:55 ale kernel: mlx4_en mlx4_core0: Activating port:1
Aug 22 19:52:55 ale mlxen0: Ethernet address: 00:02:c9:51:35:34
Aug 22 19:52:55 ale kernel: mlx4_en: mlx4_core0: Port 1: Using 12 TX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlx4_core0: Port 1: Using 12 TX rings
Aug 22 19:52:55 ale kernel: mlxen0: link state changed to DOWN
Aug 22 19:52:55 ale kernel: mlxen0: link state changed to DOWN
Aug 22 19:52:55 ale kernel: mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Using 12 TX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Using 12 TX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Using 8 RX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Using 8 RX rings
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Initializing port
Aug 22 19:52:55 ale kernel: mlx4_en: mlxen0: Initializing port

ifconfig output for the device of course at this point it's not configure with an IP but the status is “ACTIVE” and the switch can see it too.

root@ale[~]# ifconfig mlxen0
mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:51:35:34
        hwaddr 00:02:c9:51:35:34
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>)
        status: active
root@ale[~]#