オマエラ wlog
← go back

making the ODI DFP-34X-2C2 GPON SFP work on the intel 82599es

by z411 @ 2025-03-26 [ tech ]

It's been a few years since I started my own homelab and lately I've been moving to a rack-based setup. That includes a Ubiquiti 24-port switch, a custom built 1U Proxmox Server and a 2U NAS. Until now I was using an OpenWRT flashed TP-Link router but I always wanted to move to OPNsense. So I got a 1U box with Ethernet and SFP ports, and installed OPNsense on it.

Since the box had SFP ports I said - why use the ISP router if I already can use a SFP ONT on it? So I got the ODI DFP-34X-2C3 which has the popular RTL9601D chip. The problem came when the Intel SFP card didn't recognize it - I had enabled allow_unsupported_sfp and it still didn't work.

In the end I ended up hacking the ixgbe driver to force it to work, and it works flawlessly. I'll show you how.

New BKHD 1U soft router box

The quick and dirty hack

If you just want it to work (like I did) follow these steps.

Step 1: Get the Intel ixgbe driver

First we need to get the driver from Intel. Get it from the Intel Github:

git clone https://github.com/intel/ethernet-linux-ixgbe/

Step 2: Patch the driver

Now we need to edit the driver so it forcefully recognizes the stick. Open the file src/ixgbe_phy.c and look for these lines (1495 at the time of writing):

/* Verify supported 1G SFP modules */
if (comp_codes_10g == 0 &&
    !(hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core1 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core1 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core1)) {
        hw->phy.type = ixgbe_phy_sfp_unsupported;
        status = IXGBE_ERR_SFP_NOT_SUPPORTED;
        goto out;
}

Change them to this:

/* Verify supported 1G SFP modules */
if (comp_codes_10g == 0 &&
    !(hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core1 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core1 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core0 ||
      hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core1)) {
        printk(" -- custom driver patch applied.\n");
        hw->phy.type = ixgbe_phy_sfp_intel;
        hw->phy.sfp_type = ixgbe_sfp_type_1g_lx_core0;
        status = IXGBE_SUCCESS;
        goto out;
}

What we're doing here is that if the stick isn't 10G and the driver fails to recognize its type, we force it to Intel type (so it doesn't complain about unsupported brand) and 1000BASE-LX so it negotiates the speed properly.

Step 3: Compile the driver

Now we compile and install the driver, run inside the src/ directory:

make && make install

If you check the driver location with modinfo ixgbe it should show it in the updates directory.

Step 4: Loading and testing our patched driver

If everything looks as expected, unload the module and reload the one we just compiled:

rmmod ixgbe && modprobe ixgbe

If you check dmesg it should show you the debug message we added:

dmesg output

ethtool <ifname> should show us 1000baseT/Full speed, and ethtool -m <ifname> should show the ODI stick information.

ethtool output

Now let's try adding an IP to our interface and set it to up:

ip address add 192.168.1.10/24 dev <ifname>
ip link set up dev <ifname>

And after a minute or so, the interface should change its state to UP and show the link LED:

uplink

uplink led

Try pinging 192.168.1.1 (default IP address of the ODI stick) and it should reply:

working ping

If everything works as expected you should be able to configure it through the WebUI and use it by your router.

working router

What about FreeBSD/OPNsense?

The FreeBSD ix driver is basically a port of the Linux ixgbe driver so it's the same. The difference is the driver in FreeBSD is statically compiled in the GENERIC kernel rather than compiled as a module, which means you'll have to compile the kernel. This is actually rather simple in FreeBSD compared to Linux.

Please follow the instructions in the FreeBSD handbook about compiling the kernel:
FreeBSD Handbook - Chapter 10. Configuring the FreeBSD Kernel

Get the kernel source, then edit the file /usr/src/sys/dev/ixgbe/ixgbe_phy.c just like we did before, then compile it, install it, reboot, and everything should work as well.

You can see detailed information about the card and module using ifconfig -v <ifname>.

The proper fix, and the reason of why this happens

If you're interested in knowing why this has to be done, it has to do with compliance. The Intel driver follows the SFF-8472 spec closely, and address 0x06 (byte 6) is supposed to carry the compliance bit. The ODI module is a non-compliant HiSGMII module and only uses the address 0x0c (byte 12) to signal its nominal rate, which is not really understood by the Intel driver.

So if we wanted a proper fix we would need to do something like this (I haven't tested it):

diff --git a/src/ixgbe_phy.c b/src/ixgbe_phy.c
index 3d99a88..9a990f6 100644
--- a/src/ixgbe_phy.c
+++ b/src/ixgbe_phy.c
@@ -1270,6 +1270,7 @@ s32 ixgbe_identify_sfp_module_generic(struct ixgbe_hw *hw)
        u8 oui_bytes[3] = {0, 0, 0};
        u8 cable_tech = 0;
        u8 cable_spec = 0;
+       u8 br_nom = 0;
        u16 enforce_sfp = 0;

        DEBUGFUNC("ixgbe_identify_sfp_module_generic");
@@ -1313,6 +1314,14 @@ s32 ixgbe_identify_sfp_module_generic(struct ixgbe_hw *hw)

                if (status != IXGBE_SUCCESS)
                        goto err_read_i2c_eeprom;
+
+               status = hw->phy.ops.read_i2c_eeprom(hw,
+                                                    IXGBE_SFF_BITRATE_NOMINAL,
+                                                    &br_nom);
+
+               if (status != IXGBE_SUCCESS)
+                       goto err_read_i2c_eeprom;
+

                 /* ID Module
                  * =========
@@ -1393,6 +1402,13 @@ s32 ixgbe_identify_sfp_module_generic(struct ixgbe_hw *hw)
                                else
                                        hw->phy.sfp_type =
                                                ixgbe_sfp_type_1g_lx_core1;
+                       } else if (br_nom >= 12 && br_nom <= 13) {
+                               if (hw->bus.lan_id == 0)
+                                       hw->phy.sfp_type =
+                                               ixgbe_sfp_type_1g_lx_core0;
+                               else
+                                       hw->phy.sfp_type =
+                                               ixgbe_sfp_type_1g_lx_core1;
                        } else {
                                hw->phy.sfp_type = ixgbe_sfp_type_unknown;
                        }
diff --git a/src/ixgbe_phy.h b/src/ixgbe_phy.h
index b6ddb2e..d9c926e 100644
--- a/src/ixgbe_phy.h
+++ b/src/ixgbe_phy.h
@@ -18,6 +18,7 @@
 #define IXGBE_SFF_1GBE_COMP_CODES      0x6
 #define IXGBE_SFF_10GBE_COMP_CODES     0x3
 #define IXGBE_SFF_CABLE_TECHNOLOGY     0x8
+#define IXGBE_SFF_BITRATE_NOMINAL      0xC
 #define IXGBE_SFF_CABLE_SPEC_COMP      0x3C
 #define IXGBE_SFF_SFF_8472_SWAP                0x5C
 #define IXGBE_SFF_SFF_8472_COMP                0x5E

Not sure if Intel would want to handle this behavior which is why I haven't sent the patch but if anyone is interested in testing it and upstreaming it you can contact me.

Thanks

I'd like to thank the following people:

Tweet | Permalink | 12:16