Linux Predictable Interface Names

The problem

This week at work while upgrading a hypervisor from Bullseye to Bookworm, the automatic provisioning failed. Upon closer inspection, it became clear the failure was caused by the machine not having network connectivity after the first step of provisioning.

Upon closer inspection, things became clearer: the predictable interface names had changed.

On a Bullseye host of the same model we have:

2: enp65s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether f8:f2:1e:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever
3: enp65s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether f8:f2:1e:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet6 fe80::faf2:1eff:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever

On the Bookworm host we have:

2: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether f8:f2:1e:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp65s0f0
    inet6 fe80::faf2:1eff:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever
3: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether f8:f2:1e:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp65s0f1
    inet6 fe80::faf2:1eff:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever

It felt odd. I had a faint memory of having a similar issue when we went from Buster to Bullseye a few years ago, so I went to check our Puppet repository.

commit 3ce56af31f5ce8cfbbc5a6a0e96fc2b56f47fab4
Author: Luiz Amaral
Date:   Mon Jul 11 14:39:54 2022 +0200

    networking: Update R6515 interfaces on bullseye

diff --git a/manifests/server/networking/debian.pp b/manifests/server/networking/debian.pp
index a75091409..8e45d18a4 100644
--- a/manifests/server/networking/debian.pp
+++ b/manifests/server/networking/debian.pp
@@ -35,7 +35,10 @@ class ig::server::networking::debian inherits ig::server::networking {
-            'PowerEdge R6515'  => ['ens3f0', 'ens3f1'],
+            'PowerEdge R6515'  => ig::os::is_debian_at_least('bullseye') ? {
+                true  => ['enp65s0f0', 'enp65s0f1'],
+                false => ['ens3f0', 'ens3f1'],
+            },

So we have been down this path before, except that now we are back to the behavior from Buster times.

Poking around

While debugging this issue, I found an interesting page from Debian. In the section How to migrate to this scheme on upgraded systems, they provided a very useful command to grasp what is going on:

$ udevadm test-builtin net_id /sys/class/net/<if_name>

So I try that on Bookworm and get a wall of text:

$ udevadm test-builtin net_id /sys/class/net/ens3f0
Trying to open "/etc/systemd/hwdb/hwdb.bin"...
Trying to open "/etc/udev/hwdb.bin"...
Trying to open "/usr/lib/systemd/hwdb/hwdb.bin"...
Trying to open "/lib/systemd/hwdb/hwdb.bin"...
Trying to open "/lib/udev/hwdb.bin"...
=== trie on-disk ===
tool version:          252
file size:        11878541 bytes
header size             80 bytes
strings            2501253 bytes
nodes              9377208 bytes
Loading kernel module index.
Failed to read $container of PID 1, ignoring: Permission denied
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Found container virtualization none.
Using default interface naming scheme 'v252'.
Skipping overridden file '/usr/lib/systemd/network/99-default.link'.
Parsed configuration file "/etc/systemd/network/99-default.link"
Parsed configuration file "/usr/lib/systemd/network/73-usb-net-by-mac.link"
Created link configuration context.
ID_NET_NAMING_SCHEME=v252
ID_NET_NAME_MAC=enxf8f21exxxxxx
ens3f0: MAC address identifier: hw_addr=f8:f2:1e:xx:xx:xx → xf8f21exxxxxx
ID_OUI_FROM_DATABASE=Intel Corporate
sd-device: Failed to chase symlinks in "/sys/devices/pci0000:40/0000:40:01.1/0000:41:00.0/of_node".
sd-device: Failed to chase symlinks in "/sys/devices/pci0000:40/0000:40:01.1/0000:41:00.0/physfn".
ens3f0: Parsing slot information from PCI device sysname "0000:41:00.0": success
ens3f0: dev_port=0
ens3f0: PCI path identifier: domain=0 bus=65 slot=0 func=0 phys_port= dev_port=0 → p65s0f0
0000:40:01.1: Device is a PCI bridge.
ens3f0: Slot identifier: domain=0 slot=3 func=0 phys_port= dev_port=0 → s3f0
ID_NET_NAME_PATH=enp65s0f0
ID_NET_NAME_SLOT=ens3f0
Unload kernel module index.
Unloaded link configuration context.

Most important to look at, are the ones starting with ID_NET_NAME (listed here in order of priority):

ID_NET_NAME_SLOT=ens3f0
ID_NET_NAME_PATH=enp65s0f0
ID_NET_NAME_MAC=enxf8f21exxxxxx

It makes sense. The highest priority is the ID_NET_NAME_SLOT and that is what was used, all fine so far. I then took a look at the Bullseye host. The same command yielded:

$ sudo udevadm test-builtin net_id /sys/class/net/enp65s0f0
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Parsed configuration file /usr/lib/systemd/network/73-usb-net-by-mac.link
Created link configuration context.
Using default interface naming scheme 'v247'.
ID_NET_NAMING_SCHEME=v247
ID_NET_NAME_MAC=enxf8f21exxxxxx
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp65s0f0
Unload module index
Unloaded link configuration context.

Much shorter this time, and most importantly, ID_NET_NAME_SLOT is nowhere to be seen! After a more careful look, there is another important difference:

< ID_NET_NAMING_SCHEME=v252
---
> ID_NET_NAMING_SCHEME=v247

Bookworm is using a newer version than Bullseye, maybe something changed between these two versions?

The reveal

With the information on the different naming schemes, I started looking for any kind of changelogs from systemd. I somehow manage to land on systemd.net-naming-scheme.

There I encounter the following:

When a PCI slot is associated with a PCI bridge that has multiple child network controllers, the same value of the ID_NET_NAME_SLOT property might be derived for those controllers. This would cause a naming conflict if the property is selected as the device name. Now, we detect this situation and don’t produce the ID_NET_NAME_SLOT property.
Added in version 247.

And a little further down:

Since version v247 we no longer set ID_NET_NAME_SLOT if we detect that a PCI device associated with a slot is a PCI bridge as that would create naming conflict when there are more child devices on that bridge. Now, this is relaxed and we will use slot information to generate the name based on it but only if the PCI device has multiple functions. This is safe because distinct function number is a part of the device name for multifunction devices.
Added in version 251.

Wait a moment! Buster was running systemd v241, Bullseye v247 and Bookworm v252. That answers everything. ID_NET_NAME_SLOT was there in Buster, got removed in Bullseye and now is back on Bookworm.

With that said (or should I say written?), off I go to patch the Puppet code some more, in hopes that this won’t change yet again in the future.

Hacking around the issue

While investigating it, I figured out one way of making things “stable”. One could add a /etc/systemd/network/99-default.link file with the following contents:

[Match]
OriginalName=*

[Link]
NamePolicy=keep path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

This would force systemd to always use the path naming scheme, unless a persistent name is indicated somewhere else.

The problem#

Poking around#

The reveal#

Hacking around the issue#

The problem

Poking around

The reveal

Hacking around the issue