Linux hotplug events: How udev and netlink power device detection
When a USB device connects, a network interface appears, or a disk is hot-swapped, Linux needs to notify the right processes. This mechanism, rooted in netlink sockets and udev, forms the backbone of dynamic device management. Understanding these internals helps SREs debug device detection failures, automate hardware provisioning, and build reliable infrastructure automation.
What is netlink?
Netlink is a Linux-specific socket family for communication between the kernel and userspace. Unlike syscalls that userspace must initiate, netlink allows the kernel to push notifications asynchronously. It uses a datagram model similar to UDP but supports ancillary data and multicast groups.
For hardware events, the kernel uses NETLINK_KOBJECT_UEVENT to broadcast device changes. These events include information about device paths, subsystems, and actions like add, remove, or change.
The udev layer
Raw kernel events lack context that applications need. The kernel might announce a device before firmware loads or permissions are set. This is where udev steps in.
udev listens to kernel events, applies rules from /etc/udev/rules.d/, performs actions like setting permissions or creating symlinks, and then rebroadcasts the processed events. Applications using libudev receive these sanitized events, avoiding race conditions with kernel-level changes.
The event flow looks like this:
Kernel (kobject) -> Netlink -> udev daemon -> Netlink -> Applications
Listening to events directly
You can monitor both kernel and udev events without libudev. The kernel events use MONITOR_GROUP_KERNEL (group 1), while udev rebroadcasts use MONITOR_GROUP_UDEV (group 2).
# Monitor kernel events
udevadm monitor --kernel
# Monitor udev-processed events
udevadm monitor --udev
# Both together
udevadm monitor --kernel --udev
The udev stream includes a header with filtering data: subsystem hashes, device type hashes, and a bloom filter for tags. This lets applications efficiently filter events without parsing the full payload.
SRE applications
Understanding hotplug events enables powerful automation:
Dynamic storage provisioning - Detect when disks appear in cloud environments and automatically partition, format, and mount them based on naming conventions or metadata.
Network interface management - Trigger bonding reconfiguration or VLAN setup when interfaces are added or removed, essential for bare-metal Kubernetes nodes.
USB device control - Enforce security policies by detecting unauthorized devices and triggering alerts or automatic disconnection.
Container runtime integration - Pass through devices to containers only when they appear, useful for GPU workloads or specialized hardware.
Debugging tips
When devices fail to appear correctly, check the event chain:
# See what the kernel reports
udevadm monitor --kernel --property
# Test rule matching
udevadm test /sys/path/to/device
# Trigger a re-scan
udevadm trigger --subsystem-match=block
Look for udev-worker processes stuck on slow rules or external commands that block event processing.
Conclusion
Linux hotplug events through netlink and udev provide the foundation for dynamic hardware management. For SREs building automated infrastructure, understanding this event chain helps debug device detection issues and build reliable hardware provisioning workflows.
Akmatori helps SRE teams automate infrastructure operations with AI agents that understand your systems. Try Akmatori to bring intelligent automation to your infrastructure, powered by Gcore's global edge network.