Cfrtjryk

We have this log in our syslog:

udevd [ PID ]: inotify_add_watch(6, /dev/sda, 10) failed: operation not permitted

Why do we get this error and how do we solve it?

Our environment: Ubuntu 12.04; LXC; we run inside a container; and I am not sure about SELinux (I don't have access) but it's not enabled.

Ubuntu12.04: How to disable a daemon process at startup

Debian Bug report logs - #620921
udev: Please detect lxc, and don't try to start there

At the first glance udev events are supported in the container. But for
the sake of optimization, I recommend to not use it as it will trigger
the events in all the containers.

In case the above is not clear, I suggest killing it with fire. Usually it would not be desirable for udev inside a container to be even thinking about touching sda etc. Usually there would not be anything that you would want udev to do.

Reading the following, you might guess my answer is toeing the systemd party line :-). Apparently LXC had some different opinions, at least at one time: https://stgraber.org/2013/12/21/lxc-1-0-your-second-container/#comments

I believe the commenter "wwwwww" is a pseudonym (!) of the systemd lead Lennart Poettering. Either that, or someone did a great imitation matching his writing style and his position on this issue :-).

Perhaps someone more familiar with LXC would know exactly which combinations of udev and LXC setups that LXC expects to do anything useful. And what conditions might generate a warning message like this. The above link offered a date range for Ubuntu, which claims the original Ubuntu 12.04 release should be fine. However it does not say whether or not it emits any spurious warnings. (It wouldn't be the first piece of software to do so:-))

Whatever the merits, if you don't need to access any physical device from inside LXC, disabling udev would seem a simple way to avoid seeing any udev warnings. "While we wait for people to figure out exactly how a device namespace should work". The LXC developer mentions "this is far from ideal" :-). This was in 2013, and there is still no device namespace (as of Linux v4.20).

The next relevant comment seems to be "Our default configuration will let udev create device nodes but only access those that are allowed in the configuration." In that sense your LXC was working as LXC want it to: it allowed you to create a device node /dev/sda, but did not allow you to access it.

I do not know why your udev creates /dev/sda, (presumably) does not complain about being unable to run blkid on it, but does complain about not being to watch it.

The kernel (as of v4.20) does not provide isolation for devices. There is no namespace for devices. Compared e.g. to network namespaces, which allow isolating network interfaces. For the list of namespaces which can be isolated, look in man 7 namespaces or man 2 clone.

If you're curious what a principled container runtime can do, the answer is that it can disable access to all devices (except a few virtual ones like /dev/null, /dev/pts/*, etc). I am more familiar with systemd-nspawn (and its documentation). At least with cgroups v1, nspawn uses the device control group to disable access to devices. cgroups v2 eventually gained an equivalent feature. In the mean time, nspawn prevents you from creating a device node by using seccomp(), and that works pretty well. Of course this means you must trust the container filesystem image not to contain the any of the "wrong" device nodes, so the cgroup solution is better.

Current systemd-udevd.service detects that it should not run if /sys has been mounted read-only.