Upgrading the kernel should be quick and painless…
After upgrading the kernel from v5.3 to 5.7, a lot of systemd services failed (Debian 8), in particular systemd-remount-fs:
● systemd-remount-fs.service - Remount Root and Kernel File Systems
Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; static)
Active: failed (Result: exit-code) since Sun 2020-07-26 15:28:15 IDT; 17min ago
Docs: man:systemd-remount-fs.service(8)
http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
Process: 223 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE)
Main PID: 223 (code=exited, status=1/FAILURE)
Jul 26 15:28:15 systemd[1]: systemd-remount-fs.service: main process exited, code=exited, status=1/FAILURE
Jul 26 15:28:15 systemd[1]: Failed to start Remount Root and Kernel File Systems.
Jul 26 15:28:15 systemd[1]: Unit systemd-remount-fs.service entered failed state.
and indeed, the root NFS remained read-only (checked with “mount” command), which explains why so many other services failed.
After an strace session, I managed to nail down the problem: The system call to mount(), which was supposed to do the remount, simply failed:
mount("10.1.1.1:/path/to/debian-82", "/", 0x61a250, MS_REMOUNT, "addr=10.1.1.1") = -1 EINVAL (Invalid argument)
On the other hand, any attempt to remount another read-only NFS mount, which had been mounted the regular way (i.e. after boot) went through clean, of course:
mount("10.1.1.1:/path/to/debian-82", "/mnt/tmp", 0x61a230, MS_REMOUNT, "addr=10.1.1.1") = 0
The only apparent difference between the two cases is the third argument, which is ignored for MS_REMOUNT according to the manpage.
The manpage also says something about the EINVAL return value:
EINVAL A remount operation (MS_REMOUNT) was attempted, but source was not already mounted on target.
A hint to the problem could be that the type of the mount, as listed in /proc/mounts, is “nfs” for the root mounted filesystem, but “nfs4″ for the one in /mnt/tmp. The reason for this difference isn’t completely clear.
The solution
So it’s all about that little hint: If the nfsroot is selected to boot as version 4, then there’s no problem remounting it. Why it made a difference from one kernel version to another is beyond me. So the fix is to add nfsvers=4 to the nfsroot assignment. Something like
root=/dev/nfs nfsroot=10.1.1.1:/path/to/debian-82,nfsvers=4
For the record, I re-ran the remount command with strace again, and exactly the same system call was made, including that most-likely-ignored 0x61a250 argument, and it simply returned success (zero) instead of EINVAL.
As a side note, the rootfstype=nfs in the kernel command line is completely ignored. Write any junk instead of “nfs” and it makes no difference.
Another yak shaved successfully.