System, systemd on NFS (v4) fails, rpc.idmapd

March 29th, 2018

• Index
• Installation
• [solved] [FAILED] Failed to start Remount Root and Kernel File System

updates: 5 (20171209)

updates: 5 (20171210)

• mount -t nfs4 [SERVER IP]:/archlinux /mnt works.
• ss -ntp | grep 2049 the client establishes a connection to the server before systemd begins.
• NSF4 id mapper can only be used with Kerberos?

Failed to start Remount Root and Kernel File Systems

Hi when I’m trying to start ubuntu it says that it Failed to start Remount Root and Kernel File Systems. I’ve tried to start it in recovery mode and typed: sudo fsck -f /dev/sda6 but it didn’t work. My partitions look like this:

/dev/sda1 — EFI System
/dev/sda2 — Microsoft reserved
/dev/sda3 Microsoft basic data
/dev/sda4 Microsoft basic data
/dev/sda5 Windows recovery enviroment
/dev/sda6 Linux filesystem
/dev/sda7 Linux swap

please help, I’m pretty desperate..

March 29th, 2018

Failed to start Remount Root and Kernel File Systems

May be best to see details, you can run from your Ubuntu live installer or any working install, use ppa version not older Boot-Repair ISO:
Post the link to the Create BootInfo summary report. Is part of Boot-Repair:
https://help.ubuntu.com/community/Boot-Info and:
https://sourceforge.net/p/boot-repair/home/Home/

You may need fsck on your / partition.

To see all the ext4 partitions
sudo parted -l
#From liveDVD/Flash so everything is unmounted,swap off if necessary, change example shown with partition sdb1 to your partition(s)
#e2fsck is used to check the ext2/ext3/ext4 family of file systems. -p tries fixes where response not required
sudo e2fsck -C0 -p -f -v /dev/sdb1
# -y auto answers yes for fixes needing response, also see man e2fsck
sudo e2fsck -f -y -v /dev/sdb1

Поставил себе Manjaro через Manjaro Architect. И при загрузке у меня выкидывает:

Failed to start Remount Root and Kernel File Systems.

При этом система дальше грузится. Вроде всё хорошо, замечаний в работае нету.

~ >>> systemctl --failed [130] UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

~ >>> systemctl status systemd-remount-fs.service
● systemd-remount-fs.service - Remount Root and Kernel File Systems Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2017-08-25 21:56:50 EEST; 11min ago Docs: man:systemd-remount-fs.service(8) https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 461 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) Main PID: 461 (code=exited, status=1/FAILURE)
Авг 25 21:56:50 FLTTSHY systemd[1]: Starting Remount Root and Kernel File Systems...
Авг 25 21:56:50 FLTTSHY systemd-remount-fs[461]: mount: /: mount point not mounted or bad option.
Авг 25 21:56:50 FLTTSHY systemd-remount-fs[461]: /usr/bin/mount for / exited with exit status 32.
Авг 25 21:56:50 FLTTSHY systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Авг 25 21:56:50 FLTTSHY systemd[1]: Failed to start Remount Root and Kernel File Systems.
Авг 25 21:56:50 FLTTSHY systemd[1]: systemd-remount-fs.service: Unit entered failed state.
Авг 25 21:56:50 FLTTSHY systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.

При попытке запуска процесса

~ >>> sudo /usr/lib/systemd/systemd-remount-fs [130]
[sudo] пароль для alex:
mount: /: mount point not mounted or bad option.
/usr/bin/mount for / exited with exit status 32.
~ >>> 

Что это может быть?

Установил я через debootstrap Debian jessie. Только ядро и systemd взял из бэкпортов: 4.8.0-0.bpo2-amd64, systemd 230.

Всё замечательно, только система факапится на монтировании корня: Failed to mount /.

ОС у меня установлена на LVM.

В процессе дебага выяснил, что всё монтируется великолепно ещё в initramfs. Далее успешно происходит чек диска и ремаунт юнитом systemd-remount-fs.

Но юнит local-fs.target не запускается из-за фейла зависимости -.mount:

[ 4.801773] systemd[1]: -.mount: Directory / to mount over is not empty, mounting anyway.
[ 4.802659] systemd[1]: -.mount: About to execute: /bin/mount /dev/mapper/os-root / -t ext4 -o errors=remount-ro
[ 4.803013] systemd[1]: -.mount: Forked /bin/mount as 767
[ 4.803234] systemd[1]: -.mount: Changed dead -> mounting
[ 4.804544] systemd[767]: -.mount: Executing: /bin/mount /dev/mapper/os-root / -t ext4 -o errors=remount-ro
[ 4.809787] systemd[1]: Received SIGCHLD from PID 767 (mount).
[ 4.809862] systemd[1]: Child 767 (mount) died (code=exited, status=32/n/a)
[ 4.809972] systemd[1]: -.mount: Child 767 belongs to -.mount
[ 4.809990] systemd[1]: -.mount: Mount process exited, code=exited status=32
[ 4.810005] systemd[1]: -.mount: Changed mounting -> failed
[ 4.810290] systemd[1]: -.mount: Job -.mount/start finished, result=failed
[ 4.810315] systemd[1]: Failed to mount /.

Утешительный вывод: Systemd пытается на кой-то хрен смонтировать корень в тот момент, когда он уже отлично смонтирован и хорошо себя чувствует. Если я перехожу в debug shell, вижу в что всё смонтировано отлично. Но на буте команда mount ожидаемо отдаёт ненулевой код, чем ломает local-fs.target и все зависимости далее.

Если убрать рут из fstab, то всё запускается замечательно, но тогда systemd-remount-fs не сработает (по исходникам он читает /etc/fstab хардкодно).

Вот мой /etc/fstab:

/dev/mapper/os-root	/	ext4	errors=remount-ro	0	1
/dev/mapper/data-storage	/var/lib/storage/	ext4	noatime,relatime,nodev,nosuid,noexec	0	0
UUID=fa1c680a-13d2-48c2-808d-bec1966ed483	none	swap	sw	0	0
UUID=2CB2-3E54	/boot/efi/	vfat	defaults	0	1

Если ставить не на lvm, а на раздел, то ничего не меняется.

Параметры загрузки ядра:

root=/dev/mapper/os-root ro rootfstype=ext4 systemd.debug-shell=1 systemd.log_level=debug systemd.log_target=kmsg log_buf_len=100M printk.devkmsg=on quiet splash

Очень помог ман по дебагу, но что-то я уже бессилен. https://freedesktop.org/wiki/Software/systemd/Debugging/

Как мне в итоге сказать systemd игнорировать уже смонтированный раздел? Что вообще творится? Да, параметр nofail игнорируется для рутовых разделов, о чём радостно рапортует systemd-fstab-generator.

Заранее отвечу на кой хрен мне systemd из бэкпортов — нужен networkctl.

since i updated systemd a couple of days ago, i get an while booting.

This is what `systemctl status systemd-remount-fs.service` says:

[user@machine ~]$ systemctl status systemd-remount-fs.service
systemd-remount-fs.service - Remount Root and Kernel File Systems Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; static) Active: failed (Result: exit-code) since Mon 2014-03-10 22:58:51 CET; 6min ago Docs: man:systemd-remount-fs.service(8) http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Main PID: 6966 (code=exited, status=1/FAILURE)
Mar 10 22:58:50 machine systemd-remount-fs[6966]: mount: /etc/fstab: parse error: ignore entry at line 5.
Mar 10 22:58:50 machine systemd-remount-fs[6966]: mount: /etc/fstab: parse error: ignore entry at line 8.
Mar 10 22:58:50 machine systemd-remount-fs[6966]: mount: /etc/fstab: parse error: ignore entry at line 11.
Mar 10 22:58:51 machine systemd-remount-fs[6966]: mount: / not mounted or bad option
Mar 10 22:58:51 machine systemd-remount-fs[6966]: In some cases useful info is found in syslog - try
Mar 10 22:58:51 machine systemd-remount-fs[6966]: dmesg | tail or so.
Mar 10 22:58:51 machine systemd[1]: systemd-remount-fs.service: main process exited, code=exited, status=1/FAILURE
Mar 10 22:58:51 machine systemd[1]: Failed to start Remount Root and Kernel File Systems.
Mar 10 22:58:51 machine systemd[1]: Unit systemd-remount-fs.service entered failed state.

This is what blkid tells me:

/dev/sda1: LABEL="EFI" UUID="70D6-1701" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="0b59a783-8d1f-4923-8136-44151a653b9e"
/dev/sda2: UUID="0a9de25e-ec90-3e7b-ab65-5ed4b22c2d1f" LABEL="Macintosh HD" TYPE="hfsplus" PARTLABEL="Macintosh HD" PARTUUID="ffbfb815-51bb-4797-b35a-dd4a9acdb63f"
/dev/sda3: UUID="475699d6-1fac-3da8-b391-f417ce11288c" LABEL="Recovery HD" TYPE="hfsplus" PARTLABEL="Recovery HD" PARTUUID="b855bfe8-52a4-4ed1-bd86-e2f664e95107"
/dev/sda4: UUID="ab78d78c-bc47-350f-8122-010a538dad81" LABEL="booter" TYPE="hfsplus" PARTLABEL="Boot Part" PARTUUID="3769aa28-2909-487d-a406-108e969b2cc2"
/dev/sda5: UUID="27af4583-6c81-460f-a027-c2c64379971b" TYPE="ext4" PARTLABEL="Boot" PARTUUID="752d4d1d-e5ad-411e-9b18-8c029b290dd8"
/dev/sda6: UUID="b9a9f27e-04ca-4141-9b22-ae1c6bd64d3e" TYPE="swap" PARTLABEL="SWAP drive" PARTUUID="a9b32507-f88c-49e5-8566-052dc8c0abf2"
/dev/sda7: UUID="d18f76b7-96cc-48e2-854a-c361f069c197" TYPE="ext4" PARTLABEL="Root" PARTUUID="4608ccaf-ba93-4b23-8528-2878c6cc1db1"

This is how my fstab looks like:

#
# /etc/fstab: static file system information
#
# <file system>	<dir>	<type>	<options>	<dump>	<pass>
UUID=d18f76b7-96cc-48e2-854a-c361f069c197
/dev/sda7	/	ext4	rw,relatime,data=ordered	0 1
UUID=27af4583-6c81-460f-a027-c2c64379971b
/dev/sda5	/boot	ext4	rw,relatime,data=ordered	0 2
UUID=b9a9f27e-04ca-4141-9b22-ae1c6bd64d3e
/dev/sda6	none	swap	defaults	0 0

This is the interesting part of /boot/grub/grub.cfg

### BEGIN /etc/grub.d/10_archlinux ###
menuentry "Arch Linux pkg-linux kernel" --class arch-linux --class arch --class gnu-linux --class gnu --class os { if [ x$feature_all_video_module = xy ]; then insmod all_video fi set gfxpayload=keep insmod ext2 if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt5 --hint-efi=hd0,gpt5 --hint-baremetal=ahci0,gpt5 27af4583-6c81-460f-a027-c2c64379971b else search --no-floppy --fs-uuid --set=root 27af4583-6c81-460f-a027-c2c64379971b fi echo 'Loading Arch Linux pkg-linux kernel ...' linux /vmlinuz-linux root=UUID=d18f76b7-96cc-48e2-854a-c361f069c197 rw quiet rootflags=data=writeback echo 'Loading Arch Linux pkg-linux kernel initramfs ...' initrd /initramfs-linux.img
}
menuentry "Arch Linux pkg-linux kernel (fallback initramfs)" --class arch-linux --class arch --class gnu-linux --class gnu --class os { if [ x$feature_all_video_module = xy ]; then insmod all_video fi set gfxpayload=keep insmod ext2 if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt5 --hint-efi=hd0,gpt5 --hint-baremetal=ahci0,gpt5 27af4583-6c81-460f-a027-c2c64379971b else search --no-floppy --fs-uuid --set=root 27af4583-6c81-460f-a027-c2c64379971b fi echo 'Loading Arch Linux pkg-linux kernel ...' linux /vmlinuz-linux root=UUID=d18f76b7-96cc-48e2-854a-c361f069c197 rw quiet rootflags=data=writeback echo 'Loading Arch Linux pkg-linux kernel fallback initramfs ...' initrd /initramfs-linux-fallback.img
}
### END /etc/grub.d/10_archlinux ###
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-d18f76b7-96cc-48e2-854a-c361f069c197' {	load_video	set gfxpayload=keep	insmod gzio	insmod part_gpt	insmod ext2	set root='hd0,gpt5'	if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt5 --hint-efi=hd0,gpt5 --hint-baremetal=ahci0,gpt5 27af4583-6c81-460f-a027-c2c64379971b	else search --no-floppy --fs-uuid --set=root 27af4583-6c81-460f-a027-c2c64379971b	fi	echo	'Loading Linux linux ...'	linux	/vmlinuz-linux root=UUID=d18f76b7-96cc-48e2-854a-c361f069c197 rw quiet rootflags=data=writeback	echo	'Loading initial ramdisk ...'	initrd	/initramfs-linux.img
}
### END /etc/grub.d/10_linux ###

I already tried this, which is -the way I understand it- checking if the entries in fstab and grub.cfg are correct, compared to what `blkid` says.

Please help me!

Thanks a lot.

2020-05-04 20

levi

Moderator

From: Yorkshire, UK

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.

2020-05-04 10

arch322yes

Member

[solved] [FAILED] Failed to start Remount Root and Kernel File System

Hello, how can i solved this problem?

Machine: Virtualbox 6.1.6

Startup Message with Kernel and LTS-Kernel:

I can use my archlinux32 without problems.

Last edited by arch322yes (2020-05-06 14:51:32)

RFC

• RFC7530 (Network File System (NFS) Version 4 Protocol, 201503, PROPOSED STANDARD) — Updated by RFC7931
• RFC5661 (Network File System (NFS) Version 4 Minor Version 1 Protocol, 201001, PROPOSED STANDARD) — Updated by RFC8178
• RFC7862 (Network File System (NFS) Version 4 Minor Version 2 Protocol, 201001, PROPOSED STANDARD) — Updated by RFC8178 — which refers back to [RFC5661].

NFS4ERR_BADOWNER (Error Code 10039)

This error is returned when an owner or owner_group attribute value or the who field of an ACE within an ACL attribute value cannot be translated to a local representation.

The specifications discuss in Section 5.9. Interpreting owner and owner_group, I am not sure what to cite as relevant however.

NFS4ERR_SYMLINK (Error Code 10029)

The current filehandle designates a symbolic link when the current operation does not allow a symbolic link as the target.

NFS4ERR_NOENT (Error Code 2)

This indicates no such file or directory. The file system object referenced by the name specified does not exist.

The current filehandle is assumed to refer to a regular directory a named attribute directory. LOOKUPP assigns the filehandle for its parent directory to be the current filehandle. If there is no parent directory, an NFS4ERR_NOENT error must be returned. Therefore, NFS4ERR_NOENT will be returned by the server when the current filehandle is at the root or top of the server’s file tree.

The problem

I am attempting to set up a diskless node/workstation/system. The OS (4.13.12-1-ARCH) is installed on the SERVER /srv/archlinux. After a successful netboot from GRUB to NFSv4, systemd begins but fails at multiple stages, for example:

• Failed to mount Kernel Configuration File System.
• Failed to mount Kernel Debug File System.
• Failed to mount Huge Pages File System
• Failed to start Load/Save Random Seed.
• Failed to mount /tmp.
• Failed to start Rebuild Journal Catalog.
• Then ends with Not tainted 4.13.12-1-ARCH #1...

• Failed to mount POSIX Message Queue File System.
• Failed to start Remount Root and Kernel File System.
• Failed to mount Huge Pages File System.
• Failed to mount Kernel Debug File System.
• Failed to mount Kernel Configuration File System.
• Then ends with Not tainted 4.13.12-1-ARCH #1...

I suspect the failures are caused by an incorrect configuration of NFSv4 or the local network.

2020-05-07 07

arch322yes

Member

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

Hello levi, you mean line 535.

535 [ 3.972114] BTRFS info (device sda2): unrecognized mount option 'space_cache.subvol=@snapsthots/TESTING'

353 isn’t 535.

Last edited by arch322yes (2020-05-07 07:47:10)

Solution 1

Lets first check your file system for errors.

• boot to the GRUB menu
• choose Advanced Options
• choose Recovery mode
• choose Root access
• at the # prompt, type sudo fsck -f /
• repeat the fsck command if there were errors
• type reboot

• boot to a Ubuntu Live DVD/USB in “Try Ubuntu” mode
• open a terminal window by pressing Ctrl+Alt+T
• type sudo fdisk -l
• identify the /dev/sdXX device name for your «Linux Filesystem»
• type sudo fsck -f /dev/sda2, replacing sdXX with the number you found earlier
• repeat the fsck command if there were errors
• type reboot

CLIENT

/etc/hostname +(20171209) client2
/etc/hosts (exactly the same as the hosts file on the server)
/etc/resolveconf.conf name_servers=192.168.0.1
/etc/idmapd.conf (exactly the same as the idmapd.conf file on the server)
/etc/fstab # sys=sec or sys=none to correspond to server export settings. /dev/nfs / nfs rw,hard,rsize=9151,sec=sys,clientaddr=192.168.2.102 0 0 devtmpfs /dev devtmpfs defaults proc /proc proc defaults none /run tmpfs defaults sys /sys sysfs defaults run /run tmpfs defaults tmp /tmp tmpfs defaults

The fstab was defined by comparing the mounted directories on the server using findmnt -A.

2020-05-04 19

abaumann

Administrator

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

Thanks levi. There’s also a

which looks like BTRFS is missing somehow (or at least some options are not supported).

+(20171210) Is the remote directory on the server mounted to /new_root?

The mkinitcpio script uses the variable mount_handler to carry an assigned «mounting function», in this case nfs_mount_handler(), to which the «root path» is passed $1 at a later stage; /new_root.

showmount -a server -> All mount points on server: (empty)
ss -ntp | grep 2049 ->
ESTAB 0 0 192.168.2.101:2049 192.168.2.102:809 (random port)

Troubleshooting

• No firewall
• No Kerberos, LDAP, etc.
• No SELinux
• The user root exists on both SERVER and CLIENT, with the same password.

SERVER

All other relevant configuration files for NFSv4 I could identify on the SERVER.

/etc/nsswitch.conf passwd: compat mymachines systemd group: compat mymachines systemd shadow: compat publickey: files hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname networks: files protocols: files services: files ethers: files rpc: files netgroup: files
/etc/nfs.conf (all settings commented out)
/etc/conf.d/nfs-common.conf (all settings commented out)

network configuration

/etc/hosts 127.0.0.1 localhost.localdomain localhost ::1 ip6.localhost localhost 192.168.0.101 nd1.localdomain server servernd1 192.168.1.101 nd2.localdomain server servernd2 192.168.2.101 nd3.localdomain server servernd2 192.168.1.102 client1.localdomain client1 192.168.2.102 client2.localdomain client2
/etc/resolveconf.conf name_servers=192.168.0.1
# hostname -f
# nd1.localdomain
# hostname -i
192.168.0.101 192.168.1.101 192.168.2.101
# getent hosts IP -> the corresponding line in /etc/hosts
# getent ahosts HOSTNAME -> the corresponding line in /etc/hosts
# ping -c 3 server.localdomain -> 0% packet loss
# id -u root -> 0
# id -un 0 -> root
Display the system's effective NFSv4 domain name on stdout.
# nfsidmap -d -> localdomain
Display on stdout all keys currently in the keyring used to cache ID mapping results. These keys are visible only to the superuser.
# nfsidmap -l -> nfsidmap: '.id_resolver' keyring was not found.

+(20171209) Wireshark

Analyzing the Wireshark log, it is quite extensive but begins with something like:

[IP CLIENT] -> [IP SERVER] NFS 226 V4 Call ACCESS FH: [HEX VALUE], [Check: RD LU MD XT DL]
[IP SERVER] -> [IP CLIENT] NFS 238 V4 Reply (Call In 34) ACCESS, [Allowed: RD LU MD XT DL]
[IP CLIENT] -> [IP SERVER] NFS 246 V4 Call LOOKUP DH: [HEX VALUE]/archlinux

When the CLIENT requests /Id-linux-x86-64.so.2 the first errors start to appear:

[IP CLIENT] -> [IP SERVER] NFS 342 V4 Call OPEN DH: [HEX VALUE]/Id-linux-x86-64.so.2
[SERVER IP] -> [CLIENT IP] NFS 166 V4 Reply (Call In 124) OPEN Status: NFS4ERR_SYMLINK

The pattern more or less repeats itself with more frequent errors, for example, LOOKUP Status; and OPEN Status: reporting NFS4ERR_NOENT.

[SERVER IP] -> [CLIENT IP] NFS 182 V4 Reply (Call In 9562) SETATTR Status: NFS4ERR_BADOWNER

Net_nfs4

• +(20171210) NFS version on SERVER and CLIENT cat /proc/fs/nfsd/versions -> -2 +3 +4 +4.1 +4.2
• On SERVER and CLIENT cat /sys/module/nfsd/parameters/nfs4_disable_idmapping -> N.
• On SERVER echo "options nfsd nfs4_disable_idmapping=0" > /etc/modprobe.d/nfsd.conf.
• On CLIENT the /sys/module/nfs/parameters/nfs4_disable_idmapping does not exist, and not sure how to manually create it as the /sys is read only.
• +(20171210) On CLIENT echo "options nfs nfs4_disable_idmapping=0" > /etc/modprobe.d/nfs.conf.

The CLIENT IP is 192.168.2.102/24. The CLIENT network device is connected to SERVER nd2 192.168.2.101/24 (hostname: servernd2).

The network information during boot:

:: running early hook [udev]
starting version 235
:: running hook [udev]
:: Triggering uevents...
:: running hook [net_nfs4]
IP-Config: eth0 hardware address [CLIENT NETWORK DEVICE MAC] mtu 1500 DHCP
hostname client2 IP-Config: eth0 guessed broadcast address 192.168.2.255
IP-Config: eth0 complete (from 192.168.0.101): address: 192.168.2.102 broadcast: 192.168.2.255 netmask: 255.255.255.0 gateway: 192.168.2.101 dns0 : 192.168.0.1 dns1 : 0.0.0.0 host : client2 domain : localdomain rootserver: 192.168.0.101 rootpath: /srv/archlinux filename : /netboot/grub/i386-pc/core.0
NFS-Mount: 192.168.2.101:/archlinux
Waiting 10 seconds for device /dev/nfs ...
(systemd takes over from here)

+(20171210) NFS4, sec=sys and id mapper are incompatible?

Reading the doco, it looks like sec=sys and the id mapper can be used to correctly map uid/gid to name where the client and server have different mappings in /etc/passwd and /etc/group. This simply isn’t true.

That’s because with sec=sys the id mapper doesn’t come into play in the authentication part of the nfs protocol, only the file attributes part. With sec=sys authentication, nfs just passes the client uid/gid which is used directly by the server. So permissions checks will be screwed if client and server uid and gid don’t align. To confuse things further, when the client creates a new file it is the authentication credentials that are used, so the file gets created at the server with the client’s uid/gid. After that nfs uses idmap to get the file attributes, so the uid/gid (which originally came from the client) gets mapped at the server, and you end up seeing the server’s name for a client uid/gid. Borkage! On the other hand, if the file was originally created at the server, you will see the correct name at the client, even if the uid/gid differs. But permissions checking will still be broken. — kimmie — Posted: Wed Feb 20, 2013 3:14 am Post subject: — Emphasis in original

2020-05-07 19

levi

Moderator

From: Yorkshire, UK

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

I do indeed.  Might need to look into refreshing my typing fingers as well

Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.

2020-05-06 14

arch322yes

Member

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

Hello, i have solved the Problem, my /etc/fstab not good.

Thank’s for your help.

Last edited by arch322yes (2020-05-06 14:53:23)

Rpc. idmapd

/etc/idmapd.conf [General] Verbosity = 7 Pipefs-Directory = /var/lib/nfs/rpc_pipefs Domain = localdomain [Mapping] Nobody-User = nobody Nobody-Group = nobody [Translation] Method = nnswitch
/etc/exports
(printed using # exportfs -v) /srv <world>(rw,sync,wdelay,hide,no_subtree_check,fsid=0,sec=sys,no_root_squash,no_all_squash) /srv/archlinux <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,no_root_squash,no_all_squash)
(Exposed to "world" for debugging purposes)

rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling nsswitch->uid_to_name
rpc.idmapd: nfs4_uid_to_name: nsswitch->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"

If exportfs sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

+(20171209) After making sure that the /etc/hostname for the CLIENT was set to client2 (duh), if exportfs sec=none or sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling nsswitch->gid_to_name
rpc.idmapd: nfs4_gid_to_name: nsswitch->gid_to_name returned 0
rpc.idmapd: nfs4_gid_to_name: final return value is 0
rpc.idmapd: Server : (group) id "190" -> name "systemd-journal@localdomain"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

If I instead change method from nsswitch to static (UID mapping in NFS)

/etc/idmapd.conf ... [Translation] Method = static [Static] root@localdomain = root

rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: static_getpwnam: name 'root@localdomain' mapped to 'root'
rpc.idmapd: static_getpwnam: group 'root@localdomain' mapped to ' root'
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/static.so for method static
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling static->uid_to_name
rpc.idmapd: nfs4_uid_to_name: static->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"

If exportfs sec=sys, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

If exportfs sec=none, it continues like:

rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling static->gid_to_name
rpc.idmapd: nfs4_gid_to_name: static->gid_to_name returned -2
rpc.idmapd: nfs4_gid_to_name: final return value is -2
rpc.idmapd: Server : (group) id "190" -> name "nobody"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)

• NFSv4 User Mapping
• NFS user mapping
• Mapping UID and GID of local user to the mounted NFS share
• And many many more… Often related to a switch from NFSv3 to NFSv4, and rarely about netboot.

2020-05-04 18

levi

Moderator

From: Yorkshire, UK

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

For anyone looking into this after me, that error occurs in context on line 538 of that paste.  I note an informational warning from BTRFS immediately before that crash, but I have no experience using that FS yet personally, and it’s only an informational note so it may not actually be a real warning with any consequence.

Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.

Why the NSFv4 errors occur?

Server : (group) id "190" -> name "nobody"

1. You might not trust the root user of a client with root access to
   the server’s files.
2. The same username on client and server might have different
   numerical ID’s

+(20171209) rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'

According to Steve Dickson in a comment (2011-08-12 16:01:55 EDT) to a Red Hat Bugzilla – Bug 715430 report

nss_getpwnam: name '0' does not map into domain

On the Debian Mailing Lists, in an e-mail correspondence between Jonas Meurer and Christian Seiler (20150722) concerning «Kerberos-secured NFSv4» the error is explained in detail. My summary of the discussion:

When the NFS client sends nss_getpwnam: name '8' domain 'freesources.org': resulting localname '(null)'

Short answer: I have no idea.

If I understand the longer answer correctly, the problem could occur because the NFS client relies on the «kernel’s key cache». For the NFS server this should never be a problem because the «kernel’s key cache» is never used.

Since you are using just regular nsswitch via /etc/passwd, nss_getpwnam should never fail in your case, unless you do some weird stuff with /etc/passwd at the same time.

The answer also refers to an alternative method to idmapd; nfsidmap, although reading the man I cannot quite understand how it would replace idmapd.

+(20171209) nss_getpwnam: name 'root@domain.com' does not map into domain 'localdomain'

This error message does not seem to occur for me, I am however including the answer from SUSE’s support knowledgebase — 10-DEC-13 Modified Date: 12-OCT-17 — because of the description of cause, and the proposed remedy which stands in contrast to the other found discussions.

Chown failures or idmapd errors like the ones documented above are typically a result of either:

1. The username is known to the client but not known to the server, or
2. The idmapd domain name is set differently on the client than it is on the server.

The proposed remedy is to disable idmapd.

nfs.nfs4_disable_idmapping=1

2020-05-06 18

levi

Moderator

From: Yorkshire, UK

Re: [solved] [FAILED] Failed to start Remount Root and Kernel File System

Hah, that misspelling of snapshots was right there on line 353 of the log and repeated 5 times throughout, but I just didn’t see it.  Might need to upgrade vgrep

Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.

+(20171210) mount -t nfs4 [SERVER IP]

rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"
rpc.idmapd: Server : (group) id "99" -> name "nobody@localdomain"
... -> name "tty@localdomain"
... -> name "systemd-journal-upload@localdomain"
... -> name rpc@localdomain
... -> name systemd-journal@localdomain
... -> name utmp@localdomain

The result of genfstab was also different:

[SERVER IP]:/archlinux / nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,times=600,retrans=2,sec=sys,clientaddr=[CLIENT IP],local_lock=none,addr=[SERVER IP] 0 0

Nevertheless, after reboot systemd failed again with the same failures as described at the beginning of the post.

Solution 2

For me, this problem appeared after I installed a backup with wrong/old fstab UUIDs. I had to view the right UUIDs with sudo blkid and update my /etc/fstab file accordingly.

Related videos on Youtube

01 : 50

Manjaro Failed to start Remount Root and Kernel File Systems

08 : 53

How to Fix Kernel Panic Unable to Mount Root FS

07 : 27

Fix Arch Linux Boot with arch-chroot

18 : 02

Sửa lỗi File: \Boot\BCD trên Windows 7/8/10 theo 2 cách (100% THÀNH CÔNG)

02 : 25

Ubuntu: Boot Failed: Failed to start Remount Root and Kernel File Systems (2 Solutions!!)

08 : 53

How To Fix «Failed To Mount /Boot and Unknown Filsystem vfat» on Arch Linux!!!!!!!!

Comments

• I suddenly got my desktop machine to fail booting with no obvious reason. I haven’t made any significant changes/updates since the last shutdown. Here is what happens:

  Normal boot sequence throws me off to tty1. After successful login:

  /usr/lib/ubuntu-release-upgrader/release-upgrader-motd: 31: /usr/lib/ubuntu-release-upgrader/release-upgrader-motd: cannot create /usr/lib/ubuntu-release-upgrader/release-upgrader-available: Read-only file system
  /usr/lib/update-notifier/update-motd-fsck-at-reboot: 33: cannot create /usr/lib/update-notifier/update-motd-fsck-at-reboot: Read-only file system
  Welcome to Ubuntu 16.04.1 LTS! (<kernel_version>)
  0 packages can be updated
  0 updates are security updates
  *** /dev/sda2 should be checked for errors ***[email protected]:^$ _

  Note: Switching to tty7 doesn’t work. The screen blanks a little and then that’s it

  Going through recovery mode and running fsck gives:

  

  

  Any help is appreciated

• You can’t force a check while any partition is mounted. I unmounted / then forced a check. Turns out there were a couple of orphaned nodes need to be fixed and then boot resumed flawlessly. Thank you. On a side note: do you think that preventing fsck to do periodic checks might have caused the issue ?

• You don’t want to disable fsck checks. Cheers, Al

• I have the correct UUIDs in fstab, Still my system gives this error (It boots fine though)

• Thanks! This fixed my glitchy Raspberry Pi that was displaying the same error message on boot