Root file system read-only

I’ve lately been having an issue with my root filesystem becoming readonly. It happens some amount of time after boot. I don’t know exactly when it happens, as I don’t usually notice it until something such as suspending the computer or printing fails. It seems to be fairly random. Since most of my system is on that partition, I can’t re-mount it without rebooting.

After this happens, the system runs a fsck. Sometimes it prompts to fix problems; other times it apparently finds none.

I realize that the disk isn’t in the best of conditions, and I have complete backups in case of catastrophic failure. Yet the lack of errors in the logs, combined with SpinRite’s test results and the unchanged SMART data makes me think that this problem has some cause other than disk failure.

Other than disk failure, what could cause my symptoms?

My freshly installed Ubuntu Bionic 18.04 on a Dell Inspiron 15 laptop has a weird problem. After resuming from suspend mode, sometimes the root file system is mounted read-only. This happens more often (but not always) when I plug in or unplug the power supply when the system is suspended.

The above is the line in the output of the mount command pertaining to the root file system. The rw at the start of this line changes to ro once the problem occurs.

I’m fairly sure that the errors=remount-ro parameter is relevant here, but I’m not in favor of simply removing it and thereby suppress the symptoms without addressing whatever the cause of the problem is.

I am not sure how to narrow down the possible causes of this strange behavior. There’s nothing in /var/log/* because that’s on the read-only file system (or rather on the root fs which has gone ro when the problem occurs) which makes it impossible to log any relevant output.

Smartctl gives the harddisk a clean bill of health. Which stands to reason, because this is a brand new laptop. The output of upower —monitor shows nothing out of the ordinary when I unplug and replug the power supply.

What can I do to find out what’s going on here?

I am using Gentoo Linux and for a while now, the root file system is mounted read-only on booting. For obvious reasons, this is quite annoying as most services do not start up correctly (I do not use a separate file system for /var). After the system is up, I have to log in, remount the root file system read-write, fix /etc/mtab, mount all other file systems in from /etc/fstab and then start up all the missing daemons. I know that there are ways to make a system run properly with a read-only file system, but I would rather restore the old behaviour of a writable root file system.

The strange thing is that after running mount / -o remount,rw, the file system is mounted in writable mode without any errors. I suspected some problem with fsck, but now I have disabled automatic file system checks on the partition (tune2fs -c0 -i0).

When I run dmesg, only these lines mention the partition at all, although I am not sure if not something gets lost because /var/log is not writable:

The line in /etc/fstab looks like this:

/dev/sda5 / ext3 noatime 0 1

I am using the kernel 2.6.34-gentoo-r6 (the same problem existed with a previous 2.6.31 kernel). I have created it using genkernel 3.4.10.906. My grub configuration looks like this:

title=Gentoo Linux (2.6.34-gentoo-r6)
root (hd0,0)
kernel /kernel-genkernel-x86_64-2.6.34-gentoo-r6 root=/dev/ram0 real_root=/dev/sda5 vga=792 CONSOLE=/dev/tty1 resume=/dev/sda6
initrd /initramfs-genkernel-x86_64-2.6.34-gentoo-r6

Apart from that, I run baselayout 2.0.0 with openrc 0.6.3, if that is any important. sysvinit 2.87-r3 is also installed, I don’t know if it’s actually used.

Here is the output of dumpe2fs:

I have no idea what could cause this problem. I cannot find any error messages, and searching the internet, I only find manuals how to deliberately mount the root filesystem read-

My web server (Linux, Debian Wheezy) has suddenly gone read-only, MySQL have crashed (but Apache didn’t) — giving errors when pressing tab to complete file name as:

bash: cannot create temp file for here-document: Read-only file system

Also, when I restart Apache it won’t turn on anymore.

What I tried to do : I’ve replaced the hard disk , moved web files again ; error repeated after 1 week.

Logs shows nothing, one notice that I’ve seen many failed attempts to access the server through ssh.

mount -n output:

Hi,
I have Proxmox installed and it seems to be every 6 or 7 days the file system goes readonly and i cant work out why.

As the system is readonly i ran an fsck on pve-root which detected issues:-

As you can see lots or errors on the root volume.

However, if i run as smartctl on my drive it passes with no issues:-

So no disk appears to be full here, at least not after a reboot.

Can anyone help me out here — i would suspect a failing drive if not for the smart tests failing and the consistant working for a week before it fails making me thing something is triggering this.

Last edited: Jan 30, 2023

An IO error is a very good indicator that an actual IO error happened. Absent any special software shims, it usually points to a disk.
A SMART error is also a good indicator of some sort of disk problem, however its absence is not an indication of fully healthy disk.

Дополнительно:  Windows остановила это устройство код 43 — как исправить ошибку | remontka.pro

If you grep for «Discard» in your logs, you will probably find:

What could be running this service? Its not in crontab.
If you google the other messages it should lead you to fstrim. Lets check services for its presence:

So based on the presented facts we know its the fstrim operation that is being run once a week.
You have not mentioned make and model of your disk. Many cheap and obscure consumer disks are not meant to be run 24×7 with server type load.

An IO error is a very good indicator that an actual IO error happened. Absent any special software shims, it usually points to a disk. A SMART error is also a good indicator of some sort of disk problem, however its absence is not an indication of fully healthy disk.

Hi and many thanks for the response. The drive is a KINGSTON SUV500240G but this Proxmox is only used for PfSense, Pihole and a Unifi controller. Its not exaxtly working its ass off.

Given the evidence so far would you agree a replacement for the drive is the best start here?

Its not about quantity, its about quality. People dont run fstrim jobs on their windows laptops.

I would probably look first whether there are other reports about fstrim interaction with this disk. Run fstrim manually to confirm, check the firmware, run vendor testing tool if it exists. But I would also weigh that against the ease/cost of replacing the disk.

So that seemed to work — at least at the moment.

I did find a post which suggested that fstrim wasnt supported on Kingston, but it was from 2016 IIRC, maybe 2018. I also found another post suggesting a number of ways to check here — https://unix.stackexchange.com/questions/584549/how-do-i-check-if-my-ssd-supports-fstrim

However, my one does seem to support it, note the DISC-GRAN and DISC-MAX being higher than zero which is an indicator —

And in fact i found a spec sheet stating that it does at the firmware level that im running specifically — https://smarthdd.com/database/KINGSTON-SUV500240G/003056RI/

Not sure what to do here, im tempted just to get another drive at £30 and be done with it, hopefully, (Samsung 870 Evo or Crucial MX500)

Final update — New SSD installed and working. When i took the old one out and attatched it to a Laptop to clone it even failed cloning due to an IO error so it was the drive at fault. So much for SMART though and thumbsup to me for having backups of my vms and containers!

Please mark the thread as solved.

First post: For the last three days I’ve been going through the arch wiki, learning new things and installing arch for the first time.

Almost done with my installation, I tried installing intel-ucode, since I have an intel CPU.

error: failed to init transaction (unable to lock database)
error: could not lock database: Read-only filesystem

I quickly realised that my root partition file system had become read-only.

# mkdir dir_name
mkdir: cannot create directory dir_name: Read-only file system

Which is weird, since I’ve been running around, creating and viming files left and right for the last few hours.

What is more lsblk and fstab alike will assure me my root partition is not read only:

#lsblk
name maj:min rm size ro type mountpoint
nvme0n1p1 259:1 0 220G 0 part /
nvme0n1p3 259:3 0 600M 0 part /boot

The second is my efi partition, which I mounted to /boot (it was previously on /efi)

#cat /etc/fstab
UUID=uuid_here / ext4 rw,relatime 0 1
UUID=uuid_here /efi vfat rw,relatime,fmask=0022,dmas=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 2

Note that fstab has not yet been updated and still thinks the efi partition is mounted on /efi. Of course,I can’t update it if I can’t write on the filesystem, but I don’t think that has anything to do with the problem.

What do you make of this? What could have caused this?

P.S. Commands were executed from arch-chroot /mnt. I’m still booting from my installation USB. For that reason I have been hesitant to reboot before fully completing the installation.That said, the USB’s environment is not read-only, so there’s that.

Last edited by silberblume (2020-12-27 23:02:43)

There’s nothing scarier, nothing more deadly than an ambulance

File system is read only

Edit: anyone looking for information on how to use FSCK, this question quickly changed focus. This ended up being a FSTAB issue. Still worth a read if you’re having issues with your file system being read only.

My boot partition is locked on read only. Ik trying to force a fsck on it, but all the advice online seems outdated. anyone know how to do this?

Last edited by GarraTheWinnie on Sat Nov 05, 2022 8:21 pm, edited 3 times in total.

Forcing fsck

Mon Oct 31, 2022 12:56 am

kerry_s wrote: ↑

Mon Oct 31, 2022 12:41 am

boot partition on what?
if sd, it usually indicates failed sd, which locks it read only.

It’s a micro sd. Typing the code below unlocks it until I reboot. So I don’t think its a hardware issue

Mon Oct 31, 2022 1:24 am

GarraTheWinnie wrote: ↑

It shouldn’t. That will remount the root partition as rw but as it is already mounted that way it does nothing and won’t affect the boot partition.

will remount the boot partition rw but again it’s mounted that way by default.

Дополнительно:  wcifs.sys Windows process - What is it?

Usually if the (u)SD card has gone bad and switched to read only mode the OS will still think it has mounted rw and appear to allow writes but those writes are lost at shutdown.

Lastly, I know it’s inaccurate but around here micro SD and SD are used interchangably.

Knowledge, skills, & experience have value. If you expect to profit from someone’s you should expect to pay for them.

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

Mon Oct 31, 2022 1:58 am

thagrol wrote: ↑

The added «/boot/ just shoots out a error. don’t do anything.

I haven’t written to the /boot. until you showed me that command, I never even tried.

I really want to get fsck to run

Mon Oct 31, 2022 4:24 am

it’s already set to do that if somethings wrong.
if the sd is locked because of failer it’s not going to help.

2022-10-30-182210_2560x720_scrot.png (117.23 KiB) Viewed 2347 times

Mon Oct 31, 2022 10:11 am

If the Miocrosd card is the problem, and I If I img it, then restore it to a new card ,will that in theory fix my problem, or will it transfer the problem to my new card?

Mon Oct 31, 2022 12:02 pm

You sem to be confusing your partitions. boot (mounted at /boot) is the first (FAT) partition on the SD card. The root partition is the second (ext4) partition an is mounted to /.

Impossible to say. It very much depends on the nature of the damage and the method used to copy the card. Even if the copy succeeds you may still end up with corrupted data.

What I’d do is this:

Mon Oct 31, 2022 11:48 pm

When I say my boot, I really mean my main partition where all my files are.

So my /home/ directory.

It’s completely locked unless I put in that command.

I’ll give those things a try. I have another pi thah works fine, so I’ll use FSCK on the micro-as in question, using that pi setup

Tue Nov 01, 2022 12:20 am

I’m now very confused. As I suspect are you.

A directory is not equal to a partition.

Partitions get mounted on directories.

On the default RPiOS /home is a directory on the root partition. However that may not be the case on your system as I don’t know what OS you’re using or how it has been setup.

I’m unsure what further help anyone can offer without more information from you and some effort on your part. Not least of which is you learning the difference between partitions and directories, the difference between the boot and root partitions, how Linux files system owner, group, and permissions work, and how to find the error message (if any) thrown during boot.

Tue Nov 01, 2022 1:32 am

I am likely confused, but I do know the difference between a directory and a partition. I simply didn’t explain my thought well.

What I meant is: the partition that contains the /home/ directory is the one that is write protected.

I always assumed it was the boot partition, because I
Grew up using dos, and generally the partition you start on, is the same the system booted from. Obviously that isn’t the case here.

My os is Raspian Lite.not sure what version. But I installed it within the last year or so.

I use this pi without any screen hooked up to it. I make all my changes via a SSH Connection.

I’ll type in that command and get back with what it says, as well as the contents of that file.

If this makes any difference, the last thing I did before the issue occurred; was empty a MySQL table.

Also, I noticed a day or so the drive was showing as full, which was bizarre because I don’t keep a lot on it. That was actually one of the reasons I was checking up on it

Tue Nov 01, 2022 9:23 pm

So I was able to get that information

1. Whenever I try to make any kind of change, I get the error: «Read-only file system», until I do the command «»sudo mount -o remount,rw /» (attacked an image)
2. as mentioned above, I don’t have a screen attached it, but I checked » /var/log/syslog» and there is no error
3.

Code: Select all

/dev/mmcblk0p2 on / type ext4 (ro,relatime)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

screen shot Screen Shot 2022-11-01 at 5.18.17 PM.png (122.66 KiB) Viewed 2146 times

Tue Nov 01, 2022 10:08 pm

your not understanding, the remount is done in ram, not on the sd, it may look like you made it writable, but it’ll be gone on reboot.

can you post

to see the use%

Screenshot from 2022-11-01 12-07-59.png (52.73 KiB) Viewed 2143 times

Tue Nov 01, 2022 11:01 pm

That suggests the problem isn’t with disc damage or corruption or with fsck.

So your root partition has been mounted read only. The question is why? I still don’t have enough information to do more than guess. And guessing wastes both our time.

Is that your entire fstab? I’m also unsure where «Binary file /etc/fstab matches» has come from and what help you think including it will be.

/etc/fstab should not be a binary file. It should be a text file. The default one should look similar to this:

Дополнительно:  Экран цветов смерти в Windows 10 - Инструменты ошибок

Have you edited your fstab from the default?

Have you enabled the read only root overlay via raspi-config?

Have you tried running fsck in the way I suggested earlier? fsck will not run on a mounted partition.

As for you lack of space (and this could be related to the other problem) are you using a USB drive for additional storage?

I’m afraid that right now there is little more that I can add and given the obvious communication problems (or problems in terminology) I’m going to have to step back from this thread.

Tue Nov 01, 2022 11:28 pm

I posted a photo of the result of fs -l

I never touched the fstab. I have no idea why its a binary. I didn’t add that line, it was there when I got there. that was the entire contents

I don’t know how to enabled the read only root overlay via raspi-config? I wasn’t in it either, so I don’t think I enabled it by accident.

I haven’t done it yet. I needed to make an image of the sd card, and that took me a while to be in a position to do it

No, I don’t need external storage for this PI, because it only has one job, and the application isn’t big. I’ve stored everything in /home/pi

We’ll I appreciate everything you have done so far. Thanks anyway

Screen Shot 2022-11-01 at 7.19.45 PM.png (126.8 KiB) Viewed 2102 times

Wed Nov 02, 2022 5:18 am

Wed Nov 02, 2022 2:07 am

yeah, that sd card is toast.

I see nothing to suggest dead hardware.

There is no entry for / in /etc/fstab. The boot scripts will therefore neglect to mount it rw, and the kernel default is ro as set in cmdline.txt. This will also produce errors during boot, and would normally prevent the GUI from starting, but as OP is running a Lite image headless, they will only notice that they have to manually remount / each time.

The damage to /etc/fstab cannot be explained by a hardware issue either, as hardware would not know to put the name of the file it was corrupting inside the file itself. Admittedly hardware could first put binary in the file and then software further corrupt it, but equally software could do both. Google has quite a few reports of «Binary file /etc/fstab matches» inside the file, but I do not spot the common factor. The file is not one that should be commonly rebuilt in normal use.

I would correct /etc/fstab to just the essential lines, with the true device names because using the wrong PARTUUID will not work and risks breaking headless boot:

/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 noatime 0 1

I do agree to copy off any important files first, not because I think the card is failing, but because of the precarious headless boot while trying to fix boot issues.

Wed Nov 02, 2022 10:51 am

jojopi wrote: ↑

I’ll start with that. How do I change it? At the moment everything is read-only unless I mount/unmount the partition. When I reboot, the changes I made are lost.

Would putting the micro-sd into a different machine work?

Wed Nov 02, 2022 11:40 am

That is a classic symptom of a End of Life card that has gone permanently Read Only.

You can remount the FS read/write, but any changes are held in the buffers and the card ignores them.

Thu Nov 03, 2022 1:11 am

So I was able to edit the fstab file. I chanced it to the code shown above. I’d did it by using another working computer. This seemed to fix the problem. I need to do some more test to see if it’s all working.

Assuming this was the problem, anyone know how this could have happened, and how I could have prevented it?

Last edited by GarraTheWinnie on Thu Nov 03, 2022 1:39 am, edited 1 time in total.

Thu Nov 03, 2022 1:30 am

when the sd starts to die you get more and more corruption. there’s no prevention, just don’t ignore the signs.
you can google to understand sd failure.

Do we know it’s sd-failure? Is it possible the fstab file was corrupt from the pi being shut off incorrectly, or something else?

Thu Nov 03, 2022 3:09 am

It could not have been just card failure, or just unsafe shutdown. The damaged file had its own name mentioned within itself (in the form of a message from the grep command). That must have involved some faulty high-level script.

If your system is working again you can install and run debsums to verify the checkusms of system files:

sudo apt update
sudo apt install debsums
sudo debsums -c

On a good system, the last command should take a few minutes but produce no output.

Thu Nov 03, 2022 7:25 pm

So I did this, and it spit out a reference to the library used for my hardware clock. This Pi has a Real Time Clock, I’m pretty sure that’s the library that controls it

I’m not really sure what to make of that.

I reinstalled the package and re-ran debsums, and it didn’t show up again. So does this mean the card is probably fine?

0DD50A9A-C030-4684-8DF0-C5E46AF9C43E.jpeg (18.48 KiB) Viewed 1828 times

Return to “Beginners”

Оцените статью
Master Hi-technology
Добавить комментарий