Corrupt FS / Kernel AUFS / RO-Root

General discussion relating to the O2 Joggler, from the default O2 setup, to alternative operating systems and applications.
Post Reply
bobemoe
Posts: 4
Joined: Sun Sep 01, 2019 12:31 pm

Corrupt FS / Kernel AUFS / RO-Root

Post by bobemoe » Sun Sep 01, 2019 1:13 pm

Hi All

I've had 2 Jogglers for a while now, and developed some interesting software/OS's for them. Unfortunately after they have been running for a while (days) they end up failing to (re)boot, or suddenly drop to a read-only root filesystem. Looking at `dmesg` I see lots of bad blocks and errors about the root filesystem. (I will paste a copy of the errors next time it happens)

If I run `fsck` it will repair lots of problems, I then restore the corrupt files from a backup and all is good again for a few days.

I'm using ext3 mounted with noatime.

This happens with root fs on the internal mmc and also external usb sticks, I've tried cheap ones and official sandisk ones.

I've tried different OS's, initially modifying the OS from the projects at https://birdslikewires.net/squeezeplay-os and https://jwills.co.uk/projects/joggler-xubuntu/ and more recently using debootstrap to install my favourite OS Devuan (a systemd free Debian derivative) but keeping the kernels, from the above projects. I have never compiled my own kernel.

No matter what combination of OS/Kernel I use I still end up with a corrupt root FS after a day or two.

It seems to happen more frequently when writing to the disk, installing packages and such, however it has been known to happen while practically idle, though I'm sure the OS is writing especially during cron/maintenance activity.

The only thing I haven't really experimented with is using a FS other than ext3.

So this leads me to three possible paths:

1. Debug and fix the corruption issue. I have no idea where to start with this? Has anyone else experienced this? I can gather more debug info.

2. Try different filesystems. I'm not sure this will fix the issue and even if it does, I shouldn't have to stop using ext3? Maybe it'll help debugging?

3. Create a read only root FS. I like this idea of this and have started trying to modify the initrd.img from the above projects boot partition to use AUFS to do so, however I don't think the AUFS module is included, so it is not working. I see this viewtopic.php?f=2&t=477&p=7990 where a similar thing is attempted but I'm not sure the result. Should I post there? don't really want to necrobump it!

I'd really love to be using my Jogglers, so any help much appreciated :)

Thanks.

User avatar
pete
Posts: 2763
Joined: Mon Aug 01, 2011 6:33 am
Location: Time Traveler

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by pete » Sun Sep 01, 2019 1:56 pm

The eMMC is probably at the end life stage on the two Jogglers. I have this issue with a "few" older Jogglers. Watch the USB stick as it boots and you will see there is a time when the USB stick gets powered off while booting which over time will mess with the USB stick and eventually corrupt it unless you are using a powered USB hub and a USB SSD drive rather than a USB stick.

Over the years here have settled on using the Avaya boot ROM over the original O2 / Openpeak boot ROM. I have also added a realtek NIC rom to all of my O2 Jogglers, Openpeak 1 and 2 devices.

I have some 15 Openframe 1's which look much like the Joggler and have much different internals which have run fine now for more than 5 years (well close to 10 years) 24/7. These came to me as Verizon prototypes and they appear never to have been released.

These have a large metal heatsink over the entire motherboard and I am using the ZIF clip / ZIF cable to a 16Gb ZIF SSD card. These have no problems. The majority are running embedded Windows using Seabios. One has been modded to run Linux plus has an added USB hub attached to the back and an RTC with battery inside.

Here also have rewritten the BOOT partition making it double sized and using fat32 and it has worked fine.
- Pete

Auto mater

bobemoe
Posts: 4
Joined: Sun Sep 01, 2019 12:31 pm

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by bobemoe » Mon Sep 02, 2019 9:56 am

Hi, thanks for your reply.

Yeah I figured the internal flash was probably getting old, that's why I switched to using an external USB stick. I'd not noticed it getting powered off during boot, I will look out for this, sounds a bit crazy, how come that happens? If it happens while mounted I can see it causing all sorts of problems. Would just using a powered hub help, or does it need a SSD too?

I've not heard of the Avaya boot ROM before, what is the advantage of this, and where can I find one? Not much in the search engines. I see your other thread here about it, will it run on the early Jogglers too?

User avatar
pete
Posts: 2763
Joined: Mon Aug 01, 2011 6:33 am
Location: Time Traveler

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by pete » Mon Sep 02, 2019 6:21 pm

how come that happens?

Dunno. It has always happened and over time will or can trash the USB stick.

Initially here migrated to using an SSD drive, then an SSD stick but the issue with the disconnect always happens.

Many years ago switched my Dockstar Debian computers (used for automation) to USB Kingspec SSD drives and it worked for 24/7 operations.
The USB sticks would get corrupted over time.

Will try to low level format the eMMC drive on one of my 24/7 running XPe touchscreens....not sure if this will help.

hxxps://hddguru.com/software/HDD-LLF-Low-Level-Format-Tool/

I have used this tool for SD cards. Never on an eMMC but it should work.

Would just using a powered hub help, or does it need a SSD too?

Maybe. As mentioned before went to using a ZIF connected SSD drive many years ago except on Openpeak Verizon Prototypes.

Avaya boot ROM before, what is the advantage of this

There is a piece to the rom that enables SSD boots. (standard EFI boot rom just works with the eMMC and USB sticks).

The ROM is posted here in Andy's archives.
- Pete

Auto mater

User avatar
pete
Posts: 2763
Joined: Mon Aug 01, 2011 6:33 am
Location: Time Traveler

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by pete » Mon Sep 02, 2019 10:06 pm

Trying the low level format running in XP embedded on one of my touchscreens.

9/2/2019 4:04:39 PM * * * * * WARNING!!! * * * * *
9/2/2019 4:04:39 PM DURING THE LOW LEVEL FORMAT PROCESS
9/2/2019 4:04:39 PM ALL DATA WILL BE FULLY and IRREVERSIBLY ERASED!
9/2/2019 4:05:18 PM
9/2/2019 4:05:18 PM =================================================
9/2/2019 4:05:18 PM HDD Low Level Format Tool 4.40; http://hddguru.com
9/2/2019 4:05:18 PM Low level format: [1] Storage Device 0000 [1028.12 MB] (2008064 512-byte sectors)
9/2/2019 4:05:18 PM Device capacity: 2,008,064 sectors
9/2/2019 4:05:18 PM Locking device...
9/2/2019 4:05:18 PM Formatting...

I am curious if it shows errors.
- Pete

Auto mater

bobemoe
Posts: 4
Joined: Sun Sep 01, 2019 12:31 pm

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by bobemoe » Fri Sep 06, 2019 12:56 pm

Hmm, sounds like a lot of hassle if hubs and ssd's are getting involved it kind of reduces the Jogglers compactness a bit! I've got some old touchscreen laptops that I am thinking of just running my software on. Shame as I really liked its compact design, but I've not got the time to be tinkering too much at the moment.

So do you think the USB flash would still get corrupt if its mounted read only from the start?

How did your format go?

User avatar
pete
Posts: 2763
Joined: Mon Aug 01, 2011 6:33 am
Location: Time Traveler

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by pete » Sat Sep 07, 2019 7:42 am

So do you think the USB flash would still get corrupt if its mounted read only from the start?

I do not know. Best to give it a spin.

How did your format go?

The low level format completed. I cannot do a fat32 or NTFS format in embedded XP. Will have to switch the Joggler over to the EFI bios.

The Seabios'd XPe tabletop touchscreens have been running for years now 24/7.
- Pete

Auto mater

bobemoe
Posts: 4
Joined: Sun Sep 01, 2019 12:31 pm

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by bobemoe » Thu Sep 12, 2019 12:31 pm

Cool, so that brings me back to looking for a kernel with AUFS support. Any ideas on this? You mention a andys archives section to this site, sounds interesting, can't find it though, got link? Thanks

User avatar
pete
Posts: 2763
Joined: Mon Aug 01, 2011 6:33 am
Location: Time Traveler

Re: Corrupt FS / Kernel AUFS / RO-Root

Post by pete » Thu Sep 12, 2019 9:01 pm

Yes thinking the original Joggler Ubuntu build using 14.X had AUFS support. That said I have no clue today with Ubuntu 19.X. The first RPi's also had AUFS support.

It would be Buzz's first Ubuntu Links..over here...

dubdubdub.jogglerwiki.com/wiki/Linux_Distributions

I did not think there was still support for AUFS.

Back in the 1990's recall updating my Palm memory with a service in California.

You could maybe find a service to update the eMMC these days over to 8, 16 or 32 Gb and even the RAM over to 1Gb or 2Gb.

There are some You Tube videos on how to do this with a hot air gun. You have to be very patient though.
- Pete

Auto mater

Post Reply