Please help me find what stupid thing I've done now. I run a custom kernel to get the microcode for Ryzen 5, Radeon RX5600. It has worked great for almost 3 years, however, after a recent -Syu, I could not boot my new custom kernel, the standard kernel, nor my old kernel. They all do the same thing. The last line in the boot sequence says
"mount-fstab: info: starts..."
and the system hangs indefinitely. No error messages are visible on the screen. No prompt and the only thing I seem to be able to do is Ctl-Alt-Del. There is no DM involved as I boot to a CLI prompt.

I booted from a USB and set up a chroot with /mnt being identical to my normal / as far as disk mounts go, but even if I run dmesg within the chroot (by that I mean after chroot /mnt /bin/bash), it gives me the USB boot info. Is there a way to see the boot messages for the last boot attempt within the chroot? Or is there some boot option I can set to give me more info?
you should be able to access the logs via <the_installation_location>/var/logs directly even without a chroot..
The only one with the timestamp of the attempted boot is wtmp which is a binary file.
I put loglevel=7 and debug as kernel parameters. That added 2 more lines that simply mentioned mounting the last 2 partitions (both data). Been working on this for 3 days and I'm stumped. I suppose my best bet at this point is to wipe and reinstall?
utmpdump --help

Usage:
 utmpdump [options] [filename]

Dump UTMP and WTMP files in raw format.

Options:
 -f, --follow         output appended data as the file grows
 -r, --reverse        write back dumped data into utmp file
 -o, --output <file>  write to file instead of standard output
 -h, --help           display this help
 -V, --version        display version

For more details see utmpdump(1).
might help with that
although looking at mine doesn't show much beyond logins on tty
but yeah, logs are only useful for stuff that happens after logging begins.. if a failure happens before that .. ??
another thing might be to check /etc/fstab to make sure it's in order..
Thanks for the reply. Yeah, nothing of interest in wtmp. fstab hasn't changed since April 2020, and everything looks in order. I can mount everything and fsck is clean for them all.
Ctrl-Shft-F!2 should give you user access (not root)
If the error is in fstab 
% sudo mount -a
should give you an error message, you say through chroot it doesn't.
your system freaking out in such early stage may be read only 
% sudo mount -o remount,rw /dev/sdxxx /
edit your /etc/66/init.conf --> verbocity=4
reboot, go to tty12, scan /run/66/log/0/current for {unable fail error} 
grep -i unable /run/66/log/0/current
usually the first error message is the culprit.

Also, is your bootloader or kernel name linux linux-lts linux-hardened of the image changed?
Maybe use mkinitcpio to remake the image. Do you also have a headers part of the kernel?

Interesting
Thanks! "mount -a" gave no errors. "/" was already rw, verbocity was already 4, so I skipped to the grep.
"if: fatal: unable to spawn /usr/lib/66/script/modules.sh: No such file or directory"
"s6-rc: warning: unable to start service modules-system: command exited 111"
and one more that I'm too lazy to type but probably related to the above.

I put a soft link to /usr/bin/modules.sh in 66/script which kicked the can a little further down the road (that may be an American idiom) to a missing "tmpfiles.sh".
There is no tmpfiles.sh on my system so i don't know where that should have come from.
Has the pkg modules been uninstalled by any chance?
Actually the only way you can uninstall it is if boot@ -66serv is uninstalled, because both modules and opentmpfiles are dependencies of it.
mount the failing system to /mnt
sudo pacman -S modules opentmpfiles -r /mnt

Only way to have boot@ -66serv and not have opentmpfiles and modules is if you remove them by "pacman -Rdd"|
Reinstalled all those (said they were up-to-date) and ran the '66-enable -t boot -F boot@ system' as prompted. Still no tmpfiles.sh on my system.

However, on each of the installs I got a:
"Refreshing PackageKit
"Error connecting: Could not connect: No such file or directory"

In pacman.log it says:
"running '90-packagekit-refresh.hook'
and the same connector error as above.

Searching through pacman.log, nothing of significance has been removed. I removed xiphos as I always have to when icu is updated and some cruft (electrum, elixir, allegro) which I no longer use.
I'd wait for Eric to deconstruct this puzzle

Maybe his syncing of the mirror/repository run into problems while uploading new packages, and new/old packages for 66 were mixed. This sounds like behavior from the days he split and made separate opentmpfiles and modules but since 66 has been revised a couple of times.

Very strange
From the v2.4.0 version, the local-tmpfiles call the opentmpfiles.sh (not the tmpfiles.sh) script and the modules-system service call the modules.sh package.https://git.obarun.org/obmods/boot-66serv/-/commit/0f1aa679d7871ad43f39865ae18f4e4a0c4b7d4a

So, the opentmpfiles.sh and modules.sh should be installed at /usr/bin. Now, your /usr/lib/66/service/boot@ should be show you 
[main]
@ type = module
@ version = 2.4.1
@ description = "Set of services to boot a machine"
@ user = (root)

[regex]
@ infiles = (
::@ bootconf@ =/etc/66/conf/boot@ @I/version
::@ scripts@ =/usr/lib/66/script
::@ initconf@ =/etc/66/init.conf
::@ skeldir@ =/etc/66
::@ live_dir@ =/run/66
::@ tmpfiles@ =/usr/bin 
::@ modules@ =/usr/bin
::@ vers@ =2.4.1
)
note the paths for the tmpfiles and modules script. Be carefull to not using a personnal boot@ service define at /etc/66/service. In that case change the path of the opentmpfiles.sh and modules.sh at /etc/66/service/boot@ path declaration and enable again the service.
To know that you can do 
# 66-inservice -t boot -o source boot@ <whatever>

Powered by Obarun