WORKAROUND for machines that won't boot after the 0.8.0.0 upgrade

cdop

NOTE: this post has been edited to include @eric 's corrections below.

In case it helps others, here are the steps I had to take on my machine to get it to boot again after the 0.8.0.0 update.

Please read the entire thing + replies before attempting, and do this only as a last ditch effort before the inevitable reinstall, or on a clean but non-bootable install if that happens to be the case.

These steps assume some level of comfort with 66 and manipulating trees, services and configs using it. Depending on your configuration, following these steps may leave the system in a state that prevents 66 from running without errors.

The gist of it is: backup or snapshot all trees/configs/custom service frontend files, and rebuild every tree from scratch gradually, starting with boot.

Make sure ALL of the following apply to your situation:
- setting 66 VERBOSITY=4, adding break=y to/removing quiet from the kernel cmdline are not helping in finding the source of the issue;
- the machine is hanging after 66 starts to do its thing, i.e., grub/cryptsetup/lvm/filesystem/etc. problems have been ruled out (see above);
- waiting for a fix or continuing to debug on your own is not an option anymore;
- downgrading is not desirable for whatever reason.
- the next logical step would be reinstalling Obarun (or the system was just reinstalled but doesn't boot).
Boot a live Obarun iso and arch-chroot into the system's root partition.
Locate and backup all custom frontend files and modules to a safe location, or use the snapshot functionality.
Locate and backup all custom envs/configs for your services and modules to a safe location. Everyone has at least the boot@ module. Use something like 66 configure -L $SERVICE_OR_MODULE 2> $FILE. Check that $FILE has the contents you expect, or use the snapshot functionality.
Backup the output of 66 tree status -g to a safe location. It will help later when rebuilding the trees.
Double check the content of those backups, were going to delete the originals in the next steps.
Disable all trees except global, e.g. 66 tree disable root. Other trees can be re-enabled/worked on gradually from a correctly booting system later.
The global tree should be empty. 66 remove all services from it if its not. (Backups/snapshots, remember?)
Remove the boot@ module from from the boot tree. If there are other services or modules in the boot tree remove those too. E.g. 66 remove boot@system.
Force a clean config for boot@ : 66 parse -I boot@system.
Enable the boot@ module again in the boot tree. E.g. 66 enable boot@system (edited, see eric's explanation below)
Configure boot@ with 66 configure boot@system, referring to the backup or snapshot from step 4, then 66 reconfigure boot@system.
Smoke check the output of 66 tree status -g:
- global should be enabled;
- global should be empty;
- boot should be disabled (66 works like this, don't enable the boot tree);
- all other trees should also be disabled;
- boot should have only the boot@ module and it's dependents;
- all services in the boot tree should list state=Enabled;
- at least one tty-rc@ is enabled in the boot tree (this is needed for the rest of the job).

The machine should now boot correctly, allowing the other trees to be worked on. Good luck.

eric

Thanks to share your experience but some instructions are incorrect or misunderstood.
First you don't have to backup manually your system. When you trigger the update process a snapshot is taken. Also, you can create by yourself a snapshot before triggering the update. Just install the new 66 package and right after do (as root and user):

# 66 snapshot create my_backup
% 66 snapshot create my_backup

The 66 snapshot command do not trigger the upgrade process. So you can create any snapshot that you what. But again, triggering the update process will automatically create a snapshot.
You can retrieve the snapshot at /var/lib/66/.snapshot for root and $HOME/.66/.snapshotfor regular user.

cdop and do this only as a last ditch effort before the inevitable reinstall.

It not necessary to reinstall the system. This is only concern services. As long as the snapshot was taken, you can rollback the 66 ecosystem has it was before the upgrade doing 66 snapshot restore <nameOfTheSnap>. Obviously, 66 packages family need to be downgrade too.

cdop Enable the boot@ module again in the boot tree. E.g. 66 enable boot@system -t boot. Be sure to include the tree in this command

This command will not work, it should be 66 -t boot enable boot@system. Actually, you don't need to specify the tree to use for the boot@service because it use the filed InTree = ( boot ). So, even if you don't specify the tree to use the boot@ service will be automatically associated to the boot tree.

cdop Enable the boot@ module again in the boot tree. E.g. 66 enable boot@system -t boot. Be sure to include the tree in this command

Force a clean config for boot@ : 66 parse -f -I boot@system.

Actually this two steps need to be inverted because you have previously removed the boot@system. So, when you enable it, it will be automatically parsed (which is good), but it doesn't apply the -I option of the parse process with the enable command. So,

# 66 remove boot@system
# 66 tree remove boot
# 66 parse -I boot@system # no need to force as long as the service was removed previously
# 66 enable boot@system

I'm disappointed to see so many users with trouble to migrate the 66 ecosystem. I have made tremendous test before making the tag, including server on critical production on my companies (and others than mine) without any troubles.
I'm really sorry about that guys.

cdop

eric I'm disappointed to see so many users with trouble to migrate the 66 ecosystem.

Time to to take a 15 min break and re-read the list of notable systemd bugs . You'll feel better.

In my case there was a combination of factors: I needed the machine up quickly, I had not yet read the docs for the snapshot feature, one of my custom services broke the migration process, and after that some of the 66 commands would not run correctly anymore, but I also did not want to downgrade. I was not aware at the time that people were having trouble migrating clean installs too (I only read that later).

My thought process was: I don't have overly complicated trees, so it's definitely less work to recreate them than to reinstall the entire system. The instructions I posted yesterday were a post-mortem analysis of my shell history, so it's possible that they were incomplete or some of the steps were redundant.

I updated the relevant parts based on your reply.