Update to 12.3 prevents boot on Fujitsu Esprimo Q920

[asciinaut]

Describe the issue you are experiencing

Updated Home Assistant OS from 12.2 to 12.3.

After reboot system doesn't boot properly

Sometimes it boots into rescue mode automatically if no option is chosen manually after reboot. Sometimes it shows:

Booting "Slot B (recue shell)"

Trying to terminate EFI services again
error: couldn't retrieve memory map.

Failed to boot both default and fallback entries.

Choosing the slots manually neither of the slots (Slot A, Slot B nor the respective rescue slots) boot.

What operating system image do you use?

generic-x86-64 (Generic UEFI capable x86-64 systems)

What version of Home Assistant Operating System is installed?

inaccessible

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

1. Start the announced update from 12.2 to 12.3 in the Webinterface
2. wait for the reboot

Anything in the Supervisor logs that might be useful for us?

inaccessible

Anything in the Host logs that might be useful for us?

inaccessible

System information

inaccessible

Additional information

No response

[agners]

probably related to #3347 ?

Unlikely, as this is Home Assistant Yellow.

This is probably related to the GRUB revert in #3324.

What x86-64 machine are you running on?

Sometimes it boots into rescue mode automatically

What rescue mode exactly? Is ha os info working in that shell?

Probably your best way forward here is to replace the GRUB bootloader on the first partition of your boot disk (e.g. using a Ubuntu Live USB flash drive). You can find older version of the GRUB bootloader capable of booting HAOS in this comment #3305 (comment).

[asciinaut]

Thank you very much for your answer.

What x86-64 machine are you running on?

Fujitsu Esprimo Q920 - Intel Core i5 4590T

What rescue mode exactly? Is ha os info working in that shell?

This does not work in the rescue shell but I can exit the rescue shell which makes the supervisor start. In that case it is HAOS 12.2 so I assume I starts from Slot B rescue. From that point on I have access to the system and can provide additional information.

➜  ~ ha os info
board: generic-x86-64
boot: B
boot_slots:
  A:
    state: inactive
    status: bad
    version: "12.3"
  B:
    state: booted
    status: good
    version: "12.2"
data_disk: MicroFrom-256GB-SATA3-SSD-07042223E0108
update_available: true
version: "12.2"
version_latest: "12.3"

Start into this rescue mode is successfull in about 1 out of 10 boot attempts. Otherwise I only see the mentioned error.

[asciinaut]

Adding system information of 12.2 startet from Slot B rescue:

System Information

version	core-2024.5.2
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.12.2
os_name	Linux
os_version	6.6.25-haos
arch	x86_64
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.34.0
Stage	running
Available Repositories	1467
Downloaded Repositories	14
HACS Data	ok

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 12.2
update_channel	stable
supervisor_version	supervisor-2024.05.1
agent_version	1.6.0
docker_version	25.0.5
disk_total	234.0 GB
disk_used	13.5 GB
healthy	true
supported	true
board	generic-x86-64
supervisor_api	ok
version_api	ok
installed_addons	Z-Wave JS (0.5.0), File editor (5.8.0), Z-Wave JS UI (3.6.0), Network UPS Tools (0.13.0), Advanced SSH & Web Terminal (17.2.0), Piper (1.5.0), Whisper (2.0.0), openWakeWord (1.10.0), RaspberryMatic CCU (3.75.7.20240420), Assist Microphone (1.2.0), Mosquitto broker (6.4.0)

Dashboards

dashboards	8
resources	3
views	18
mode	storage

Recorder

oldest_recorder_run	1. Mai 2024 um 06:00
current_recorder_run	8. Mai 2024 um 20:45
estimated_db_size	278.83 MiB
database_engine	sqlite
database_version	3.44.2

[agners]

Hm, weird, so the new GRUB is able to boot HAOS 12.2, but not 12.3? 🤔 I wonder if boot slot A (or the kernel partition thereof) is somehow corrupted. When in boot slot B, can you just try to install HAOS 12.3 again? It should try to install it to the boot slot A again:

ha os update --version 12.3

[asciinaut]

➜  ~ ha os update --version 12.3
Processing... Done.

Command completed successfully.

After the reboot the problem persists.

I will transfer an image of the SSD to another identical esprimo and downgrade the production one to 12.2.

If I can reproduce the problem on the other esprimo, I will try older grub images as suggested in #3305 (comment) starting with the 32-bit EFI files and then 64-bit ones to check if this is the same behaviour.

[sairon]

@asciinaut Can you also check if you're using the latest BIOS (as it sometimes resolves some weird UEFI boot issues) and try booting a fresh 12.3 install e.g. from an USB thumb drive?

[asciinaut]

@sairon booting from a fresh 12.3 USB thunb drive has the same issues on two identical esprimo.

However both have a BIOS update pending. Will update one to see if the issue persists.

[asciinaut]

A BIOS update did not fix the problem. I'm waiting for the images to finish and then continue with the older GRUB images.

For reference the Q920 BIOS versions tested:

• R1.47.0
• R1.46.0

[kimzeuner]

Just wanted to report that i had exactly the same behaviour with my Q920. BIOS updates didnt work for me too. Only solution was to replace the GRUB Files via ubuntu. Unfortunately i have already replaced both an can not report if only changing one of them would help.

[Botschafter]

Just wanted to report that i had exactly the same behaviour with my Q920. BIOS updates didnt work for me too. Only solution was to replace the GRUB Files via ubuntu. Unfortunately i have already replaced both an can not report if only changing one of them would help.

Can you please describe how to replace the GRUB Files via ubuntu? I have the same Problems with my Q920 since Updating to HAOS.

[kimzeuner]

Sure. I downloaded the latest ubuntu version and flashed it on an usb stick with balena etcher. Put the stick into your esprimo and start it. Use F12 to open the boot menu and select the usb stick as boot device. After some time ubuntu will start.
Open the firefox browser and download the "old" GRUB files from here
Opened the terminal with a right click on the desktop and "open in terminal"
create a new folder with
sudo mkdir /mount01
Then mount the /EFI/BOOT folder to your created folder with
sudo mount /dev/sda1 /mount01
(maybe the /dev/sda1 ist different on your sytem but i think it will be the same as you also have an esprimo.)
Cooy the downloaded files with
sudo cp command
In my case it was something like
sudo cp /home/ubuntu/downloads/grub/xxx.efi /mount01
Repeat that for both files (you will not see a message that it was succesfull)

That way is also described here

Shutdown ubuntu and restart the esprimo. It will take some minutes till your ha is available again.

[Botschafter]

....

Shutdown ubuntu and restart the esprimo. It will take some minutes till your ha is available again.

Thank You Very much. Got Control Back again.

[asciinaut]

@agners I have tested the 12.1 grub files and can confirm that the 64-bit grub image fixed the problem. The 32-bit version have the same problem as before.

For that reason it think it is indeed related to #3305 (comment).

What irritates me, is that unlike described in the other issue, the grub loader from version 12.2 started without any problems. So it doesn't seem to be the exact same problem. @Botschafter, @kimzeuner can you confirm that the Grub from version 12.2 also booted smoothly on your Q920?

[kimzeuner]

Yes, i can confirm that. Im currently running 12.2 without any problems.

[xmancz]

I try it, but no works.

I don't know if I'm using the right /dev/sda...

How do I know which one is the right one?

[kimzeuner]

Im not an expert but as i have read in the other issue it should always be the first partition with 32M so /dev/sda1 should be the right one for you. I think in my system it looked similar to yours in the screenshot