vIDM node failed to boot after applying patch CSP-95247

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After installing VMware Identity Manager patch CSP-95247 found here, the VM does no longer boot and is stuck with "Error 15: File not found"
• This VMware Identity Manager node was not deployed in version 3.3.7 but has been upgraded from an earlier version 

Environment

Only upgraded VMware Identity Manager appliances are impacted which are using Grub version 0.97

Cause

The installed Grub version was not changed and the Installation is still using Grub version 0.97.

The patch updates the Grub configuration for version 2 but not earlier version and therefor the boot configuration is pointing to the old binaries.

Resolution

Option 1: To restore access:

1. To be able to boot the broken vIDM appliance please follow KB article 376435, Step 1-4.
2. Then edit the kernel package from /vmlinuz-4.19.269-3.ph3 into /vmlinuz-4.19.315-1.ph3 and confirm with Enter
3. To proceed to boot the appliance press b
4. Once the appliance is up & running, open a shell session to it.
5. Create a backup up of the file /boot/grub/menu.lst
6. Edit /boot/grub/menu.lst file using vim, hit letter "I" for Insert Mode
7. Update /vmlinuz-4.19.269-3.ph3 into /vmlinuz-4.19.315-1.ph3
8. Update /initrd.img-4.19.269-3.ph3 into /initrd.img-4.19.315-1.ph3
9. Exit Insert Mode with Escape button, then save the changes using ":wq"

Note:

• If you are planning to install the Patch CSP-95247 then please follow above steps 4-9 and only then install the patch.
• Check if grub2 boot file exists, using  ls -ltr /boot/grub*  on each IDM node. If so skip the install-ph-grub2

Option 2: To avoid the issue from happening again with further patches please upgrade the installed grub installation using the attached bundle:

1. Before proceeding please take a Backup of the vIDM environment.
2. Copy the bundle install-ph-grub2.tar.xz to the impacted vIDM appliance to either /tmp or /root
3. Decompress the archive using command:
   

   tar -xvf install-ph-grub2.tar.xz

4. Navigate to the new folder: cd install-ph-grub2 
5. Run the script as below
   

   ./migrate-to-photon-grub2.sh /dev/sda

6. Reboot the system, or if vIDM is deployed through Aria Suite Lifecycle Manager, perform a Power Off / Power On operation from within the LCM interface.

Additional Information

Here are two ways to verify if the appliance is using grub2.

• You can see it when you boot the VM. The Grub2 (2.0x) has a different layout versus the older Grub (0.97) version.

• ssh as root to the appliance.

  • ls -la /boot shows grub is a symbolic link to /boot/grub2 

    lrwxrwxrwx  1 root root    5 Jan 24  2023 grub -> grub2
drwxr-xr-x  5 root root 1.0K Jan 24  2023 grub2

  • /boot/grub2/grub.cfg contains references to grub2, for example.

    # Begin /boot/grub2/grub.cfg

    set default=0
set timeout=5
search -n -u XXXXXX-XXXX-XXXX-XXXX-XXXXXX -s
loadfont /grub2/ascii.pf2