Inconsistent VM boot behavior while creating instance from Identical Image

Products

• VMware Integrated OpenStack

Issue/Introduction

• Experiencing an issue related to VM image registration and VM boot behavior.
• When attempted to register the same VMDK image file, twice using the "openstack image create" command.  Although both registrations completed without any errors and the debug logs showed no significant differences,  the resulting behavior was inconsistent.
• The command was used with following parameters.

#openstack image create --disk-format vmdk --file xxx-image-xxxxxx-xxx-xxx-001-1.vmdk --private \

   --property vmware_adaptertype=paraVirtual \
   --property vmware_disktype=streamOptimized \
   --property vmware_template_disk_type=thin \
   --property hw_firmware_type=uefi \
   --property os_secure_boot=required \
   --property hw_machine_type=q35 \
   --property vmware_ostype=rhel8_64Guest \
   --property vmware_create_template=true \
   --property vmdk_skip_conversion=false \
   xxx-image-xxxxxx-xxx-xxxx-001_prod-debug2 --debug

 

• First registration (xxx-image-xxxxxx-xxxx-xxxx-001_prod-debug2, ID: xxxxxxx-xxxx-xxx-xxxx-xxxxxxxx) → All five instances created from this image failed to boot the OS.

#openstack image create --disk-format vmdk --file /mnt/nfj/xxxx/images/xxx-image-xxxxx-xxx-xxxx-001/20251031/xxx-image-xxxxxx-xxx-xxx-001-1.vmdk --private --property vmware_adaptertype=paraVirtual --property vmware_disktype=streamOptimized --property vmware_template_disk_type=thin --property hw_firmware_type=uefi --property os_secure_boot=required --property hw_machine_type=q35 --property vmware_ostype=xxxx_64Guest --property vmware_create_template=true --property vmdk_skip_conversion=false xxx-image-xxxxxx-xxx-xxx-001_prod-debug2 --debug

 

• Second registration (xxx-image-xxxxxxx-xxxx-xxxx-001_prod-debug3, ID: xxxxxxx-xxxx-xxx-xxxx-xxxxxxxx)) → All five instances created from the same image booted successfully.

# openstack image create --disk-format vmdk --file /mnt/nfj/xxxx/images/xxx-image-xxxxxxx-xxxx-xxxx-001/20251031/xxx-image-xxxxxxx-xxxx-xxxx-001-1.vmdk --private --property vmware_adaptertype=paraVirtual --property vmware_disktype=streamOptimized --property vmware_template_disk_type=thin --property hw_firmware_type=uefi --property os_secure_boot=required --property hw_machine_type=q35 --property vmware_ostype=rhel8_64Guest --property vmware_create_template=true --property vmdk_skip_conversion=false xxx-image-xxxxxxx-xxxx-xxxx-001_prod-debug3 --debug

Environment

 

• VMware Integrated Openstack  7.3

Cause

• The openstack image create command may sometimes result in a incomplete image registration, even though the command and debug output appear successful.
• In the failed case, instances created from the image show symptoms of partial OS corruption, as if the latter portion of the image is missing or unreadable.
• Prime suspect is that, the image that failed to boot has an incomplete flat vmdk created and the failed image would have the same content of the good ones in the beginning, but missing the part close to the end or having something different close to the end.
• There are no known issues of corrupting image upload however there were instances where issues was observed in cinder-backup during restore which uses a shared component (oslo.vmware) to upload vmdk to vSphere using NFC.

Resolution

• Validate and compare the flat vmdk files
• Check the corresponding glance image (a VM template) on vCenter under Monitor --> Task and Events --> Events for any error or warning eventts.
• If there are no warnings or error and if the vmdk flat files are intact and no issues observed, please use the following attached script.

• Please find attached one-off patch for VIO 7.3 :vio-patch-7.3.0-pr3298866.tgz

1. Copy file vio-patch-7.3.0-pr3298866.tgz to vio-manager
2. Add the patch

#viocli oneoffpatch add -l vio-patch-7.3.0-pr3298866.tgz

3.Execute command

#viocli oneoffpatch install -p vio-patch-7.3.0-pr3298866"

Note : If need to uninstall the patch, execute the command "viocli oneoffpatch uninstall -p vio-patch-7.3.0-pr3298866"

 

•  If the error is still reproducible after installing the one-off patch, before uninstalling the one-off patch, please create a VIO support bundle and collect rw_handles.py generated by:

#osctl exec -ti deploy/glance-api -- cat /usr/lib/python3.7/site-packages/oslo_vmware/rw_handles.py > rw_handles.py

before uninstalling the one-off patch.