Image extra_config has invalid format error when resizing an instance

Products

• VMware Integrated OpenStack

Issue/Introduction

• When resizing an instance you are getting this error:
  Error: Failed to perform requested operation on instance "example-01", the instance has an error status: Please try again later [Error: Invalid input received: Image extra_config has invalid format].
• In nova-compute log you will see a Traceback with messages like the following for the instance in question:
  ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.7/site-packages/nova/virt/vmwareapi/vmops.py", line 692, in _get_extra_specs
ERROR oslo_messaging.rpc.server json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 255 (char 254)
• Instance metadata comes from the image metadata that was used to build the instance

 

Environment

7.x

Cause

In the Nova database, the value field in the tables instance_system_metadata and shadow_instance_system_metadata  is set to 255 characters (varchar(255) and data is being cut off.

Example:

MariaDB [nova]> select * from instance_system_metadata where instance_uuid="<instance uuid>" and key="image_vmware_extra_config";

{"smbios.assetTag":"OpenTelekomCloud","parallelX.present":"FALSE","serialX.present":"FALSE","svga.vgaOnly":"TRUE","pciPassthru*.present":"FALSE","RemoteDisplay.vnc.enabled":"FALSE","tools.setInfo.sizeLimit":"1048576","Net.BlockGuestBPDU":"1","floppyX.pre

Note: There should be "floppyX.present":"FALSE"}' at the end but the line is cut off.

As the data has been truncated, further investigation will be needed to see what can be done to repair the data.

Resolution

Impact: This is making a change to the database.  Please make sure to take a backup before running these statements.

• Backup the Nova database to the /tmp directory on the management server
  osctl exec -ti mariadb-server-0 -- mysqldump --defaults-file=/etc/mysql/admin_user.cnf nova > /tmp/nova.sql
  
  

If the instance metadata is stored in the image, we can do the following:

• In the Nova database, set the type for instance_system_metadata text.

osctl exec -it mariadb-server-0 bash  
mysql --defaults-file=/etc/mysql/admin_user.cnf
use nova;

MariaDB [nova]> ALTER TABLE instance_system_metadata MODIFY value TEXT;
MariaDB [nova]> ALTER TABLE shadow_instance_system_metadata MODIFY value TEXT;


• Update the instance with correct values from the image (glance database)
  Note:  You will need the instance UUID and the image UUID

MariaDB [nova]> UPDATE nova.instance_system_metadata SET value = (SELECT value FROM glance.image_properties WHERE image_id = '<uuid>') WHERE  instance_uuid = '<uuid>';
MariaDB [nova]> UPDATE nova.shadow_instance_metadata SET value = (SELECT value FROM glance.image_properties WHERE image_id = '<uuid>') WHERE  instance_uuid = '<uuid>';