Cannot connect host to vCenter or vpxd crash due to error: string too large for database
search cancel

Cannot connect host to vCenter or vpxd crash due to error: string too large for database

book

Article ID: 327022

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSAN

Issue/Introduction

This article is created for the purpose of resolving and preventing 'string too large for database' errors in vCenter.

Symptoms:

Attempting to connect an ESXi host to vCenter times out or vpxd crash with coredump and you see error similar to:

string too large for database

In the vpxd.log file,you can see lines similar to


error vpxd[7FC341374700] [Originator@6876 sub=Default opID=AddHostWizard-apply-10967-ngc:70003754-72] [Vdb::VdbField] Invalid value written to column DEVICE_INFO_SUMMARY in table VPX_VM_VIRTUAL_DEVICE
error vpxd[7FC341374700] [Originator@6876 sub=Default opID=AddHostWizard-apply-10967-ngc:70003754-72] [Vdb::VdbField] String too large: 514 > max(510)
error vpxd[7FC341374700] [Originator@6876 sub=MoHost opID=AddHostWizard-apply-10967-ngc:70003754-72] [HostMo::Reconnect] Got method fault: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
[context]zKq7AVECAAAAAIStcgAednB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAGCAHF2cHhkAAF4D3EBGuJ9AcALfgEUs64BmHStAcqDrQHTzHwBQc58AcVvfQEjdH0BdM95AQx7egEqg3oB7NBYAbuNWYLD4QUBbGlidmltLXR5cGVzLnNvAAEuw3QBlbpzAQqwcwF71HMBsid0AEesIwAuryMAN
7crA1R0AGxpYnB0aHJlYWQuc28uMAAEHYwObGliYy5zby42AA==[/context]
2018-05-09T14:38:37.309Z info vpxd[7FC341374700] [Originator@6876 sub=vpxLro opID=AddHostWizard-apply-10967-ngc:70003754-72] [VpxLRO] -- FINISH task-108756
2018-05-09T14:38:37.309Z info vpxd[7FC341374700] [Originator@6876 sub=Default opID=AddHostWizard-apply-10967-ngc:70003754-72] [VpxLRO] -- ERROR task-108756 -- host-20 -- vim.HostSystem.reconnect: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "string too large for database"
--> msg = ""
--> }
--> Args:
-->
--> Arg cnxSpec:
--> (vim.host.ConnectSpec) {
--> hostName = "<esxi_fqdn>",
--> port = -1,
--> sslThumbprint = <unset>,
--> userName = "root",
--> password = (not shown),
--> vmFolder = 'vim.Folder:########-####-####-####-########135d:group-v3',
--> force = true,
--> vimAccountName = "vpxuser",
--> vimAccountPassword = (not shown),
--> managementIp = <unset>,
--> lockdownMode = <unset>,
--> hostGateway = (vim.host.GatewaySpec) null
--> }
--> Arg reconnectSpec:


In the vpxa.log on the esxi host you see entries similar to:

verbose vpxa[740EB70] [Originator@6876 sub=vpxaVmprovUtil opID=AddHostWizard-apply-6826-ngc:70001896-a0-d4] [TranslateDatastoreMoref] vim.Datastore:vsan:52dc0a41########-######3f1cb6db1d -> vim.Datastore:ds://
/vmfs/volumes/vsan:52dc0a41########-######3f1cb6db1d/
verbose vpxa[740EB70] [Originator@6876 sub=vpxaMoVm opID=AddHostWizard-apply-6826-ngc:70001896-a0-d4] Existing ISO file name: SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588_01130f0e-0b02
-461b-8c67-321155940509.ISO
verbose vpxa[740EB70] [Originator@6876 sub=PropertyProvider opID=AddHostWizard-apply-6826-ngc:70001896-a0-d4] RecordOp ASSIGN: info.progress, session[########-####-####-####-########0b91]52308476-734d-7a42-31d
d-8806d9b711de. Applied change to temp map.

 

Environment

VMware vCenter Server 6.x
VMware vCenter Server 6.5.x
VMware vSAN 6.x
VMware vCenter Server 6.7.x

Cause

This issue is triggered when there are ISOs with lengthy names mounted to VMs when using Content Library on VSAN Datastores.

The vCenter Database table 'VPX_VM_VIRTUAL_DEVICE' has a column called 'DEVICE_INFO_SUMMARY' which is only designed to handle a certain number of characters.

This column contains the paths to the devices that are mounted on virtual machines.

By default, the VSAN datastore path is longer than the paths to VMFS or NFS.

Content Library also adds an extension to ISO files. 

When using both of the above, having naturally large named ISOs can surpass the column character limit for 'DEVICE_INFO_SUMMARY'.

Resolution

  • Upgrade vCenter to version 7.0 or higher.
  • You can also use one of the following workarounds: 
    • Workaround1 : Remove the configuration causing the crash 
      • STEP 1: Identify the virtual machines:

                        PowerCLI:  get the names & paths to the virtual machines with ISOs mounted 

Get-VM | FT Name, @{Label="ISO file"; Expression = { ($_ | Get-CDDrive).ISOPath }}​


                     ESXi CLI (SSH/ESXi Shell):

grep -i ISO /vmfs/volumes/*/*/*.vmx


                   GUI (HOST UI):

Check each VM by right clicking one at a time and going to 'Edit Settings'.
Check the CD/DVD drive to see if there is an ISO mounted.

      • STEP 2: Remove the iso files

        ESXi CLI (SSH/ESXi Shell):
        • Type the following command to edit vmx file using the paths found in STEP 1 (BACKUP THE VMX FILE FIRST - cp VM_NAME.vmx VM_NAME.vmx.bak):
          vi /vmfs/volumes/VSAN_DATASTORE/VM_DIRECTORY/VM_NAME.vmx

        • Type 'i' for INSERT mode.

          Remove the ISO name by going to the end of the line and using backspace.

                            GUI (HOST UI): 

        • Right Click VM in question and select 'Edit Settings'.
        • Change the CD/DVD drive to Client Device

 

      • STEP 3: Rename ISO

                 Once the ISO has been unmounted from the VM, you can rename the ISO to a shorter name so it can be used in the vSAN environment. 

  •  
  •  
    • Workaround 2: increase DEVICE_INFO_SUMMARY column of table VPX_VM_VIRTUAL_DEVICE in VCDB. 
      • STEP 1:  Stop the vpxd service on vCenter by running service-control --stop vpxd
      • STEP 2: To connect to the database - Get SSH/Console to VCSA and type:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

      • STEP 3: Run the following queries on VCDB database:


alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (514);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (514);

      • STEP 4: Add the following to /etc/vmware-vpx/vpxd.cfg between <config> and </config>:

   <stressOption>
      <vdb>
         <schemaLoader>
             <forceODBC>1</forceODBC>
         </schemaLoader>
      </vdb>
   </stressOption>

      • STEP 5: Restart the vpxd service

service-control --start vpxd

      • STEP 6:  Remove the vpxd.cfg snippet added in step 4.

Additional Information

Impact/Risks:
This issue will render the ESXi hosts disconnected/unusable in vCenter until action is taken.

Powered by Wolken Software