Error caused vCenter Server disconnect - java.lang.NullPointerException when integrating vCenter deployed on Azure with VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to integrate a vCenter Server deployed on Microsoft Azure VMware Solution (AVS) with a newly deployed VMware Cloud Director (VCD) instance, the vCenter Server disconnects.
• An attempt to integrate the same vCenter with a different VCD instance also results in the same error.
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on the Cloud Director cell, you see entries similar to:
  
  Caused by: java.lang.NullPointerException
      at com.vmware.ssdc.library.vim.VimUtil.convertToValRef(VimUtil.java:137)
      at com.vmware.vcloud.inventory.impl.DvPortgroupInventoryRecordImpl.getInfo(DvPortgroupInventoryRecordImpl.java:177)
      at com.vmware.vcloud.inventory.impl.DvPortgroupInventoryRecordImpl.getInfo(DvPortgroupInventoryRecordImpl.java:35)
      at com.vmware.vcloud.inventory.impl.ObjectUpdateProcessor.insertInventory(ObjectUpdateProcessor.java:279)
      at com.vmware.vcloud.inventory.impl.ObjectUpdateProcessor.process(ObjectUpdateProcessor.java:169)
      at com.vmware.vcloud.inventory.impl.InventoryServiceDistributedImpl.handleUpdate(InventoryServiceDistributedImpl.java:245)
      at com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader$ReadPhase.processData(PropertyCollectorReader.java:1136)
  
  
• In /opt/vmware/vcloud-director/logs/vclistener.log, you see multiple failed attempts to create a NAS datastore:
  
  2025-03-17 11:33:37,439 | DEBUG    | pc-activity-pool-727870   | PropertyMapDao                 | [vcId = #####-####-####-####-#######] Inserting dvportgroup-#### in inventory DB, blob size 5,906 bytes. | activity=(com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader,urn:uuid:#####-####-####-####-#######)
  2025-03-17 11:33:37,439 | DEBUG    | Thread-143 (ActiveMQ-client-global-threads) | TaskManager                    | Handling completion update from MessageBusAdapter for task [vcId=#####-####-####-####-#######, moref=task-42287], state ERROR, cell=vcd01.example.com |
  2025-03-17 11:33:37,439 | DEBUG    | Thread-143 (ActiveMQ-client-global-threads) | TaskManager                    | Returning pre-existing wait handle for task [vcId=#####-####-####-####-#######, moref=task-42287] |
  2025-03-17 11:33:37,439 | DEBUG    | Thread-143 (ActiveMQ-client-global-threads) | WaitHandle                     | Skipping update for task [valref = [vcId=#####-####-####-####-#######, moref=task-42287], state = ERROR, taskName = CreateNasDatastore, progress = null, entityName = esx01-########.example.azure.com, errorMessage = An error occurred during host configuration: .] as it is in complete state. 
  
  
• vCenter tasks reveals multiple failed attempts to create a NAS datastore on specific ESXi host(s).
  
  
  
  
• A NAS datastore (e.g., named Datastore1) might be present in vCenter but is in inaccessible state.
  
  

Environment

VMware Cloud Director 10.x
vCenter Server 8.x

Cause

The vCenter inventory is stored in the VCD database to enable faster provisioning. However, in this case, the issue arose because the distributed port group information could not be retrieved, as it was being held by a locked datastore creation task.

Resolution

The resolution for this issue requires intervention within the vCenter Server environment on Azure VMware Solution. It is recommended to engage Microsoft Azure Support to address the underlying problem.

In this scenario described, Microsoft Azure Support resolved the issue by performing the following actions:

1. Removal of the stale and inaccessible NAS datastore (e.g., Datastore1) from the vCenter Server inventory.
2. Deletion of the associated iSCSI distributed portgroup from the vSphere Distributed Switch configuration within vCenter Server.