Data Solutions Controller Manager is in a crash loop state post upgrade
search cancel

Data Solutions Controller Manager is in a crash loop state post upgrade

book

Article ID: 417199

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

  • VMware Cloud Director (VCD) Data Solutions (DS) Controller Manager is in a crash loop state.
  • vcd-ds-controller-manager-<controller-id> displays a CrashLoopBackOff status.
  • csi-vcd-controllerplugin-<plugin-id> displays a CrashLoopBackOff status.
  • VCD Data Solutions Controller is failing to communicate with the VMware Cloud Director API.
  • VCD data-solutions-operator is failing to communicate with the VMware Cloud Director API.
  • Cloud Director has recently been upgraded from version 10.5.x
  • Kubernetes Container Storage Interface driver for VMware Cloud Director (CSI) version 1.5.0 is installed in the cluster.
  • PostgreSQL instances are showing as disconnected.
  • Logs for container manager of pod vcd-ds-controller-manager-<controller-id> in the vcd-ds-system namespace display an error similar to the following:

2025-10-30T14:09:47.491Z        ERROR   rdeadapter.watcher      Failed to list objects  {"error": "failed to list entities: [500 INTERNAL_SERVER_ERROR] [<request-id>] Unable to perform this action. Contact your cloud administrator., entity: [ ]"}
vcloud.vmware.com/data-solutions-operator/rde/adapter/watcher.(*Watcher).Run vcloud.vmware.com/data-solutions-operator/rde/adapter/watcher/watcher.go:86

2025-10-30T14:09:47.499Z        ERROR   informer        listWatch loop  {"context": "informer.Run", "type": "*dsprovisioning.Client", "error": "processList: listing: failed to list entities: [500 INTERNAL_SERVER_ERROR] [ <request-id> ] Unable to perform this action. Contact your cloud administrator., entity: [ ]", "errorVerbose": "failed to list entities: [500 INTERNAL_SERVER_ERROR] [
<request-id> ] Unable to perform this action. Contact your cloud administrator., entity: [ ]\nlisting\n<hostname>/awesome/informer/pkg/cache.(*GenericInformer).processList\n\t<hostname>/awesome/[email protected]/pkg/cache/informer.go:500\n<hostname>/awesome/informer/pkg/cache.(*GenericInformer).listWatch.func3\n\<hostname>/awesome/[email protected]/pkg/cache/informer.go:408\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:169
5\nprocessList\n<hostname>/awesome/informer/pkg/cache.(*GenericInformer).listWatch.func3\n\t<hostname>/awesome/[email protected]/pkg/cache/informer.go:409\ngolang.org/x
/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1695"}
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection.ConditionFunc.WithContext.func1
        k8s.io/[email protected]/pkg/util/wait/wait.go:222
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext

  • 500 INTERNAL_SERVER_ERROR is generated.
  • On the Cloud Director cell within the log file /opt/vmware/vcloud-container-debug.log the following error is displayed:

    2025-10-30 14:09:47.499 | ERROR    | pool-jetty-197132         | DefaultExceptionMapper         | Internal server exception | requestId=<request-id>,request=GET https://<cloud.example.com>/cloudapi/1.0.0/entities/types/vmware/dsProvisioning/0.1,requestTime=1761831355543,remoteAddress=<ipaddress>,userAgent=Go-http-client/1.1,accept=application/*+json;version 38.1 applicat
    ion/json;version 38.1
    java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Enum (java.lang.String and java.lang.Enum are in module java.base of loader 'bootstrap')
            at org.hibernate.type.EnumType.nullSafeSet(EnumType.java:134)
            at org.hibernate.type.CustomType.nullSafeSet(CustomType.java:179)
            at org.hibernate.loader.Loader.bindPositionalParameters(Loader.java:1728)

Environment

VMware Cloud Director 10.6.1.1

VMware Container Service Extension 4.2.3

VMware Cloud Director extension for Data Solutions 1.6

Cause

Kubernetes Container Storage Interface driver for VMware Cloud Director (CSI) 1.5.0 version is not compatible with the VCD 10.6.1.  

Resolution

The CSI driver needs to be upgraded to version 1.6 or above.

Refer to the instructions outlined in the following link on how to upgrade the cluster components:
Upgrade Kubernetes Components in VMware Cloud Director Container Service Extension Clusters

Additional Information

Kubernetes Container Storage Interface Driver for VMware Cloud Director Compatibility

Manage the VMware Cloud Director extension for Data Solutions Operator

Product Interoperability Matrix

Powered by Wolken Software