vSAN Data Protection - Take snapshot task failed - vCenter is in Enhanced Linked Mode (ELM)

Products

VMware vSAN VMware vSAN 8.x

Issue/Introduction

VMware vSAN Data Protection - Take snapshot task failed

vCenter is in Enhanced Linked Mode (ELM)

/storage/log/snapservice/snap-service.log shows

"level":"debug","timestamp":"2025-05-01T06:03:19.346Z","C":"sync/thread_pool.go:448","message":"Waiting for job","opID":"thread-pool-VmSnapshotTPX-tp-24-worker-1","worker":"VmSnapshotTPX-tp-24-worker-1"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"fxevent/zap.go:122","message":"invoking","function":"gitlab.eng.vmware.com/core-build/vsan_snapshot_service/pkg/initializer.CreateServiceAccount()"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"vcenter/service_accounts.go:239","message":"Find the credential file /secrets/svc_account_credential"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"vcenter/service_accounts.go:413","message":"Scheduling password rotation for ","username:":"com.vmware.vsan.snapservice","nextUpdateTime:":"2025-05-19T02:24:39.433Z","lastUpdateTime":"2025-04-09T02:24:39.433Z"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"vcenter/service_accounts.go:139","message":"successfully initialized a service account object","svc account user":"com.vmware.vsan.snapservice"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"fxevent/zap.go:122","message":"invoking","function":"fx.Annotate(gitlab.eng.vmware.com/core-build/vsan_snapshot_service/pkg/initializer.RemoveAdminCredential(), fx.ParamTags([\"name:\\\"configFile\\\"\"])"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"fxevent/zap.go:122","message":"invoking","function":"gitlab.eng.vmware.com/core-build/vsan_snapshot_service/pkg/initializer.StartTaskMonitor()"}
{"level":"info","timestamp":"2025-05-01T06:03:19.349Z","C":"auth/client.go:246","message":"Creating new VC client using svc account","opID":"tasks-monitor"}
{"level":"debug","timestamp":"2025-05-01T06:03:19.349Z","C":"auth/client.go:592","message":"skip reading certs from sub-directory","opID":"tasks-monitor","Path:":"/etc/ssl/certs/nginx"}
{"level":"info","timestamp":"2025-05-01T06:03:19.371Z","C":"auth/client.go:565","message":"Using","vim.version":"8.0.3.0"}
{"level":"info","timestamp":"2025-05-01T06:03:19.371Z","C":"auth/client.go:513","message":"Find the svc account credential file","filename:":"/secrets/svc_account_credential"}
{"level":"info","timestamp":"2025-05-01T06:03:19.371Z","C":"auth/client.go:413","message":"trying to renew saml token","opID":"tasks-monitor"}
{"level":"info","timestamp":"2025-05-01T06:03:19.371Z","C":"auth/client.go:418","message":"obtained lock, trying to renew saml token","opID":"tasks-monitor"}
{"level":"info","timestamp":"2025-05-01T06:03:19.371Z","C":"auth/client.go:420","message":"starting to renew saml token","opID":"tasks-monitor"}
{"level":"info","timestamp":"2025-05-01T06:03:19.396Z","C":"auth/client.go:565","message":"Using","opID":"tasks-monitor","vim.version":"8.0.3.0"}
{"level":"error","timestamp":"2025-05-01T06:03:19.746Z","C":"auth/client.go:359","message":"Failed to get token from cert/key pair","opID":"tasks-monitor","error":"ns0:FailedAuthentication: Invalid

Environment

VMware vSAN 8.x
VMware Snapshot Service appliance (All Versions)

Cause

Service account "com.vmware.vsan.snapservice" is locked after the VMware Snapshot Service appliance was rebooted. This is because only one user with the same name can exist in vCenter when using ELM.

So upon reboot of the Snapshot Service Appliance the service account "com.vmware.vsan.snapservice" password becomes invalid and the account gets locked.

Resolution

Upgrade to VCF/VVF 9.0

If upgrade is not possible for vCenters in an Enhanced Linked Mode (ELM) configuration, then deploy only one instance of vSAN Snapshot Service appliance on the primary vCenter. Not an instance of vSAN Snapshot Service appliance for each vCenter in the ELM configuration.

If the service account "com.vmware.vsan.snapservice" is already locked open a case with VMware vSAN support for further assistance.