WCP Service fails to start with "panic: crypto/cipher: input not full blocks"

Products

• VMware vCenter Server

Issue/Introduction

In vCenter Server environment, the Workload Control Plane (WCP) service fails to start. You observe the following symptoms:

• Running service-control --start wcp returns an error:
  
  Error executing start on service wcp. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "wcp"
            ],
            "localized": "An error occurred while starting service 'wcp'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "wcp"
            ],
            "localized": "An error occurred while starting service 'wcp'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}


• In journalctl -xe, the process terminates with Signal 6 (ANOM_ABEND):
<month> <date> <time - hh:mm:ss> audit[985]: ANOM_ABEND auid=4294967295 uid=1021 gid=59000 ses=4294967295 subj=unconfined pid=985 comm="wcpsvc" exe="/usr/lib/vmware-wcp/wcpsvc" sig=6 res=1

• In /var/log/vmware/wcp/stdstream.log.stderr, you see a Go runtime panic:

  time="YYYY-MM-DDTHH:MM:SS" level=info msg="Initializing Wcp Service. pid=985 build=22634669 change=11998729"
panic: crypto/cipher: input not full blocks

  goroutine 1 [running]:
panic({0x4183da0, 0x4e41578})
        /build/mts/release/bora-22634669/compcache/cayman_go/ob-20968754/linux64/src/runtime/panic.go:941 +0x397 fp=0xc0010ef0e8 sp=0xc0010ef028 pc=0x1838ab7
crypto/internal/boring.(*aesCBC).CryptBlocks(0x4e62be0?, {0xc00052aa68?, 0xc0009b5c80?, 0x10?}, {0xc0009b5c90?, 0xc0009d3660?, 0xc0010ef1a0?})
        /build/mts/release/bora-22634669/compcache/cayman_go/ob-20968754/linux64/src/crypto/internal/boring/aes.go:140 +0xe5 fp=0xc0010ef128 sp=0xc0010ef0e8 pc=0x1976b65
server/crypto.decryptCBC({0x4e724f8?, 0xc00078d0e0?}, {0xc0009b5c80, 0x28, 0x28}, {0xc000524c00, 0x20, 0x200})
        /build/mts/release/bora-22634669/bora/vpx/wcp/wcpsvc/src/server/crypto/cryptography.go:175 +0x17e fp=0xc0010ef1b0 sp=0xc0010ef128 pc=0x240621e
server/crypto.Decrypt.func1()
        /build/mts/release/bora-22634669/bora/vpx/wcp/wcpsvc/src/server/crypto/cryptography.go:148 +0x1ac fp=0xc0010ef270 sp=0xc0010ef1b0 pc=0x2405fac
server/crypto.Decrypt({0x4e724f8, 0xc00078d0e0}, {0xc0009b5c80, 0x28, 0x28}, {0xc000524c00, 0x20, 0x200})

• In /var/log/vmware/wcp/wcpsvcs.log, decryption errors are present: error wcp [crypto/cryptography.go:201] [opID=vCLS] Unable to decrypt string: cipher: message authentication failed

YYYY-MM-DDTHH:MM:SS info wcp [dataprovider/clustervm_password.go:125] [opID=vCLS] Successfully executed: SELECT password, last_update_timestamp FROM hdcs.hdcs_clustervm_password;
YYYY-MM-DDTHH:MM:SS error wcp [crypto/cryptography.go:201] [opID=vCLS] Unable to decrypt string: cipher: message authentication failed
YYYY-MM-DDTHH:MM:SS error wcp [crypto/cryptography.go:147] [opID=vCLS] AES decryption GCM Mode failed due to: <nil>, cipher: message authentication failed. Retrying CBC

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

The clustersvc component within WCP attempts to decrypt the vCLS (vSphere Cluster Services) or Cluster VM password stored in the HDCS database. This panic occurs when the ciphertext stored in the database is empty, corrupted, or formatted incorrectly, preventing the AES-CBC decryption logic from processing the data blocks.

Resolution

Verify the corruption in the database.

1. Log in to the vCenter Server Appliance via SSH as root.

2. Run the built-in decryption script to check the health of the stored password:
   /usr/lib/vmware-wcp/decrypt_clustervm_pw.py

3. If the output is empty or returns a Traceback: The database record is corrupted. 

4. If the output returns a cleartext password: The issue may lie elsewhere

5. Inspect the current database record:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT * FROM hdcs.hdcs_clustervm_password;"

Because resolving this issue requires manual manipulation of the vCenter Server Database  and can impact environments utilizing Tanzu (Workload Management), users are advised not to attempt manual DB manupulation.

If you encounter the symptoms outlined above, contact Broadcom Technical Support for further assistance, as the resolution involves modifying vCenter Server database instance, which should be performed with caution.