WCP and vstat service may not start when a custom CA signed certificate is applied to vCenter
Article ID: 325222
Updated On:
Products
Issue/Introduction
The change of custom CA certificate on the vCenter which included a certificate policy with a larger OID segment value caused issue for the go interpreter in vCenter,
As a result WCP and vStat services fail to authenticate and start their services.
Symptoms:
The issue occurs during a vCenter update when the WCP service fails to start.
SSL Certificate replacement with Custom Certificate has been successful recently on vCenter.
The wcp service is in a stopped state.
Checking logs for /var/log/vmware/wcp/wcpsvc.log we see following
"tls failed to parse certificate from server: x509: invalid certificate policies"
below is example date and time sample and not exact match for what you may observe.
2023-11-10T11:00:41.466Z error wcp [ssolib/ssoadmin.go:110] Failed to create ssoadmin client; VIM url: https://vcenter:443/lookupservice/sdk <https://vcenter/lookupservice/sdk> , err: Post "https://vcenter:443/sso-adminserver/sdk <https://vcenter/sso-adminserver/sdk> ": tls: failed to parse certificate from server: x509: invalid certificate policies
Environment
VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.3
Cause
The change of custom CA certificate on the vCenter which included a certificate policy with a larger OID segment value caused issue for the go interpreter in vCenter,
vCenter 7 and 8 currently used go interpreter < 1.22 which has a bug which only allows 4 bytes for a value for the Certificate policy OID segment values. When a larger value is used on the certificate header it causes the go interpreter to fail with error "tls failed to parse certificate from server: x509: invalid certificate policies"
The issue is related to the custom CA signed certificate used on the vCenter.
When a custom CA certificate is created it has headers for certificate policies which are encoded to the certificate.
Example below from using openssl to decode the headers OIDs
OIDs under Certificate Policies as headers on the certificate are below
Policy: 1.3.9999.712312.5.1.1
Policy: 1.3.9999.712312.5.2.11
Policy: 1.3.9999.712312.5.3.1
Each of those OIDs segment values (1.3.9999.x.x.x ) is interpreted as an int and Go developers decided to use 4 bytes for it always to behave the same on 32 and 64 bit systems. So, in this example above the oid segment value, 712312 is encoded in 4 bytes and because of that Go has no issue with this certificate.
The maximum number the 4 bytes can take is 4,294,967,295 value .... if the encoding of the certificate policies used a value larger than this it will fail to be read by the go library when parsing the certificate.
Example of a certificate policy OID segments with a part with value greater than 4294967295
X509v3 Certificate Policies:
Policy: 1.3.9999.871051157387.5.1.1
Policy: 1.3.9999.871051157387.5.2.11
Policy: 1.3.9999.871051157387.5.3.1
We see the error below for the wcp service log wcpsvc.log when attempting to read the certificate to authenticate
2023-11-10T11:00:41.466Z error wcp [ssolib/ssoadmin.go:110] Failed to create ssoadmin client; VIM url: https://vcenter:443/lookupservice/sdk <https://vcenter/lookupservice/sdk> , err: Post "https://vcenter:443/sso-adminserver/sdk <https://vcenter/sso-adminserver/sdk> ": tls: failed to parse certificate from server: x509: invalid certificate policies
Resolution
vCenter 8.0u3 will not see this problem as it will use a different method to authenticate the wcp service.
Only workaround for now is to regen custom CA certs without the certificate policies or to values less than 4,294,967,295 per OID segment.
Workaround:
Nothing that can be done in this moment since it is a limitation of the go interpreter which is currently in use on vCenter 7 and 8.
To check if the custom CA certificate is affected by this.
Test the custom CA certificate using openssl to extract the values of the OID segments related to Certificate policies.
openssl x509 -text -in ca.cer | grep -A20 'Certificate Policies'
You should see something like below for certificate policy OID segment with value less than 4294967295
X509v3 Certificate Policies:
Policy: 1.3.9999.712312.5.1.1
Policy: 1.3.9999.712312.5.2.11
Policy: 1.3.9999.712312.5.3.1
Value of OID segment 712312 is less than 4294967295 value and no issue for go interpreter. wcp and vstat services can authenticate no problem and services start as expected.
For a certificate policy OID segment with value greater than 4294967295
X509v3 Certificate Policies:
Policy: 1.3.9999.871051157387.5.1.1
Policy: 1.3.9999.871051157387.5.2.11
Policy: 1.3.9999.871051157387.5.3.1
Value of OID segment 871051157387 is larger than 4294967295 causes issue for go interpreter and wcp to fail to start and vstat service also due to failed authentication.
A new version of go interpreter is expected in 2024 to be applied to vCenter 7 and 8 which will solve issue.
Additional Information
A new version of go interpreter is expected in 2024 to be applied to vCenter 7 and 8 which will solve issue.
Impact/Risks:
WCP and vStat services cannot start as they cannot authenticate using the vCenter CA custom certificate via go interpreter as its fails to read the Certificate policies OID segments successfully and stops.
Feedback
