Error: certificate_unknown(46) during VCF Operations for Networks 9.1 upgrade

Products

VCF Operations for Networks VMware Cloud Foundation

Issue/Introduction

While upgrading VCF from version 9.0.x to version 9.1, or while applying an express patch to 9.1 the process fails while running pre-checks against VCF Operations for Networks.

An error message similar to one of the following is displayed in the UI:

An unexpected error occurred in step nodes_health_check. Reference Code: D6CFEA3A. Please contact support with this reference code. Detail: 401 Unauthorized:

An unexpected error occurred in step nodes_health_check. Reference Code: 14926D9A. Please contact support with this reference code. 

Detail: I/O error on GET request for "https://<VCF Operations for Networks IP>/api/management/nodes": Failed to obtain authentication token: 

Failed to obtain OpsNetworks authentication: I/O error on POST request for any api on vRNI for example: "https://<VCF Operations for Networks IP>/api/auth/login": certificate_unknown(46)

The Task failed errors with Reference Codes such as 1A93F68A, D6CFEA3A, or 14926D9A.

Messages similar to the following appear in the /var/log/containers/vcf-sddc-upgrade-service-sddcupgrade-####-sddc-lcm-_sddcupgrade-####.log file on the VMSP worker node:

YYYY-MM-DD2T14:24:08.555238002Z Failed to obtain authentication token: Failed to obtain OpsNetworks 91x+ authentication: I/O error on POST request for "https://<IP>/api/ni/auth/token": certificate_unknown(46)

Environment

VMware Cloud Foundation 9.1.x
VCF Operations for Networks 9.0.x, 9.1.x

Cause

The SSL handshake fails because the VCF Operations for Networks certificate is not trusted by the VCF Operations fleet runtime platform. This prevents the lifecycle management service from obtaining the necessary authentication tokens and it returns with certificate_unknown(46) error seen on Fleet Lifecycle task/subtask.

Resolution

Note: These steps are only to be used during a failed upgrade from VCF Operations for Networks 9.0.x to 9.1.x or when installing an express patch for 9.1.x when VCF Operations for Networks is still at the previous version.

Obtain the VCF services runtime FQDN from the VCF Operations UI (Build > Lifecycle > VCF Management > Components).
Download the add_cert_to_vcf_services_runtime_platform_trust.sh script (attached to this article).
Copy the script to the SDDC Manager VM as the vcf user.
Log in to the SDDC Manager via SSH and switch to the root user:
```
su -
```
Ensure the vcf user home directory is the current directory by running
```
cd /home/vcf
```

Download the certificate from the primary VCF Operations for Networks platform node by replacing <NODE_IP> with the IP address of the VCF Operations for Networks primary platform node and execute the command from one of the following options based on the certificate type:

For Self-Signed Certificates:

openssl s_client -connect <NODE_IP>:443 -showcerts </dev/null 2>/dev/null | openssl x509 -outform PEM > /home/vcf/ops-network-signer.pem

For CA-Signed Certificates:

openssl s_client -connect <NODE_IP>:443 -showcerts </dev/null 2>/dev/null | awk '/-----BEGIN CERTIFICATE-----/ { n++; in_cert=1; cert="" } in_cert { cert = cert $0 "\n" } /-----END CERTIFICATE-----/ { in_cert=0; if (n==2) { printf "%s", cert; exit } }' > /home/vcf/ops-network-signer.pem

Replace <RUNTIME_FQDN> with the value obtained in Step 1 and run the following command to execute the import script:

bash /home/vcf/add_cert_to_vcf_services_runtime_platform_trust.sh --vcf-services-runtime-fqdn <RUNTIME_FQDN> --ca-cert-file /home/vcf/ops-network-signer.pem

Follow the on-screen prompts to trust the certificate and restart the Software Depot and SDDC Lifecycle services.
Note: you will need to the password for VCF service runtime when running this script.

After the command is executed, output similar to the following should be seen:

Password for VCF service runtime admin: 

=> Checking connectivity...
  Checking VCF services runtime (########:443)... reachable (HTTP response received)

==> Verifying TLS certificate (Trust On First Use)...
  Fetching TLS certificate from VCF services runtime (########>:443)... ok (leaf certificate fetched)

Certificate will not expire
Certificate will not expire
  Certificate details:
    Host:       ########:443
    Subject:    C=US, ST=CA, L=Palo Alto, O=Broadcom, OU=vcfms, CN=########
    Issuer:     CN=CA, DC=vsphere, DC=local, C=US, ST=#####, O=########, OU=VMware Engineering
    Not Before: Jun 20 07:39:22 2026 GMT
    Not After:  Jun 19 07:39:22 2028 GMT
    SHA-256:    ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
  Verify the fingerprint above matches the certificate on ######## before accepting.

  Trust the TLS certificate presented by VCF services runtime?
  Proceed? [y/N]: y

==> [1/5] Resolving certificate...
  Using '/home/vcf/ops-network-signer.pem' as cert.

Certificate will not expire
Certificate will not expire
  Certificate details:
    File:       /home/vcf/ops-network-signer.pem
    Subject:    CN=CA, DC=vsphere, DC=local, C=US, ST=######, O=########, OU=VMware Engineering
    Issuer:     CN=CA, DC=vsphere, DC=local, C=US, ST=#######, O=########, OU=VMware Engineering
    Not Before: Jun 28 21:49:58 2026 GMT
    Not After:  Jun 28 21:49:58 2027 GMT
    SHA-256:    ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
    Type:       self-signed
  Certificate resolved from /home/vcf/ops-network-signer.pem

  Add the certificate above to the VCF services runtime platform trust and restart Software depot and SDDC lifecycle?
  Proceed? [y/N]: y

==> [2/5] Authenticating as admin at ########...
  Authentication successful at ########.

==> [3/5] Adding certificate to VCF services runtime platform trust...
  Cert-add task submitted: ########
  Polling task ######## [add certificate]...
    [0s] status: Pending
    [10s] status: Pending
    [20s] status: Running
    <truncated>
    [60s] status: Succeeded
  Task ######## [add certificate] succeeded.

==> [4/5] Restarting Software depot, as SDDC lifecycle depends on it...
  Found Software depot component: ID=16dd36bf-####-####-####-c3020744c9c5
  Restart task submitted: ########
  Polling task ######## [restart Software depot]...
    [0s] status: Pending
    [10s] status: Pending
    [20s] status: Running
   <truncated>
    [560s] status: Succeeded
  Task ######## [restart Software depot] succeeded.

==> [5/5] Restarting SDDC lifecycle...
  Found SDDC lifecycle component: ID=ba2cbdf2-####-####-####-7c309afa1104
  Restart task submitted: ########
  Polling task ######## [restart SDDC lifecycle]...
    [0s] status: Pending
    [10s] status: Pending
    [20s] status: Running
   <truncated>
    [510s] status: Succeeded
  Task ######## [restart SDDC lifecycle] succeeded.

==> Done. Certificate trusted and services restarted successfully.

The VCF Operations for Networks pre-check can be retried or a new one can be started in the VCF Operations UI.
Pre-check should be successful.
VCF Operations for Networks upgrade can be initiated.

If further assistance is needed, see Contact Broadcom Support.

Additional Information

Subscribe to this article to receive updates on permanent fixes (Reference: KB 275360).
If you are contacting Broadcom support about this issue, provide the following:

- SDDC Lifecycle support bundle (In the VCF Operations GUI, Build > Lifecycle > VCF Management > Components > SDDC Lifecycle > Generate Support Bundle)
- Text of any error messages seen in the VCF Operations GUI.

Refer to handling Log Bundles for offline review with Broadcom support:

Attachments

add_cert_to_vcf_services_runtime_platform_trust.sh get_app