Upgrade to NCP v4.0.1.x Fails due to Missing Common Name with NSX-T Manager Certificate

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When attempting to upgrade the NCP tile to v4.0.1x and above, the TAS apply change fails on the diego_database showing the below error output.

Task 63695 | 21:57:33 | L starting jobs: diego_database/2ad0d234-2371-4778-b407-f49b05e5bd08 (0) (canary) (00:05:36)
                      L Error: 'diego_database/2ad0d234-2371-4778-b407-f49b05e5bd08 (0)' is not running after update. Review logs for failed jobs: ncp
Task 63695 | 22:02:34 | Error: 'diego_database/2ad0d234-2371-4778-b407-f49b05e5bd08 (0)' is not running after update. Review logs for failed jobs: ncp

Task 63695 Started  Thu May  2 21:53:49 UTC 2024
Task 63695 Finished Thu May  2 22:02:34 UTC 2024
Task 63695 Duration 00:08:45
Task 63695 error


Updating deployment:
  Expected task '63695' to succeed but state is 'error'
Exit code

In the ncp-stdout.log, you will find the process exiting repeatedly.

2024-05-08T17:42:58.910Z a436fbe9-56a4-4934-9fba-9f680805f8f1 NSX 7937 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO" security="True"] nsx_ujo.common.nsx_log_adaptor Initialized log configuration
2024-05-08T17:42:58.919Z a436fbe9-56a4-4934-9fba-9f680805f8f1 NSX 7937 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.main Receive signal for handling 15
2024-05-08T17:42:58.920Z a436fbe9-56a4-4934-9fba-9f680805f8f1 NSX 7937 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.main Main process is exiting, terminate election process!

The error is not seen in logs because it's sent to the stderr output:

  File "/usr/local/lib/python3.8/dist-packages/nsx_ujo/ncp/nsx/nsx_wrapper.py", line 108, in is_self_signed_cert
    if ((self.subject_cn == self.issuer_cn) and
  File "/usr/local/lib/python3.8/dist-packages/nsx_ujo/ncp/nsx/nsx_wrapper.py", line 67, in subject_cn
    return self.cert.get_subject().CN.strip()
AttributeError: 'NoneType' object has no attribute 'strip'

Cause

A new capability was introduced in v4.0.1.x to support multiple NSX CA certificates. This capability needs to look into the certificate CN (Common Name), which is missing for the certificate offered by NSX.

The Common Name (CN) field in a certificate serves as a key identifier for the entity represented by the certificate. It's one of the most significant components of the Distinguished Name (DN) within the certificate, used mainly for specifying the domain or system that the certificate is intended to authenticate.

Resolution

To confirm the issue:

1. Navigate to the NSX tile --> NSX Manager tab --> NSX Manager CA Cert field.
2. Copy the certificate contents in the field.
3. Use any online decoder (sslshopper) or openssl to decode the content of the certificates

   • openssl x509 -in certificate.pem -text -noout

4. Confirm whether or not the CN field is populated.
5. If it is not populated, the server certificate will need to be updated and the subject's CN field will need to be populated with a non-null value.

Importing and Replacing Certificates