Negative Serial Number Certificate presented by tca-proxy service when CNF is trying to reach to the TCA Manager

Products

• VMware Telco Cloud Automation

Issue/Introduction

During CNF upgrade, the deployment manager fails to communicate with the Kubernetes cluster. The operation fails with a TLS handshake error indicating a certificate parsing issue.

The deployment_manager.log contains the following error:

[ERROR]: Error: Kubernetes cluster unreachable: Get "https://<IP_REDACTED>:8500/version": tls: failed to parse certificate from server: x509: negative serial number

Environment

3.4

Cause

The issue is caused by a malformed X.509 certificate generated by the TCA-M proxy component. The certificate possesses a negative serial number, which violates integer encoding rules expected by the Go-based TLS client in k8s clusters. This results in the client rejecting the connection during the upgrade pre-check.

Issue is fixed in TCA 3.4, however given the 60-day renewal cycle for proxy certificates, it is possible that the certificate expired and may have not rotated on the machines following the upgrade. tca-proxy cert is persisted on the tca-manager

Resolution

1. SSH into TCA-M VM
   
   
2. Go to TCA postgres database:

kubectl -n tca-mgr exec -it postgres-0 – /bin/bash psql -d tca

     3. Delete the TCA certificate record from database:

delete from proxy_service_certificates where type = 'Tca';

     4. Restart proxy service from TCA manager 9443 Validate that re-genearted cert does not have negative Serial Number openssl s_client -connect <TCA IP / FQDN>:8500 2>&1 | openssl x509 -text | grep 'Serial Number'

Note: openssl command should be run from a different terminal not from the TCA manager shell

     5. If the serial number is still negative than repeat step 1-5 few times(3-4) and confirm the cert has been renewed.