Unable to perform Certificate Replacement in VCF Operations for Networks 9.x Using Custom CA

Products

• VCF Operations for Networks
• VCF Operations

Issue/Introduction

The VCF Operations for Networks certificate replacement failed with error LCMVRNICONFIG9019.

From the VCF Fleet Management logs located at /var/log/vrlcm/vmware_vrlcm.log, a similar log message is shown as below.

YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [pool-3-thread-91] [c.v.v.l.p.c.v.d.h.VRNIUtils]  -- Successfully fetched certificates from vRNI
YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [pool-3-thread-91] [c.v.v.l.p.c.v.d.h.VRNIUtils]  -- Calling update certificate API - path /api/ni/settings/certificates
/##################TA= - request - {"certificate":"-----BEGIN CERTIFICATE-----\nMI

[...]

rY=\n-----END CERTIFICATE-----\n","private_key":null}
YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [pool-3-thread-91] [c.v.v.l.u.CustomTrustManager]  -- Certificate chain trusted
YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [pool-3-thread-91] [c.v.v.l.p.c.v.d.h.VRNIUtils]  -- Response code - 400 and data - {"code":400,"message":"Bad Request","details":[{"code":400,"message":"Either private key or certificate is missing in the request.","target":[]}]}
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR vrlcm[1307] [pool-3-thread-91] [c.v.v.l.p.c.v.d.h.VRNIUtils]  -- Failed to update certificate
YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [pool-3-thread-91] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVRNICONFIG9019', Retry : 'true', Causing Properties : '{ CAUSE ::  }'
com.vmware.vrealize.lcm.plugin.core.vrni.common.exception.VRNIPlatformException: Failed to update certificate. Try again
        at com.vmware.vrealize.lcm.plugin.core.vrni.certificate.ApplyVRNICertificateTask.execute(ApplyVRNICertificateTask.java:201) [vmlcm-vrniplugin-core-9.0.1.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62) [vmlcm-engineservice-core-9.0.1.0-SNAPSHOT.jar!/:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
YYYY-MM-DDTHH:MM:SS.SSSZ INFO vrlcm[1307] [scheduling-1] [c.v.v.l.r.c.RequestProcessor]  -- Updating the Environment request status to FAILED for request ID : ########-####-####-####-########8f with request type : REPLACE_PRODUCT_CERTIFICATE.

 

Additionally, the VCF Operations for Networks Appliance API logs under /var/log/arkin/restapilayer/api_request.log show that the system returned a 400 error code when processing API PUT requests to update the certificate. Please refer to the log snippet below:

YYYY-MM-DDTHH:MM:SS.SSSZ INFO restapilayer 1074 [netw@4413 class="restapilayer.api.AuditLoggerService" thread="dw-15375 - PUT /ni/settings/certificates/##################TA=" method="logApiRequest" line="31"] [<Source IP address>] PUT 400 0 /ni/settings/certificates/##################TA=

 

Environment

VCF Operations for Networks 9.x

VCF Operations 9.x

Cause

This issue occurred because the private key was missing and was not provided to the VCF Operations for Networks appliance node, resulting in a 400 error code. The API PUT request originated from Fleet Management.

Resolution

If you need to replace the VCF Operations for Networks certificate with a custom CA, please open a case via Broadcom Support for assistance.

Additional Information

The VCF 9.0 developer portal shows the PUT API request for updating certificates on VCF Operations for Networks appliance requires the private key to be included in the request body.

Reference link: VCF Operations for Networks API – PUT Certificates