(403 Forbidden)" error while accessing NSX Intelligence

Products

VMware vDefend Firewall

Issue/Introduction

While accessing NSX Intelligence 1.0.0 from Discover & Take Action page in Plan & Troubleshoot tab in NSX-T Datacenter 2.5 using CA signed certificates, you see these errors similar to:

The application server is unable to fulfill your request due to insufficient privileges. You do not have the privileges to access NSX Intelligence. (403 Forbidden)

OR

Failed to load requested application. Please try refreshing the browser or contact support if the problem persists.
In the /var/log/kafka/server.log file of the NSX Intelligence appliance, you see entries similar to:

[2019-12-18 23:00:05,259] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR1_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-12-18 23:00:05,645] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR2_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-12-18 23:00:05,763] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR3_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

This issue occurs due to an incomplete or incorrect certificate exchange with NSX Intelligence appliance when NSX Manager uses CA-signed certificates.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.0.0 with NSX Intelligence 1.1.0

Workaround:
To work around this issue if the NSX-T Manager is using the CA signed certificate:

For NSX-T Data Center 2.5.0 with NSX Intelligence 1.0.0

Step 1

Contact Broadcom Support and reference this KB article (321203) requesting the FTP link to download the WAR files (proton and policy).

Step 2

(a) Download nsxapi.war and policy.war files from the FTP link and copy to all the 3 NSX-T Manager nodes using root and do the following steps.

  (i) Make a copy of the existing war files:

           # cp /opt/vmware/proton-tomcat/webapps/nsxapi.war /root/nsxapi.war.firestar.bak
           # cp /opt/vmware/policy-tomcat/webapps/policy.war /root/policy.war.firestar.bak

  (ii) Stop the manager and policy services:

           # systemctl stop proton
           # systemctl stop nsx-policy-manager

  (iii) For example, if the new nsxapi.war and policy.war files are under /tmp directory, use these commands to replace the existing war files:

           # cp /tmp/nsxapi.war /opt/vmware/proton-tomcat/webapps/
           # cp /tmp/policy.war /opt/vmware/policy-tomcat/webapps/
           # chown uproton:uproton /opt/vmware/proton-tomcat/webapps/nsxapi.war
           # chown uproton:uproton /opt/vmware/policy-tomcat/webapps/policy.war
           # chmod 644 /opt/vmware/policy-tomcat/webapps/policy.war
# chmod 644 /opt/vmware/proton-tomcat/webapps/nsxapi.war

(iv) Restart the manager and policy services:

           # systemctl start proton
           # systemctl start nsx-policy-manager

(b) After the management cluster status is stable, obtain the pem_encoded field from the cluster certificate API:

  (i) Use the following API GET:

          GET https://{{NSX_MANAGER_VIP/IP}}/api/v1/trust-management/certificates?type=cluster_api_certificate

(ii) From the resulting JSON, copy the value for field pem_encoded, excluding the double quotes.

(d) Add the new certificate to the client truststore and kafka broker truststore on the NSX Intelligence appliance.

(i) Using SSH, log in to the NSX Intelligence appliance.

# ssh root@<nsx-intelligence-appliance-ip>
           # export NEW_CERT_FILE=/root/new_cert.pem
           # export HTTP_CERT_PWD_FILE=/config/http/.http_cert_pw
           # export HTTP_CERT_PW=$(cat $HTTP_CERT_PWD_FILE)
           # export CLIENT_TRUSTSTORE_FILE="/home/secureall/secureall/.store/.client_truststore"
           # export KAFKA_TRUSTSTORE_FILE="/home/secureall/secureall/.store/.kafka_broker_truststore"

(ii) Paste in the new pem_encoded field from the JSON:

           # echo -e "contents copied from step b(ii)" > $NEW_CERT_FILE

(iii) In case the cluster certificate is a CA signed certificate, there will be multiple "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" in $NEW_CERT_FILE. Split them into separate files. Skip this step if the cluster certificate is a self-signed certificate.

           # csplit -f 'new_cert_' -b '%01d.pem' -z -s new_cert.pem -- '/-----BEGIN CERTIFICATE-----/' '{*}'
            Typically there will be three or more new files generated. Export all file paths accordingly:

# export NEW_CERT_FILE_0=/root/new_cert_0.pem
           # export NEW_CERT_FILE_1=/root/new_cert_1.pem
           # export NEW_CERT_FILE_2=/root/new_cert_2.pem

(iv) If the cluster certificate is a CA signed certificate, insert all the certificates into client truststore and kafka broker truststore. The alias should be prefixed with nsx-manager-certificate.

           # keytool -import -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt

           If this command fails with error: "keytool error: java.lang.Exception: Certificate not imported, alias <nsx-manager-certificate mp-cluster 0> already exists", delete the existing certificate using the below command and run the above command again.

           # keytool -delete -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
           # keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 1" -file $NEW_CERT_FILE_1 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
           # keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 2" -file $NEW_CERT_FILE_2 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
           # keytool -import -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt

           If this command fails with this error: "keytool error: java.lang.Exception: Certificate not imported, alias <nsx-manager-certificate mp-cluster 0> already exists", delete the existing the certificate using below command and run the above command again.

           # keytool -delete -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt
           # keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 1" -file $NEW_CERT_FILE_1 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt
           # keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 2" -file $NEW_CERT_FILE_2 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt

(v) Verify the certificates were successfully added:

           # keytool -list -v -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
           # keytool -list -v -keystore $KAFKA_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt

(e) Restart proxy, kafka, nsx-config and processing services in the NSX Intelligence appliance.

           # systemctl restart proxy
           # systemctl restart kafka
# systemctl restart nsx-config
           # systemctl restart processing

You should now be able to refresh the "Plan & Troubleshoot" page and view the flow information as before in the NSX-T Manager User Interface (UI).

Note: If the above workaround does not resolve the issue, you may have to delete the NSX Intelligence appliance using direct REST API.

Obtain the deployment ID (denoted by vm_id field in the API response) for the NSX Intelligence appliance:

GET https://<NSXMGR_IP>/api/v1/intelligence/nodes/deployments
Delete the NSX Intelligence appliance using the following API:

POST https://<NSXMGR_IP>/api/v1/intelligence/nodes/deployments/<deployment_id_from_prev_step>?action=delete
Deploy a new NSX Intelligence appliance and follow the steps in the KB above to resolve the issue.

For NSX-T Data Center 2.5.0 with NSX Intelligence 1.1.0.

Delete the failed deployment of NSX Intelligence.
Follow the steps above up-to Step 2 (a).
Redeploy NSX Intelligence 1.1.0.