Cross-cloud connectivity through Public Cloud Gateway lost
Article ID: 330600
Updated On:
Products
VMware NSX
Issue/Introduction
- You're attempting cross-cloud communication through Public Cloud Gateway (PCG).
- You observe similar entries in PCG under the respective directory:
/var/log/pcm/pcm.log
2023-08-28T20:45:48.326Z ERROR task-executor-1 RestClientManager 2500 - [nsx@6876 comp="nsx-public-cloud-gateway" errorCode="PCG50555" level="ERROR" subcomp="pcm"] Failed to receive response from resource https://127.0.0.1:2081/policy/api/v1/infra/tier-1s/<tier-1-gw>/segments/<segment>/ports/<port-id>, Unable to reach Manager resource
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "https://127.0.0.1:2081/policy/api/v1/infra/tier-1s/<tier-1-gw>/segments/<segment>/ports/<port-id>": Received fatal alert: certificate_unknown; nested exception is javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.8.jar:5.3.8]
2023-08-28T20:45:48.326Z ERROR task-executor-1 RestClientManager 2500 - [nsx@6876 comp="nsx-public-cloud-gateway" errorCode="PCG50555" level="ERROR" subcomp="pcm"] Failed to receive response from resource https://127.0.0.1:2081/policy/api/v1/infra/tier-1s/<tier-1-gw>/segments/<segment>/ports/<port-id>, Unable to reach Manager resource
org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "https://127.0.0.1:2081/policy/api/v1/infra/tier-1s/<tier-1-gw>/segments/<segment>/ports/<port-id>": Received fatal alert: certificate_unknown; nested exception is javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.8.jar:5.3.8]
Environment
VMware NSX-T Data Center 3.1.3.5
Resolution
Currently, the workaround is to replace Principal Identity (PI) certificates for Cloud Service Manager (CSM) and its peers.
Workaround:
If you believe you're hitting this issue please follow the below steps:
1) Replace CSM certs
1.Log into the CSM as root.
2. Create an empty script file via the following command: vi csm_script.sh
3. Paste the below content into the script and save it.
2. Create an empty script file via the following command: vi csm_script.sh
3. Paste the below content into the script and save it.
#!/usr/bin/bash
set -eux
keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
keytool -delete -alias tomcat -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
export CN=`hostname`
/usr/bin/timeout --kill-after=1 120 /opt/vmware/nsx-node-api/bin/make_cert.py /usr/java/jre/bin/keytool -noprompt -genkeypair -alias tomcat -keyalg RSA -keysize 2048 -storepass `cat /config/http/.http_cert_pw` -keypass `cat /config/http/.http_cert_pw` -keystore /home/secureall/secureall/.store/.bluelane_keystore -dname "CN=$CN, OU=NSX, O=VMware Inc., L=Palo Alto, ST=CA, C=US" -validity 3650
echo "Press Enter to Continue ..." && read c
thumbprint=$(keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw` 2>/dev/null | awk 'NR==17{ print; exit }' | awk -F'SHA256: ' '{print $2}' | sed 's/:/''/g'| tr '[:upper:]' '[:lower:]')
keytool -export -alias tomcat -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw` -rfc -file csm.cert
echo "Press Enter to Continue ..." && read c
cert=`cat csm.cert`
echo $cert
echo ""
echo ""
echo $thumbprint
set -eux
keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
keytool -delete -alias tomcat -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
export CN=`hostname`
/usr/bin/timeout --kill-after=1 120 /opt/vmware/nsx-node-api/bin/make_cert.py /usr/java/jre/bin/keytool -noprompt -genkeypair -alias tomcat -keyalg RSA -keysize 2048 -storepass `cat /config/http/.http_cert_pw` -keypass `cat /config/http/.http_cert_pw` -keystore /home/secureall/secureall/.store/.bluelane_keystore -dname "CN=$CN, OU=NSX, O=VMware Inc., L=Palo Alto, ST=CA, C=US" -validity 3650
echo "Press Enter to Continue ..." && read c
thumbprint=$(keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw` 2>/dev/null | awk 'NR==17{ print; exit }' | awk -F'SHA256: ' '{print $2}' | sed 's/:/''/g'| tr '[:upper:]' '[:lower:]')
keytool -export -alias tomcat -v -keystore /home/secureall/secureall/.store/.bluelane_keystore -storepass `cat /config/http/.http_cert_pw` -rfc -file csm.cert
echo "Press Enter to Continue ..." && read c
cert=`cat csm.cert`
echo $cert
echo ""
echo ""
echo $thumbprint
4. Run the script to update tomcat certificate on the CSM. From the response, make note of thumbprint and csm.cert
2) Manager Steps
1. Copy the csm.cert from 1.4, and import it onto the NSX Manager via UI through System > Certificates > Import > Certificate. (make sure the service-certificate toggle is turned off)
2. Once the cert is imported onto NSX Manager, it should appear on the UI under System > Certificate. Expand the newly imported cert and copy the "id" field to be used in step
3. Run the following API via Postman/curl GET https://<mgr>/api/v1/trust-management/principal-identities, to get the list of PIs and their associated information.
4. Copy the CSM's principal ID. Look for name = "nsx-cloud-service-manager", resource_type="principal identity". In here copy corresponding "id" in this section. This will be the "principal_identity_id" needed for step
5. Run this API via Postman/curl POST https://<mgr>/api/v1/trust-management/principal-identities?action=update_certificate with below payload
{
"principal_identity_id": "<pi-id>", <== from 2.4
"certificate_id": "<new-cert-id>" <== from 2.2
}
"principal_identity_id": "<pi-id>", <== from 2.4
"certificate_id": "<new-cert-id>" <== from 2.2
}
Once the Post API is successful, the newly imported certificate will show the extended expiry date on the NSX Manager UI.
3) PCG steps
1. Log into the PCG as root.
2. Copy the cert "csm.cert" (generated in step 1.4) as a file from CSM.
3. Create an empty script file via the following command: vi pcg_script.sh
4. Paste the below content into the script file and save it.
2. Copy the cert "csm.cert" (generated in step 1.4) as a file from CSM.
3. Create an empty script file via the following command: vi pcg_script.sh
4. Paste the below content into the script file and save it.
#!/usr/bin/bash
set -eux
keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
keytool -delete -alias csmcert -v -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
/usr/java/jre/bin/keytool -import -alias csmcert -file csm.cert -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw` -noprompt
set -eux
keytool -list -v -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
keytool -delete -alias csmcert -v -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw`
echo "Press Enter to Continue ..." && read c
/usr/java/jre/bin/keytool -import -alias csmcert -file csm.cert -keystore /home/secureall/secureall/.store/.bluelane_truststore -storepass `cat /config/http/.http_cert_pw` -noprompt
5. Run the above created script.
6.Restart PCM service by running the below command
6.Restart PCM service by running the below command
service nsx-public-cloud-manager restart
4) Go back to CSM root shell and restart CSM service by doing the follow command
service nsx-cloud-service-manager restart
Feedback
Yes
No
Powered by
