Upgrading to vRealize Automation 7.3.1, 7.4 and 7.5 fails while running post-install scripts on the replica appliance nodes
Article ID: 328302
Updated On:
Products
VMware
Issue/Introduction
Symptoms:
- Upgrading to vRealize Automation 7.3.1, 7.4, 7.5 replica appliance fails while running post-install scripts.
- You see this issue while upgrading from vRA 7.1, 7.2, 7.3 to vRA 7.3.1, 7.4, 7.5
- In the /opt/vmware/var/log/vami/updatecli.log file, you see entries similar to:
+ echo 'Script /etc/bootstrap/postupdate.d/09-90-prepare-psql failed, error status 1'
+ exit 1
+ rm -f /tmp/postupdate-err-log
+ exit 1
+ trapfunc
+ excode=1
+ test 1 -gt 0
+ vami_update_msg set post-install 'Post-install: failed'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'
+ sleep 1
+ test 1 -gt 0 -o 0 -gt 0
+ vami_update_msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'
+ exit 1
+ rm -f /tmp/postupdate-err-log
+ exit 1
+ trapfunc
+ excode=1
+ test 1 -gt 0
+ vami_update_msg set post-install 'Post-install: failed'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'
+ sleep 1
+ test 1 -gt 0 -o 0 -gt 0
+ vami_update_msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'
- You have replaced the default self-signed vRealize Automation VAMI certificate.
Cause
This issue occurs when the /opt/vmware/etc/lighttpd/server.pem file contains chain certificate.
It should contain only one certificate and only one private key entry, otherwise it will cause upgrade failure during the post-install scripts for replica appliance nodes.
It should contain only one certificate and only one private key entry, otherwise it will cause upgrade failure during the post-install scripts for replica appliance nodes.
Resolution
This issue is resolved in VMware vRealize Automation 7.4 available at VMware Downloads.
Workaround:
You should see output similar to:
Workaround:
1. Revert the environment to the state before upgrade.
2. Back up your current certificate file on all virtual appliance nodes:
cp /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem-bak
3. Remove the chain certificate from /opt/vmware/etc/lighttpd/server.pem for all virtual appliance nodes.
4. Run the following commands:
service vami-lighttp restart
service haproxy restart
5. Wait for 10 minutes to allow the PostgreSQL database to be updated with correct certificates data.
6. Verify that the certificates stored in the PostgreSQL database are correct :
sudo -u postgres -i -- /opt/vmware/vpostgres/current/bin/psql vcac -c "SELECT node_host,node_cert from cluster_nodes where node_type='VA';"
2. Back up your current certificate file on all virtual appliance nodes:
cp /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem-bak
3. Remove the chain certificate from /opt/vmware/etc/lighttpd/server.pem for all virtual appliance nodes.
4. Run the following commands:
service vami-lighttp restart
service haproxy restart
5. Wait for 10 minutes to allow the PostgreSQL database to be updated with correct certificates data.
6. Verify that the certificates stored in the PostgreSQL database are correct :
sudo -u postgres -i -- /opt/vmware/vpostgres/current/bin/psql vcac -c "SELECT node_host,node_cert from cluster_nodes where node_type='VA';"
You should see output similar to:
node_host | node_cert
MASTER_VA.domain.com | MIIDNDCCAhwCCQDAYlr48T0k0DANBgkqhkiG9w0BAQUFADBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwHhcNMTUxMDI4MTIyOTI2WhcNND
MwMzE1MTIyOTI2WjBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKdM8FkNjK065gC0Dp9/3WdXNSrQnRpUfkTAiOlMvFpzbfuZfP8cW
aWjEPLri+yd+iUDLUYWBfxJajdnnG5NacjHbBxjFAkX5BF7ymtJrdfal0nw9furNNXRVLzbXZUcwvABxoZTldOWgezul0KeNjcEyHMwSGRhQ/uaQvfHRb2Y/I7FGpmB/EBu9Q4ehYBeE69oHvH7foHNBPtsMwcQnbm+pn1zGoBe78auSwPKwBV6C+Q/Y11h22lZnU/cxFcEfnYHUtbqPPHlOZXInHjiSir4aWykEZ+U8/
1dZy8ty+Xt5+ka/d5NZd4/2aneyLUn/CEeXCwP2GK6nb6HVn70lXAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBACvebOo1jFsRevrjuAn0Vu+Qav/g1KcgNFSr7w0phcE8uxBxzyheo9vBU64Q/ckSI4BDcekt/JEMlDIMIvI3Ld7LkDo4xa3e+qjSwwfe2NrxbMSqSRWRUGk/hUuePhYUxcLNpfO3PfkHiKo9AX+r/UGM7
hy/B75ZIbHk6EsV23rTrW/muB8NAXKKwVpOQeUYMKhgMhh6aLE7Q4e4wrXQQNQSD0IXBh232HcYrEQorhwutkdc8fNMg/YwwknRSlA8n4T89vPjYLF6Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGrNf7sZdT5DYo=
MwMzE1MTIyOTI2WjBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKdM8FkNjK065gC0Dp9/3WdXNSrQnRpUfkTAiOlMvFpzbfuZfP8cW
aWjEPLri+yd+iUDLUYWBfxJajdnnG5NacjHbBxjFAkX5BF7ymtJrdfal0nw9furNNXRVLzbXZUcwvABxoZTldOWgezul0KeNjcEyHMwSGRhQ/uaQvfHRb2Y/I7FGpmB/EBu9Q4ehYBeE69oHvH7foHNBPtsMwcQnbm+pn1zGoBe78auSwPKwBV6C+Q/Y11h22lZnU/cxFcEfnYHUtbqPPHlOZXInHjiSir4aWykEZ+U8/
1dZy8ty+Xt5+ka/d5NZd4/2aneyLUn/CEeXCwP2GK6nb6HVn70lXAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBACvebOo1jFsRevrjuAn0Vu+Qav/g1KcgNFSr7w0phcE8uxBxzyheo9vBU64Q/ckSI4BDcekt/JEMlDIMIvI3Ld7LkDo4xa3e+qjSwwfe2NrxbMSqSRWRUGk/hUuePhYUxcLNpfO3PfkHiKo9AX+r/UGM7
hy/B75ZIbHk6EsV23rTrW/muB8NAXKKwVpOQeUYMKhgMhh6aLE7Q4e4wrXQQNQSD0IXBh232HcYrEQorhwutkdc8fNMg/YwwknRSlA8n4T89vPjYLF6Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGrNf7sZdT5DYo=
REPLICA_VA.domain.com | MIIDNDCCAhwCCQDAYlr48T0k0DANBgkqhkiG9w0BAQUFADBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwHhcNMTUxMDI4MTIyOTI2WhcNND
MwMzE1MTIyOTI2WjBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKdM8FkNjK065gC0Dp9/3WdXNSrQnRpUfkTAiOlMvFpzbfuZfP8cW
aWjEPLri+yd+iUDLUYWBfxJajdnnG5NacjHbBxjFAkX5BF7ymtJrdfal0nw9furNNXRVLzbXZUcwvABxoZTldOWgezul0KeNjcEyHMwSGRhQ/uaQvfHRb2Y/I7FGpmB/EBu9Q4ehYBeE69oHvH7foHNBPtsMwcQnbm+pn1zGoBe78auSwPKwBV6C+Q/Y11h22lZnU/cxFcEfnYHUtbqPPHlOZXInHjiSir4aWykEZ+U8/
1dZy8ty+Xt5+ka/d5NZd4/2aneyLUn/CEeXCwP2GK6nb6HVn70lXAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBACvebOo1jFsRevrjuAn0Vu+Qav/g1KcgNFSr7w0phcE8uxBxzyheo9vBU64Q/ckSI4BDcekt/JEMlDIMIvI3Ld7LkDo4xa3e+qjSwwfe2NrxbMSqSRWRUGk/hUuePhYUxcLNpfO3PfkHiKo9AX+r/UGM7
hy/B75ZIbHk6EsV23rTrW/muB8NAXKKwVpOQeUYMKhgMhh6aLE7Q4e4wrXQQNQSD0IXBh232HcYrEQorhwutkdc8fNMg/YwwknRSlA8n4T89vPjYLF6Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGrNf7sZdT5DYo=
MwMzE1MTIyOTI2WjBcMQ8wDQYDVQQDEwZWTXdhcmUxFTATBgNVBAoTDFZNd2FyZSwgSW5jLjElMCMGA1UECxMcdkNBQyBTZWxmIFNpZ25lZCBDZXJ0aWZpY2F0ZTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKdM8FkNjK065gC0Dp9/3WdXNSrQnRpUfkTAiOlMvFpzbfuZfP8cW
aWjEPLri+yd+iUDLUYWBfxJajdnnG5NacjHbBxjFAkX5BF7ymtJrdfal0nw9furNNXRVLzbXZUcwvABxoZTldOWgezul0KeNjcEyHMwSGRhQ/uaQvfHRb2Y/I7FGpmB/EBu9Q4ehYBeE69oHvH7foHNBPtsMwcQnbm+pn1zGoBe78auSwPKwBV6C+Q/Y11h22lZnU/cxFcEfnYHUtbqPPHlOZXInHjiSir4aWykEZ+U8/
1dZy8ty+Xt5+ka/d5NZd4/2aneyLUn/CEeXCwP2GK6nb6HVn70lXAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBACvebOo1jFsRevrjuAn0Vu+Qav/g1KcgNFSr7w0phcE8uxBxzyheo9vBU64Q/ckSI4BDcekt/JEMlDIMIvI3Ld7LkDo4xa3e+qjSwwfe2NrxbMSqSRWRUGk/hUuePhYUxcLNpfO3PfkHiKo9AX+r/UGM7
hy/B75ZIbHk6EsV23rTrW/muB8NAXKKwVpOQeUYMKhgMhh6aLE7Q4e4wrXQQNQSD0IXBh232HcYrEQorhwutkdc8fNMg/YwwknRSlA8n4T89vPjYLF6Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGrNf7sZdT5DYo=
...
Note:
a. The fixed server.pem file should contain only 1 key and 1 certificate section.
It should be clear of newlines and incorrect end-of-line symbols. Like below sample:
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAynTPBZDYytOuYAtA6ff91nVzUq0J0aVH5EwIjpTLxac237mX
....
YoPfKhEPoH9zpZN3/6ttUaIBx+8W+j8dVLMkqYpGZREulXUUeafEoQ==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDNDCCAhwCCQDAYlr48T0k0DANBgkqhkiG9w0BAQUFADBcMQ8wDQYDVQQDEwZW
....
Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGr
Nf7sZdT5DYo=
-----END CERTIFICATE-----
b. The node_cert field in the database should contain only the VAMI certificate configured in server.pem, without extra BEGIN/END CERTIFICATE lines. It should also be clear of newlines and incorrect symbols.
It should be clear of newlines and incorrect end-of-line symbols. Like below sample:
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAynTPBZDYytOuYAtA6ff91nVzUq0J0aVH5EwIjpTLxac237mX
....
YoPfKhEPoH9zpZN3/6ttUaIBx+8W+j8dVLMkqYpGZREulXUUeafEoQ==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDNDCCAhwCCQDAYlr48T0k0DANBgkqhkiG9w0BAQUFADBcMQ8wDQYDVQQDEwZW
....
Gy0l0ouRhRI/rgMiYIOoAp4hdPh0HSz8wh0UWQpc6ZhYISc9EkSTB0VAWZIxaZGr
Nf7sZdT5DYo=
-----END CERTIFICATE-----
b. The node_cert field in the database should contain only the VAMI certificate configured in server.pem, without extra BEGIN/END CERTIFICATE lines. It should also be clear of newlines and incorrect symbols.
7. Login to the vRealize Automation Management Console and navigate to vRA Settings > Cluster page. Verify that all IaaS nodes have recent "Last Connected" time.
8. Retry the upgrade.
8. Retry the upgrade.
Feedback
Yes
No
Powered by
