Using RDU the "vCenter upgrade pre-validaton" fails in SDDC UI

search cancel

Using RDU the "vCenter upgrade pre-validaton" fails in SDDC UI

book

Article ID: 418500

calendar_today

Updated On:

Products

VMware SDDC Manager VMware vCenter Server 8.0

Issue/Introduction

In SDDC, /var/log/vmware/vcf/lcm/lcm.log

YYYY-MM-DDTmm:ss.684+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=1f##60-####-4cf6-####-89####7b0dd,resourceType=VCENTER,resourceId=41###721-###-4d2b-####-a84f###5a715,bundleElementId=8a###a5d-####-4MMa-####-ba6##09f055] [c.v.e.s.l.p.i.v.VCenterRDUUpgrader,Upgrade-5] RDU Error object : {"errorType":"ERROR","messages":[{"id":"vcenter.deployment.migration_upgrade.operation_suspended_error","defaultMessage":"The operation has been suspended due to a service\u0027s restart.","args":[],"localized":"The operation has been suspended due to a service\u0027s restart."},{"id":"vcenter.deployment.migration_upgrade.operation_suspended_resolution","defaultMessage":"Execute the operation again","args":[],"localized":"Execute the operation again"}]}

YYYY-MM-DDTmm:ss:56.685+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=1f##60-####-4cf6-####-89####7b0dd,resourceType=VCENTER,resourceId=41###721-###-4d2b-####-a84f###5a715,bundleElementId=8a###a5d-####-4MMa-####-ba6##09f055] [c.v.e.s.l.p.i.v.VCenterRDUUpgrader,Upgrade-5] RDU Notifications : {"warnings":[{"id":"vcenter.deployment.migration_upgrade.failed_to_delete_target_error","time":"YYYY-MM-DDTmm:50:ss.68####57Z[UTC]","message":{"id":"vcenter.deployment.migration_upgrade.failed_to_delete_target_error","defaultMessage":"Failed to delete the target appliance.","localized":"Failed to delete the target appliance."},"resolution":{"id":"vcenter.deployment.migration_upgrade.failed_to_delete_vm.resolution","defaultMessage":"Manually delete VM \<FQDN_OF_vCenter>-upgrade-target-appliance_693##fd-###-4150-####-c56###595610\u0027.","params":{"0":{"s":"vcenter-upgrade-target-appliance_69####d-8##d-###-ae41-c56####610"}},"localized":"Manually delete VM \<FQDN_OF_vCenter>-upgrade-target-appliance_69####d-8##d-###-ae41-c56####610\u0027."}}]}

In vCenter server, /var/log/vmware/applmgmt/Patchrunner.log

YYYY-MM-DDThh:mm:ss.624Z INFO vmware_b2b.patching.data.model Additional info from UserData received : sourceManagementAddress, sourceManagementPassword, sourceManagementPort, sourceManagementUsername, targetManagementAddress, targetManagementPassword, targetManagementPort, targetManagementUsername, targetPlacement
YYYY-MM-DDThh:mm:ss.632Z vmafd-patch:Validation ERROR vmware_b2b.patching.executor.hook_executor Component's script 'vmafd-patch' is not regular python module
Traceback (most recent call last):
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 54, in executeHook
  File "/usr/lib/python3.10/imp.py", line 297, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'vmafd-patch'
YYYY-MM-DDThh:mm:ss.637Z ERROR __main__ Validate vCSA components got unhandled exception
Traceback (most recent call last):
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 115, in executeHook
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 135, in get_nowait
    return self.get(False)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 116, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/py/vmware_b2b/patching/phases/validator.py", line 197, in validate
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/py/vmware_b2b/patching/phases/validator.py", line 73, in _validateComponents
patch_errors.InternalError: Cannot execute the component hook

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/PatchRunner.py", line 376, in main
  File "/storage/lifecycle/vlcm/current/stageDir2381650550/scripts/patches/py/vmware_b2b/patching/phases/validator.py", line 220, in validat
  File "/usr/lib/python3.10/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)

From vCenter server, /var/log/vmware/vlcm.log

YYYY-MM-DDThh:mm:ss.601Z info vlcm [locerrors/vapierrors.go:63] [opID=6910bb81] Error type: ERROR
YYYY-MM-DDThh:mm:ss.601Z info vlcm [ndu/executor.go:207] State Initialize is already SUCCEEDED
YYYY-MM-DDThh:mm:ss.601Z info vlcm [ndu/executor.go:207] State Precheck is already FAILED
YYYY-MM-DDThh:mm:ss.601Z info vlcm [ndu/executor.go:228] Autocancel will be used for this run
YYYY-MM-DDThh:mm:ss.601Z info vlcm [locerrors/vapierrors.go:36] [opID=6910bf5e] Parsing vAPI error json to real object
YYYY-MM-DDThh:mm:ss.601Z info vlcm [locerrors/vapierrors.go:45] [opID=6910bf5e] Error type: map[Data:null ErrorType:"ERROR" Messages:[{"Id":"vcenter.deployment.migration_upgrade.operation_suspended_error","DefaultMessage":"The operation has been suspended due to a service's restart.","Args":[],"Params":null,"Localized":"The operation has been suspended due to a service's restart."},{"Id":"vcenter.deployment.migration_upgrade.operation_suspended_resolution","DefaultMessage":"Execute the operation again","Args":[],"Params":null,"Localized":"Execute the operation again"}]]
YYYY-MM-DDThh:mm:ss.601Z info vlcm [locerrors/vapierrors.go:63] [opID=6910bf5e] Error type: ERROR

YYYY-MM-DDThh:mm:ss.601Z warning vlcm [vcrestlib/helper.go:192] Request to service failed; POST, url: https://<FQDN_OF_VCENTER>:443/rest/vcenter/tokenservice/token-exchange, Code: 401, Body: '{"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","messages":[{"args":[],"default_message":"Authentication required.","id":"com.vmware.vapi.endpoint.method.authentication.required"}],"challenge":"Basic realm=\"VAPI endpoint\",SIGN realm=bab9de############5763d6fcff078b4089c4132f12f526fad,service=\"VAPI endpoint\",sts=\"https://<FQDN_OF_VCENTER>/sts/STSService/vsphere.local\""}}'
YYYY-MM-DDThh:mm:ss.601Z info vlcm [vapiurl/url.go:118] Returning local URL: https://<FQDN_OF_VCENTER>:443/openidconnect/jwks/vsphere.local
YYYY-MM-DDThh:mm:ss.636Z info vlcm [auth/handlers.go:288] Fetching token issuer
YYYY-MM-DDThh:mm:ss.651Z info vlcm [exec/exec.go:123] Run command [/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost] (duration: 15ms)
...
...
...
YYYY-MM-DDThh:mm:ss.761Z info vlcm [upgraderepo/downloadmanifest.go:29] [opID=5554a97f-###-40a3-###-837####e37fd-####] Download manifest
YYYY-MM-DDThh:mm:ss.761Z info vlcm [upgraderepo/downloadfile.go:41] [opID=5554a97f-###-40a3-###-837####e37fd-####] Download file [https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/4###67721-####-4d2b-b####045a715/iso/repos/patcher_repo/package-pool/rpm-manifest.json] to [/tmp/custom-repo-updates-1342364941/rpm-manifest.json]
YYYY-MM-DDThh:mm:ss.761Z info vlcm [serviceconfig/proxy.go:67] [opID=5554a97f-###-40a3-###-837####e37fd-####] Proxy is disabled
YYYY-MM-DDThh:mm:ss.848Z error vlcm [exec/exec.go:115] [opID=5554a97f-###-40a3-###-837####e37fd-####] Run command [/usr/bin/curl --capath /etc/ssl/certs -L -C - --retry 0 --verbose --fail -w vlcm_http_code: %{http_code}\n --connect-timeout 15 --tlsv1.2 --tls-max 1.3 --tls13-ciphers TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256 --ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AE
S128-GCM-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256 --curves prime256v1:secp384r1:secp521r1 -o /tmp/custom-repo-updates-1342364941/rpm-manifest.json https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/41b67721-6241-4d
2b-b04f-a84f1045a715/iso/repos/patcher_repo/package-pool/rpm-manifest.json] failed (duration: 87ms)
YYYY-MM-DDThh:mm:ss.849Z error vlcm [logger/multiline.go:17] [opID=5554a97f-###-40a3-###-837####e37fd-####] curl failed
   025-11-09T17:39:25.849Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] ERR:001 exit status 22
YYYY-MM-DDThh:mm:ss.849Z error vlcm [logger/multiline.go:51] [opID=5554a97f-###-40a3-###-837####e37fd-####] ERR:002-
YYYY-MM-DDThh:mm:ss.849Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:001   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
...
...
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:042 * The requested URL returned error: 404
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:043   0   146    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:044 * Closing connection 0
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:045 * TLSv1.3 (OUT), TLS alert, close notify (256):
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:046 curl: (22) The requested URL returned error: 404
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:43] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:047 vlcm_http_code: 404
YYYY-MM-DDThh:mm:ss.851Z error vlcm [logger/multiline.go:51] [opID=5554a97f-###-40a3-###-837####e37fd-####] OUT:048-
YYYY-MM-DDThh:mm:ss.851Z info vlcm [curl/curl.go:283] [opID=5554a97f-###-40a3-###-837####e37fd-####] Got status code [404], retries should stop
YYYY-MM-DDThh:mm:ss.851Z info vlcm [retry/retry.go:115] [opID=5554a97f-###-40a3-###-837####e37fd-####] retry: done after 1 retries with error
YYYY-MM-DDThh:mm:ss.851Z error vlcm [upgraderepo/downloadfile.go:77] [opID=5554a97f-###-40a3-###-837####e37fd-####] Download file [https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/4###67721-####-4d2b-b####045a715/iso/repos/patcher_repo/package-pool/rpm-manifest.json] failed: download: exit status 22
YYYY-MM-DDThh:mm:ss.851Z error vlcm [upgraderepo/downloadmanifest.go:35] [opID=5554a97f-###-40a3-###-837####e37fd-####] Download manifest [https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/4###67721-####-4d2b-b####045a715/iso/repos/patcher_repo/package-pool/rpm-manifest.json] failed: download: exit status 22
YYYY-MM-DDThh:mm:ss.852Z error vlcm [updateprovider/custom_repo.go:278] [opID=5554a97f-###-40a3-###-837####e37fd-####] Error while downloading the manifest file from url https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/4###67721-####-4d2b-b####045a715/iso/repos/patcher_repo/package-pool/rpm-manifest.json to destination /tmp/custom-repo-updates-1342364941: download: exit status 22
YYYY-MM-DDThh:mm:ss.852Z info vlcm [updateprovider/default_repo.go:225] [opID=5554a97f-###-40a3-###-837####e37fd-####] Cleaning up temp directory: /tmp/custom-repo-updates-1342364941
YYYY-MM-DDThh:mm:ss.852Z error vlcm [updateprovider/custom_repo.go:184] [opID=5554a97f-###-40a3-###-837####e37fd-####] Error while downloading updates from custom URL https://<FQDN_OF_SDDC>/vmware/vcf/bundle/VMware/vRack/lcm/69###2fd-8##d-4##0-a##41-c5###95610/4###67721-####-4d2b-b####045a715/iso/repos/patcher_repo for file rpm-manifest.json: download: exit status 22

In the vCenter server, /var/log/vmware/vapi/endpoint/endpoint.log

YYYY-MM-DDThh:mm:ss.044Z | ERROR | sso6                      | SoapBindingImpl                | 0####cf-c##8-4##c-9##4c-9a9###71 | Error communicating to the remote server http://localhost:1080/sts/system-STSService
com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 503: Service Unavailable
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.checkStatusCode(HttpTransportPipe.java:310) ~[?:1.8.0_392]
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.createResponsePacket(HttpTransportPipe.java:259) ~[?:1.8.0_392]
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:217) ~[?:1.8.0_392]
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130) ~[?:1.8.0_392]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
YYYY-MM-DDThh:mm:ss.050Z | WARN  | sso6                      | StsFacade                      | 0####cf-c##8-4##c-9##4c-9a9###71 | Call to STS failed 
com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server http://localhost:1080/sts/system-STSService
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:996) ~[libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) ~[libwstclient.jar:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]

In the vCenter server, /var/log/vmware/sso/vmware-identity-sts-default.log

YYYY-MM-DDThh:mm:ss.852 ERROR sts-default[32:Thread-8] [CorId=55fb78eb-2a40-4967-b05f-9b5321d52237 OpId=] [com.vmware.identity.idm.server.provider.PooledLdapConnectionFactory] com.vmware.identity.interop.ldap.ServerDownLdapException: Can't contact LDAP server\nLDAP error [code: -1]
YYYY-MM-DDThh:mm:ss.852 ERROR sts-default[32:Thread-8] [CorId=55fb78eb-2a40-4967-b05f-9b5321d52237 OpId=] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldap://<FQDN_OF_VCENTER>:389] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use
 secondary URIs, if applicable
YYYY-MM-DDThh:mm:ss.852 ERROR sts-default[32:Thread-8] [CorId=55fb78eb-2a40-4967-b05f-9b5321d52237 OpId=] [com.vmware.identity.idm.server.provider.BaseLdapProvider] com.vmware.identity.interop.ldap.ServerDownLdapException: Can't contact LDAP server\nLDAP error [code: -1]
YYYY-MM-DDThh:mm:ss.852ERROR sts-default[32:Thread-8] [CorId=55fb78eb-2a40-4967-b05f-9b5321d52237 OpId=] [com.vmware.identity.idm.server.IdentityManager] Failed to find solution user [machine-01####7c-0##5-4##2-8##9-32a####344d] in tenant [vsphere.local]

Environment

VMware Cloud Foundation 5.X

Cause

Old sts cert is retained in the STS_INTERNAL_SSL_CERT store.

Resolution

To resolve the issue, delete STS_INTERNAL_SSL_CERT from VESC store.

Refer below mentioned KB and follow the instruction to remove the STS STS_INTERNAL_SSL_CERT

Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH

Retry patching of the vCenter via SDDC UI (NDU method)

Feedback

thumb_up Yes

thumb_down No