During VCF upgrade from 4.2.1 to 4.4.x fails on vRLI/vRA with error "Failed to create snapshots for product vrli from environment vrops and vrli with error [{"messageid" "Invalid credentials provided in product while logging to VC"

Products

• VMware Aria Suite
• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• During upgrade snapshot for Log Insight fails with Error Messages observed: Failed to create snapshots for product vrli from environment vrops and vrli with error [{"messageid" "Invalid credentials provided in product while logging to VC
• Upgrade logs shows entries as below

 "-[{"name":"Upgrade - VRLI "FQDN"","description":"Upgrade -VRLI         FQDN","status":"FAILED","errors":       [{"errorCode":"COMPLETED_WITH_FAILURE","message":"Failed to create snapshot for product vrli from environment cross-region-env with error [{\"messageId\":\"LCMCOMMON80006\",\"message\":\"Invalid credentials provided in Product while logging to VC.\",\"eventId\":\"a82a20c8-a53a-43c9-ac4b-debcb8a41460\",\"retry\":true,\"exceptionMessage\":\"Cannot complete login due to an incorrect user name or password.\",\"exceptionStackTrace\":\"(vim.fault.InvalidLogin)

 

• lcm.log shows similar messages as below

2022-04-06T23:28:42.135+0000 INFO [vcf_lcm,51273c7eed54d41d,5410] [c.v.e.sddc.lcm.audit.AuditScheduler,pool-7-thread-3] Last upgrade status for software type VRA and resourceId 9f2065d5-c41f-42dd-81c0-80a11f712fe6:vra is: {"id":"36","bundleId":"f607104a-fffc-4e8a-a579- 7d4c84bfac76","bundleElementId":"1f52e42e-2808-44cf-a0ec-962445dd4fb0","bundleElementOrder":1,"resourceType":"VRA","resourceId":"9f2065d5-c41f-42dd-81c0-80a11f712fe6:vra","upgradeId":"9ce0a114-d0df-415e-adc4-d13a40365636","upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"VREALIZE_UPGRADE","errorCode":"COMPLETED_WITH_FAILURE","errorDescription":"Timed out while checking request vraUpgradePre1649273730507","metadata":"\nVerify you can log and perform REST API calls to the vRealize Suite Lifecycle Manager appliance at ********.\nCheck for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm/.\nManually revert the vRealize Automation virtual machines to the snapshot with prefix vRa_LCM_Upgrade_Backup.\n Please retry the upgrade once the upgrade is available again.","referenceToken":"JHPL1N"},"startTime":1649273416904,"endTime":1649286359177,"numRetriesLeft":2,"upgradeStage":"[\n \"VREALIZE_UPGRADE_PRECHECK\",\n \"VREALIZE_UPGRADE_PREUPGRADE_CHECK\",\n \"VREALIZE_UPGRADE_CREATE_SNAPSHOT\",\n \"VREALIZE_UPGRADE_PREPARATION\",\n \"VREALIZE_UPGRADE_TRANSFER_PACKAGES\",\n \"VREALIZE_UPGRADE\"\n]","backupInfoList":[],"upgradeStages":"[\n \"VREALIZE_UPGRADE_PRECHECK\",\n \"VREALIZE_UPGRADE_PREUPGRADE_CHECK\",\n \"VREALIZE_UPGRADE_PREPARE_BACKUP\",\n \"VREALIZE_UPGRADE_CREATE_SNAPSHOT\",\n \"VREALIZE_UPGRADE_PREPARATION\",\n \"VREALIZE_UPGRADE_TRANSFER_PACKAGES\",\n \"VREALIZE_UPGRADE\",\n \"VREALIZE_UPGRADE_POST_CONFIGURATION\",\n \"VREALIZE_UPGRADE_POSTCHECK\",\n \"VREALIZE_UPGRADE_CLEAN_BACKUP\"\n]","metadata":"{\n \"upgradeStatus\": \"INPROGRESS\",\n \"primitiveSpec\": {\n \"upgradeId\": \"9ce0a114-d0df-415e-adc4-d13a40365636\",\n \"resourceId\": \"9f2065d5-c41f-42dd-81c0-80a11f712fe6:vra\",\n \"resourceType\": \"VRA\",\n \"bundleElementOrder\": 1,\n \"patchElement\": {\n \"id\": \"1f52e42e-2808-44cf-a0ec-962445dd4fb0\",\n \"type\": \"VRA\",\n \"vendor\": \"VMware\",\n \"version\": \"8.5.0-18472703\",\n \"previousVersion\": \"8.4.1-18054500\",\n \"file\": \"f607104a-fffc-4e8a-a579-7d4c84bfac76/bundle-47700/vra-upgrade-18527839.tar\",\n \"checksum\": \"e2fc1ff243f1b113a4153f254a08f3460b79e768d8023a3c4abbb5953f950ffd\"\n },\n \"patchElements\": [],\n \"slaType\": \"SLOW\",\n \"endTime\": 1649291416901\n },\n \"stage\": \"VREALIZE_UPGRADE\",\n \"stagePrepared\": true,\n \"repository\": {\n \"stringDataByTypeMap\": {\n \"BACKUP_SNAPSHOT_PREFIX\": \"vRa_LCM_Upgrade_Backup\",\n \"CREATE_SNAPSHOT_REQUEST_ID\": \"0c601cae-e280-461d-b3e1-873d9cc9fd44\",\n \"REPOSITORY_URL\": \"/data/productlinks/vra/8.5.0-18472703/patch/Prelude_VA-8.5.0.18046-18472703-updaterepo.iso\",\n \"START_PRECHECK_REQUEST_ID\": \"614f60f0-1613-402f-bd7c-201a80900a43\",\n \"START_UPGRADE_REQUEST_ID\": \"vraUpgradePre1649273730507\"\n }\n }\n}","stages":[]}

 

Note:The preceding log excerpts are only examples. Date,time and environmental variables may vary depending on your environment.

Environment

VMware vRealize Log Insight 8.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 4.2.x
VMware Cloud Foundation 4.4.x

Cause

The service account in this case is [email protected] was changed manually.Service accounts that are automatically generated during bring-up, host commissioning, and workload creation. Service accounts have a limited set of privileges and are created for communication between products. Passwords for service accounts are randomly generated by SDDC Manager. Do not change the passwords for system accounts and the [email protected] account outside SDDC Manager. This can break your VMware Cloud Foundation system

Resolution

Update the credentials of service accounts by remediating the password.

Please check the VCF documentation for more information -

Additional Information

Impact/Risks:

Password remediation syncs the password of the account stored in the SDDC Manager with the updated password in the component.