During update to vCenter server 7.0 U3+, patch update fails while installing vmware-autodeploy rpm
Article ID: 381956
Updated On:
Products
Issue/Introduction
-
Patching vCenter Server to 7.0 U3+ fails while installing "vmware-autodeploy-7.0.3.#####-########.noarch.rpm"
-
Error message on VAMI "Installation of one or more RPMs failed".
- Log file
/var/log/vmware/applmgmt/update_microservice.logwill show entries similar to:
24811 - functions:: runScript: 206 - INFO - Execution history of RPM script /storage/seat/software-updateikqf6d4k/stage/pre-vmware-autodeploy-7.0.3.#####-########.noarch.rpm.sh is
[+ groupadd -f -r deploy
+ useradd -r -g deploy -d /var/lib/rbd -c 'Deploy User' -s /sbin/nologin -u 1019 deploy
+ :
+ rm -rf /storage/service-state/rbd
]
[]
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {UPDATE meta SET _prescriptendtime = DATETIME('now'), _statusid = 180 WHERE _name = ? } with parameters ('vmware-autodeploy-7.0.3.#####-########.noarch.rpm',)
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {INSERT INTO progress VALUES(NULL, ?, DATETIME('now'), ?, ?, ?)} with parameters ('install', 'Pre-script of RPM vmware-autodeploy-7.0.3.#####-########.noarch.rpm executed successfully', 26, 70)
24811 - rpmfunctions:: runPreScriptForRpm: 423 - INFO - Pre-script /storage/seat/software-updateikqf6d4k/stage/pre-vmware-autodeploy-7.0.3.#####-########.noarch.rpm.sh of RPM vmware-autodeploy-7.0.3.#####-########.noarch.rpm executed successfully
24811 - rpmfunctions:: installRpm: 558 - INFO - Installing RPM {/storage/seat/software-updateikqf6d4k/stage/vmware-autodeploy-7.0.3.#####-########.noarch.rpm}
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {UPDATE meta SET _installstarttime = DATETIME('now'), _statusid = 200 WHERE _name = ? } with parameters ('vmware-autodeploy-7.0.3.#####-########.noarch.rpm',)
24811 - functions:: runInstallRpmTransaction: 94 - INFO - Installing RPM [/storage/seat/software-updateikqf6d4k/stage/vmware-autodeploy-7.0.3.#####-########.noarch.rpm]
24811 - functions:: runCommand: 48 - DEBUG - Invoking command [/bin/bash --login -c /bin/rpm -Uhv --nopre --nopost --nodeps --force '/storage/seat/software-updateikqf6d4k/stage/vmware-autodeploy-7.0.3.#####-########.noarch.rpm']
24811 - functions:: runCommand: 65 - DEBUG - Process completed. Return code [1], stdout [Preparing... ########################################
Updating / installing...
vmware-autodeploy-7.0.3.#####-################################################
], stderr [error: unpacking of archive failed on file /var/lib/rbd/cache: cpio: File from package already exists as a directory in system
error: vmware-autodeploy-7.0.3.#####-########.noarch: install failed
error: vmware-autodeploy-7.0.3.#####-########.noarch: erase skipped
]
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {UPDATE meta SET _installendtime = DATETIME('now'), _statusid = 220 WHERE _name = ? } with parameters ('vmware-autodeploy-7.0.3.#####-########.noarch.rpm',)
24811 - rpmfunctions:: installRpm: 565 - ERROR - Install RPM transaction failed.
24811 - rpmfunctions:: installRpm: 571 - ERROR - Failed to install RPM
24811 - rpmfunctions:: installRpm: 572 - ERROR - RPM installation failed
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {UPDATE meta SET _statusid = 280 WHERE _name = ? } with parameters ('vmware-autodeploy-7.0.3.#####-########.noarch.rpm',)
24811 - rpmfunctions:: installPendingRpms: 468 - ERROR - RPM {vmware-autodeploy-7.0.3.02200-14372979.noarch.rpm} installation failed
24811 - update_b2b:: executeScriptsAndInstallRpms:2142 - ERROR - Install of one or more RPMs failed. Halting patching
24811 - update_b2b:: executeScriptsAndInstallRpms:2143 - ERROR - RPM installation failed
24811 - task_manager:: update: 80 - DEBUG - UpdateTask: status=RESUMABLE, progress=75, message={'id': 'com.vmware.appliance.plain_message', 'default_message': '%s', 'args': ['Installation of one or more RPMs failed']}
24811 - update_b2b:: installRpms:2387 - ERROR - Installation of one or more RPMs failed. Patching cannot proceed
24811 - update_b2b:: installRpms:2388 - ERROR - RPM installation failed
24811 - dbfunctions:: executeDML: 54 - DEBUG - Executing Query {INSERT INTO progress VALUES(NULL, ?, DATETIME('now'), ?, ?, ?)} with parameters ('failed', 'Installation of one or more RPMs failed. Patching cannot proceed. Installation failed', 100, 95)
24811 - update_b2b:: install:2674 - ERROR - Installation of RPMs phase failed.
Environment
- vCenter Server 7.0.x
Cause
This issue is caused due to invalid 'cache' symlink created under /var/lib/rbd.
Resolution
To resolve this issue, remove and recreate symlink. In this specific scenario symlink created with junk characters 'cache-@#4321-56#8'.
Important process to be followed before preforming the resolution steps below :
- Please take offline snapshot of the vCenter VM from the respective ESXi host it is currently running on, if the vCenter is in the same Single Sign-on Domain, replicating in Enhanced Linked Mode (ELM), there is high potential of corruption of the domain if snapshots of the appliances are taken while they are in running state.
- Use of offline snapshots in ELM deployments is very strongly recommended for a safe rollback point. This means all appliances should be gracefully shut down, and snapshots need to be taken while the VCSAs are in powered off state (at the same time).
- If any change must be reverted, restore all of the nodes in the ELM deployment to this offline/consistent snapshot state. Only start powering the restored nodes back on after all of them have been restored from the snapshots.
Resolution steps :
1. Connect to vCenter Server for which the issue is encountered through SSH as the root user.
2. Stop the vmware-rbd-watchdog service
- service-control --stop vmware-rbd-watchdog
3. Navigate to /var/lib/rbd/ and remove invalid symlink file.
- rm -rf "cache-@#4321-56#8"
4. Re-create symlink by running the following command.
- ln -s /storage/autodeploy/cache cache
5. Start autodeply service and retry patch update.
- service-control --start vmware-rbd-watchdog
Feedback
