vCenter update fails with "Test RPM transaction failed" when /storage/core or /storage/log filesystem is full

Products

• VMware vCenter Server

Issue/Introduction

When updating vCenter in the VAMI, the update/patching will hang and produce the error below

Installation failed
Test RPM transaction failed. Collect the logs for diagnostics

Environment

vCenter 7.0.x

Cause

The issue is caused by a lack of free space in one of the vCenter /storage partitions. For example, /storage/core or /storage/log

The amount of free space can be checked by running the command below in an SSH on the vCenter:

# df -h

 

You can also check the update_microservices.log or PatchRunner.log to identify which /storage partition is full 

update_microservices.log

[YYYY-MM-DDTHH:MM:SS] - 36294 -     update_functions::      runCommandAndCheckResult: 373 -    DEBUG - runCommandAndCheckResult failed: '\tinstalling package vmware-e
sx-netdumper-7.0.3.02100-0.0.13975548.i386 needs 4KB on the /storage/core filesystem\n'

[YYYY-MM-DDTHH:MM:SS] - 36294 -         task_manager::                        update:  80 -    DEBUG - UpdateTask: status=RESUMABLE, progress=40, message={'id': 'com.vmware.appliance.rpm_transaction_failed', 'default_message': 'Test RPM transaction failed. Collect the logs for diagnostics', 'args': []}
[YYYY-MM-DDTHH:MM:SS] - 36294 -           update_b2b::                    preInstall:2349 -    DEBUG - Returning from preInstall(), failure in preinstall

 

PatchRunner.log

[YYYY-MM-DDTHH:MM:SS] - 11761 -        update_script::           collectRequirements: 645 -    DEBUG - Transform patch runner requirements results: {
    "requirements": {
        "requiredDiskSpace": {
            "/storage/core": 22.60932327270508,
            "/storage/seat": 19.084375
        },
        "rpmIgnoreList": [],
        "questions": [],
        "rebootRequired": false
    },
    "patchInfo": {
        "timeToInstall": 0,
        "patchSummary": null
    },
    "mismatches": [
        {
            "text": {
                "id": "patch.rpm.no.storage.install.error.text",
                "translatable": "Insufficient storage space on partition \"%(0)s\".",
                "args": [
                    "/storage/log"
                ],
                "localized": "Insufficient storage space on partition \"/storage/log\"."
            },
            "description": {
                "id": "patch.rpm.no.storage.install.error.description",
                "translatable": "As part of the patch product rpms need to be installed on the system, however this won't be possible due to insufficient storage.",
                "localized": "As part of the patch product rpms need to be installed on the system, however this won't be possible due to insufficient storage."
            },
            "resolution": {
                "id": "patch.rpm.no.storage.install.error.resolution",
                "translatable": "Ensure that on partition \"%(0)s\" there is at least %(1)s MB free.",
                "args": [
                    "/storage/log",
                    "1"
                ],
                "localized": "Ensure that on partition \"/storage/log\" there is at least 1 MB free."
            },
            "problemId": null,
            "severity": "error",
            "relatedUserDataId": null
        }
    ]
}

 

Resolution

To resolve this issue, clear space in the impacted /storage partition by removing old files or dump files that are no longer required.

Example: 

rm vmkernel-zdump.x

Alternatively dumpfile can be moved to an alternate location should they wish to be saved 

mv vmkernel-zdump.x /tmp

Additionally, we can delete any of the older core dump files. Below is an example of one of them.

After clearing space in the impacted /storage partition resume the vCenter upgrade.

Additionally refer the articles below to help identify which partition is full and how to clean up.

https://knowledge.broadcom.com/external/article/318953/