vCenter upgrade failing during stage 2 "internal error occurs during export of vmware service lifecycle"
search cancel

vCenter upgrade failing during stage 2 "internal error occurs during export of vmware service lifecycle"

book

Article ID: 381494

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

When upgrading vCenter there is failure during stage 2 stating:

internal error occurs during export of vmware service lifecycle

When checking the logs on source vCenter node in /var/log/vmware/upgrade you see:

in export-upgrade-runner.log

YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export: 2024-10-23T13:26:46.117Z ERROR __main__ Upgrade Phase 'vmon:Export' failed. Exception: Expecting value: line 1 column 1 (char 0)
YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/componentPhaseLauncher.py", line 461, in main
YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export:     executionResult = systemExtension(exeContext)
YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/libs/sdk/extensions.py", line 94, in __call__
YYYY-MM-DDTHH:MM:SS.24Z INFO upgrade.states.component_states vmon:Export:     result = self.extension(*args)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/libs/sdk/extensions.py", line 110, in _func
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     return func(*args)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 221, in vmon_export
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     _export_services_runstate_cfg(context)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 160, in _export_services_runstate_cfg
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     services_runstate_cfg = _read_services_runstate_cfg(context)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 137, in _read_services_runstate_cfg
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     service_name = _parse_vmon_cfg(servicecfg_filepath, context).get(
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 54, in _parse_vmon_cfg
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     return json.loads(data)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     return _default_decoder.decode(s)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:   File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export:     raise JSONDecodeError("Expecting value", s, err.value) from None
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export: STDERR: /tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/componentPhaseLauncher.py:86: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export: STDERR:   import imp
YYYY-MM-DDTHH:MM:SS.25Z INFO upgrade.states.component_states vmon:Export: Script completed for 1.0084128379821777 secs with return-code='1', and executionId=b8cf8e2b-7e36-44c2-8544-384deda95b62
YYYY-MM-DDTHH:MM:SS.26Z ERROR upgrade.states.component_states vmon:Export: Remote script failed with an error [InternalError()]
YYYY-MM-DDTHH:MM:SS.26Z ERROR upgrade.states.component_states vmon:Export: failed with internal error. For details take a look at Export_com.vmware.vmon_2024_10_23_14_26.log.
YYYY-MM-DDTHH:MM:SS.35Z WARNING wf.wf_processor State Export com.vmware.vmon has failed.
YYYY-MM-DDTHH:MM:SS.35Z ERROR wf.wf_processor Workflow execution has failed

In Export_com.vmware.vmon_2024_10_23_14_26.log

YYYY-MM-DDTHH:MM:SS.113Z INFO extensions Executing upgrade phase '<module 'vmon' from '/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py'>:Export' with context {'sourceVcVersion': '7.0', 'sourceSsoVersion': '7.0', 'sourcePlatform': 'Linux', 'destinationVcVersion': '8.0', 'destinationVcVersionFull': '8.0.3.00300', 'destinationSsoVersion': '8.0', 'destinationPlatform': 'Linux', 'locale': 'en', 'systemExportDir': '/usr/lib/vmware/cis_upgrade_runner/cis-export-folder/system-data', 'resourceDir': '/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/upgrade_working_directory', 'devMode': False, 'exportDir': '/usr/lib/vmware/cis_upgrade_runner/cis-export-folder/vmon', 'sourceSsoUsername': None, 'sourceSsoPassword': 'CENSORED', 'vcUsername': '[email protected]', 'vcPassword': 'CENSORED', 'vcdb.migrateSet': 'core', 'sourceManagementAddress': 'vCenter_FQDN', 'sourceManagementUsername': 'root', 'sourceManagementPassword': 'CENSORED', 'sourceManagementPort': 443, 'sourceManagementThumbprint': '', 'targetManagementAddress': 'vCenter_FQDN', 'targetManagementUsername': 'root', 'targetManagementPassword': 'CENSORED', 'targetManagementPort': 443, 'targetManagementThumbprint': '', 'logDir': '/var/log/vmware/upgrade'}.
YYYY-MM-DDTHH:MM:SS.117Z ERROR __main__ Upgrade Phase 'vmon:Export' failed. Exception: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/componentPhaseLauncher.py", line 461, in main
    executionResult = systemExtension(exeContext)
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/libs/sdk/extensions.py", line 94, in __call__
    result = self.extension(*args)
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/libs/sdk/extensions.py", line 110, in _func
    return func(*args)
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 221, in vmon_export
    _export_services_runstate_cfg(context)
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 160, in _export_services_runstate_cfg
    services_runstate_cfg = _read_services_runstate_cfg(context)
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 137, in _read_services_runstate_cfg
    service_name = _parse_vmon_cfg(servicecfg_filepath, context).get(
  File "/tmp/vmware-upgrade-temp-dirOFTg1Hd8HA/tmpyxQOmLfuTe/payload/component-scripts/vmon/__init__.py", line 54, in _parse_vmon_cfg
    return json.loads(data)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Environment

  • VMware vCenter Server 8.x
  • VMware vCenter Sever 7.x

Cause

This occurs if there is any foreign file located at /etc/vmware/vmware-vmon/svcCfgfiles in the vCenter.

This location should contain only default json files and any .bak or non json files located hear with disrupt the upgrade process.

Resolution

Removal of foreign non standard json files from the directory solves the issue.

The following is an example of the default files located in /etc/vmware/vmware-vmon/svcCfgfiles for a vCenter VMware VirtualCenter 7.0.3 build-24026615

Any foreign files such as for example a .bak file with same leading name as one of the standard jsons which may have been left over from previous troubleshooting or error files/logs.

When checking for comparison compare the location against a healthy working vCenter of same version.

Powered by Wolken Software