vCenter server upgrade from 8.x to 9.0 fails at Stage 2 at 20% during "import copied data to target vCenter Sever"
search cancel

vCenter server upgrade from 8.x to 9.0 fails at Stage 2 at 20% during "import copied data to target vCenter Sever"

book

Article ID: 422291

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
1) The vCenter Server upgrade process fails during Stage 2. An error message is displayed: "An error occurred while importing copied data to the target vCenter Server."
2) The upgrade may complete successfully, but the vCenter Server remains in a "hung" or unresponsive state.
3) In the vSphere Client, hosts within the cluster report an operational status of "vSphere HA Agent Unreachable."

vpxd.log:
YYYY-MM-DD info vpxd[794669] [Originator@6876 sub=NfcLib opID=DasRetryMgrPeriodic-########-####-6d73] [NFC INFO]netSendLatencyStats: count 118 min/max/avg 0/421/181 usec
YYYY-MM-DD info vpxd[794669] [Originator@6876 sub=NfcLib opID=DasRetryMgrPeriodic-########-####-6d73] [NFC INFO]netRecvLatencyStats: count 7 min/max/avg 6/113496/54127 usec
YYYY-MM-DD info vpxd[794375] [Originator@6876 sub=vmomi.soapStub[208]] SOAP request returned HTTP failure; <<cs p:00007f1250004060, TCP:localhost:1080>, /sms/sdk>, method: queryProvider; code: 500(Internal Server Error); fa
ult: (sms.fault.QueryExecutionFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.pbm.pbmFault.locale",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "summary",
-->                value = "Upgrade is ongoing."
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ]
-->    msg = "Received SOAP response fault from [<<cs p:00007f1250004060, TCP:localhost:1080>, /sms/sdk>]: queryProvider
--> Upgrade is ongoing."
YYYY-MM-DD warning vpxd[794375] [Originator@6876 sub=MoCluster] Failed to get storage providers from SMS; N3Sms5Fault19QueryExecutionFault9ExceptionE(Fault cause: sms.fault.QueryExecutionFault
--> [context]zKq7AVECAQAAAA7SfAEXdnB4ZAAAyzdIbGlidm1hY29yZS5zbwAArDA3AA9VOAHV5w1saWJzbXMtdHlwZXMuc28AAV72DQIoWSFsaWJ2bW9taS5zbwACcIUhAov8IINxv14BdnB4ZACDtA5UAoOTFFQCAkfVGgFIPQ2DqFIKAoMrVgoCg+B7qwKDn3yrAgD/YC0ABPwsAA9VLQDrKkYExJAIbGliYy5zby42AARskRA=[/context]
YYYY-MM-DD info vpxd[794700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-456531@14684661-5433dddc] [VpxLRO] -- BEGIN lro-10035 --  -- SpecSyncLRO.Synchronize --
YYYY-MM-DD info vpxd[794700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-456531@14684661-5433dddc] [VpxLRO] -- FINISH lro-10035
YYYY-MM-DD info vpxd[794330] [Originator@6876 sub=vpxLro opID=PodCrxMgr-domain-c114-24] [VpxLRO] -- BEGIN lro-10036 --  -- HdcsPodCrxManage

/var/log/vmware/upgrade/upgrade-import.log:
YYYY-MM-DD INFO __main__ ---This is migration/upgrade to version 9.0---
YYYY-MM-DD INFO __main__ Creating configuration file --/usr/lib/vmware/cis_upgrade_runner/upgrade-import-config.json
YYYY-MM-DD INFO __main__ ==========================
YYYY-MM-DD INFO install_param_utils Retrieve install-parameter deployment.node.type=embedded
:
:
YYYY-MM-DD INFO __main__ ==========================
YYYY-MM-DD INFO base_commands Executing command --['/usr/bin/python3', '/usr/lib/vmware/cis_upgrade_runner/UpgradeOrchestrator.py', '-m', 'import', '-f', 'upgrade-import-config.json', '-o', '/var/log/vmware/upgrade/import.json', '--logDir', '/var/log/vmware/upgrade', '--logFileName', 'import-upgrade-runner.log', '--cancelFile', '/var/tmp/upgrade_cancel.op', '-l', 'en', '--logLevel', 'INFO', '--silent', '--disableScreenLog'] on localhost
YYYY-MM-DD ERROR transport Command ['/usr/bin/python3', '/usr/lib/vmware/cis_upgrade_runner/UpgradeOrchestrator.py', '-m', 'import', '-f', 'upgrade-import-config.json', '-o', '/var/log/vmware/upgrade/import.json', '--logDir', '/var/log/vmware/upgrade', '--logFileName', 'import-upgrade-runner.log', '--cancelFile', '/var/tmp/upgrade_cancel.op', '-l', 'en', '--logLevel', 'INFO', '--silent', '--disableScreenLog'] exit-code=1, stdout=, stderr=
YYYY-MM-DD ERROR __main__ ERROR: Fatal error during upgrade IMPORT. For more details take a look at: /var/log/vmware/upgrade/import-upgrade-runner.log
YYYY-MM-DD INFO root Exiting with exit-code 1

/var/log/vmware/upgrade/import-upgrade-runner.log:
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: YYYY-MM-DD INFO applmgmt applmgmt import started
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: YYYY-MM-DD INFO applmgmt Generating /var/lib/vmware-envoy-system-proxy/config.json from /storage/seat/cis-export-folder/applmgmt/proxy
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: YYYY-MM-DD ERROR __main__ Upgrade Phase 'applmgmt:Import' failed. Exception: While reading from '<???>' [line 31]: option 'no_proxy' in section 'DEFAULT' already exists   ============>>>>>>>>>> Duplicate entry
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: Traceback (most recent call last):
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:   File "/usr/lib/vmware/cis_upgrade_runner/payload/componentPhaseLauncher.py", line 461, in main
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:     executionResult = systemExtension(exeContext)
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:   File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/extensions.py", line 94, in __call__
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:     result = self.extension(*args)
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:              ^^^^^^^^^^^^^^^^^^^^^
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:     self._read(f, source)
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:   File "/usr/lib/python3.11/configparser.py", line 1113, in _read
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import:     raise DuplicateOptionError(sectname, optname,
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: configparser.DuplicateOptionError: While reading from '<???>' [line 31]: option 'no_proxy' in section 'DEFAULT' already exists    ============>>>>>>>>>> Duplicate entry
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: STDERR: /usr/lib/vmware/cis_upgrade_runner/payload/componentPhaseLauncher.py:86: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: STDERR:   import imp
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: STDERR: touch: cannot touch '/usr/lib64/python2.7/site-packages/zope/__init__.py': No such file or directory
YYYY-MM-DD INFO upgrade.states.component_states applmgmt:Import: Script completed for 1.0073275566101074 secs with return-code='1', and executionId=4d9b1156-b0a5-43a2-a947-890a503aa1fd
YYYY-MM-DD ERROR upgrade.states.component_states applmgmt:Import: Remote script failed with an error [InternalError()]
YYYY-MM-DD ERROR upgrade.states.component_states applmgmt:Import: failed with internal error. For details take a look at Import_com.vmware.applmgmt_YYYY_MM_HH.log.
YYYY-MM-DD WARNING wf.wf_processor State Import com.vmware.applmgmt has failed.
YYYY-MM-DD ERROR wf.wf_processor Workflow execution has failed
YYYY-MM-DD WARNING root stopping status aggregation...
YYYY-MM-DD INFO UpgradeRunner Persisting upgrade workflow context..
YYYY-MM-DD INFO deployer.migration_env_deployer Cleaning-up migration environment...
YYYY-MM-DD INFO deployer.migration_env_deployer All targets were already unregistered as extensions
YYYY-MM-DD INFO deployer.migration_env_deployer Cleaning up target: None target with address=127.0.0.1
YYYY-MM-DD ERROR UpgradeRunner Upgrade mode import failed

cat /etc/sysconfig/proxy
# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="no"
# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""
# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY=", localhost, 127.0.0.1"
NO_PROXY=", localhost, 127.0.0.1"    ============>>>>>>>>>> Duplicate entry

service-control --status --all
Running:
 applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-envoy-system-proxy vmware-hvc vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui
Stopped:
 vmcam vmware-content-library vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vstats vtsdb wcp   

Environment

VMware vCenter Server 8.x
VMware vCenter 9.0.0

Cause

Redundant or duplicate configuration lines in the /etc/sysconfig/proxy file prevent the OS from correctly parsing network environment variables during the upgrade migration or service startup.

Resolution

1) ssh to vCenter server.
2) Navigate to "/etc/sysconfig"
3) Take a backup of the file "proxy" run "cp proxy proxy.bkup"
4) Open file "proxy" via vi editor.
5) Remove the duplicate entry and save the file.

Powered by Wolken Software