VCenter OVA deployment via content library fails when using ESXi Hosts in a dual stack configuration with IPv4 and IPv6 or with multiple IPv4 DNS entries
search cancel

VCenter OVA deployment via content library fails when using ESXi Hosts in a dual stack configuration with IPv4 and IPv6 or with multiple IPv4 DNS entries

book

Article ID: 344912

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • vCenter OVA deployment via Content Library fails.
  • vSphere Client reports the following error message: 
    Transfer failed: Error during transfer of ds:///vmfs/volumes/<Datastore_UUID>//<VM_NAME>/<VM_NAME>.vmdk: 
A specified parameter was not correct: spec.source.cnxSpec.host.
  • the vCenter Server service log /var/log/vmware/vpxd/vpxd.log contains the following errors: 
    ..
2023-04-03T17:59:27.137+02:00 info vpxd[07160] [Originator@6876 sub=vpxLro opID=c1c37d81-ae7b-4901-9ff3-b0299c91d384-c2-bf] [VpxLRO] -- BEGIN task-2561223 -- nfcManager -- nfc.NfcManager.copy -- 522b0ee1-9065-001e-29c4-680d59bc6973(5241cb5c-0191-8105-34c8-5b5686ceeeb4)
2023-04-03T17:59:27.138+02:00 error vpxd[07160] [Originator@6876 sub=NfcMgr opID=c1c37d81-ae7b-4901-9ff3-b0299c91d384-c2-bf] Host: <IPV4 address> not found
..
2023-04-03T17:59:52.357+02:00 info vpxd[07019] [Originator@6876 sub=Default opID=50ae248c-01] [VpxLRO] -- ERROR task-2561222 -- <template name>-qrubu -- ResourcePool.ImportVAppLRO: vim.fault.OvfImportFailed:
--> Result:
--> (vim.fault.OvfImportFailed) {
-->              faultCause = (vmodl.fault.SystemError) {
-->                      faultCause = (vmodl.MethodFault) null,
-->                      faultMessage = (vmodl.LocalizableMessage) [
-->                                    (vmodl.LocalizableMessage) {
-->                                           key = "com.vmware.ovfs.ovfs-main.ovfs.transfer_failed",
-->                                           arg = (vmodl.KeyAnyValue) [
-->                                                        (vmodl.KeyAnyValue) {
-->                                                                key = "0",
-->                                                                value = "Error during transfer of ds:///vmfs/volumes/5f2989da-681a2b76-e07b-9440c9f2eff6//<VM-name>/<VM-NAME>.vmdk: A specified parameter was not correct: spec.source.cnxSpec.host"
-->                                                        }
-->                                           ],
-->                                           message = "Transfer failed: Error during transfer of ds:///vmfs/volumes/5f2989da-681a2b76-e07b-9440c9f2eff6//<VM-name>/<VM-NAME>.vmdk: A specified parameter was not correct: spec.source.cnxSpec.host"
-->                                   }
-->                      ],
-->                      reason = ""
-->                      msg = "Transfer failed: Error during transfer of ds:///vmfs/volumes/5f2989da-681a2b76-e07b-9440c9f2eff6//<VM-name>/<VM-NAME>.vmdk: A specified parameter was not correct: spec.source.cnxSpec.host"
-->              },
-->              faultMessage = <unset>
-->              msg = ""
--> }
--> Args:
-->
..
  • the Content Library log /var/log/vmware/content-library/cls.log contains: 
    2023-04-03T17:59:27.070+02:00 | DEBUG | c1c37d81-ae7b-4901-9ff3-b0299c91d384 | transferService-pool-7-thread-29 | NfcHostSelectionStrategy       | Creating NFC tickets for specs: [CreateSpec (com.vmware.vcenter.files.nfc.tickets.datastore.create_spec) => {
    datastore = datastore-166786:2ddca27f-a93d-426a-b24b-4c0ea3089d69,
    hosts = [host-171767:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-173181:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-183139:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-319901:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-171173:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-160144:2ddca27f-a93d-426a-b24b-4c0ea3089d69, host-171761:2ddca27f-a93d-426a-b24b-4c0ea3089d69]     =======================> goes through all the HOST IDs will fail with the error from VPXD from above ["Host: <IPV4 address> not found"]
  • When querying the vCenter database, you may see the hosts registered with an IPv6 address: 
    # psql -d VCDB -U postgres -c "SELECT ID, IP_ADDRESS FROM VPX_HOST"
  • If the environment does not use IPv6, the ESXi hosts might have duplicate DNS entries with different IPs. You can confirm this in /var/log/vmware/dnsmasq.log
Aug 29 21:00:05 dnsmasq[1391]: cached esxi-01.example.lab is 10.10.0.222
Aug 29 21:00:05 dnsmasq[1391]: cached esxi-01.example.lab is 10.20.0.222


Environment

VMware vSphere ESXi 7.0.x
VMware vCenter Server 7.0.x

Cause

This is caused by a mismatch of an ESXi host's resolved IP address between vpxd and content library.

Resolution

Currently, there is no resolution to this issue in dual stack IPv6/IPv4 environments, though VMware engineering is aware of this problem and working towards a solution. Use the workaround below for now.

In environments where there are duplicate DNS entries for the ESXi hostname, you must remove or change the entries that are not associated with the management IP address.

Please subscribe to this article to be informed when the status changes.

Workaround:

Note: Before attempting to apply the steps outlined below, please ensure that a fresh backup or an offline snapshot of the vCenter Server Appliance exists. If the vCenter Server Appliance is part of an ELM replication group, please be aware that offline snapshots or backups need to be created for all members of the ELM replication. In case a restore is required, all other ELM member must be restored as well.
Furthermore please be aware that snapshots are not supported for vCenter Server Appliances where VCHA is enabled (see https://kb.vmware.com/s/article/2148003).

To work around the issue in a dual IPv4/IPv6 environment

  1. Create  a backup copy the vdcs.json file: 
    # cp /etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json /storage/core/vdcs.json_orig
  2. Edit the file using the VI editor: 
    # vi /etc/vmware/vmware-vmon/svcCfgfiles/vdcs.json
  3. Find the following content: 
        "StartCommandArgs": [
        "-Dcom.sun.management.jmxremote=true",
        "-Dcom.sun.management.jmxremote.host=localhost",
        "-Dcom.sun.management.jmxremote.port=13038",
        "-Dcom.sun.management.jmxremote.ssl=false",
        "-Dcom.sun.management.jmxremote.authenticate=true",
        "-Dcom.sun.management.jmxremote.access.file=/etc/vmware/vmware-vmon/svcCfgfiles/jmx/content-library.access",
        "-Dcom.sun.management.jmxremote.password.file=/etc/vmware/vmware-vmon/svcCfgfiles/jmx/content-library.password",
  4. Right after "StartCommandArgs": [ insert a  new line "-Djava.net.preferIPv6Addresses=true", (ensure to end the line with a comma): 
        "StartCommandArgs": [
        "-Djava.net.preferIPv6Addresses=true",
        "-Dcom.sun.management.jmxremote=true",
        "-Dcom.sun.management.jmxremote.host=localhost",
        "-Dcom.sun.management.jmxremote.port=13038",
        "-Dcom.sun.management.jmxremote.ssl=false",
        "-Dcom.sun.management.jmxremote.authenticate=true",
        "-Dcom.sun.management.jmxremote.access.file=/etc/vmware/vmware-vmon/svcCfgfiles/jmx/content-library.access",
        "-Dcom.sun.management.jmxremote.password.file=/etc/vmware/vmware-vmon/svcCfgfiles/jmx/content-library.password",
  5. Restart the Content Library service: 
    # service-control --restart vmware-content-library
  6. Attempt again to deploy using the Content Library


Additional Information

Impact/Risks:
Unable to use Content Library when using IPV6 for ESXi Hosts 
Manual OVA upload/ deployment works via H5 CLient.

Mostly impacting NSX -T and Content Library - see documentation below.
https://avinetworks.com/docs/latest/avi-nsx-t-integration/

Powered by Wolken Software