VM migration between clusters fail with error "Changing or applying VM Storage Policies with Data Service capabilities during migrate operations is disallowed"
search cancel

VM migration between clusters fail with error "Changing or applying VM Storage Policies with Data Service capabilities during migrate operations is disallowed"

book

Article ID: 430477

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Unable to migrate Virtual Machines between clusters in a vCenter Server environment. The migration fails with the error: "Changing or applying VM Storage Policies with Data Service capabilities during migrate operations is disallowed."

  • When attempting a vMotion or Storage vMotion of a virtual machine, the operation fails at the validation or initialization phase with the following error in the vSphere Client:

    "The operation is not supported on the object. Changing or applying VM Storage Policies with Data Service capabilities during migrate operations is disallowed. VM Storage Policies with Data Service capabilities can be assigned to the provisioned VM after the migrate operation has been completed and before the VM has been powered on."

  • In the vCenter Server logs, entries similar to the following is observed. 

    /var/log/vmware/vpxd/vpxd.log 

    error vpxd[#####] [Originator@6876 sub=vpxLro opID=...] [VpxLRO] -- ERROR task-####: vmodl.fault.NotSupported:
    --> key = "com.vmware.vim.vpxd.iofilter.policyChangeDisallowed"

  • In the source ESXi host logs, you see the below log entries. The host-side daemon (iofiltervpd) cannot validate the vCenter's certificate as it cannot find a matching trusted certificate in its store.

    /var/log/iofiltervpd.log

    YYYYMM-DDThh:mm:ss No(29) iofiltervpd[######]: IOFVPSSL_VerifySSLCertificate:423:Client certificate can't be verified
    YYYYMM-DDThh:mm:ss No(29) iofiltervpd[######]: IOFVPSSL_VerifySSLCertificate:423:Client certificate can't be verified
    YYYYMM-DDThh:mm:ss Wa(28) iofiltervpd[######]: run:199:SSL Connection error 30 : SSL_ERROR_SSL
    YYYYMM-DDThh:mm:ss Wa(28)[+] iofiltervpd[######]: error:0A000416:SSL routines::sslv3 alert certificate unknown
    YYYYMM-DDThh:mm:ss No(29) iofiltervpd[######]: HTTPGet:73:pathStr: /version.xml
    YYYYMM-DDThh:mm:ss No(29) iofiltervpd[######]: IOFVPSSL_VerifySSLCertificate:378:Client certificate not presented
    YYYYMM-DDThh:mm:ss No(29) iofiltervpd[######]: IOFVPSSL_RemoveCertFromTrustStore:630:Failed to remove client certificate: Unable to find certificate in the ssl cert store. It may be already deleted.

  • In the vCenter Server logs, entries similar to the following is observed. The vCenter Storage Policy Service (SPS) fails to communicate with the VASA provider. The following error is observed in the logs, indicating that the version.xml file is unreachable.

    /var/log/vmware/vmware-sps/sps.log

    YYYY-MM-DDThh:mm:ss [pool-27-thread-5] ERROR opId=sps-Main-614169-823 com. vmware.vim. sms. provider.vasa.alarm. AlarmDispatcher - Error: org.apache. axis2.AxisFault: self-signed certificate occured as provider: <HOST FQDN>:9080, version.xml is offline
    YYYY-MM-DDThh:mm:ss [pool-27-thread-4] ERROR opId=sps-Main-614169-823 com. vmware. vim. sms. provider.vasa.alarm. AlarmDispatcher - Error: org. apache.axis2.AxisFault: self-signed certificate occured as provider: <HOST FQDN>:9080 version.xml is offline
    YYYY-MM-DDThh:mm:ss [pool-27-thread-2] ERROR opId=sps-Main-614169-823 com.vmware.vim.sms.provider.vasa.alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self-signed certificate occured as provider: <HOST FQDN>:9080 version.xml is offline

Environment

VMware vCenter Server 

Cause

Synchronization failure between the vCenter Storage Management Service (SMS) and the IOFilter providers on the ESXi hosts, typically triggered by a stale `sms_self_signed` certificate in the SMS trust store.

 

Resolution

To resolve this issue, the SSL trust between the vCenter Storage Provider service (sps) and the ESXi hosts must be re-established.

Note: Take a snapshot of the vCenter Server in a powered-off state if the vCenter Servers are in ELM before following the steps below. Refer the kb- Snapshot Best practices for vCenter Server Virtual Machines.

  1. Identify the Offline Provider

    1. Log in to the vSphere Client.
    2. Navigate to vCenter Server > Configure > Storage Providers.
    3. Look for providers with a status of Offline or Disconnected. Note the "Last Sync Time" (it may be significantly outdated)


  2. Re-register the Storage Provider

    1. Download unreg_vasa.py attached to this KB. Use WinScp to move the script to the vCenter Server.
    2. Take a SSH session to the vCenter server and login as "root" user.
    3. Run the command below.  This command stores the cert file signed with sha1WithRSAEncryption.
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_Before.out
    4. Run the unreg_vasa.py script as instructed below.
      Note: Replace the <VC_IP> with the IP of the VC. The script is used to automatically unregisters all the IOFilter providers which are offline from the vCenter Server.
      python unreg_vasa.py -s <VC_IP> -d
    5. Provide the administrator credentials when requested by the prompt given by the script.
      Note: The script identifies the bad providers. This should show the total number of IOFilters that are in a disconnected state.
    6. Hit 'Y' when the prompt asks to unregister the bad providers(IOFilters). All the IOFilter providers will be unregistered one by one.
    7. After the script execution finishes successfully, run the command below which shall delete the cert from SMS trust store.  
      /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed
      Expected Output:
      Warning: This operation will delete entry [sms_self_signed] from store [sms]
Do you wish to continue? Y/N [N]
Y
Deleted entry with alias [sms_self_signed] in store [sms] successfully
    8. Run the command below to restarts SPS service 
      vmon-cli -r sps
      Note: IOFilter provider registration starts automatically when SPS restarts. Wait for some time until SPS has finished initialization and check vCenter→ Configure → Storage Providers to verify that the IOFilters are now online.

  3. Verify Migration

    1. Once the provider status returns to Online and the "Last Sync Time" is current, re-attempt the Virtual Machine migration. The compatibility check should now pass without the "Data Service capabilities" error.

Additional Information


Even if the Storage Policy is not being changed during migration, vCenter performs a "re-apply" validation of the policy on the destination host. Often due to a stale SSL trust/certificate mismatch, if the destination host's I/O filter provider is in an offline or unsynced state, vCenter Server incorrectly identifies the operation as an unsupported policy change during migration and blocks the task.

Attachments

unreg_vasa.py get_app
Powered by Wolken Software