"Socket Read Timeout" error when performing a migration from a Cloud Site to an On-Premises Site in vCloud Availability

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• When migrating a vApp or VM from a Cloud Site to an On-Premises Site, the migration fails with the error:

Socket Read Timeout

• In /opt/vmware/h4/cloud/log/cloud.log on the vApp Replication Manager, you see similar messages:

2019-09-25 12:17:50.983  WARN - [########-####-####-####-##########] [c4-scheduler-2] c.v.h.c.v.i.LocalVcloudInventoryService  : Unable to refresh storage profiles (Major code = 403, minor code = ACCESS_TO_RESOURCE_IS_FORBIDDEN) - Access is forbidden
2019-09-25 12:17:57.399  WARN - [########-####-####-####-##########] [health-check-1] c.v.r.health.HealthCheckingFactory       : Resource is broken: Vcloud{site='Site-A', description='', defaultSite=true, lsUrl='https://psc.corp.local/lookupservice/sdk', lsThumbprint='SHA-256:E3:C3:09:12:4T:W4:##:##:##:##:##:##:D4:B0:E1:18:12:F5:18:77:52:70:0O:U8:5D:57:06:##:##:##:##:##', ssoUser='[email protected]', ssoPassword=[censored], vcdUrl='https://vcd.corp.local/api', vcdThumbprint='SHA-256:V5:C5:D6:78:88:98:90:C2:3E:7C:TT:9L:F2:CF:LL:11:JK:JN:MH:VX:9F:98:40:45:377:02', vcdSystemUser='administrator@system', vcdSystemPassword=[censored]}
com.vmware.vcloud.client.exception.VcloudException: (Major code = 403, minor code = ACCESS_TO_RESOURCE_IS_FORBIDDEN) - Access is forbidden
        at com.vmware.vcloud.client.VcloudClient.lambda$defaultErrorDeserializer$0(VcloudClient.java:196)
2019-09-25 12:17:57.399  WARN - [########-####-####-####-##########] [health-check-1] c.v.h.c.vcloud.admin.AdminHealthChecker  : Admin session error for Site-A(https://vcd.corp.local/api)
com.vmware.vcloud.client.exception.VcloudException: (Major code = 403, minor code = ACCESS_TO_RESOURCE_IS_FORBIDDEN) - Access is forbidden
2019-09-25 12:17:57.400 DEBUG - [########-####-####-####-##########] [health-check-1] c.v.h.c.vcloud.admin.AdminHealthChecker  : Disposing admin vCloud connection Site-A(https://vcd.corp.local/api)
2019-09-25 12:18:02.764  WARN - [########-####-####-####-##########] [health-check-2] c.v.r.health.HealthCheckingFactory       : Resource is broken: VcloudSiteSettings{site='Site-A'} VcloudSettings{apiVersion='27.0', queryPageSize=100} RestSettings{baseURI=https://https://vcd.corp.local/api, timeout=70000, maxConn=10, sniHostname='null', hostnameVerifier=null, clientCertificate=null, trustManager=com.vmware.rest.client.security.ShaTrustManager@445608593, authenticator=com.vmware.h4.cloud.session.CredentialsVerificationService$2@4200456}
com.vmware.vcloud.client.exception.VcloudException: (Major code = 403, minor code = ACCESS_TO_RESOURCE_IS_FORBIDDEN) - Access is forbidden
        at com.vmware.vcloud.client.VcloudClient.lambda$defaultErrorDeserializer$0(VcloudClient.java:196)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Availability 3.0.x

Cause

This issue occurs when vCloud Availability requested a cookie from vCloud Director to perform the migration just as the vCloud Director cookie expires.

Resolution

To resolve this issue, perform a new migration and vCloud Availability will obtain a new cookie from vCloud Director allowing the migration task to complete successfully.