Upload compatibilityMatrix to SDDC manager fails with "java.net.SocketTimeoutException: timeout" error

Products

• VMware SDDC Manager

Issue/Introduction

• Uploading compatibilityMatrix to SDDC manager with Offline Bundle Transfer Utility (OBTU) or VCF download tool fails with below timeout error:
   
  Failed to upload the metadata into SDDC Manager
Failed to upload the compatibility data into SDDC Manager
com.vmware.evo.sddc.lcm.model.error.LcmException: Api call to v1/compatibility-matrices failed.
      at com.vmware.evo.sddc.lcm.tools.common.adapter.impl.CompatibilityUploadApiAdapterImpl.uploadCompatibilityMetadataFileToLCM(CompatibilityUploadApiAdapterImpl.java:170)
...
Caused by: com.vmware.vcf.rest.api.runtime.ApiException: java.net.SocketTimeoutException: timeout
...
Caused by: java.net.SocketTimeoutException: timeout
...
Caused by: java.net.SocketException: Socket closed


• SDDC manager /var/log/vmware/vcf/lcm/lcm-activity.log shows the API call completes successfully with return code of 200:
  
  {"username":"[email protected]","timestamp":"YYYY-MM-DDTHH:MM:SS.XXXZ","clientIP":"127.0.0.1","userAgent":"Swagger-Codegen/1.0.0/java","api":"/v1/compatibility-matrices","httpMethod":"PUT","httpStatus":200,"operation":"Update CompatibilityMatrix for a source with JSON file","remoteIP":"127.0.0.1"}
  
  
• /var/log/nginx/access.log shows the call returns with HTTP status 499 which indicates client closes connection before receiving response from upstream server:
  
  "PUT /v1/compatibility-matrices?compatibilityMatrixSource=VMWARE_COMPAT HTTP/1.1" 499 0 "-" "Swagger-Codegen/1.0.0/java" "-" 

Environment

VMware SDDC manager 5.x
VMware SDDC manager 9.0

Cause

The reason for SocketTimeout exception is a very slow response of SDDC manager REST API call when uploading compatibility data.

Resolution

Disable compatibility check in LCM as a workaround to speed up API handling and overcome the issue:

1. Login to SDDC manager and change user to root
2. Edit the file:
   vi /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
3. Change vcf.compatibility.controllers.compatibilityCheckEnabled from true to false
4. Restart LCM service:
   systemctl restart lcm

Continue with OBTU or VCF download tool workflow and after all the metadata and bundles are uploaded, re-enable compatibility check by executing the steps but set vcf.compatibility.controllers.compatibilityCheckEnabled back to true.