VCF Import fails at NSX Deployment Stage

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

Brownfield VCF Import fails - NSX deployment fails with error stack :

[2024-12-26 08:56:33,336] [INFO] nsx_deployment_spec: Extracting files to: /nfs/vmware/vcf/nfs-mount/bundle
[2024-12-26 08:57:27,709] [INFO] nsx_deployment_spec: Extracted files as: ['bundle-133764.manifest', 'bundle-133764.tar', 'bundle-133764.manifest.sig']
[2024-12-26 08:57:27,715] [INFO] nsx_deployment_spec: Added group read permissions to : /nfs/vmware/vcf/nfs-mount/bundle/bundle-133764.tar
[2024-12-26 08:57:27,715] [INFO] nsx_deployment_spec: Added group read permissions to : /nfs/vmware/vcf/nfs-mount/bundle/bundle-133764.manifest
[2024-12-26 08:57:27,715] [INFO] nsx_deployment_spec: Added group read permissions to : /nfs/vmware/vcf/nfs-mount/bundle/bundle-133764.manifest.sig
[2024-12-26 08:57:27,715] [INFO] sddc_manager_helper: Using cached SDDC Manager token header
[2024-12-26 08:57:27,899] [INFO] sddc_manager_helper: Upload API response: 500 ; {"errorCode":"VCF_ERROR_INTERNAL_SERVER_ERROR","arguments":[],"message":"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.","causes":[{"type":"com.vmware.evo.sddc.lcm.model.error.LcmException"}],"referenceToken":"JAGTBT"}
[2024-12-26 08:57:27,899] [ERROR] request_helper: Result status code from bundle initiate upload: {'errorCode': 'VCF_ERROR_INTERNAL_SERVER_ERROR', 'arguments': [], 'message': 'A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.', 'causes': [{'type': 'com.vmware.evo.sddc.lcm.model.error.LcmException'}], 'referenceToken': 'JAGTBT'} : 500
[2024-12-26 08:57:27,899] [INFO] vcf_brownfield: Could not complete convert domain operation for vCenter: vCenter_FQDN
[2024-12-26 08:57:27,900] [INFO] vcf_brownfield: Operation convert completed on target:vCenter_FQDN with status: FAIL in 220.5s

 

Cause

NSX deployment fails due to file permission issue.
In SDDC Manager the files get default permissions of 600 instead of 644 so we need to add more permissions to the newly created files. This way the service users will be able to read the files.

Log snippets: lcm-debug.log - /var/log/vmware/vcf/lcm 

2024-12-26T08:57:27.778+0000 INFO  [vcf_lcm,676dxxxxxxxxxxxxxxxxx,2755] [c.v.v.l.r.a.c.v.b.BundleController,http-nio-127.0.0.1-7400-exec-9] Bundle Upload Specification BundleUploadSpec(bundleFilePath=/nfs/vmware/vcf/nfs-mount/bundle/bundle-1 33764.tar, manifestFilePath=/nfs/vmware/vcf/nfs-mount/bundle/bundle-xx.manifest, signatureFilePath=/nfs/vmware/vcf/nfs-mount/bundle/bundle-xx.manifest.sig, compatibilitySetsFilePath=null, partnerExtensionSpec=null) 
2024-12-26T08:57:27.778+0000 ERROR [vcf_lcm,676dxxxxxxxxxxxxxxxxx,2755] [c.v.e.s.l.s.i.BundleUploadServiceImpl,http-nio-127.0.0.1-7400-exec-9] /nfs/vmware/vcf/nfs-mount/bundle/bundle-xx.manifest file can not read by LCM due to insufficient permissions. vcf_lcm user must have read and write access to /nfs/vmware/vcf/nfs-mount/bundle directory or upload bundle files from any directory where vcf_lcm user have read and write access. 
2024-12-26T08:57:27.779+0000 ERROR [vcf_lcm,676d676dxxxxxxxxxxxxxxxxx,2755] [c.v.v.l.r.a.c.v.b.BundleController,http-nio-127.0.0.1-7400-exec-9] Bundle Upload Failed com.vmware.evo.sddc.lcm.model.error.LcmException: /nfs/vmware/vcf/nfs-mount/bundle/bundle-xx.manifest file can not read by  LCM due to insufficient permissions. vcf_lcm user must have read and write access to /nfs/vmware/vcf/nfs-mount/bundle directory or upload bundle files from any directory where vcf_lcm user have read and write access.

Resolution

The issue is resolved with VCF 5.2 release. 

In the interim, below workaround could be followed. 

1. Take a snapshot of the SDDC manager 
2. Login to SDDC SSH with vcf user and elevate to root 
3. Navigate to bundle directory - 

cd /nfs/vmware/vcf/nfs-mount/bundle

4. Change the file permissions and ownership (if required) of the bundle files -  

chown vcf:vcf filename 
chmod 644 filename