Async patch enablement fails with BUNDLE_UPLOAD_UNKNOWN_ERROR in SDDC manager / Compatibility matrix upload fails with "Compatibility data failed to update the file with exception {}"
search cancel

Async patch enablement fails with BUNDLE_UPLOAD_UNKNOWN_ERROR in SDDC manager / Compatibility matrix upload fails with "Compatibility data failed to update the file with exception {}"

book

Article ID: 388829

calendar_today

Updated On: 06-03-2025

Products

VMware SDDC Manager

Issue/Introduction

Async patch enablement fails: 

  •  /var/log/vmware/vcf/lcm/tools/asyncpatchtool/async_patch_tool.xxxxx.xxxxxxx.log

2025-02-21 00:41:48.331 [main] INFO  [com.vmware.evo.sddc.lcm.tools.common.BundleUploadValidationUtil]
                  Current upload status is: FAILED for bundleId: 7#######################-apTool
2025-02-21 00:41:48.333 [main] INFO  [com.vmware.evo.sddc.lcm.tools.telemetry.asyncpatch.AsyncPatchToolTelemetryAdapter]
                  Telemetry data collection is not enabled
2025-02-21 00:41:48.333 [main] ERROR [com.vmware.evo.sddc.lcm.tools.asyncpatch.AsyncPatchToolMain]
                  Error while initiating required functionality: INPUT_SPEC_WORKFLOW
2025-02-21 00:41:48.911 [main] ERROR [com.vmware.evo.sddc.lcm.tools.asyncpatch.AsyncPatchToolMain]
                  BUNDLE_UPLOAD_UNKNOWN_ERROR
-- Error Message: Failed uploading bundle for upload id: d#######################
-- Remedy: null
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed uploading bundle for upload id: d#######################

  • Log file /var/log/vmware/vcf/lcm/lcm-debug.log reports the following: 

2025-02-21T05:56:39.037+0000 ERROR [vcf_lcm,69ce985d9434c3c4,9e55] [c.v.e.s.l.s.i.BundleUploadServiceAsyncHandler,Async-4] bundle upload 77xxxxxxxxxxxxxxxxxxxxxxxxxxx : Invalid manifest error { "errorCode": "BUNDLE_MANIFEST_INVALID", "errorMessage": "Invalid bundle manifest", "stackTraceStr": "java.io.IOException: Destination \u0027/nfs/vmware/vcf/nfs-mount/bundle/7xxxxxxxxxxxxxxxxxxxxxxxxxxx-apTool/manifest\u0027 directory cannot be created\n\tat org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1070)\n\tat org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:1000)\n\tat org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:961)\n\tat com.vmware.evo.sddc.lcm.services.impl.BundleUploadServiceAsyncHandler.uploadBundle(BundleUploadServi
ceAsyncHandler.java:135)\n\tat com.vmware.evo.sddc.lcm.services.impl.BundleUploadServiceAsyncHandler$$FastClassBySpringCGLIB$$23861d69.invoke(\u003cgenerated\u003e)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)\n\tat org.springframework.aop.framew
ork.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)\n\tat org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)\n\tat org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgro
undThread(TraceAsyncAspect.java:64)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:5
66)\n\tat org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)\n\tat org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)\n\tat org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocat
ion.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.spr
ingframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)\n\tat org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:64)\n\tat java.
base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n" }
2025-02-21T05:56:39.042+0000 DEBUG [vcf_lcm,69ce985d9434c3c4,9e55] [c.v.e.s.l.d.c.b.BundleClientImpl,Async-4] Updating bundle 7xxxxxxxxxxxxxxxxxxxxxxxxxxx-apTool to status FAILED

For compatibly matrix upload : 

  • /var/log/vmware/vcf/lcm/lcm-debug.log

2025-05-14T03:50:43.489+0000 DEBUG [vcf_lcm,682413139826fc255acd20b48f72e8f0,e567] [c.v.v.c.c.common.ControllerFileUtil,http-nio-127.0.0.1-7400-exec-9] Creating file /nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json.tmp to store compatibility data
2025-05-14T03:50:43.489+0000 ERROR [vcf_lcm,682413139826fc255acd20b48f72e8f0,e567] [c.v.v.c.c.i.v.VvsCompatibilityController,http-nio-127.0.0.1-7400-exec-9] Compatibility data failed to update the file with exception {}
java.nio.file.AccessDeniedException: /nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json.tmp
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)

2025-05-14T03:50:43.490+0000 ERROR [vcf_lcm,682413139826fc255acd20b48f72e8f0,e567] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-9] [OPHQ8K] COMPATIBILITY_MATRIX_CONTENT_UPDATE_FAILED Updating the content to the compatibility controller failed.
com.vmware.evo.sddc.common.core.error.InvalidInputException: Updating the content to the compatibility controller failed.
        at com.vmware.vcf.lcm.rest.api.controller.v1.common.ExceptionAssemblers.toRestExceptionFromLcmException(ExceptionAssemblers.java:187)
        at com.vmware.vcf.lcm.rest.api.controller.v1.compatibility.CompatibilityMatrixController.updateCompatibilityMatrix(CompatibilityMatrixController.java:55) 

Caused by: com.vmware.evo.sddc.lcm.model.error.LcmException: Updating the content to the compatibility controller failed.

Environment

VCF 4.x 

VCF 5.x

Cause

The issue occurs due to permission and ownership on the Bundle directory and compatibility directory 

The expected permissions on the /nfs/vmware/vcf/nfs-mount directory will be: 

drwxrwxr-x 10 root vcf 4096 Jul 29 21:59 .
drwxr-xr-x 3 root root 4096 Jul 12 03:41 ..
drwxrwxr-x 5 vcf_lcm backup 4096 Jul 18 17:04 backup
drwxrwxr-x 4 vcf vcf 4096 Jul 18 16:53 base-install-images
drwxrwxr-x 18 vcf_lcm vcf 4096 Aug 8 15:53 bundle
drwxr-xr-x 2 vcf_lcm vcf 4096 Aug 8 21:39 compatibility
drwxrwxr-x 2 vcf vcf 4096 Jul 25 14:05 dlbundles
drwxr-xr-x 2 vcf_lcm vcf 4096 Jul 31 20:32 vsan-hcl

 

Resolution

Changing the ownership and permission as required will resolve the issue. 

  • To change the ownership run: 

chown -R vcf_lcm:vcf bundle compatibility

  • To change the permissions: 

chmod 775 -R compatibility bundle

Note: For VCF 4.x the compatibility directory would not be present,updating only the bundle will be sufficient. 

 

Powered by Wolken Software