In a VCF 9.0 environment, post deployment, recurring 'Downloading Esx metadata, vibs and vendor add-ons' alerts are seen in the SDDC UI
search cancel

In a VCF 9.0 environment, post deployment, recurring 'Downloading Esx metadata, vibs and vendor add-ons' alerts are seen in the SDDC UI

book

Article ID: 415136

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

 

  • The "Downloading Esx metadata, vibs and vendor add-ons" task constantly spins:
  • The subtask 'TASK INFO' has a status of IN_PROGRESS and a TYPE  of UMDS_SYNC.
  • The /var/log/vmware/vmware-updatemgr/umds/vmware-downloadService.log on the relevant vCenter shows that the UMDS download has in fact succeeded in obtaining an access token:

    YYYY-MM-DDTHH:MM:SS.MS+HH:MM info vmware-downloadService[2733771] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 2649] Succesfully fetched auth access token [MASKED TOKEN] VVS endpoint: [URL]
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM info vmware-downloadService[2733771] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 2241] >VVS access token [MASKED TOKEN] received.
  • The umds-commandOutput.json on the SDDC shows the download task as 'SUCCEEDED':

        },
        "########-####-####-####-############": {
            "taskID": "########-####-####-####-############",
    "status": "SUCCEEDED",
            "startTime": "YYYY-MM-DDTHH:MM:SS.MS+HH:MM",
            "operation": "vmware-umds -D -m --info-level error --task-id ########-####-####-####-############ ",
            "endTime": "YYYY-MM-DDTHH:MM:SS.MS+HH:MM"
        }
    }
  • The lcm-debug.log on the SDDC shows a '200' status for the download task:

    YYYY-MM-DDTHH:MM:SS.MS+HH:MM INFO  [vcf_lcm,########################,####] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-###.#.#.#-7400-exec-5] {"username":null,"timestamp":"YYYY-MM-DDTHH:MM:SS.MS+HH:MM","clientIP":"###.#.#.#","userAgent":"Apache-HttpClient/5.3.1 (Java/17.0.12)","api":"/lcm/internal/umds/tasks/sync/########-####-####-####-############","httpMethod":"GET","httpStatus":200,"operation":"Get UMDS sync task specified by task Id","remoteIP":"###.#.#.#","duration":2}
  • The lcm-debug.log will show that a file permission operation was successfully initiated:

    2025-09-30T12:36:35.549+0000 INFO  [vcf_lcm,68da5eefdfcce5ff087fa5306deeaeab,2225] [c.v.e.s.l.f.c.FilePermissionUtil,Scheduled-8] Modified file permissions to 775 on the file: /nfs/vmware/vcf/nfs-mount/umds/patch-store
  • A jstack trace shows a Thread.State of 'WAITING'. See Additional Information section below on the jstack procedure:

    "Scheduled-9" #53 prio=5 os_prio=0 cpu=4364.41ms elapsed=596019.72s tid=0x00007f2ecb77fda0 nid=0x8ad in Object.wait()  [0x00007f2e1a4e5000]
       java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait([email protected]/Native Method)
            - waiting on <no object reference available>
            at java.lang.Object.wait([email protected]/Object.java:338)
            at java.lang.ProcessImpl.waitFor([email protected]/ProcessImpl.java:434)
            - locked <0x00000007487beeb8> (a java.lang.ProcessImpl)

Environment

  • Product: VCF 9.0
  • Configuration: Connected online environment

Cause

  • This issue is caused by UMDS sync operations are hanging indefinitely due to file permission issues in the patch store directory.
  • Permission changes were not applied successfully to all files.
  • The `FilePermissionUtil.setReadWriteExecuteForGroupAndReadExecuteForOthersPermissions` method waits indefinitely for a `chmod` command that never returns output.
  • All subsequent UMDS sync operations will hang indefinitely.

 

Resolution

  1. To work around this issue manually change ownership of patch store to `vcf_lcm:vcf`and set permissions to 775 on the patch store directory:
    chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/umds/patch-store
    chmod -R 775 /nfs/vmware/vcf/nfs-mount/umds/patch-store
  2. Restart the lcm service
    systemctl restart lcm

Additional Information

To generate a jstack trace proceed as below:

  1. Login to SDDC Manager as vcf user
  2. Elevate to root user
  3. Run systemctl status lcm

    root@######## [ /var/log/vmware/vcf/lcm ]# systemctl status lcm

    lcm.service - VMware Cloud Foundation Lifecycle Management Service
         Loaded: loaded (/etc/systemd/system/lcm.service; enabled; preset: enabled)
         Active: active (running) since Mon 2025-09-29 10:44:19 UTC; 2 weeks 2 days ago
        Process: 1473 ExecStartPost=/usr/bin/sh -c /bin/test -f /var/log/vmware/vcf/lcm/lcm.out && /bin/chown vcf_lcm:vcf /var/log/vmware/vcf/lcm/lcm.out (code=exited, status=0/SUCCESS)
        Process: 1478 ExecStartPost=/usr/bin/sh -c /bin/test -f /var/log/vmware/vcf/lcm/lcm.err && /bin/chown vcf_lcm:vcf /var/log/vmware/vcf/lcm/lcm.err (code=exited, status=0/SUCCESS)
       Main PID: 1472 (java)
          Tasks: 94 (limit: 19134)
         Memory: 6.1G
            CPU: 4h 54min 49.697s
         CGroup: /system.slice/lcm.service
                 └─1472 /usr/lib/jvm/openjdk-java17-headless.x86_64/bin/java -Dorg.bouncycastle.fips.approved_only=true -Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true -XX:+UseParallelGC --a…
  4. In the example above the MAIN PID is 1472. It will differ in your specific environment.
  5. Run the following:  jstack 1472 > jstack.txt
  6. Review the jstack.txt file
  7. This issue is resolved in VCF 9.0.2

 

Powered by Wolken Software