NVAIE 7.x (vGPU 580.x.x) Guest Driver Download fails in DLVM 9.0.1.0

search cancel

NVAIE 7.x (vGPU 580.x.x) Guest Driver Download fails in DLVM 9.0.1.0

book

Article ID: 425067

calendar_today

Updated On:

Products

VCF Private AI Services

Issue/Introduction

^{In environments where NVIDIA NVAIE 7.x (corresponding to the vGPU 580 guest driver branch) is already installed on the ESXi host, DLVM 9.0.1.0 will fail to download vGPU guest driver with the following error in /var/log/vgpu-install.log:}

^{VERSION="9.0.1.0"}
^{BUILD_NUMBER="24882593"}
^{vGPU host driver version detected: 580.105.06}
^...
^{2025-12-18 14:26:29,309 - urllib3.connectionpool - _new_conn[DEBUG]: Starting new HTTPS connection (1): api.ngc.nvidia.com:443}
^{2025-12-18 14:26:30,285 - urllib3.connectionpool - _make_request[DEBUG]: https://api.ngc.nvidia.com:443 "GET /v2/org/nvidia/team/vgpu/resources/vgpu-guest-driver-7/versions/7.3/files/NVIDIA-Linux-x86_64-580.105.08-grid.run HTTP/1.1" 403 111}
^{2025-12-18 14:26:30,372 - dlvminit.logging_config - error[ERROR]: ERROR Download failed: 403 Client Error: Forbidden for url: https://api.ngc.nvidia.com/v2/org/nvidia/team/vgpu/resources/vgpu-guest-driver-7/versions/7.3/files/NVIDIA-Linux-x86_64-580.105.08-grid.run}

Environment

^{VMware Cloud Foundation 9.0.0.0}

^{VMware Cloud Foundation 9.0.1.0}

^{VMware Private AI Foundation 9.0.1.0 (DLVM 9.0.1.0)}

Cause

^{Starting with the NVAIE 7.0 / vGPU 580 branch, NVIDIA introduced a breaking change to the resource naming convention for guest drivers hosted on the NGC catalog:}

^{Legacy Resource Name (NVAIE 6.x and earlier): vgpu-guest-driver}

^{New Resource Name (NVAIE 7.x / vGPU 580.x.x): vgpu-for-compute-guest-driver}

^{The internal deployment logic of DLVM 9.0.1.0 uses hardcoded strings that match the legacy naming convention. When the DLVM attempts to fetch a 580-series guest driver using the outdated resource path, the NGC API returns a 403 Forbidden error.}

Resolution

^{DLVM 9.0.2.0 has been released and includes a fix for the NVIDIA guest driver naming convention change. It is strongly recommended to use DLVM 9.0.2.0 or later, which resolves this issue without any manual intervention.}

^{For environments that run DLVM 9.0.1.0, to work around the issue, manually overriding the resource name mapping using a cloud-init script during the deployment process.}

^{Step 1: Prepare the Cloud-Init Configuration}

^{Use the following YAML configuration to patch the internal constants.py file within the DLVM. This patch redirects BASE_GUEST_DRIVER_URL to the correct vgpu-for-compute-guest-driver resource.}

^{#cloud-config}

^write_files:

^{- encoding: b64}

^{path: /opt/dlvm/dlvminit/constants.py}

^{owner: root:root}

^{permissions: '0755'}

^{content: IyBDb3B5cmlnaHQgKGMpIDIwMjUgQnJvYWRjb20uIEFsbCBSaWdodHMgUmVzZXJ2ZWQuDQojIEJyb2FkY29tIENvbmZpZGVudGlhbC4gVGhlIHRlcm0gIkJyb2FkY29tIiByZWZlcnMgdG8gQnJvYWRjb20gSW5jLg0KIyBhbmQvb3IgaXRzIHN1YnNpZGlhcmllcy4NCg0KZnJvbSBlbnVtIGltcG9ydCBFbnVtDQoNCiMgT1ZGIFByb3BlcnR5IEtleXMNCk9WRl9QUk9QRVJUSUVTID0gew0KICAgICJJTlNUQU5DRV9JRCI6ICJpbnN0YW5jZS1pZCIsDQogICAgIkhPU1ROQU1FIjogImhvc3RuYW1lIiwNCiAgICAiU0VFREZST00iOiAic2VlZGZyb20iLA0KICAgICJQVUJMSUNfS0VZUyI6ICJwdWJsaWMta2V5cyIsDQogICAgIlVTRVJfREFUQSI6ICJ1c2VyLWRhdGEiLA0KICAgICJQQVNTV09SRCI6ICJwYXNzd29yZCIsDQogICAgIlZHUFVfTElDRU5TRSI6ICJ2Z3B1LWxpY2Vuc2UiLA0KICAgICJOR0NfQVBJX0tFWSI6ICJuZ2MtYXBpLWtleSIsDQogICAgIk5WSURJQV9QT1JUQUxfQVBJX0tFWSI6ICJudmlkaWEtcG9ydGFsLWFwaS1rZXkiLA0KICAgICJWR1BVX0hPU1RfRFJJVkVSX1ZFUlNJT04iOiAidmdwdS1ob3N0LWRyaXZlci12ZXJzaW9uIiwNCiAgICAiVkdQVV9VUkwiOiAidmdwdS11cmwiLA0KICAgICJSRUdJU1RSWV9VUkkiOiAicmVnaXN0cnktdXJpIiwNCiAgICAiUkVHSVNUUllfVVNFUiI6ICJyZWdpc3RyeS11c2VyIiwNCiAgICAiUkVHSVNUUllfUEFTU1dEIjogInJlZ2lzdHJ5LXBhc3N3ZCIsDQogICAgIlNFQ09OREFSWV9SRUdJU1RSWV9VUkkiOiAicmVnaXN0cnktMi11cmkiLA0KICAgICJTRUNPTkRBUllfUkVHSVNUUllfVVNFUiI6ICJyZWdpc3RyeS0yLXVzZXIiLA0KICAgICJTRUNPTkRBUllfUkVHSVNUUllfUEFTU1dEIjogInJlZ2lzdHJ5LTItcGFzc3dkIiwNCiAgICAiSU1BR0VfT05FTElORVIiOiAiaW1hZ2Utb25lbGluZXIiLA0KICAgICJET0NLRVJfQ09NUE9TRV9VUkkiOiAiZG9ja2VyLWNvbXBvc2UtdXJpIiwNCiAgICAiQ09ORklHX0pTT04iOiAiY29uZmlnLWpzb24iLA0KICAgICJDT05EQV9FTlZJUk9OTUVOVF9JTlNUQUxMIjogImNvbmRhLWVudmlyb25tZW50LWluc3RhbGwiLA0KICAgICJETFZNQ09OU09MRV9FTkFCTEUiOiAiZGx2bWNvbnNvbGUtZW5hYmxlIiwNCn0NCg0KRU5WSVJPTk1FTlRTX1ZBUlMgPSB7DQogICAgIyBleHBvcnQgRExWTV9ERUJVRz10cnVlIGZvciBkZWJ1ZyBtb2RlDQogICAgIkRFQlVHX01PREUiOiAiRExWTV9ERUJVRyIsDQp9DQoNCk5HQ19WR1BVX0RSSVZFUl9DQVRBTE9HX1VSTCA9ICJodHRwczovL2FwaS5uZ2MubnZpZGlhLmNvbS92Mi9yZXNvdXJjZXMvb3JnL252aWRpYS90ZWFtL3ZncHUvdmdwdV9kcml2ZXJfY2F0YWxvZy9sYXRlc3QvZmlsZXM/cmVkaXJlY3Q9dHJ1ZSZwYXRoPXZncHVEcml2ZXJDYXRhbG9nLnlhbWwiDQpCQVNFX0dVRVNUX0RSSVZFUl9VUkwgPSAoDQogICAgImh0dHBzOi8vYXBpLm5nYy5udmlkaWEuY29tL3YyL29yZy9udmlkaWEvdGVhbS92Z3B1L3Jlc291cmNlcy8iDQogICAgInZncHUtZm9yLWNvbXB1dGUtZ3Vlc3QtZHJpdmVyLXttYWpvcl9udmFpZV9yZWxlYXNlfS92ZXJzaW9ucy97bnZhaWVfcmVsZWFzZX0vZmlsZXMvIg0KICAgICJOVklESUEtTGludXgteDg2XzY0LXtndWVzdF9kcml2ZXJfdmVyc2lvbn0tZ3JpZC5ydW4iDQopDQpOVklESUFfUFJPRFVDVCA9ICdudmFpZScNCkhPU1RfVFlQRSA9ICJob3N0Ig0KR1VFU1RfVFlQRSA9ICJndWVzdCINCkRSSVZFUl9NQVRDSF9SVUxFUyA9IHsNCiAgICAnaG9zdCc6IFsNCiAgICAgICAgeyd0eXBlJzogJ2hvc3QnfSwNCiAgICAgICAgeydoeXBlcnZpc29yJzogJ0VTWGknfQ0KICAgIF0sDQogICAgJ2d1ZXN0JzogWw0KICAgICAgICB7J3R5cGUnOiAnZ3Vlc3QnfSwNCiAgICAgICAgeydvcyc6ICdMaW51eCd9DQogICAgXQ0KfQ0KIyBNYWdpYyBzdHJpbmcgY29uc3RhbnRzIGZvciB2Z3B1IGRyaXZlciBjYXRhbG9nDQpjbGFzcyBDYXRhbG9nS2V5czoNCiAgICBEUklWRVIgPSAiZHJpdmVyIg0KICAgIEJSQU5DSCA9ICJicmFuY2giDQogICAgUFJPRFVDVCA9ICJwcm9kdWN0Ig0KICAgIFJFTEVBU0UgPSAicmVsZWFzZSINCiAgICBWRVJTSU9OID0gInZlcnNpb24iDQogICAgVFlQRSA9ICJ0eXBlIg0KICAgIE5BTUUgPSAibmFtZSINCiAgICBBTExPVyA9ICJhbGxvdyINCg0KTE9DQUxfTUFOSUZFU1RfRklMRSA9ICIvb3B0L2Rsdm0vZGx2bWluaXQvdmdwdS1uZ2MtbWFuaWZlc3QudHh0Ig0KDQpHdWVzdEJvb3RzdHJhcF9LRVkgPSAiZ3Vlc3RpbmZvLnZtc2VydmljZS5ib290c3RyYXAuY29uZGl0aW9uIg0KDQpQUk9YWV9FTlZTID0gWw0KICAgICdodHRwX3Byb3h5JywgJ0hUVFBfUFJPWFknLCANCiAgICAnaHR0cHNfcHJveHknLCAnSFRUUFNfUFJPWFknLCANCiAgICAnZnRwX3Byb3h5JywgJ0ZUUF9QUk9YWScsIA0KICAgICdub19wcm94eScsICdOT19QUk9YWScsIA0KICAgICdhbGxfcHJveHknLCAnQUxMX1BST1hZJywNCl0NCg0KU1lTVEVNX1VTRVIgPSAidm13YXJlIg0KDQpjbGFzcyBIYXNoQWxnb3JpdGhtKEVudW0pOg0KICAgIE1ENSA9ICdtZDUnDQogICAgU0hBMSA9ICdzaGExJw0KICAgIFNIQTIyNCA9ICdzaGEyMjQnDQogICAgU0hBMjU2ID0gJ3NoYTI1NicNCiAgICBTSEEzODQgPSAnc2hhMzg0Jw0KICAgIFNIQTUxMiA9ICdzaGE1MTInDQoNCkxPR0dBQkxFX0ZJTEVfUEFUSFMgPSBbDQogICAgIi9vcHQvZGx2bS9lbnZpcm9ubWVudCIsDQogICAgIi9ldGMvc3lzdGVtZC9zeXN0ZW0vcmMtbG9jYWwuc2VydmljZS5kL2Vudmlyb25tZW50LmNvbmYiLA0KICAgICIvZXRjL3N5c3RlbWQvc3lzdGVtL2RvY2tlci5zZXJ2aWNlLmQvcHJveHkuY29uZiIsDQogICAgIi9ldGMvbnZpZGlhL2dyaWRkLmNvbmYiDQpdDQoNCkRFRkFVTFRfQ0FfQlVORExFID0gJy9ldGMvc3NsL2NlcnRzL2NhLWNlcnRpZmljYXRlcy5jcnQnDQpCQVNFX1RNUF9ESVIgPSAnL29wdC9kbHZtL3RtcCc=}

^{Step 2: Applying the Workaround}

^{Alternative: Manual Fix

If the cloud-init workaround above still does not resolve the issue, you can apply the fix manually by directly editing the configuration file on the DLVM.}

^{SSH into your DLVM as a user with sudo privileges.}
^{Open the file /opt/dlvm/dlvminit/constants.py for editing:}

^{sudo vi /opt/dlvm/dlvminit/constants.py}

^{Locate the BASE_GUEST_DRIVER_URL variable (typically between lines 39–43). It will look similar to:}

^{BASE_GUEST_DRIVER_URL = (}
^{"https://api.ngc.nvidia.com/v2/org/nvidia/team/vgpu/resources/"}
^{"vgpu-guest-driver-{major_nvaie_release}/versions/{nvaie_release}/files/"}
^{"NVIDIA-Linux-x86_64-{guest_driver_version}-grid.run"}
⁾

^{Change the resource name from vgpu-guest-driver to vgpu-for-compute-guest-driver. After editing, the variable should read:}

^{BASE_GUEST_DRIVER_URL = (}
^{"https://api.ngc.nvidia.com/v2/org/nvidia/team/vgpu/resources/"}
^{"vgpu-for-compute-guest-driver-{major_nvaie_release}/versions/{nvaie_release}/files/"}
^{"NVIDIA-Linux-x86_64-{guest_driver_version}-grid.run"}
⁾

^{Save and close the file.}
^{Apply the change by either rebooting the DLVM or manually executing the vGPU driver download script:}

^{sudo /opt/dlvm/get-vgpu-driver.sh}

^{Check /var/log/vgpu-install.log to confirm that the guest driver download completes successfully.}

^{To apply the fix, follow these steps to inject the configuration into your deployment environment:}

^{If deploying DLVM via vCenter Server,}
- ^{Copy the YAML code block from Step 1}
- ^{Base64 encode the entire YAML block}
- ^{Paste the resulting Base64 string into the 'Encoded user-data'field in the DLVM OVF parameter}
^{If deploying DLVM via VMService,}
- ^{Copy the YAML code block from Step 1}
- ^{Base64 encode the entire YAML block}
- ^{Paste the resulting Base64 string into the 'user-data'field in the DLVM OVF parameter}
^{If If deploying DLVM via VCFA,}
- ^{Check the 'Custom cloud-init'}
- ^{Paste the raw YAML script (from Step 1) directly into the provided text area.}

Feedback

thumb_up Yes

thumb_down No