Error: "NAT rule has an invalid vnic value 0" when powering on a vApp or Virtual Machine in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Powering on a vApp or Virtual Machine (VM) fails with the error below:
  

  [ ########-####-####-########1008 ] NAT rule ##### has an invalid vnic value 0 . The value should be a vnic index number.


• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the error below is present:
  
  DATE TIME| DEBUG    | Backend-activity-pool-119545 | NsxTVAppNetwork                | Exception occurred while deploying and attaching network to vApp edge EDGE_NAME with id ########-####-####-########4a3e. | requestId=########-####-####-########1008,request=POST https://cloud.example.com/api/vApp/vapp-########-####-####-########fccc/action/deploy,requestTime=1
748616245118,remoteAddress=##.##.##.##:53060,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 39.0.0-alpha vcd=########-####-####-########aa98
,task=########-####-####-########ddd6 activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-########ddd6) activity=(com.vmware.ssdc.backend.services.im
pl.DeployVAppActivity,urn:uuid:########-####-####-########5b43) activity=(com.vmware.ssdc.backend.DeployVAppVmSubsetActivity,urn:uuid:########-####-####-########638e)
com.vmware.vcloud.api.presentation.service.BadRequestException: NAT rule ##### has an invalid vnic value 0. The value should be a vnic index number.
[...]
  
  AND
  
  DATE TIME | ERROR    | Backend-activity-pool-119545 | DeployVAppVmSubsetActivity     | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.ssdc.backend.
DeployVAppVmSubsetActivity$DeployNetworksPhase@185a067b - Handle: urn:uuid:########-####-####-########638e, Current Phase: DeployVAppVmSubsetActivity$DeployNetworksPhase | requestId=########-####-####-########008,request=POST https://cloud.example.com/api/vApp/vapp-########-####-####-########fccc/action/deploy,requestTime=1748616245118,remoteAddress=##.##.##.##:53060,userAge
nt=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 39.0.0-alpha vcd=########-####-####-########aa98,task=########-####-####-########ddd6 activity=
(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-########ddd6) activity=(com.vmware.ssdc.backend.services.impl.DeployVAppActivity,urn:uuid:########-####-####-########f5b43) activity=(com.vmware.ssdc.backend.DeployVAppVmSubsetActivity,urn:uuid:########-####-####-########638e)
java.util.concurrent.ExecutionException: com.vmware.vcloud.api.presentation.service.BadRequestException: NAT rule 65,544 has an invalid vnic value 0. The value should be a vnic index number.
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:136)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vcloud.api.presentation.service.BadRequestException: NAT rule ##### has an invalid vnic value 0. The value should be a vnic index number.
[...]


• There are NAT rules configured on the Edge Gateway for the routed vApp network.
• If a VM with a NAT rule is moved to another vApp, the NAT rule stays in the source vApp Network and does not get moved to the new vApp along with the VM.
• Ghost VMs on vApp networks preventing VMs from powering on.

Environment

• VMware Cloud director 10.5.1.1
• VMware Cloud Director 10.6.1.1

Cause

The issue is caused because of the presence of some ghost entries in the NAT settings of the certain vApp networks. The issue is also caused because of The NAT rule that has invalid vnic "0" index.

Resolution

Engineering are aware of this issue and will be fixed in a future release of VMware Cloud Director.

Please Subscribe to this KB Article to be notified with any updates.

As a workaround, the below SQL Query helps identify which vApps are facing this behaviour.

Warning: Modifications to the VCD Database are not supported without Broadcom Global Support.

1. Take a database backup before proceeding as per documentation at Backup and Restore of Your VMware Cloud Director Appliance
2. SSH the VCD primary cell and connect to the DB running the command below:
   
   sudo -i -u postgres psql vcloud
   
   
3. Run the following SQL Query to identify which vApps are facing this behaviour:
   

   WITH network_nat_details AS (
    SELECT
        ln.id AS logical_network_id,
        ln.name AS logical_network_name,
        ln.scope_id AS vapp_sg_id,
        COALESCE((xpath('/object/void[@property="vmNicIndex"]/int/text()', nat_spec.rule_xml))[1]::text, '0') AS nic_number,
        (xpath('/object/void[@property="vappScopedVmId"]/string/text()', nat_spec.rule_xml))[1]::text AS vapp_scoped_vm_id,
        (xpath('/object/void[@property="ruleId"]/int/text()', nat_spec.rule_xml))[1]::text AS natrule_id,
        gw.id AS gateway_id
    FROM
        logical_network ln
    JOIN
        gateway_interface gi ON ln.id = gi.logical_network_id
    JOIN
        gateway gw ON gi.gateway_id = gw.id
    CROSS JOIN LATERAL (
        SELECT unnest(xpath('//object[@class="com.vmware.vcloud.fabric.net.gateway.services.impl.NatRuleSpecImpl"]', gw.service_config::xml)) AS rule_xml
    ) AS nat_spec
    WHERE
        ln.scope_type = 3
        AND gw.service_config IS NOT NULL
)

   SELECT 
    nd.logical_network_id, 
 nd.logical_network_name,
 nd.vapp_sg_id,
 nd.gateway_id,
 nd.vapp_scoped_vm_id,
 nd.nic_number as nic_index,
 nd.natrule_id as natrule,
 vapp.name as vapp_name,
 org.name as org_name,
 ovdc.name as ovdc_name
 
FROM 
    network_nat_details nd 
 left join vm_container vapp on nd.vapp_sg_id=vapp.sg_id
 left join organization org on org.org_id = vapp.org_id
 left join org_prov_vdc ovdc on ovdc.id=vapp.org_vdc_id
where  vapp.sg_type=1 and not exists (select 1 from vapp_vm  vm where vm.vapp_scoped_vm_id=nd.vapp_scoped_vm_id and vm.vapp_id=nd.vapp_sg_id);

4. On the Tenant portal, browse to the affected vApp and the NAT rules.
5. Identify the NAT rules from the output of the above query in step. 3
6. Take note of the NAT rule configuration.
7. Delete and (recreate - if neccessary) the NAT rule.
8. Power on the vApp.