Firewall preparation shows as busy on the cluster

Products

• VMware vDefend Firewall

Issue/Introduction

Symptoms:

• In the Host Preparation section, the whole cluster is spinning as busy.
• In the Firewall section, you may see a Last publish error on one host of the cluster.
• In the vsm.log, the host-id is reported as not found.

2017-07-27 11:04:37.018 GMT  INFO taskScheduler-13 FirewallForceSyncHostTask:44 - ForceSync Host task run for : host-44668
2017-07-27 11:04:37.019 GMT ERROR taskScheduler-13 MessagingHelper:196 - No host found with this id host-44668

• In the vsm.log, the host-id seems to be known:

2017-07-27 11:07:35.839 GMT  INFO messagingTaskExecutor-1 FirewallInstallManagerImpl:322 - Firewall Enabled for context host-44668 : true
2017-07-27 11:07:35.845 GMT  INFO messagingTaskExecutor-1 EventBsdFtrMgrImpl:282 - Preparing to transactionally update resource: host-44668, with feature status: com.vmware.vshield.firewall
2017-07-27 11:07:35.846 GMT  INFO messagingTaskExecutor-1 EventBsdFtrMgrImpl:305 - Resource host-44668 had a Feature com.vmware.vshield.firewall status entry. Updating existing entry.
2017-07-27 11:07:35.848 GMT  INFO messagingTaskExecutor-1 EventBsdFtrMgrImpl:284 - Transactionally updated resource: host-44668, with feature status: [resourceId : null, featureId : com.vmware.vshield.firewall, featureVersion : 5.5, status : YELLOW, installed :  true, errorStatus :  ]

• Querying the NSX Manager for the hosts shows correct details of the host.

HQ-NSX-01a.nsx.gss> show cluster all
No.  Cluster Name   Cluster Id               Datacenter Name   Firewall Status
1    HQ-Prod-01a    domain-c14               HQ-NSX-DC         Enabled
2    HQ-Mgmt-01a    domain-c36               HQ-NSX-DC         Not Ready
3    HQ-Edge-01a    domain-c34               HQ-NSX-DC         Enabled HQ-NSX-01a.nsx.gss> show cluster domain-c14 Datacenter: HQ-NSX-DC
Cluster: HQ-Prod-01a
No.  Host Name                  Host Id                  Installation Status
1    hq-esxi-prod-01a.nsx.gss   host-44668               Enabled
2    hq-esxi-prod-02a.nsx.gss   host-14                  Enabled
HQ-NSX-01a.nsx.gss> show host host-44668
Datacenter: HQ-NSX-DC
Cluster: HQ-Prod-01a
Host: hq-esxi-prod-01a.nsx.gss
[snip]

• In NSX Manager database, the records are inconsistent about that host-id.
  The table firewall_appliance doesn't have a record of that host.
  The table firewall_status_host has a record of that host but the current_generation_number is empty.
  The table messaging_client has a proper record of that host.

 

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x

Resolution

To resolve this issue:

1. Enter Maintenance Mode on the host.
2. Remove the host out of the cluster.
3. Reboot the host.
4. Add the host back in the cluster.
5. Exit Maintenance Mode on the host.