SDDC workflow to add hosts to cluster fails. NSX shows an error, "Install Failed" on a cluster that has Malware Prevention enabled

Products

• VMware SDDC Manager
• VMware NSX
• VMware vCenter Server

Issue/Introduction

Attempting to add hosts to a SDDC managed cluster fails.

The /var/log/proton/nsxapi.log on the associated NSX Managers shows entries similar to:

YYYY-MM-DDTHH:MM:SS.###Z INFO http-nio-127.0.0.1-7440-exec-35 InstallFailed 77558 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="<id>" subcomp="manager" username="<username>"] Issue INSTALL_FAILED observed for node : <nodeid>. Msg : Failed to install software on host. Health Check for '<hostfqdn>' failed Host '<hostfqdn>' was not processed, the reason: 'Health Check for '<hostfqdn>' failed'

The vCenter /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-###.log shows entries similar to:

 [vSphere ESX Agent Manager] [com.vmware.eam.ehp.host.hostInitialization.<guid>] returned status: NOT_OK

YYYY-MM-DDTHH:MM:SS.###Z error vmware-vum-server[377520] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:<id>. Task Failed. Error: Error:
--> com.vmware.vapi.std.errors.error
--> Messages:
--> com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed<Health Check for '<hostfqdn>' failed>
-->

The /var/log/vmware/eam/eam.log on the vCenter shows entries similar to:

YYYY-MM-DDTHH:MM:SS.###Z | ERROR | host-agent-2 | AuditedJob.java | 106 | JOB FAILED: [#1464561176] InstallHostAgentJob(HostAgent(ID: <id>))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for HostAgent(ID: <id>) due to AgentVmDatastore or AgentVmNetwork missing.

New issues:
 [
eam.issue.NoCustomAgentVmDatastore {
   time = YYYY-MM-DDTHH:MM:SS.###,
   description = <unset>,
   key = 199,
   agency = 'Agency:<id>:null',
   agencyName = '_NSXT_<id>_domain-c####',
   solutionId = 'com.vmware.nsx.management.nsxt',
   solutionName = 'com.vmware.nsx.management.nsxt',
   agent = 'Agent:<id>:null',
   agentName = '<id>',
   host = 'HostSystem:host-####:<id>',
   hostName = '<hostfqdn>',
   customAgentVmDatastore = vmodl.ManagedObjectReference [
      'Datastore:datastore-####:<id>',
   ],
   customAgentVmDatastoreName = String [
      'datastore-####',
   ],

Viewing Malware Prevention in the NSX UI shows the Service VM Deployment Status: "Deployment failed"

Resolution

Open a case with Broadcom support and mention this article to investigate the cause of the SVM deployment failure.