EAM fails to complete VIB operation due to failed VIB payload publish

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• EAM operations with VIBs fail.
  
  
• EAM folder within the TMP directory gets deleted and NSX manager is unable to prepare the host for VXLAN.
  
  
• When a new host is added or a host is rebooted it fails to prepare the host.
  
  
• In /var/log/vmware/eam/eam.log* , you see entries similar to:
  

  YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | host-X-Y | VibJob.java | 214 | Cannot access vib package at <VIB_PACKAGE_URL>
  com.vmware.eam.EamIOException: Failed to publish stream content into /tmp/eam6690660256643099065/72a126f8-9ec3-4352-9d37-ebd3b8247660
  at com.vmware.eam.fileserver.FileServer.publish(FileServer.java:91)
  at com.vmware.eam.vib.impl.PublishingBuilder.url(PublishingBuilder.java:27)
  at com.vmware.eam.vib.impl.AbstractBuilder.addDescriptor(AbstractBuilder.java:53)
  at com.vmware.eam.vib.impl.Loader.onMetadata(Loader.java:95)
  at com.vmware.eam.vib.impl.Loader.load(Loader.java:83)
  at com.vmware.eam.vib.impl.Loader.load(Loader.java:59)
  at com.vmware.eam.vib.impl.PackageDepotImpl.loadFrom(PackageDepotImpl.java:149)
  at com.vmware.eam.vib.impl.PackageDepotImpl.importPackage(PackageDepotImpl.java:52)
  at com.vmware.eam.job.VibJob.updateVibs(VibJob.java:212)
  at com.vmware.eam.job.VibJob.call(VibJob.java:154)
  at com.vmware.eam.job.VibJob.call(VibJob.java:48)
  at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
  at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: java.nio.file.NoSuchFileException: /tmp/eam6690660256643099065/72a126f8-9ec3-4352-9d37-ebd3b8247660
  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
  at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
  at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
  at java.nio.file.Files.newOutputStream(Files.java:216)
  at com.vmware.eam.util.ByteSource.copyTo(ByteSource.java:96)
  at com.vmware.eam.fileserver.FileServer.publish(FileServer.java:89)
  
  
  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Cause

PhotonOS has introduced auto clean up policy which cleans /tmp folder on regular basis.
This issue occurs because the ESX Agent Manager uses the /tmp folder for storing published VIB payloads. This leads to the removal of folder structure used by EAM to store published payloads causing the EAM operation to fail.

Resolution

This is a known issue affecting vSphere 6.5 deployments using PhotonOS as underlying OS of vCenter Server.

This issue is resolved in vSphere 6.5 U1, latest releases available at Broadcom VCF Product Downloads.

To work around this issue, restart the EAM service by running these commands:

service-control --stop eam

service-control --start eam