HCX Network tasks failing due to problem to create new files: "Err: open /var/log/fscheck#: read-only file system"

Products

• VMware HCX

Issue/Introduction

• While performing an HCX network task, such as extension, un-extension, or enabling MON (Mobility Optimized Networking), the following error is observed in /common/logs/admin/app.log:

<Date> UTC [InterconnectService_SvcThread-####, IX:####-###-###-###-#####, J:####, , TxId: ####-###-###-###-#####] WARN  c.v.v.h.s.i.GenerateA
ndPostConfiguration- Failed to post the config using VIX for the appliance ####-###-###-###-#####. Retrying the operation.
java.lang.RuntimeException: Failed to upload file to guest @ /tmp/newConfig.proto for vm <NE-Appliance-Name>
        at com.vmware.vchs.hybridity.adapters.vcenter.GuestOperationsFacet.uploadFile(GuestOperationsFacet.java:487)
[...]
Caused by: com.vmware.vim.binding.vim.fault.CannotAccessFile: Unable to access file /tmp/newConfig.proto

• Network un-extension operation fails with error: POST_CONFIG_VIX
• From the HCX Manager UI, under Interconnect -> Service Mesh, when viewing appliances and clicking the "i - info" icon, you see the alarm:
  

  System state is critical
  Partition root has problem to create new files. Err: open /fscheck######: read-only file system
  partition log has problem to create new files. Err: open /var/log/fscheck######: read-only file system

 

Cause

Environmental factors, such as unexpected power-off events might cause the partition problem affecting Fleet appliances and most common factors are unrelated to HCX.

Resolution

• If the NE appliance is standalone: From the HCX Manager UI, under Interconnect -> Service Mesh> Appliances> Select the affected appliance and click Redeploy, use Force option: Manage Service Mesh Appliances
• If the NE appliance is part of a NE HA group:  From the HCX Manager UI, under Interconnect -> Service Mesh> View Appliances> HA Management, on the affected pair select Redeploy, Use Force Redeploy option: Managing Network Extension High Availability