We have a deployment that has been stopped because it was running far too long.
On analysis we see the error in the logs.
Agent Logs:
2017-03-31 15:27:33,094 [New I/O client worker #1-2] ERROR (com.nolio.nimi.comm.impl.nettysupport.BasicHandler:57) - NimiConnectionImpl
lastAccessedTime=1490966853094}:java.lang.IllegalStateException: FAT not created
java.lang.IllegalStateException: FAT not created
at com.nolio.nimi.persistency.filepersister.PersistencyFile.setPosition(PersistencyFile.java:613)
at com.nolio.nimi.persistency.filepersister.PersistencyFile.deleteEntryInFileTable(PersistencyFile.java:336)
at com.nolio.nimi.persistency.filepersister.PersistencyFile.delete(PersistencyFile.java:193)
Agent logs may be overwritten with lot of such error and we may lost the data and actual cause of the issue. The first occurrence of this issue is very informative which reflect that the above error occurred due to not having enough disk space.
2017-03-30 23:31:45,578 [HealthMonitor] ERROR (com.nolio.nimi.persistency.filepersister.PersistencyFile:101) - problem creating new fat file
java.io.IOException: There is not enough space on the disk
Make sure there is enough space on the agent server and re-ran the deployment.