Symptoms:
- Virtual Machines lose Northbound connectivity after vMotion (North/South).
- Affected North/South VM traffic is flowing through Hardware VTEP.
- East/West traffic is not affected.
- NSX Controller ToR Agent logs (show log cloudnet/toragent.log) show errors similar to:
2019-06-14 12:40:32,689 | DEBUG | nioEventLoopGroup-3-1 | JsonRpcEndpoint | Response : {"id":"34c38828-5a11-4310-bf0a-dd15194cb5cc","result":[{"count":1},{},{"details":"Table Ucast_Macs_Remote column locator row d2e2f641-b925-42d7-90a8-6a875d75eb52 references nonexistent row 22054b92-7c1c-4622-a21c-d81dcebb77d3 in table Physical_Locator.","error":"referential integrity violation"}],"error":null}
2019-06-14 12:40:32,690 | ERROR | pool-6-thread-9 | TorInorderMessageProcessor | Error when processing a QueueElement
com.vmware.toragent.tormgr.util.TransactionException: ExceptionMessage: Error updating rowOperationsRequested: 2OperationsExecuted: 3
at com.vmware.toragent.tormgr.southbound.TorClient.executeTransaction(TorClient.java:2012)
at com.vmware.toragent.tormgr.southbound.TorClient.executeTransaction(TorClient.java:1984)
at com.vmware.toragent.tormgr.southbound.UcastMacEvent.updateUcastMacRemoteRow(UcastMacEvent.java:132)
at com.vmware.toragent.tormgr.southbound.UcastMacEvent.processUpdate(UcastMacEvent.java:73)
at com.vmware.toragent.tormgr.southbound.SouthboundEvent.processElement(SouthboundEvent.java:79)
at com.vmware.toragent.tormgr.lib.TorInorderMessageProcessor$TorInorderQueueElement.processElement(TorInorderMessageProcessor.java:261)
at com.vmware.toragent.tormgr.lib.QueueElementRunner.run(TorQueueElementProcessor.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.