VNA Bulk Deletion and Re-add of SDN_NetworkInterface Models Due to Partial Viptela vManage Inventory Fetch (HTTP 503)

Products

• Network Observability
• Spectrum
• Virtual Network Assurance

Issue/Introduction

A customer reported that nearly 100% of their SDN_NetworkInterface models (approximately 31,000 models) were unexpectedly destroyed and immediately recreated within a five-minute period in Spectrum.
This occurred with a VNA (Virtual Network Assurance) integration with Viptela.

The database logs (from the DDM db) confirmed the events:

• Destroys (Event Type 0x10202): 30,844 models were destroyed

• Creates (Event Type 0x10201): 30,980 models were created

Environment

any supported Spectrum until 25.4.3 + VNA (Viptela)

Cause

The bulk deletion and subsequent re-add were traced to an issue during the VNA inventory poll cycle due to a temporary external factor:

 

1. Viptela vManage Unreachability: During the inventory poll cycle that began at 06:40:00 on November 28th, the Viptela vManage server became unreachable, returning HTTP 503 ("Service Unavailable") errors.

2. Partial Inventory Fetch: Because the server returned a "Service Unavailable" response midway through the fetch (pagination failure), the collector only retrieved a partial inventory (approximately 20,000 of the 30,980 records).

3. VNA Misinterpretation & Bulk Delete: VNA interpreted the missing records as having been removed from the environment, which triggered a bulk delete of 30,844 entities and 114,141 relations.

   • Supporting Log Snippet (Gateway.log): The log recorded the delete event: Inventory changes detected (Entity changes: 0 added, 0 updated, 30844 deleted - Relation changes: 0 added, 114141 deleted) at 2025-11-28 06:40:59,979.

4. Bulk Re-add: The subsequent poll at 06:50:00 was successful, retrieving the full inventory, which resulted in the immediate bulk re-add.

   • Supporting Log Snippet (Gateway.log): The log recorded the re-add event: Inventory changes detected (Entity changes: 30980 added, 0 updated, 0 deleted - Relation changes: 30980 added, 0 deleted) at 2025-11-28 06:52:13,299.

Resolution

The deletion was deemed "expected behavior" in the current VNA logic for partial inventory processing but was not the desired outcome for this scenario.

A defect was created (DE180214) to prevent this in the future. The fix will involve improving the logic so that VNA does not publish partial inventory messages if pagination fails due to external factors (such as network unreachability or Service Unavailable errors).

Defect fixed in Spectrum 25.4.4