Index out of sync error in stateDTO in Federation environment

Products

• VMware NSX

Issue/Introduction

• This is a known issue with NSX-T-3.2.2
• NSX-T Manager not listing any Hosts and Virtual Machine inventory over the UI in Federation environment

Fabric -> Hosts/Clusters and Inventory -> Virtual Machines

• Below errors are seen in the UI:

"system is still in uncertain state please reload."

"Error: Index out of sync, please resync via 'start search resync policy'"

Environment

VMware NSX-T Data Center

Cause

Exception thrown in stateDTO at the time of indexing in UI which caused the error in UI indexing. When you performed resync, it just recalculated the state again and it came out of the error.

Resolution

This is fixed with VMware NSX 3.2.3 and above.

Workaround:

• Perform the below workaround only if API output returns records with "nsx_search_manager_DeploymentProgress" OR "nsx_search_policy_DeploymentProgress"

  "_id" : "nsx_search_manager_DeploymentProgress"
"_id : "nsx_search_policy_DeploymentProgress"

• Use the below API to check the stream tag for errors so that you can restart accordingly (search resync policy or search resync manager)

curl -XGET "http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty" -H 'Content-Type: application/json'

 

sample:

application/json'

{

"took" : 10,

"timed_out" : false,

"_shards" : {

"total" : 4,

"successful" : 4,

"skipped" : 0,

"failed" : 0,

 },

"hits" : {

"total" : {

"value" : 1,

"relation" : "eq"

},

"max score" : 0.0013670542,

"hits" : [

 {

  "_index" : "policy table_indexing position",

  "type" : "_doc",

  "_id" : "nsx_search_manager_DeploymentProgress", <<<<<<<<<<<<

  "_score" : 0.0013670542,

  "source" : {

   "namespace" : "nsx",

"streamTag” : "search_manager", <<<<<<<<

"tableName" : "DeploymentProgress"

"lastIndexingPosition" : 428318533,

"lastIndexingEpoch" : 130,

"hasFailures" : false,

"hasExhaustedFailures" : true,

}

  }

  ]

 }

}

 

Perform a resync on the affected local manager node using below command.

start search resync manager

OR 

start search resync policy

if "streamTag” : "search_policy",
 

 

Additional Information

Impact/Risks:

Unable to view the inventories (Hosts, virtual machines) in Local Managers (LMs)