Index out of sync error in stateDTO in Federation environment

Products

• VMware NSX-T Data Center

Issue/Introduction

• NSX-T Manager is not listing any Hosts and Virtual Machine inventory over the UI in the Federation environment

                     Fabric -> Hosts/Clusters and Inventory -> Virtual Machines

• Below errors are seen in the UI:

              "system is still in uncertain state please reload."

              "Error: Index out of sync, please resync via 'start search resync policy'"

Environment

VMware NSX-T Data Center

Cause

An exception was thrown in the stateDTO at the time of indexing in UI, which caused the error in UI indexing.

When you performed resync, it just recalculated the state again, and it came out of the error.

Resolution

• This is fixed with VMware NSX 3.2.3 and above.

Workaround:

• Perform the below workaround only if the API output returns records with "nsx_search_manager_DeploymentProgress" OR "nsx_search_policy_DeploymentProgress"

                  "_id" : "nsx_search_manager_DeploymentProgress"
                  "_id : "nsx_search_policy_DeploymentProgress"

• Use the below API to check the stream tag for errors so that you can restart accordingly (search resync policy or search resync manager)

curl -X GET "http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty" -H 'Content-Type: application/json'

sample:

application/json'

{

"took" : 10,

"timed_out" : false,

"_shards" : {

"total" : 4,

"successful" : 4,

"skipped" : 0,

"failed" : 0,

 },

"hits" : {

"total" : {

"value" : 1,

"relation" : "eq"

},

"max score" : 0.0013670542,

"hits" : [

 {

  "_index" : "policy table_indexing position",

  "type" : "_doc",

  "_id" : "nsx_search_manager_DeploymentProgress", <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  "_score" : 0.0013670542,

  "source" : {

   "namespace" : "nsx",

"streamTag” : "search_manager", <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

"tableName" : "DeploymentProgress"

"lastIndexingPosition" : 428318533,

"lastIndexingEpoch" : 130,

"hasFailures" : false,

"hasExhaustedFailures" : true,

}

  }

  ]

 }

}

 

• Perform a resync on the affected local manager node using the command below.

                start search resync manager

                OR 

              start search resync policy

if "streamTag” : "search_policy",
 

 

Additional Information

Impact/Risks:

Unable to view the inventories (Hosts, virtual machines) in Local Managers (LMs)