Host transport node shows install failed after manually changing NSX vibs on the host

Products

• VMware NSX

Issue/Introduction

• Host transport node shows install failed after manually changing NSX vibs on the host.
• Host overlay networking can be impacted. 
• You can see the following error in the API response.

GET /api/v1/transport-nodes/state

      {
        ...
        "maintenance_mode_state": "DISABLED",
        "node_deployment_state": {
          "details": [
            {
              "failure_code": 26080,
              "failure_message": "Node has invalid version <upgraded-version> of software nsx-monitoring.Node has invalid version <upgraded-version> of software nsx-vdpi.Node has invalid version <upgraded-version> of software nsx-shared-libs.Node has invalid version <upgraded-version> of software nsx-proxy.Node has invalid version <upgraded-version> of software nsx-netopa.Node has invalid version <upgraded-version> of software nsx-python-utils.Node has invalid version <upgraded-version> of software nsx-nestdb.Node has invalid version <upgraded-version> of software nsx-esx-datapath.Node has invalid version <upgraded-version> of software nsx-python-logging.Node has invalid version <upgraded-version> of software nsx-context-mux.Node has invalid version <upgraded-version> of software nsx-exporter.Node has invalid version <upgraded-version> of software vsipfwlib.Node has invalid version <upgraded-version> of software nsx-ids.Node has invalid version <upgraded-version> of software nsx-sfhc.Node has invalid version <upgraded-version> of software nsx-opsagent.Node has invalid version <upgraded-version> of software nsxcli.Node has invalid version <upgraded-version> of software nsx-cpp-libs.Node has invalid version <upgraded-version> of software nsx-proto2-libs.Node has invalid version <upgraded-version> of software nsx-adf.Node has invalid version <upgraded-version> of software nsx-platform-client.Node has invalid version <upgraded-version> of software nsx-cfgagent.Node has invalid version <upgraded-version> of software nsx-host.Node has invalid version <upgraded-version> of software nsx-mpa.",
              "state": "failed",
              "sub_system_id": "########-####-####-####-############"
            }
          ],
          "state": "failed"
        },
        "state": "success",
        "transport_node_id": "########-####-####-####-############"
      },

Environment

VMware NSX-T Data Center
VMware NSX

Cause

User manually upgraded vibs on the host without starting NSX upgrade procedure.

Resolution

Host needs to be put in Maintenance mode, cleaned up and then re-prepared as Transport Node.