NSX-T Host cluster update fails with Exception: Failed to load module
search cancel

NSX-T Host cluster update fails with Exception: Failed to load module

book

Article ID: 322631

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • NSX-T host upgrade fails with error message Exception:
Failed to load module nsx-esx-70u3/nsxt-vsip-19801961: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19801961): Failure
  • Unable to upgrade some host clusters and the environment is running on mixed NSX-T versions.
  • On the ESXi host, in /var/run/log/vmkernel.log, you would observe entries similar to the below:
2023-02-16T05:35:52.465Z cpu72:2115258)NetDVS: 188: A client exists for data com.vmware.port.extraConfig.serviceInsertion.gvm on portset .
2023-02-16T05:35:52.465Z cpu72:2115258)NetX Props: Failed to register com.vmware.port.extraConfig.serviceInsertion.gvm callback
2023-02-16T05:35:52.465Z cpu72:2115258)NetX Global: IOChain Initialization failed: Already exists
...
2023-02-16T05:35:53.998Z cpu72:2115258)nsxt-vsip-19801961 failed to load.
2023-02-16T05:35:53.998Z cpu72:2115258)WARNING: Elf: 3139: Kernel based module load of nsxt-vsip-19801961 failed: Failure <Mod_LoadDone failed>
2023-02-16T05:35:54.032Z cpu72:2115267)Mod: 4921: Unloading module <nsxt-rtm-19801961> ...
  • Running the command net-dvs -l | grep com.vmware.port.extraConfig.serviceInsertion.gvm we see one or more ports with this extraconfig enabled.
...
com.vmware.port.extraConfig.serviceInsertion.gvm = true , propType = POLICY
com.vmware.port.extraConfig.serviceInsertion.gvm = true , propType = POLICY
com.vmware.port.extraConfig.serviceInsertion.gvm = true , propType = POLICY
...
  • If we check the port, it has extraConfig is attached to and we see it is a vmkernel port:
port f5d3c4c8-fba3-4376-ba4f-b1408899421e:
                com.vmware.common.port.alias = f5d3c4c8-fba3-4376-ba4f-b1408899421e , propType = CONFIG
                com.vmware.common.port.connectid = 1628250306 , propType = CONFIG
                com.vmware.common.port.block = false , propType = CONFIG
                com.vmware.port.extraConfig.vnic.external.id = b5250000-0200-0000-0000-00000000003f:vmk2 , propType = CONFIG <<<< vmkernel port
                com.vmware.port.extraConfig.opaqueNetwork.id = 7fe555bf-bdb4-4c6b-8158-73a087bfc9a5 , propType = CONFIG
                com.vmware.vswitch.port.security = 0x 5. 0 <repeats 3 times>
                        propType = POLICY
                com.vmware.vswitch.port.vlan = 0x 0. 0.26. 3. 0 <repeats 4 times>
                        propType = POLICY
                com.vmware.vswitch.port.macManagement = 0x 1. 0. 0. 0.10. 0 <repeats 6 times>
                        propType = POLICY
                ...
                com.vmware.net.vxlan.traffic.marking = 0x 1. 0 <repeats 3 times>
                        propType = POLICY
                com.vmware.net.port.fc.enabled = 0x 3. 0 <repeats 3 times>
                        propType = POLICY
                com.vmware.port.extraConfig.serviceInsertion.gvm = true , propType = POLICY <<<< SI extraConfig


Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T

Cause

An object with the property “com.vmware.port.extraConfig.serviceInsertion.gvm” is assigned in the environment and consequently the vsip module install is failing. This issue occurs when service insertion used and push's the SI extraConfig to the vmkernel ports.

Resolution

To resolve this issue upgrade to NSX 3.2.2 available at VMware downloads.

Workaround:
  • Make sure the host is not in Maintenance Mode in NSX or vCenter.
  • Find the vmk2 port number from "net-stats -l | grep vmk2" command, we know it is vmk2 from the results of the net-dvs -l as per the symptoms above.
  • For example:
#net-stats -l | grep vmk2
100663324            3       0 DvsPortset-0     00:50:56:62:9a:36  vmk2
  • Get the net-dvs -l from each host and search for the vmk2 portnumber (from net-stats above 100663324) and look for:
    • com.vmware.port.extraConfig.serviceInsertion.gvm = true , propType = POLICY
    • com.vmware.common.port.volatile.status = inUse linkUp portID=100663324
Example:
port ebc60084-55ab-4b04-8dfa-eb147b352ed4:
        com.vmware.common.port.alias = ebc60084-55ab-4b04-8dfa-eb147b352ed4 ,  propType = CONFIG
        com.vmware.common.port.connectid = 1628661712 ,     propType = CONFIG
        com.vmware.common.port.block = false , propType = CONFIG
        com.vmware.port.extraConfig.vnic.external.id = b5250000-2400-0000-0000-00000000002b:vmk2 ,   propType = CONFIG
        com.vmware.port.extraConfig.opaqueNetwork.id = ff97e4b6-5f6c-4723-ac49-cb07936b65fd ,  propType = CONFIG
        ...
        com.vmware.port.extraConfig.global.id = ebc60084-55ab-4b04-8dfa-eb147b352ed4 , propType = POLICY
        com.vmware.port.opaque.network.id = ff97e4b6-5f6c-4723-ac49-cb07936b65fd ,   propType = RUNTIME
        com.vmware.port.opaque.network.type = nsx.LogicalSwitch ,    propType = RUNTIME
        ...
        com.vmware.common.port.volatile.status = inUse linkUp portID=100663324 propType = RUNTIME
        com.vmware.common.port.volatile.vlan = VLAN 888
            propType = RUNTIME
        ...
        com.vmware.port.extraConfig.serviceInsertion.gvm = true ,    propType = POLICY
  • Take note of the port alias ebc60084-55ab-4b04-8dfa-eb147b352ed4.
Note: An entry for a port in the net-dvs command's result's, start with port <alias ID> e.g.  port ebc60084-55ab-4b04-8dfa-eb147b352ed4:, as can be seen by the lesser indent of the line, and continues until the next port <alias ID>, all in between belongs to the port <alias ID> above it.
  • Scroll up slowly and find the switch where this port <alias ID>, is present. If there is only one switch, then all ports are listed under it. If there are multiple switches, you need to find the switch where this port alais comes under. The same indentation rule is true for switch as is for port, that is to say ports under a switch will have a greater indent:
Example
switch ce dc b2 87 c5 c8 43 a6-a2 7f 32 80 86 bc 5c 1e (vswitch)
max ports: 9216
global properties:
com.vmware.common.opaqueDvs = true , propType = CONFIG
com.vmware.common.alias = nvds.DVswitch01-Switch-A , propType = CONFIG
com.vmware.common.uplinkPorts:
Uplink 1, Uplink 2
propType = CONFIG
...
...
port ebc60084-55ab-4b04-8dfa-eb147b352ed4:
        com.vmware.common.port.alias = ebc60084-55ab-4b04-8dfa-eb147b352ed4 ,  propType = CONFIG
        com.vmware.common.port.connectid = 1628661712 ,     propType = CONFIG
        com.vmware.common.port.block = false , propType = CONFIG
        com.vmware.port.extraConfig.vnic.external.id = b5250000-2400-0000-0000-00000000002b:vmk2 ,   propType = CONFIG
        com.vmware.port.extraConfig.opaqueNetwork.id = ff97e4b6-5f6c-4723-ac49-cb07936b65fd ,  propType = CONFIG
        ...
        com.vmware.port.extraConfig.global.id = ebc60084-55ab-4b04-8dfa-eb147b352ed4 , propType = POLICY
        com.vmware.port.opaque.network.id = ff97e4b6-5f6c-4723-ac49-cb07936b65fd ,   propType = RUNTIME
        com.vmware.port.opaque.network.type = nsx.LogicalSwitch ,    propType = RUNTIME
        ...
        com.vmware.common.port.volatile.status = inUse linkUp portID=100663324 propType = RUNTIME
        com.vmware.common.port.volatile.vlan = VLAN 888
            propType = RUNTIME
        ...
        com.vmware.port.extraConfig.serviceInsertion.gvm = true ,    propType = POLICY
  • Note down the switch alias from com.vmware.common.alias information from the above output.
  • Run the "net-dvs -u <extraConfig NAME> -p <PORT_ID> <SWITCH_NAME>" to unload the SI extraConfig from the port on the switch. This will need to be repeated for each port found and on each host failing.
Example 
net-dvs -u com.vmware.port.extraConfig.serviceInsertion.gvm -p ebc60084-55ab-4b04-8dfa-eb147b352ed4 nvds.DVswitch01-Switch-A


Powered by Wolken Software