VMware NSX Controller service is down on NSX-T Managers with ccp_oom core dumps.

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• Controller service is down on the NSX manager appliance.

Controller Status:
Group Type: CONTROLLER
Group Status: UNAVAILABLE
Members:
UUID FQDN IP STATUS
########-####-####-####-######### ABC123 192.xxx.xx.xx DOWN
########-####-####-####-######### ABC123 192.xxx.xx.xx DOWN
########-####-####-####-######### ABC123 192.xxx.xx.xx DOWN
 

 

The rule count in the controller dump doesn’t match the realized number of rules in desired_state_manager.json. We see this in the desired state manager rule count and the following errors in the corfu-compactor-audit.log and the /var/log/corfu/corfu-compactor-audit.log on the NSX manager.
 

 },
  "/nsxapi/api/v1/firewall/rules": "Exception: timed out",
  "/nsxapi/api/v1/firewall/sections": "Exception: timed out",
  "/nsxapi/api/v1/firewall/sections/summary": {
    "last_compute_time": 1681850115214,
    "sections_summary": [
      {
        "rule_count": 1,
        "section_count": 1,
        "section_type": "L2DFW"
      },
      {
        "rule_count": 10968,
        "section_count": 358,
        "section_type": "L3DFW"
      },
      {
        "rule_count": 118,
        "section_count": 118,
        "section_type": "L3LOGICALROUTERFW"
      }
    ]
 

/var/log/corfu/corfu-compactor-audit.log

 

1892:2023-04-15T20:36:45.586Z INFO main SMRObject - ObjectBuilder: open Corfu stream nsx$FirewallRule id #######-####-####-####-#########
1895:2023-04-15T20:36:45.626Z INFO main CheckpointWriter - appendCheckpoint: Started checkpoint for #######-####-####-####-######### at snapshot Token(epoch=1178, sequence=4360116475)
3420:2023-04-15T20:43:03.916Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for #######-####-####-####-#########, entries(690295), cpSize(298894826) bytes at snapshot Token(epoch=1178, sequence=4360116475) in 378290 ms
4475:2023-04-15T20:47:53.906Z INFO main SMRObject - ObjectBuilder: open Corfu stream nsx$FirewallRule id #######-####-####-####-#########
4478:2023-04-15T20:47:53.934Z INFO main CheckpointWriter - appendCheckpoint: Started checkpoint for #######-####-####-####-######### at snapshot Token(epoch=1178, sequence=4360175529)
6117:2023-04-15T20:54:38.649Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for #######-####-####-####-#########, entries(690295), cpSize(298894826) bytes at snapshot Token(epoch=1178, sequence=4360175529) in 404715 ms
22111:2023-04-15T21:05:35.660Z INFO main SMRObject - ObjectBuilder: open Corfu stream nsx$FirewallRule id #######-####-####-####-#########

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

The rule count in the controller dump doesn’t match the realized number of rules in the desired_state_manager.json file.

Resolution

This issue is resolved in VMware NSX-T 3.2.3.1 and 4.1.1

Workaround:

• If you believe you have encountered this issue, please open a support request with Broadcom Support and refer to this KB article.

Additional Information

Impact/Risks:

NSX-T Manager Controller connectivity is down on the NSX manager appliance.