After redeploying the failed cell, the number of nodes in the VCD runtime topology does not match the number of running nodes.

Products

• VMware Cloud Director

Issue/Introduction

After a VCD cell failure and redeployed the node, the number of nodes output to /opt/vmware/vcloud-director/logs/cell-runtime.log may not match the number of running nodes.

2025-10-05 13:21:38,181 | DEBUG    | Message Broker Topology Snapshotter | BrokerContext                  | Updated the last effective date of the latest topology state from 2025-10-05T13:20:48.033854Z | 
2025-10-05 13:21:38,966 | DEBUG    | Message Broker Topology Logger | BrokerContext                  | 
defaultTopology=topology on Topology@5a9665be[owner=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=###-###-###-###, TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=###-###-###-###, TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=###-###-###-###, TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=###-###-###-###], discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='vcd-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=1000}]]:
        b493####-####-####-####-########ddf3 => TopologyMember[id=b493####-####-####-####-########ddf3, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-12, b=null], backupGroupName=null, scaleDownGroupName=null]
        209d####-####-####-####-########21e1 => TopologyMember[id=209d####-####-####-####-########21e1, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-15, b=null], backupGroupName=null, scaleDownGroupName=null]
        bf65####-####-####-####-########61b5 => TopologyMember[id=bf65####-####-####-####-########61b5, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-11, b=null], backupGroupName=null, scaleDownGroupName=null]
        cf72####-####-####-####-########6dff => TopologyMember[id=cf72####-####-####-####-########6dff, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-13, b=null], backupGroupName=null, scaleDownGroupName=null]
        ba65####-####-####-####-########5568 => TopologyMember[id=ba65####-####-####-####-########5568, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-11, b=null], backupGroupName=null, scaleDownGroupName=null]
        3796####-####-####-####-########f52c => TopologyMember[id=3796####-####-####-####-########f52c, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-14, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=6 members=6,
-----------------------------------
vcd-cluster=topology on Topology@ccb6a32[owner=ClusterConnectionImpl@400456800[nodeUUID=b493####-####-####-####-########ddf3, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=###-###-###-###, address=, server=ActiveMQServerImpl::name=b493####-####-####-####-########ddf3]]:
        b493####-####-####-####-########ddf3 => TopologyMember[id=b493####-####-####-####-########ddf3, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-12, b=null], backupGroupName=null, scaleDownGroupName=null]
        209d####-####-####-####-########21e1 => TopologyMember[id=209d####-####-####-####-########21e1, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-15, b=null], backupGroupName=null, scaleDownGroupName=null]
        bf65####-####-####-####-########61b5 => TopologyMember[id=bf65####-####-####-####-########61b5, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-11, b=null], backupGroupName=null, scaleDownGroupName=null]
        cf72####-####-####-####-########6dff => TopologyMember[id=cf72####-####-####-####-########6dff, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-13, b=null], backupGroupName=null, scaleDownGroupName=null]
        ba65####-####-####-####-########5568 => TopologyMember[id=ba65####-####-####-####-########5568, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-11, b=null], backupGroupName=null, scaleDownGroupName=null]
        3796####-####-####-####-########f52c => TopologyMember[id=3796####-####-####-####-########f52c, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)?port=61616&host=172-###-###-14, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=6 members=6 | 

In this example, there is one primary cell, two standby cells, and two application cells, so the number of nodes should be 5, but the number of nodes in cell-runtime.log is 6.

Environment

VMware Cloud Director 10.5.x
VMware Cloud Director 10.6.x

Cause

If you set the same IP address of a removed failure node to a redeployed node, both the topology of the removed node and the redeployed node may be detected between cells, which can lead to an inconsistency in the node count.
In the log example in Issue/Introduction, the cell with the IP address 172-###-###-11 was redeployed.

This issue may cause a degradation in VCD service performance.

Resolution

Broadcom is aware of this issue and working to fix it in a future release.

 

Workaround:

There are two workarounds to fix the number of nodes in cell-runtime.log : 

1. Configure the redeployed node's IP address to be different from the failure node's.

or

2. Stop all cells, then start all cells

 

The procedure for stopping and starting all nodes is described in the following document: 

Perform an Orderly Shutdown and Startup of Your VMware Cloud Director Appliance Cluster