clock_global VMs unable to start

Products

• VMware Tanzu Platform Core

Issue/Introduction

clock_global VMs unable to start
Customer has  clock_global 3 VMs configured, but only 1 is running.  Customer tried to recreate, but that failed with the error listed below.

 L Error: 'clock_global/xxxxxxxx (0)' is not running after update. Review logs for failed jobs: leadership-election

In the logs for VM clock_global/xxxxxxxx  we see
2025/09/05 15:32:58 failed to create raft TCP transport: listen tcp xxx.xxx.xxx.xxx:xxxx bind: cannot assign requested address

Resolution

Asked the customer to run following command on both the running and failing clock global VM

$ip address

also asked to collect the bpm config for each, file is located here: /var/vcap/jobs/leadership-election/config

Need to compare these values between the success and failing vm

    env:
      PORT: xxxx
      HEALTH_PORT: xxxx
      NODE_INDEX: "0"
      NODE_ADDRS: "xxx.xxx.xxx.xxx:xxxx"

------------------------------------------------------------------------------------------------------------------------
Running:
clock_global/xxxxxxxx

clock_global/xxxxxxxx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet xxx.xxx.xxx.xxx/8 scope host lo
       valid_lft forever preferred_lft forever
    inet xxx.xxx.xxx.xxx/32 scope global lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    altname ens160
    inetxxx.xxx.xxx.xxx/24 brdxxx.xxx.xxx.xxx scope global eth0
       valid_lft forever preferred_lft forever

clock_global/xxxxxxxx:/var/vcap/jobs/leadership-election/config# more bpm.yml

processes:
  - name: leadership-election
    executable: /var/vcap/packages/leadership-election/leadership-election
    env:
      PORT: xxxx
      HEALTH_PORT: xxxx
      NODE_INDEX: "0"
      NODE_ADDRS: "xxx.xxx.xxx.xxx:7100,xxx.xxx.xxx.xxx:7100,xxx.xxx.xxx.xxx:7100"
      CA_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election_ca.crt"
      CERT_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election.crt"
      KEY_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election.key"

      METRICS_PORT: xxxx
      METRICS_CA_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics_ca.crt"
      METRICS_CERT_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics.crt"
      METRICS_KEY_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics.key"

-----------------------------------------------------------------------------------------------------------

Failing:

clock_global/xxxxxxxx

clock_global/xxxxxxxx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet xxx.xxx.xxx.xxx/8 scope host lo
       valid_lft forever preferred_lft forever
    inet xxx.xxx.xxx.xxx/32 scope global lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    altname ens160
    inet xxx.xxx.xxx.xxx/24 brd xxx.xxx.xxx.xxx scope global eth0
       valid_lft forever preferred_lft forever

clock_global/xxxxxxxx :~# more /var/vcap/jobs/leadership-election/config/bpm.yml

processes:
  - name: leadership-election
    executable: /var/vcap/packages/leadership-election/leadership-election
    env:
      PORT: xxxx
      HEALTH_PORT: xxxx
      NODE_INDEX: "2"
      NODE_ADDRS: "xxx.xxx.xxx.xxx:7100,xxx.xxx.xxx.xxx:7100,xxx.xxx.xxx.xxx:7100"
      CA_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election_ca.crt"
      CERT_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election.crt"
      KEY_FILE: "/var/vcap/jobs/leadership-election/config/certs/leadership_election.key"

      METRICS_PORT: xxxx
      METRICS_CA_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics_ca.crt"
      METRICS_CERT_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics.crt"
      METRICS_KEY_FILE_PATH: "/var/vcap/jobs/leadership-election/config/certs/metrics.key"

----------------------------------------------------------------------------------------------------

DASH STATE:

clock_global/xxxxxxxx

/xxxxxxxx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inetxxx.xxx.xxx.xxx/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:ad:4c:dc brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    altname ens160
    inetxxx.xxx.xxx.xxx/24 brd xxx.xxx.xxx.xxx scope global eth0
       valid_lft forever preferred_lft forever

** no bpm.yml

 Customer does not see any reference to IPs in the manifest.

 Re-deploy of the manifest has resolved the issue.