Deploying vCenter High Availability fails with error: A general system error occurred: Failed to run pre-setup

Products

VMware vCenter Server

Issue/Introduction

Deploying vCenter High Availability fails with error: A general system error occurred: Failed to run pre-setup
- Logs present in /var/log/vmware/vcha/prepare-vcha.log matches the below snippet
  
  YYYY-MM-DDTHH:MM:SSZ ERROR prepare-vcha Failed to create vcha user, stderr: You are required to change your password immediately (root enforced)
  useradd: PAM: Authentication token is no longer valid; new one required
  code: 1
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha *** START: Vcha pre-setup ***
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha embedded DB Type can be vcha enabled.
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Found Eth1 interface with IP ##.###.##.##.###/##
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Completed Vcha Network config saved to vchaeth.cfg.
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha vcha.aware file created
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Opening firewall ports for VCHA...
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Reloading firewall...
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Saving clean pg_hba.conf...
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Saving clean pg-firewall..
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Completed setup of firewall for vcha.
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Installing the vcha-hacheck boot script...
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Setting interface eth0 to manual mode...
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Completed updating /etc/systemd/network/10-eth0.network to manual
  YYYY-MM-DDTHH:MM:SSZ INFO prepare-vcha Successfully set interface eth0 to manual mode.
  YYYY-MM-DDTHH:MM:SSZ ERROR prepare-vcha Failed to create vcha user, stderr: You are required to change your password immediately (root enforced)
  useradd: PAM: Authentication token is no longer valid; new one required
  code: 1
- Command chage -l root returns - Authentication token is no longer valid
  
  root@hostname [ ~ ]# chage -l root
  You are required to change your password immediately (root enforced)
  chage: PAM: Authentication token is no longer valid; new one required

vCenter High Availability configuration can fail due to incorrect value or has extra dot in IP address with same Error- A general system error occurred: Failed to run pre-setup
- Logs present in /var/log/vmware/vpxd/vpxd.log matches the below snippet
  
  YYYY-MM-DDTHH:MM:SSZ info vpxd[07076] [Originator@6876 sub=vpxLro opId=########-####-####-####-############] com.vmware.vcenter.vcha.cluster.deploy: taskId task-######:######-####-####-####-############
  YYYY-MM-DDTHH:MM:SSZ info vpxd[06741] [Originator@6876 sub=vpxLro opId=########-####-####-####-############] [VpxLRO] -- BEGIN task-######-- Datacenters -- com.vmware.vcenter.vcha.cluster.deploy --
  YYYY-MM-DDTHH:MM:SSZ error vpxd[06741] [Originator@6876 sub=vpxUtil opId=########-####-####-####-############] /usr/bin/python failed with error [1] and output []
  YYYY-MM-DDTHH:MM:SSZ error vpxd[06741] [Originator@6876 sub=vpxUtil opId=########-####-####-####-############] /usr/bin/python failed with error [1] and output [Traceback (most recent call last):
  --> File "/usr/lib/vmware-vcha/scripts/ipSubnet.py", line 62, in <module>
  --> main()
  --> File "/usr/lib/vmware-vcha/scripts/ipSubnet.py", line 57, in main
  --> if matchSubnet(ip1, ip2):
  --> File "/usr/lib/vmware-vcha/scripts/ipSubnet.py", line 33, in matchSubnet
  --> sameSubnet = checkSameSubnet(ip1, ip2)
  --> File "/usr/lib/vmware-vcha/scripts/ipSubnet.py", line 22, in checkSameSubnet
  --> ipnet2 = IPNetwork(ip2, strict=False)
  --> File "/usr/lib/python3.7/ipaddress.py", line 84, in ip_network
  --> address)
  --> ValueError: '##.###.###.### / ###.###..###.###' does not appear to be an IPv4 or IPv6 network
  --> ]
  YYYY-MM-DDTHH:MM:SSZ info vpxd[06741] [Originator@6876 sub=FailoverClusterConfigurator opId=######:######-####-####-####-############] Pre-setup: vcha.aware, SSH keys and PG
  YYYY-MM-DDTHH:MM:SSZ error vpxd[06741] [Originator@6876 sub=vpxUtil opId=########-####-####-####-############] /usr/bin/python failed with error [1] and output [logs available at: /var/log/vmware/vcha
  --> Traceback (most recent call last):
  --> File "/usr/lib/vmware-vcha/scripts/preSetup.py", line 479, in setupVchaEthCfg
  --> witnessIp, selfGatewayIp, peerGatewayIp, checkIps)
  --> File "/usr/lib/vmware-vcha/scripts/createVchaEthCfg.py", line 185, in createVchaEthCfg
  --> wtnsIp, node1GwIp, node2GwIp)
  --> File "/usr/lib/vmware-vcha/scripts/createVchaEthCfg.py", line 99, in checkAllIPFormat
  --> raise ValueError('node2Ip - %s: %s' % (err, IP_ERRS.get(err)))
  --> ValueError: node2Ip - 6: Expected valid Netmask or CIDR prefix. --> During handling of the above exception, another exception occurred: YYYY-MM-DDTHH:MM:SSZ info vpxd[06741] [Originator@6876 sub=FailoverClusterOperator opId=########-####-####-####-############] com.vmware.vcenter.vcha.cluster.deploy Request: --> },
  --> "deployment": "AUTO",
  --> "passive": {
  --> "failover_ip": null,
  --> "ha_ip": {
  --> "default_gateway": null,
  --> "dns_servers": null,
  --> "ip_family": "IPV4",
  --> "ipv4": {
  --> "address": "'##.###.###.###",
  --> "prefix": null,
  --> "subnet_mask": "###.###..###.###"
  --> }

Environment

VMware vCenter Server Appliance 6.x

VMware vCenter Server Appliance 7.x

Cause

The deployment fails because of root password being expired.
The deployment fails when the Subnet IP address is incorrect or has additional extra dot or value (example below)

Script at /usr/lib/vmware-vcha/scripts/createVchaEthCfg.py
- The failure is triggered at line 99, where the subnet mask for the Passive node IP is being validated. The value passed (###.###..###.#) contains an extra dot, which causes the script to raise a ValueError: 99 raise ValueError('node2Ip - %s: %s' % (err, IP_ERRS.get(err)))
- This corresponds with the deployment spec in the log snippet below:
  - "passive": {
    "ha_ip": {
    ...
    "ipv4": {
    "address": "##.###.###.###",
    "prefix": null,
    "subnet_mask": "###.###..###.#"
    }
    }
    }

Resolution

Resolution 1: Authentication Issue Observed in prepare-vcha.log

If an authentication error is encountered during VCHA deployment and the log indicates the root password needs to be changed, follow the steps below:

Check the root password status:
- root@hostname [ ~ ]# chage -l root
  You are required to change your password immediately (root enforced)
  chage: PAM: Authentication token is no longer valid; new one required
Reset the root password:
- root@hostname [ ~ ]# passwd
Verify the password status again:
- root@hostname [ ~ ]# chage -l root
  Last password change : Oct ##, ####
  Password expires : Jan ##, ####
  Password inactive : never
  Account expires : never
  Minimum number of days between password change : 0
  Maximum number of days between password change : 90
  Number of days of warning before password expires : 7
Once the password is successfully updated, retry the VCHA deployment.

Resolution 2: Invalid or Malformed IP Address Observed in /var/log/vmware/vpxd/vpxd.log

If the vpxd.log indicates an invalid IP address (e.g., extra dots or incorrect value in IP or Subnet), restart the required services:

SSH to the vCenter Server and run the following commands:
- service-control --restart --vsphere-ui
- service-control --restart --vmware-vpxd
Once the services are restarted, retry the VCHA deployment with correct entries in IP Address/Subnet and follow through the steps in the official documentation: Configure vCenter HA With the vSphere Client