vCenter Server repeatedly generates "A general system error occurred: Internal error" during "Update option values" tasks - VMWARE vSphere.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The vCenter frequently generates failed tasks mentioned below.
  
  Task Name                Target                            Status                                              Initiator      
Update option values     ##-####-##,###########.#####      A general system error occurred: Internal error     ########.#####\###-##-##-########-####-####-####-############-################-#
Update option values     ##-####-##,###########.#####      A general system error occurred: Internal error     ########.#####\###-##-##-########-####-####-####-############-################-#

• The /var/run/log/hostd.log file report failures in setting the syslog configuration because DNS was unable to resolve the FQDN of the remote syslog server:

  YYYY-MM-DDTHH:MM:SS.sssZ Er(163) Hostd[2101311]: [Originator@6876 sub=Hostsvc.SyslogConfigProvider opID=########-##-#### sid=######## user=vpxuser:########.#####\###-##-##-########-####-####-####-############-################-#] Failed to set global syslog configuration : Failed to resolve remote host ##################.
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101311]: [Originator@6876 sub=Solo.Vmomi opID=########-##-#### sid=######## user=vpxuser:########.#####\###-##-##-########-####-####-####-############-################-#] Activation finished; <<########-cad5-3a6c-4d2e-4059559f21cc, <TCP '###.#.#.# : 8307'>, <TCP '###.#.#.# : 55230'>>, ha-adv-options, vim.option.OptionManager.updateValues, <vim.version.v9_
0_0_0, internal, 9.0.0.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE]>
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101311]: [Originator@6876 sub=Solo.Vmomi opID=########-##-#### sid=######## user=vpxuser:########.#####\###-##-##-########-####-####-####-############-################-#] Arg changedValue:
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: --> (vim.option.OptionValue) [
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: -->    (vim.option.OptionValue) {
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: -->       key = "Syslog.global.logHost",
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: -->       value = "udp://###.###.##.##:514,tcp://##################:514,tcp://##.###.#.##:514,tcp://##################:514?formatter=RFC_5424&framing=octet_counting"
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: -->    }
YYYY-MM-DDTHH:MM:SS.sssZ Db(167) Hostd[2101298]: --> ]
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101311]: [Originator@6876 sub=Solo.Vmomi opID=########-##-#### sid=######## user=vpxuser:########.#####\###-##-##-########-####-####-####-############-################-#] Throw vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101311]: [Originator@6876 sub=Solo.Vmomi opID=########-##-#### sid=######## user=vpxuser:########.#####\###-##-##-########-####-####-####-############-################-#] Result:
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101298]: --> (vmodl.fault.SystemError) {
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101298]: -->    reason = "Internal error",
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101298]: -->    msg = "",
YYYY-MM-DDTHH:MM:SS.sssZ In(166) Hostd[2101298]: --> }

• The /var/log/vmsyslogd.log file reports DNS resolution failures when attempting to resolve the FQDN of the syslog server:
  
  YYYY-MM-DDTHH:MM:SS.sssZ vmsyslog.loggers.network : ERROR   ] getaddrinfo: host ################################# port 514: [Errno -3] Temporary failure in name resolution
YYYY-MM-DDTHH:MM:SS.sssZ vmsyslog.loggers.network : ERROR   ] failed to establish connection to remote syslog server #################################:514

Environment

• VMware vSphere ESXi 8.x
• VMware vSphere ESXi 9.x

Cause

The issue is primarily caused by a failure in DNS name resolution on the affected ESX hosts. This can stem from two root scenarios:

• Firewall Misconfiguration: Firewall rules on the network or the ESXi host (vmk0) are preventing the necessary traffic (ICMP/DNS) between the ESX host and the DNS server.
• DNS Infrastructure Configuration: The DNS servers are not correctly configured to resolve the specific FQDN required by the ESXi host, or the ESX host is unable to reach the configured DNS server IP.

 

Resolution

To resolve this issue, ensure the ESXi host has functional connectivity to the DNS servers and that the DNS records are correctly configured.

1. Verify DNS Connectivity: Verify the ESXi host can reach the configured DNS servers (e.g., via ping or nslookup if available in your version).

2. Check Firewall Rules: Ensure that network-level firewall rules allow traffic from the ESXi management interface (vmk0) to the DNS server IP addresses.

   • Confirm that DNS (typically UDP/TCP port 53) traffic is permitted in both directions.
   • If using the ESXi internal firewall, you can temporarily test connectivity by disabling it to isolate the issue:
     esxcli network firewall set --enabled false
   • Note: Remember to re-enable the firewall after testing:
     esxcli network firewall set --enabled true

3. Validate DNS Records: Work with your network administrator to verify that the DNS A or CNAME records for the syslog server FQDN are correctly configured and resolvable by the network segment where the ESXi hosts reside.

Additional Information

Configuring syslog on ESXi