new installation of proxy/da's consul is not working. No ACL key generated during installation.
Article ID: 233906
Updated On:
Products
Issue/Introduction
The install logs are clear, I reinstalled the proxy, nothing is working.
Seeing the following errors in the systemctl status output for consul
Feb 03 15:33:27 <SYSTEM>.<COMPANY>.com consul[31690]: * Failed to join XXXX:XXXX:XXX:1:XXXX:XXXX:XXXX:421f: dial tcp [XXXX:XXXX:XXX:1:XXXX:XXXX:XXXX:421f:]:8301: connect: no route to host
Feb 03 15:33:27 <SYSTEM>.<COMPANY>.com consul[31690]: * Failed to join XXX.XX.XXX.XXX: dial tcp XXX.XX.XXX.XXX:8301: connect: connection refused
Environment
Release : 21.2
Component :
Cause
The consul process was using DNS and not using the hosts file as specified in the nsswitch file. The DNS entry for one of the systems had the incorrect IPv6 address which caused connection failures.
Resolution
The resolution was to update DNS with the correct IPv6 address and then uninstall the DA's and proxy and reinstall.
Additional Information
The system was using DNS and not the hosts file. DNS had a old IPv6 address and that was being used for communication and failing. Although the systems nsswitch file was configured to use the hosts file first, IPv6 uses a new tag for resolution in the nsswitch file and will default to dns if the new tag is not there.
IPv4 resolution in nsswitch:
hosts: files dns myhostname
IPv6 resolution nsswitch:
ipnodes: files dns myhostname
After the ipnodes line was added to the nsswitch file the consul was still using DNS. It seems consul does not use the nsswitch file for resolution.
The resolution was to update DNS with the correct IPv6 address.
Feedback
