Cloud Proxies becoming dysfunctional after upgrading to vRealize Operations 8.10 or above

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• After upgrading to vRealize Operations 8.10, Cloud Proxy not completing the upgrade.
• If there is an Environmental Network Overlap with 172.17.x.x or 172.18.x.x
• On Cloud Proxy, the command service httpd-north status shows that it failed to start with an error similar to:

Failed to resolve server name for 172.17.0.1 (check DNS) -- or specify an explicit ServerName

Oct 28 14:48:38 localhost httpd[8746]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 172.17.0.1:443 

 

Environment

VMware vRealize Operations 8.10.x and above

Cause

While running custom adapters in containers, there were changes in /etc/httpd-north/httpd.conf to listen 172.17.0.1, which is docker's default gateway in case of a bridged network.
 

However, if the 172.17.X.X subnet is used, the docker service will use another subnet for the bridged network, e.g. 172.18.X.X (usually it increments the second byte).  As in httpd.conf it is hard coded to listen on 172.17.0.1, and the docker uses 172.18.0.1, it causes the could not bind to address 172.17.0.1:443 and fails.

Resolution

This issue is resolved in vRealize Operations 8.10.1 and later.

Workaround 1:

In case of the mentioned issues with the mentioned symptoms, as a workaround, via ifconfig command should be checked the docker's gateway.

Example: 
docker0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.18.0.1 Bcast:172.18.255.255 Mask:255.255.0.0

After, in /etc/httpd-north/httpd.conf file change all 172.17.0.1 appearances to 172.18.0.1 (IP shown in if config) and restart httpd-north service, i.e. service httpd-north restart.

Note: In case of a CP reboot, the mentioned manual changes will be reverted, so there will be a need to repeat the workaround.

 

Workaround 2:

If the issue is an overlap with customer’s network environment with 172.18.x.x then the following Workaround can be performed:

1. Manually increment the internal docker network address so that it does not overlap any adapter endpoint network address space.
   1. Configure a non-overlapping network ip address for the Cloud Proxy internal docker network
      1. Add a docker network bridge IP address that does not conflict with the IP Addresses of the adapter endpoints. This is achieved by adding attribute "bip" to existing /etc/docker/daemon.json. Note that the "bip" attribute will be removed from the /etc/docker/daemon.json file after reboot, but the internal docker network configuration will be preserved.
         
         vi /etc/docker/daemon.json
         
         {
         "iptables": true,
         "log-opts":
         {"max-size": "200m", "max-file": "1"}
         ,
         "log-level": "info" ,
         "live-restore": true,
         "bip": "172.18.0.1/16" <=== Network 172.18.0.0/16 does not conflict with endpoint network 172.17.10.0/24
         }
         
         
   2. Reboot the Cloud Proxy VM
      1. reboot -f
          
2. Verify the internal docker network has been configured with the new address
   1. Example:
      root@localhost [ ~ ]# route
      Kernel IP routing table
      Destination Gateway Genmask Flags MSS Window irtt Iface
      default **.***.**.*** 0.0.0.0 UG 0 0 0 eth0
      **.***.**.* * 255.255.252.0 U 0 0 0 eth0
      172.18.0.0 * 255.255.0.0 U 0 0 0 docker0 <===
      
      root@localhost [ ~ ]# ip route
      default via 10.126.87.253 dev eth0 proto static
      **.***.**.*/22 dev eth0 proto kernel scope link src **.***.**.**
      172.18.0.0/16 dev docker0 proto kernel scope link src 172.18.0.1 <===

Note: In case of a CP upgrade and reboot, Workaround 2 will be retained