Antrea OVS Container crashes on vSphere with Tanzu TKR versions 1.24.11 through 1.26.12

Products

• VMware vCenter Server
• vSphere with Tanzu

Issue/Introduction

Symptoms:
Workload clusters using Antrea CNI and TKR 1.24.11 will experience a crash on the antrea-ovs container once per approximately 24 hours.

The logging for the antrea-ovs container will look similar to the following:
 

2023-11-19T00:49:04.342123483Z stdout F [[1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: Starting ovsdb-server
2023-11-19T00:49:04.430417367Z stdout F Starting ovsdb-server.
2023-11-19T00:49:04.486011017Z stdout F Configuring Open vSwitch system IDs.
2023-11-19T00:49:04.487188085Z stderr F /usr/share/openvswitch/scripts/ovs-ctl: line 46: hostname: command not found
2023-11-19T00:49:04.501822087Z stdout F Enabling remote OVSDB managers.
2023-11-19T00:49:04.503623773Z stdout F [1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: Started ovsdb-server
2023-11-19T00:49:04.505142698Z stdout F [[1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: Starting ovs-vswitchd
2023-11-19T00:49:04.52037129Z stdout F [[1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: ovs-vswitchd set hw-offload to false
2023-11-19T00:49:04.693833452Z stdout F Starting ovs-vswitchd.
2023-11-19T00:49:04.693885184Z stderr F /usr/share/openvswitch/scripts/ovs-ctl: line 46: hostname: command not found
2023-11-19T00:49:04.711645343Z stdout F Enabling remote OVSDB managers.
2023-11-19T00:49:04.714087917Z stdout F [[1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: Started ovs-vswitchd
2023-11-19T00:49:04.715942076Z stdout F [[1;34m2023-11-19T00:49:04Z [0;32mINFO [0;36mantrea-ovs[0m]: Started the loop that checks OVS status every 30 seconds
2023-11-20T00:49:57.748308432Z stderr F error: failed to compress log /var/log/openvswitch/ovs-vswitchd.log.1
 

 

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

This is a known issue within the release of antrea included in this TKR: antrea:v1.7.2_vmware.3. The log-rotate feature in the antrea-ovs container is configured to rotate the logs daily but the image does not ship with gzip. Thus, the container crashes at regular intervals.

Resolution

This issue is resolved in TKR version 1.27.6, which is included in vCenter Server 7.0 Update 3P and later.

If the customer is using vCenter 8.x, it is recommended to upgrade to at least vCenter Server 8.0 Update 2c or later, which supports TKR 1.27.6.

https://knowledge.broadcom.com/external/article/323454

Additional Information

Impact/Risks:
The container will show a restart, but there has been no noticeable effect to production systems experiencing this issue.