HCX - Log rotate functionality is broken for SGW/SDR appliances

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Identify and remediate a known issue with OSAM SGW/SDR appliance log rotation workflow.

Symptoms:
During runtime, "/var/log/" directory may filled up unexpectedly on a given Sentinel Gateway(SGW) OR Sentinel Receiver(SDR) appliance used for HCX OS Assisted Migration Services.

Cause

This has happened due to incorrect permissions got assigned to log rotation script running in the backend for SGW/SDR appliances, which results "/var/log" utilization reaches 100%.

Note: The file system space under "/var/log" referred by other processes may not be used to store new logs, once the utilization reaches to its maximum value.

Resolution

The issue will be fixed in the upcoming HCX release.

Workaround:
IMPORTANT:  Below steps can be performed on both SGW and SDR appliances at HCX Connector & Cloud manager individually to remediate the log rotation functionality.

SGW:

• Login to HCX Connector via SSH as admin user.
• Execute "ccli" and "list" in order to print a list of all the fleet appliances.

$ ccli
Welcome to HCX Central CLI
 
[admin@hcx] list
|---------------------------------------------------------------------|
| Node               | Id | Address            | State     | Selected |
|---------------------------------------------------------------------|
| DX-IGW-NE-I1       | 0  | X.X.X.X:9443 | Connected |          |
|---------------------------------------------------------------------|
| DX-IGW-SGW-I1      | 1  | Y.Y.Y.Y:9443 | Connected |          |
|---------------------------------------------------------------------|

• Use the "Id" for SGW appliance obtained through CCLI and execute "go <SGW appliance Id>".

go 1
Switched to node 1.
[admin@hcx:DX-IGW-SGW-I1]  

• Execute "ssh" command.

[admin@hcx:DX-IGW-SGW-I1] ssh
Welcome to HCX Central CLI
Last login: Thu Jan 19 08:06:09 2023 from 127.0.0.1
root@DX-IGW-SGW-I1 [ ~ ]# 

• Verify the permissions associated with the file "/usr/bin/log_cleanup.sh".

root@DX-IGW-SGW-I1 [ ~ ]# stat /usr/bin/log_cleanup.sh
  File: /usr/bin/log_cleanup.sh
  Size: 572       	Blocks: 8          IO Block: 4096   regular file
Device: 832h/2098d	Inode: 31422       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Note: Based on the above output, it is set to "0644" whereas it should be set to "0755".

• Use below command to modify the permission as "0755".

root@DX-IGW-SGW-I1 [ ~ ]# chmod 0755 /usr/bin/log_cleanup.sh

• Execute below command to remove the OLD logs under "/var/log".

root@DX-IGW-SGW-I1 [ ~ ]# /usr/bin/log_cleanup.sh

SDR:
 

• Login to HCX Cloud via SSH as admin user.
• Execute "ccli" and "list" in order to print a list of all the fleet appliances.

$ ccli
Welcome to HCX Central CLI

[admin@hcx] list
|---------------------------------------------------------------------|
| Node               | Id | Address            | State     | Selected |
|---------------------------------------------------------------------|
| DX-IGW-NE-R1       | 0  | A.A.A.A:9443 | Connected |          |
|---------------------------------------------------------------------|
| DX-IGW-SDR-R1      | 1  | B.B.B.B:9443 | Connected |          |
|---------------------------------------------------------------------|

• Use the "Id" for SDR appliance obtained through CCLI and execute "go <SDR appliance Id>".

go 1
Switched to node 1.
[admin@hcx:DX-IGW-SDR-R1 ]  

• Execute "ssh" command.

[admin@hcx:DX-IGW-SDR-R1] ssh
Welcome to HCX Central CLI
Last login: Thu Jan 19 08:06:09 2023 from 127.0.0.1
root@DX-IGW-SDR-R1  [ ~ ]# 

• Verify the permissions associated with the file "/usr/bin/log_cleanup.sh".

root@DX-IGW-SDR-R1 [ ~ ]# stat /usr/bin/log_cleanup.sh
  File: /usr/bin/log_cleanup.sh
  Size: 572       	Blocks: 8          IO Block: 4096   regular file
Device: 832h/2098d	Inode: 31422       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Note: Based on the above output, it is set to "0644" whereas it should be set to "0755".

• Use below command to modify the permission as "0755".

root@DX-IGW-SDR-R1  [ ~ ]# chmod 0755 /usr/bin/log_cleanup.sh

• Execute below command to remove the OLD logs under "/var/log".

root@DX-IGW-SDR-R1  [ ~ ]# /usr/bin/log_cleanup.sh

Additional Information

Impact/Risks:

• This will be applicable to all HCX deployments running SGW/SDR appliances for OSAM service, post 4.4 or later.
• There will be NO impact to ANY migration workflow.
• There will be NO impact to Network Extension services.