Aria Operations for Networks version 6.11 appliances fail to rotate the log causing /var/log partition to grow beyond 90 Percent
search cancel

Aria Operations for Networks version 6.11 appliances fail to rotate the log causing /var/log partition to grow beyond 90 Percent

book

Article ID: 324415

calendar_today

Updated On:

Products

VMware Aria Operations for Networks

Issue/Introduction

Symptoms:

Logrotate service has issues in rotating logs for nginx, syslog, warn, etv.

Log files of syslog, warn and nginx grows and consumes disk space causing disk out of space issues. 

  1. syslog keeps writing to syslog.1 instead of syslog file. (Location is /var/log/)
  2. syslog keeps writing to warn.1 instead of warn file. (Location of log is /var/log/)
  3. nginx keep writing to access.log.1 instead of access.log file (Location of log is /var/log/nginx/)

Since the service is writing to file with '1', logrotate never performs the log rotation, because it checks the main file for rotation and its always empty.
It doesn't perform rotation on empty files and hence syslog.1, warn.2, access.log.1 files grows bigger and bigger.

 

 

Environment

VMware Aria Operations for Networks 6.11.0

Cause

After log rotation is performed it triggers a reload call to the respective service to reload the log files, so that it will continue to write to correct file.
This mechanism is broken and as a result files grow huge in size making /var/log >90%

Resolution

Aria Operations for Networks Engineering team is aware of this and a patch on top of Aria Operations for Networks Version 6.11 is made available which fixes the log rotation issue.
Download Aria Operations for Networks Version 6.11 P2 patch from the attachment section in this Knowledge Base Article.

File Name: VMware-AriaOpNetworks.6.11.0.P2.1699450092.patch.bundle
File Size: 668.2 MiB
CheckSums Values:

  • MD5SUM: CBE80EB278AD7A351ED84D1B0B7EC933
  • SHA1SUM: 1AD8B3352E1F6606B7E5EF7435E16A2DE23197C8
  • SHA-256: 9444734D2CB24AA5D78E107A676F30D81FFFA533CCEDA9759D817BB29F146781

Note:

If the 6.11.0-P1 patch is already applied there is no need for this work around but if this is the first patch then use the following steps:

  1. Login to each Appliance VM as username support
  2. Execute the following commands:
    ub
    sudo su
    mkdir -p /usr/local/lib/python3.6/dist-packages/cli
    ln -sf /usr/local/lib/python3.8/dist-packages/cli/tool_manager_runner.py /usr/local/lib/python3.6/dist-packages/cli/tool_manager_runner.py
  3. Upload patch bundle from Aria Operations for Networks GUI.

Procedure to apply patch bundle via Aria Operations for Networks GUI:

  1. Download the update patch file and save the file on your local system.
  2. Log into the vRealize Network Insight GUI as an Administrator user with username admin@local
  3. Navigate to Settings > Install and Support > Overview and Updates, then under Product, select Click here
  4. Click Browse to select the locally downloaded patch file and click Upload.
    Notes:
    • When the upload is complete, Aria Operations for Networks shows the Bundle Upload Complete message notification within 2-3 minutes and the bundle processing happens in the background.
    • Until the upload of the package happens, ensure that the session is not closed. If the session ends, you have to restart the upload process.
    • Do not refresh the page after bundle upload, until you see the Update Available message notification.
  5. In the Bundle Available message notification, click View details
    Note: The Aria Operations for Networks Update screen appears. Read the Before you proceed instruction and click Continue.
  6. Wait for the pre-checks to complete, which verifies:
    • the disk space, including the space required for migration
    • the version
    • the NTP sync status
    • the bundle checksum
  7. Click Install Now.
    Note:
    You can see the approximate time required to complete the update process on your setup.
  8. Once the update process begins, the Aria Operations for Networks Update screen provides the status of the update process.
    Notes:
    • If a node becomes inactive, the update process does not continue. The update will not resume until the node becomes active again.
    • Once the platforms are updated, you can resume your normal Aria Operations for Networks operations even though the collector update happens in parallel. Until the update process is completely over, the Node Version Mismatch detected the message is shown in the Install and Support page.
  9. Upon the completion of the update process, you a confirmation message similar to the following:

    "Validate from GUI that  all the platform and the collector nodes are updated."



Additional Information

 

 

Attachments

VMware-AriaOpNetworks.6.11.0.P2.1699450092.patch.bundle get_app