Preventive logging disk clean up in vRA 8.x

Products

• VMware Aria Suite

Issue/Introduction

All versions

This article can also be used in a precautionary manner prior to upgrades to 8.6 and above.

Symptoms:

For vRA 8.5 and below

• Upgrading to vRA 8.6 or later fails with the error

  Upgrade terminated due to critical error

• A large number of files exist in the /data/fluentd/local-storage-buffer and /data/fluentd/vrli-buffer directories
• The commands du -hs /services-logs/ and du -hs /var/log show that the sum of content inside /services-logs and /var/log exceed 80% the capacity of /services-logs as displayed by df -h /services-logs/.

For vRA 8.6 and 8.7

• The commands du -hs /services-logs/ and du -hs /var/log show that the sum of content inside /services-logs and /var/log exceed 80% the capacity of /services-logs as displayed by df -h /services-logs/.
• Prune-logs service timeouts when started and fails to free space

Environment

VMware vRealize Automation 8.x

Cause

For vRA 8.5 and below

• During upgrades to vRA 8.6 and later, the contents of /var/log are migrated into the services-logs partition
•  If the sum of contents inside /var/log and /services-logs exceed more than 80% of the total capacity of services-logs, it will cause the upgrade to fail
• Due to a bug in fluentd 1.6.3 (the fluentd version used until vRA 8.7), excessive amounts of buffer chunk files might be generated, causing inode exhaustion and a very large /services-logs/logging/fluentd.log file

For vRA 8.6 and 8.7

• prune-logs.py is too inefficient on bigger disks with a large number of files.

Resolution

​​​​​​These issues are resolved in VMware vRealize Automation 8.8 and above.

Workaround:

Prerequisites

• Please take simultaneous non-memory snapshots of each virtual appliance(s) in the cluster.
• You have access to root user and password.
• You have SSH or console access to each virtual appliance.

Procedure

1. Run the following command

   base64 -d <<< '/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4BTeBuVdABGIQkY99Bhqpmevep9yj5xO8pryhIbYBrANP1nqEwwojrlc0V3onJueaMwEyVFUkRMyolwUWpxUTOaX53X8dFuLgyLgKG/DUeQrjsVd2gV5ipWwuUHtJGd+IHiiAmb1J+DeenWW4HN3lpJLcH09j6Kzz3r92mce/XRuCGNbhue00HiuFluBmM5jmtufdaO0cwRZZR+LQw32YJyKukAle84ny/3CWUeGDtwJvN2DxqclPZyWua+/70F+rAo8Njsa+pPyr5N5JOnlNouPsBIfGnRJME8ioKyO6fbS/FsWxwKBbUpbkmShayG3akFWVJXA38//hA0owDMJ6G/aWo8tdCRdrUVTeBC5URpcSBITOZhH1epumgod5d1ExM9svztHGVkjsmcFN1TOTu3a/eblrd/BqLKdQYet98d2esL6nNGyclRWDb2t6km6ygqoUQeegX4ingfLl/OcrXomcdQGA5qipSt9n2Vg/5h7xENJgFAbicE4L3w7BBsXzevPV9wlTy/SKMh4bk0jI5SO01E/c02bXfa7shCgBmxxpN0vm4BvXvt69kSE2Ty23G6miqFYLxF2TYPNUH6IBpkIFsVax5oIYuhzypXFnHJIBG4+Sr9grECGJSik7T3wHrB+j9bbWZ0pGU2QdsiS4+VVWsfwXfshH/si9wzCL/bxWnax8aRMW3hcF2l8D+c4NBrJalyQ6pyiZURT0DLfCp8Oqv90wlboEamzQuCIKxa2lFRf1b9g6tvSY772Y+r5Xo9h0hsOqVxt6hfoPTG1boigN4nR8i2l+93F7q55rB1OcbrGhu2GAzPt+jJd51xkNbxPpt8sCX8z49EdFqtkEJQnlhW7VdRwUwYRzFSsSjD2YVB+3My87nhy7eI+EvC3iR9w12BxYpMcP7snJzGXLw+d5Aa5PWgqpTdQvBmS+9tGjgSni36qFMIrYi5FppknEh5ed90qsCO1IzVRfgNr7L+jv4b8gzHpd6hsgksiDTwdJrtKEj+wl6w1ihth16cZp3urM3Y9PqCHM7FCPEFbhv6PbSa9S1XQMFppkgchY8OQvSQtTuXyowEd0S25fXLHAUyIHkH6g65Cw1nIQ2jucEH8PmNy/StvJ2I5vpjOEc9EM733ic9q8l8CIhniPfOg/50IcSFZoE9G8VRUgip5nWGQMLR7Lto+dJKWKp8Xk+hABE+KganCiCgi/2kzLo3d/UUwReVMNq4XF/aTdkyNY0S7fC3fEJpzSn6e4f5rFPZfdjUKeb8W/MouAGKQ8Jn7Z7k7z3+gl+PM11ej2BqypfuHr63kWbcZ6FfL6U397JjUnEudMU/7LhB/Zo4yhA/72/NBen4033H+0bVtmjG/2XGGuuFBfV567pyAI2HqVxwSsENwDWpIE3NLy1/b2BOsWUvGkCwF8m2NysWuCJMOj1sfeAMhKSv2jVDlMLtvCY1L0xexSwIhaJkks7TD8JMM54YMyDQT/RXTg8WbFXDyve4zMxwTDcfO8VDTf1uHNk2ko0gJtcLLN4paUwDyQBc3qwXKdVCYjDujlx6uH38VZNV6yz9OiBkGYni5JhKgd/SD1zD1iujThv0d1ch0PUyyNuKQ28Sze31LNV3u/6UZX4sf50TpNfkrgpMhLcXytmss/Fn8zrx4Giv/3G70bg5202SpgHb6N9IN8v+ehiKAHbSA2LRx4eFlEIJxzW9Pw/E7THshDmMsDWWywRYJERHXw7BkhR+ZiRr1zJSTQPRGidoCBmBy7qdxXQkPo/WX3zD36M3gaF+WNQq9MA/aTCm08Hn+r+oLro8K9STkb5RVEFDHrymUhhHd1VcZIEpqrxwN7goJMQVczTW8+5F0QDMUi32ufyKLovGN3lKlR+ECUBH6rwMdxSL403x+S9bxKmtR7et2kzdowerx8ATJ+LDFy82l+Ea9ErnQZd1apGJtky+Nsd9YXSHJ1vHEqqlFYbdB8QbtOaxKAldjy4lss2TLBtjqUvTjCyhC9qVq6eoeXHUbSDh+ZU9NQD3yLF/zWOEfONqCsGUOLQ0yn2IZOi4SHVH7Jhn7JrdpjRAIDlry5GaA0wwtfSB0UT8lSjFBcCuxnLYFb0zBU4kkLAnRFtvAWaWdu1Vj2qUbXt6spGPxvIUHKVm6S+L33tHY2cutB8WiQxyUz/hUxgn2evx1hOa+JGBDwOvYw/5Be5aY9WCwo397vZzBirHrdHxghdhp7XuF72YcIJeOlmPuYFnxrzLkAmJiZ53E9mFeYbeUw68iO5to9cv6P+waQUNDImoFd831Fi7TaRwK56mfU3oFX4KrKmazBVJImHei/gecGWTaEklVJkFoy4zWEE+454UYKK/7ymM1lpbuILF0grsAAAAA6SZ4ABP2hmAAAYEO3ykAAGF5IuCxxGf7AgAAAAAEWVo=' | xz -d | bash -
   

Note: Logs can be found at /var/log/vmware/prelude

Additional step on vRA 8.6 and 8.7

1. If the original prune-logs.py script needs to be restored, it is backed up at /data/patch-backup/prune-logs-*/prune-logs.py,
   1. Run the following command (replacing timestamp with the actual timestamp)

      mv /data/patch-backup/prune-logs-[timestamp]/prune-logs.py /opt/python-modules/vra_logging/prune-logs.py