Aria Operations for Logs service will not start and is in failed state
search cancel

Aria Operations for Logs service will not start and is in failed state

book

Article ID: 312255

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

To Replace corrupted /usr/lib/loginsight/application/etc/loginsight-config-base.xml file on Aria Operations for Logs appliances with working file from working node or from other environment with similar version.

Symptoms:
  • Web UI will be unavailable.
  • When checking 'service loginsight status' you will see 'Failed' status instead of 'running'.
  • Main log files in /var/log/vmware/loginsight/ will not be written to for a while.
  • When checking journalctl logs you will see entries similar to: 
    -- The start-up result is RESULT.
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: Traceback (most recent call last):
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/loginsight/application/bin/loginsight", line 60, in <module>
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     strata = StrataServer(STRATA_HOME)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/loginsight/application/lib/python/StrataServer.py", line 108, in __init__
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     self.config = StrataConfig(os.path.join(home, "application", "etc", "loginsight-config-base.xml"))
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/loginsight/application/lib/python/StrataServer.py", line 27, in __init__
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     self.doc = xmlparse(path)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/python3.7/xml/dom/minidom.py", line 1958, in parse
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     return expatbuilder.parse(file)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/python3.7/xml/dom/expatbuilder.py", line 911, in parse
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     result = builder.parseFile(fp)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:   File "/usr/lib/python3.7/xml/dom/expatbuilder.py", line 207, in parseFile
Oct 10 09:09:24 vrli-fqdn loginsight[16546]:     parser.Parse(buffer, 0)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 0


Environment

VMware Aria Operations for Logs 8.12.x

Cause

Corrupted /usr/lib/loginsight/application/etc/loginsight-config-base.xml file on appliance node.

Resolution

Take a cluster snapshot before proceeding:
  • Stop loginsight service on all nodes by running:
    • service loginsight stop
  • If there a node in the cluster where you don't see the error and logs are being updated, Copy /usr/lib/loginsight/application/etc/oginsight-config-base.xml from working node and replace it on the corrupted nodes.
    • Use WinSCP or another similar tool to copy the file from working to not working node.
    • Copying from WinSCP should maintain the file permissions but verify after copying that the file has similar permissions as that of the file from the working node.
  • If all nodes are corrupted get a non corrupted file from another vRLI cluster node of similar version.
  • Start loginsight service on all the nodes by running:
    • service loginsight start
  • Verify that the service ls running properly by running 
    • service loginsight status       -- verify that the service is in active state.
      • if you see log message regarding strata_writer.lock. stop loginsight service on that node and run: rm /storage/core/loginsight/cidata/store/strata_writer.lock


Additional Information

Impact/Risks:
  • web UI goes down
  • Main Loginsight service is not running


Powered by Wolken Software