Aria Operations for Logs service will not start and is in failed state
book
Article ID: 312255
calendar_today
Updated On:
Products
VMware Aria Suite
Issue/Introduction
To Replace corrupted /usr/lib/loginsight/application/etc/loginsight-config-base.xml file on Aria Operations for Logs appliances with working file from working node or from other environment with similar version.
Symptoms:
Web UI will be unavailable.
When checking 'service loginsight status' you will see 'Failed' status instead of 'running'.
Main log files in /var/log/vmware/loginsight/ will not be written to for a while.
When checking journalctl logs you will see entries similar to:
-- The start-up result is RESULT.
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: Traceback (most recent call last):
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/loginsight/application/bin/loginsight", line 60, in <module>
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: strata = StrataServer(STRATA_HOME)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/loginsight/application/lib/python/StrataServer.py", line 108, in __init__
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: self.config = StrataConfig(os.path.join(home, "application", "etc", "loginsight-config-base.xml"))
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/loginsight/application/lib/python/StrataServer.py", line 27, in __init__
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: self.doc = xmlparse(path)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/python3.7/xml/dom/minidom.py", line 1958, in parse
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: return expatbuilder.parse(file)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/python3.7/xml/dom/expatbuilder.py", line 911, in parse
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: result = builder.parseFile(fp)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: File "/usr/lib/python3.7/xml/dom/expatbuilder.py", line 207, in parseFile
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: parser.Parse(buffer, 0)
Oct 10 09:09:24 vrli-fqdn loginsight[16546]: xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 0
Environment
VMware Aria Operations for Logs 8.12.x
Cause
Corrupted /usr/lib/loginsight/application/etc/loginsight-config-base.xml file on appliance node.
Resolution
Take a cluster snapshot before proceeding:
Stop loginsight service on all nodes by running:
service loginsight stop
If there a node in the cluster where you don't see the error and logs are being updated, Copy /usr/lib/loginsight/application/etc/oginsight-config-base.xml from working node and replace it on the corrupted nodes.
Use WinSCP or another similar tool to copy the file from working to not working node.
Copying from WinSCP should maintain the file permissions but verify after copying that the file has similar permissions as that of the file from the working node.
If all nodes are corrupted get a non corrupted file from another vRLI cluster node of similar version.
Start loginsight service on all the nodes by running:
service loginsight start
Verify that the service ls running properly by running
service loginsight status -- verify that the service is in active state.
if you see log message regarding strata_writer.lock. stop loginsight service on that node and run: rm /storage/core/loginsight/cidata/store/strata_writer.lock