/storage/log partition full due to large stream.log-x.stderr files

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• /storage/log full due to stream.log-x.stderr filling out with authz logs.
• Accessing the web-client results in below error

• /storage/log/vmware/wcp/stream.log-x.stderr logs with large size and displaying multiple entries like the one below:

• Below error are observed in /var/log/vmware/wcp/stdstream.log-x.stderr
 
YYYY-MM-DD HH:MM:SS User name and domain: vsphere-webclient-xxxxxxx-xxxx-xxxx-xxx-xxxxxxx
time="YYYY-MM-DDTHH:MM:SS" level=debug msg="Validating permissions; operation: com.vmware.cdc.change_log.get_current_sequence, requested: map[{isOperation:false id:com.vmware.cdc resourceType:}:[] {isOperation:true id:com.vmware.cdc.change_log.get_current_sequence resourceType:}:[System.Anonymous]]" opID=wcp-AuthzFilter
time="YYYY-MM-DDTHH:MM:SS" level=debug msg="Requesting authz svc; user: vsphere.local\\vsphere-webclient-xxxxxxx-xxxx-xxxx-xxx-xxxxxxx, groups: [[email protected] [email protected] [email protected] 

Environment

vSphere vCenter Server 7.0.x

Cause

Some authz logs are being redirected to stderr causing storage issues on VCSA

Resolution

This issue has been fixed in 7.0.3 P08

Workaround:

1. Make a backup copy of the wcpsvc-vmonconfig.json config file

cp /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json /tmp/wcpsvc-vmonconfig.json.bak

Note: Files in /tmp are removed once vCenter is rebooted. Therefore, you might want to save this in your computer.

2. Edit the wcpsvc-vmonconfig.json file 

 

vim /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json

3. Remove the line beginning with StreamRedirectFile and save the file.

 

{
    "Name" : "wcp",
    "StartCommand" : "/usr/lib/vmware-wcp/wcpsvc.launcher",
    "StartCommandArgs": ["--port", "8920", "--logfile", "/var/log/vmware/wcp/wcpsvc.log",
                         "--configfile", "/etc/vmware/wcp/wcpsvc.yaml",
                         "--audit-logfile", "/var/log/vmware/wcp/wcp-audit.log",
                         "--incident-logfile", "/var/log/vmware/wcp/wcp-incident.log"],
    "PreStartCommand" : "%VMWARE_PYTHON_BIN%",
    "PreStartCommandArgs" : ["/usr/lib/vmware-wcp/wcpsvc-prestart.py"],
    "StreamRedirectFile" : "/var/log/vmware/wcp/stdstream.log",                                -----------------------> # line to be modified
    "RecoveryActionProfiles" :
    {
        "DEFAULT" :
        {
            "CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
            "HEALTHFAIL" : ["NO_ACTION"]
        }
    },
    "WorkingDir" : "/usr/lib/vmware-wcp/",
    "DependsOn" : ["vpxd", "eam", "vmware-vpostgres", "lookupsvc", "cis-license", "vapi-endpoint", "sts", "trustmanagement"],
    "ApiHealthCommand" : "%VMWARE_PYTHON_BIN%",
    "ApiHealthCommandArgs" : ["/usr/lib/vmware-vmon/vmonApiHealthCmd.py", "-n", "wcp", "-u", "/wcp/health", "-t", "10"],
    "StartTimeout" : 120,
    "StopTimeout" : 30
}

 

4. Restart wcp service

 

service-control --stop wcp && service-control --start wcp

NOTE: If we are already on the 7.0.3 P08 we can clear the content of the large file under /var/log/vmware/wcp/stdstream.log-x.stderr. Command: echo "" > stdstream.log-x.stderr

Additional Information

vCenter Server Appliance disk space is full

Impact/Risks:
High, vCenter might crash due to full partitions.