Scheduled backup not taken for vCenter Server
Article ID: 401282
Updated On:
Products
Issue/Introduction
Some scheduled vCenter backups are intermittently missing.
- No error can be seen on the VAMI page.
- Certain scheduled backup dates are absent from the backup list.
Environment
vCenter Server 7.0
vCenter Server 8.0
Cause
When a scheduled backup is triggered, it first performs a validation of the backup destination and the system state.
Simultaneously, the applmgmt health API triggered as part of routine health monitoring. In this case, the health check failed due to a timeout, as the applmgmt service was still processing the backup validation.
After three consecutive timeout failures, the vmon service deemed applmgmt unhealthy and initiated a service restart. As a result, the backup operation did not complete and failed with the following error: RemoteDisconnected('Remote end closed connection without response')
backupScheduler.log:
yyyy-mm-ddThh:mm:ss.000 [0] [MainProcess:PID-20291] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 288, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
vmon.log:
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Health of service failed. Health data:
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Recover from service api health check failure. Fail count 0
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Restarting service.
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata applmgmt,UNHEALTHY,HEALTHY,1
...
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 Client info Uid=0,Gid=0,Pid=22202,Comm=(vmon-cli),PPid=35749,Comm=(python),PPid=2340,Comm=(vmon),PPid=1,Comm=(systemd),PPid=0
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 Client info Uid=0,Gid=0,Pid=22212,Comm=(vmon-cli),PPid=35749,Comm=(python),PPid=2340,Comm=(vmon),PPid=1,Comm=(systemd),PPid=0
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Sysprocess clean stop timed out. Force kill. Pid 35749
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Service exited. Exit code 1
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt-prestart> Constructed command: /usr/lib/applmgmt/support/scripts/prestart-applmgmt.sh
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Service pre-start command's stderr: Removed /etc/systemd/system/applmgmt.service.
yyyy-mm-ddThh:mm:ss.000Z Wa(03)+ host-2340
yyyy-mm-ddThh:mm:ss.000Z Wa(03) host-2340 <applmgmt> Service pre-start command's stderr: Created symlink /etc/systemd/system/applmgmt.service /dev/null.
yyyy-mm-ddThh:mm:ss.000Z Wa(03)+ host-2340
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Service pre-start command completed successfully.
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Constructed command: /usr/lib/applmgmt/applmgmt.launcher
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Running the API Health command as user
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt-healthcmd> Constructed command: /usr/bin/python /usr/lib/applmgmt/applmgmt_vmonhealth.py
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Re-check service health since it is still initializing.
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Running the API Health command as user
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt-healthcmd> Constructed command: /usr/bin/python /usr/lib/applmgmt/applmgmt_vmonhealth.py
yyyy-mm-ddThh:mm:ss.000Z In(05) host-2340 <applmgmt> Service STARTED successfully.
Resolution
This issue has been acknowledged by VMware Engineering and is planned to be addressed in a future release.
Workaround:
To avoid this conflict between the backup and health monitoring tasks, adjust the scheduled backup time by at least 10 minutes.
For example, if the backup is currently scheduled at 05:00, change it to 05:10.
Feedback
