Upgrading to Windows stemcell >2019.88 results in agent timeout after 600 seconds
Article ID: 410536
Updated On:
Products
Issue/Introduction
When upgrading Windows stemcell to >2019.88, Apply Changes fails with:
Error: Timed out pinging VM 'compilation-xxxx-xxxx-xxx-xxx-x-pas-windows-xxxxx' with agent 'xxxxx' after 600 seconds
Error: Agent exited with error: Running bootstrap: Setting up tmp dir: Creating symlink from C:\Windows\TEMP to \var\vcap\data\tmp: Removing new path at C:\Windows\TEMP: remove C:\Windows\TEMP\vmware-vmsvc-SYSTEM.log: The process cannot access the file because it is being used by another process.
Environment
Cause
Background
With Windows Update KB5053596 present on the Windows base VM, the stricter handling of GetTempPath(2) validation required a change to stembuild handling beginning in Windows stemcell release version 2019.86 and may cause the open IPC handles from vmtoolsd.exe to prevent the relink stage of the stembuild process.
Windows Update KB5053596 (March 11, 2025) for Windows Server 2019 introduced additional kernel level hardening around file handle integrity and symbolic link creation in protected system directories such as %WINDIR%\Temp and %SystemRoot%\Installer. Prior to this update, Windows allowed rename and re-link operations on directories even when certain file handles were open. After KB5053594/6 those operations are now blocked.
As part of preparing a Windows stemcell, stembuild replaces C:\Windows\Temp with a junction pointing to the ephemeral disk; the relink step deletes the existing Temp directory and creates a new symbolic link.
With the KB5053596 handle enforcement, this operation can, under certain circumstances e.g. undetected corruption on the base VM operating system, fail if any process including background system agents, has open file handles in C:\Windows\Temp. The resulting issue presents itself as an error originating from a configuration vm and appear as the BOSH agents failure to connect to the Director.
Caveats
VMware Tools inter-process communication (IPC) artifacts hold open file handles in C:\Windows\Temp such as C:\Windows\Temp\vmware-vmusr-Administrator. VMTools is required for the stembuild construct stage of the stemcell build process and thus should not be stopped manually prior to the completion of this stage.
During the construction of a Windows stemcell, stembuild can produce a completed stemcell while still failing Apply Changes. The reason for this is because stembuild does not check Windows operating system integrity or perform any post-build stemcell sanity checks. For example corruption in Windows Update servicing stack, WinHTTP, cryptography stores, WMI, or filesystem metadata will not be detected during the stembuild process but could cause failure during BOSH bootstrap.
Resolution
Option 1 (Fastest/preferred/reliable)
1. Rebuild the base VM from a licensed version of Microsoft Server 2019
2. Install all recommended Windows Updates on the new base VM
- Login to the new base vm
- Open a command shell and type powershell to open a PS session
- At the command prompt, type sconfig
- Select "6" from the menu
- Select "(A)ll"
- Select "(A)ll" to begin the download and installation of discovered updates
3. Kickoff a new stembuild
Option 2 (When option #1 is not possible)
1. Get the detailed list of current updates by setting your search context and writing out the results to a file for easy searching:
- $Session = New-Object -ComObject Microsoft.Update.Session
- $Searcher = $Session.CreateUpdateSearcher()
- $HistoryCount = $Searcher.GetTotalHistoryCount()
- $Searcher.QueryHistory(0, $HistoryCount) | Out-File -FilePath "C:\Windows\Temp\WU-out.txt"
2. Open notepad from the command line by typing notepad:
- Use the notepad menu to open a file browser and browse to C:\Windows\Temp. Double click the file you created to begin your search.
- Search for "Cumulative" in the file you created in step one and verify the expected KB article is included (actual cumulative KB# results may vary depending on when the WU was run).
- You are looking for the latest cumulative update that includes either update KB5053596 or KB50535964 which will be validate in the next steps
- The GetTempPath(2) update is installed only as part of a cumulative update and will not appear in the search results as a standalone update**
Example output from C:\Windows\Temp\WU-out.txt:
- Check for the required KB50535964 or KB50535964 update via the Microsoft Update catalog
- NOTE: the linked url includes the KB found in the Windows Update for this specific environment/lab. The cumulative KB may vary and thus the link will need to be modified to reflect the latest Windows Update
- If the required Windows Update is not installed, proceed to Option 1, step 2 to resolve.
3. Kickoff a new stembuild
- If the required Windows Update is installed, then it's likely there is corruption somewhere within the operating system. Proceed to Option 1, step 1 to resolve.
Additional Information
If Option 2 is used prior to Option 1, and the BOSH agent is still failing bootstrap, it likely means that the base OS is corrupt. This can happen for a variety of reasons that the stembuild process will not catch. In that case, Option 1 will need to be used.
Feedback
