Guest OS Customization Fails with "Cannot Complete Customization" error

Products

• VMware vCenter Server

Issue/Introduction

• When deploying a virtual machine from a template and applying a Guest OS Customization, the task fails with the following error:
  Cannot complete customization.
  
  
• The /var/log/vmware/vpxd/vpxd.log file on the vCenter Server Appliance (VCSA) contains error entries indicating a failure to create temporary files during the checkCustomizationSpec operation:

YYYY-MM-DDThh:mm:ssZ info vpxd[#####] [Originator@#### sub=vpxLro opID=#####] [VpxLRO] -- BEGIN task-##### -- vm-<ID> -- vim.VirtualMachine.checkCustomizationSpec -- #####-#####-#####-#####-#####(#####-#####-#####-#####-#####)
YYYY-MM-DDThh:mm:ssZ info vpxd[#####] [Originator@#### sub=VmCustomizer opID=#####] hostVersion = 8.0.#, Tools version = #####
YYYY-MM-DDThh:mm:ssZ error vpxd[#####] [Originator@#### sub=VmCustomizer opID=#####] Creating tmp file: 13
YYYY-MM-DDThh:mm:ssZ warning vpxd[#####] [Originator@#### sub=Vmomi opID=#####] VMOMI activation LRO failed; <<#####-#####-#####-#####-#####, <TCP '127.0.0.1 : ####'>, <TCP '127.0.0.1 : #####'>>, vm-<ID>, vim.VirtualMachine.checkCustomizationSpec, <vim.version.v8_0_#_0, internal, 8.0.#.0>, {stm: {<io_obj p:0x#####, h:###, <TCP '127.0.0.1 : #####'>, <TCP '127.0.0.1 : #####'>>, id: #####, state(in/out): #/#}, session: <#####-#####-#####-#####-#####, <TCP '127.0.0.1 : #####'>, <TCP '127.0.0.1 : #####'>>, req: {POST, /sdk/vimService}}>, N3Vim5Fault18CustomizationFault9ExceptionE(Fault cause: vim.fault.CustomizationFault
--> )
YYYY-MM-DDThh:mm:ssZ info vpxd[#####] [Originator@#### sub=vpxLro opID=#####] [VpxLRO] -- FINISH task-#####
YYYY-MM-DDThh:mm:ssZ error vpxd[#####] [Originator@#### sub=Default opID=#####] [VpxLRO] -- ERROR task-#####-- #####-#####-#####-#####-#####(#####-#####-#####-#####-#####) -- vm-<ID>-- vim.VirtualMachine.checkCustomizationSpec: :vim.fault.CustomizationFault
--> Result:
--> (vim.fault.CustomizationFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = ""
--> }

Environment

vCenter Server 8.x

Cause

• The vCenter Server service (vpxd) requires write access to the /tmp directory to generate the customization deployment package (containing Sysprep.xml or cust.cfg) before it is pushed to the ESXi host and the target VM.
• The error "Creating tmp file: 13" translates to a Permission Denied error at the operating system level on the VCSA.
• This issue occurs if the default permissions on the /tmp directory have been modified.

Resolution

To resolve this issue, restore the default permissions to the /tmp directory on the vCenter Server Appliance.

1. Take an offline snapshot of the vCenter Server Appliance to ensure a valid rollback point before making operating system changes.
   Note: If the vCenter Server is in Enhanced Linked Mode (ELM), all vCenter Servers in the SSO domain must be shut down and snapshotted simultaneously.
   Refer to Snapshot Best practices for vCenter Server Virtual Machines
   
   
2. Log in to the vCenter Server Appliance via SSH as the root user.
   
   
3. Verify the current permissions of the /tmp directory:
   ls -ld /tmp
   
   
4. Restore the required permissions to drwxrwxrwt:
   chmod 1777 /tmp


5. Verify the change is successful:
   ls -ld /tmp
   
   
6. The output must be as below:
   drwxrwxrwt ## root root #### MMM DD hh:mm /tmp
   
   
7. Restart the vpxd service using the below command:
   service-control --stop vpxd
service-control --start vpxd


8. Retry the VM deployment or guest customization task.