vCenter patch failure with error 'lookupsvc:Patch' failed

Products

• VMware vCenter Server

Issue/Introduction

• /var/log/vmware/applmgmt/PatchRunner.log
  
  =========STDERR=============
2023-04-13 19:26:04,611.611Z lookupsvc:Patch INFO patch_01 2023-04-13T19:26:03.751Z ERROR starting lookupsvc rc: 2, stdout: , stderr: Start service request failed. Error: Service crashed while starting
2023-04-13T19:26:03.752Z Lookup Service firstboot failed
2023-04-13 19:26:04,611.611Z lookupsvc:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'lookupsvc:Patch' failed.
• /var/log/vmware/applmgmt/update_microservice.log
  
  2023-04-13 19:26:05,632 - 37072 -      extensions::        __call__: 107 -  DEBUG - The component script returned 'None'
2023-04-13 19:26:05,636 - 21351 -      update_b2b::        _runScriptHook: 389 -  DEBUG - update script result file '{
  "end_time": "2023-04-13T19:26:05.616Z",
  "error": {
     "componentKey": "lookupsvc:Patch",
     "detail": [
        {
          "args": [
             ""
          ],
          "id": "patch.internal.error.text",
          "localized": "Internal error occurs during execution of update process .",
          "translatable": "Internal error occurs during execution of update process %(0)s."
        }
     ],
     "problemId": null,
     "resolution": {
        "id": "patch.internal.error.resolution",
        "localized": "Send upgrade log files to VMware technical support team for further assistance.",
        "translatable": "Send upgrade log files to VMware technical support team for further assistance."
     }
  },
  "info": [],
  "progress": 5,
  "progress_message": {
  "componentKey": "lookupsvc:Patch",
  "detail": [
     {
     "args": [
        ""
       ],
       "id": "patch.internal.error.text",
       "localized": "Internal error occurs during execution of update process .",
       "translatable": "Internal error occurs during execution of update process %(0)s."
     }
  ],
  "problemId": null,
  "resolution": {
     "id": "patch.internal.error.resolution",
  }
  
  
• /var/log/vmware/lookupsvc/lookupsvc.log for the same time states permission denied:
  
  Starting service process with pid: 43199.
/usr/java/jre-vmware/bin/lookupsvc: line 86: /storage/vmware-vmon/lookupsvc.start.cmd: Permission denied

 

Environment

VMware vCenter Server 7.0.x

Cause

Inadequate file permissions on /storage/vmware-vmon/lookupsvc.start.cmd

File /storage/vmware-vmon/lookupsvc.start.cmd is created as part of the pre-start script from /etc/vmware/vmware-vmon/svcCfgfiles/lookupsvc.json

Umask basically sets the default permissions for newly created files and directories.

Enter "umask" on ssh console, the default value is 0022 for vCenter Server 7.0 if the value is any different the new files created would have an issue with permissions. 

umask might have been modified if they have been applied custom security hardening such as:

The Photon operating system must configure a secure umask for all shells

Resolution

Take appropriate snapshot and backup of the vCenter Server before proceeding.
 

1. SSH to vCenter Server.
2. Run the below command to change the umask to original value 022  (else section of the file)
   
   # vi /etc/profile.d/umask.sh
   
   Example :
   
   # By default, the umask should be set.
if [ "$(id -gn)" = "$(id -un)" -a $EUID -gt 99 ]; then
  umask 002
else
  umask 022
fi
   
   
3. Execute "source /etc/profile.d/umask.sh"
4. Terminate and create a new SSH session
5. Check umask, expected output is 0022
6. Can also test new file creation using,
   
      $VMWARE_CLOUDVM_RAM_SIZE -J lookupsvc -O /tmp/2.txt
   stat /tmp/2.txt
   
   
7. The new file should have the correct permissions of 644
8. Take fresh snapshot and proceed to update the vCenter Server