Upgrade errands all cluster failed with "500 Internal Server Error"

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• You see an error similar to the following after clicking  "Apply changes" in Opsmanager:

4c710d54c0d2] Starting to process service instance 2 of 3  
           [upgrade-all-service-instances] 2019/03/08 17:44:42.392633 [upgrade-all] [6d6c8d57-76db-45e5-891a-4c710d54c0d2] Result: operation accepted  
           [upgrade-all-service-instances] 2019/03/08 17:44:42.392648 [upgrade-all] [6d6c8d57-76db-45e5-891a-4c710d54c0d2] Waiting for operation to complete: bosh task id 5616  
Errand 'upgrade-all-service-instances' completed with error (exit code 1)
Exit code 1
           [upgrade-all-service-instances] 2019/03/08 17:55:43.065075 [upgrade-all] FINISHED PROCESSING Status: FAILED; Summary: Number of successful operations: 1; Number of service instance orphans detected: 0; Number of deleted instances before operation could happen: 0; Number of busy instances which could not be processed: 0; Number of service instances that failed to process: 1 [6d6c8d57-76db-45e5-891a-4c710d54c0d2]  
           [upgrade-all-service-instances] 2019/03/08 17:55:43.065091 error getting last operation: HTTP response status: 500 Internal Server Error  
             
Stderr     -  

1 errand(s)

===== 2019-03-08 17:56:37 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=10.132.10.21 --deployment=pivotal-container-service-4deebd68e725789b41de run-errand upgrade-all-service-instances"; Duration: 1389s; Exit Status: 1
Exited with 1.

 

• You see a similar ncp error in  /var/vcap/sys/log/ncp/ncp.stderr.log:

ValueError: unsupported format character 'S' (0x53) at index 10
Logged from file ingress_mixin.py, line 941
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
    return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/nsx_ujo/common/nsx_log_adaptor.py", line 79, in format
    return self._base_formatter.format(record)
  File "/usr/local/lib/python2.7/dist-packages/nsx_ujo/common/nsx_log_adaptor.py", line 63, in format
    return self._base_formatter.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
ValueError: unsupported format character 'S' (0x53) at index 10
Logged from file ingress_mixin.py, line 941

Environment

VMware PKS 1.x

Resolution

This issue is resolved in NSX-T 2.3.2

Workaround:

•  Comment the following two lines on all the NCP master in the /var/vcap/data/ncp/rootfs/usr/local/lib/python2.7/dist-packages/nsx_ujo/k8s/ingress_mixin.py file

 # LOG.info('Handling rule transfer from VS %s to VS %S for ingress %s',
 #               from_vs, to_vs, ingress_cache)
 
 

• Apply changes on the ops manager