NSX Controller Service Fails to Start with Error "Protocol message had invalid UTF-8"
search cancel

NSX Controller Service Fails to Start with Error "Protocol message had invalid UTF-8"

book

Article ID: 448965

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The get cluster status command reports the Overall Status: DEGRADED and the CONTROLLER Group Status: UNAVAILABLE.
    root@nsx-manager:/var/log/cloudnet# su -l admin -c get cluster status
<Timestamp_in_UTC>
Cluster Id: <Cluster-Id>
Overall Status: DEGRADED
Group Type: CONTROLLER
Group Status: UNAVAILABLE

Members:
    UUID                     IPV4                         IPV6                                             STATUS           FQDN
    <NSX_Manager_UUID>       <NSX_Manager_IP_Address>     not configured                                   DOWN             <NSX_Manager_FQDN>

------ 
VIP: <NSX_Manager_VIP_Address>
API Leader: <NSX_Manager_UUID>  = <NSX_Manager_Hostname>= <NSX_Manager_VIP_Address>: UP
Service install-upgrade is enabled on <NSX_Manager_IP_Address>
Cluster:
CLUSTER_BOOT_MANAGER  STABLE _____ <NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
CM-INVENTORY ________ STABLE _____ <NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
CONTROLLER __________ UNAVAILABLE  <NSX_Manager_Hostname>= <NSX_Manager_IP_Address>: DOWN <--------
CORFU_NONCONFIG _____ STABLE _____ <NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
DATASTORE ___________ STABLE _____ <NSX_Manager_Hostname>= <NSX_Manager_IP_Address>: UP
HTTPS _______________ STABLE _____<NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
IDPS_REPORTING ______ STABLE _____<NSX_Manager_Hostname>= <NSX_Manager_IP_Address>: UP
MANAGER _____________ STABLE _____ <NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
MESSAGING-MANAGER ___ STABLE _____<NSX_Manager_Hostname>= <NSX_Manager_IP_Address>: UP
MONITORING __________ STABLE _____ <NSX_Manager_Hostname> = <NSX_Manager_IP_Address>: UP
  • Controller service status is UP and running.
    # su -l admin -c get service controller
Mon Jul 06 2026 UTC 05:32:26.632
Service name:       controller
Service state:      running <---
Listen address:

# /etc/init.d/nsx-ccp status <----
nsx-ccp.service - nsx-ccp: VMware NSX Controller daemon
     Loaded: loaded (/etc/init.d/nsx-ccp; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-07-06 05:31:36 UTC; 1min 8s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 24213 ExecStart=/etc/init.d/nsx-ccp start $ARG1 (code=exited, status=0/SUCCESS)
   Main PID: 24326 (wrapper)
      Tasks: 3 (limit: 28744)
     Memory: 5.9M
        CPU: 1min 50.569s
     CGroup: /system.slice/nsx-ccp.service
             └─24326 /usr/tanuki/bin/wrapper /usr/tanuki/bin/../conf/nsx-ccp-wrapper.conf wrapper.syslog.ident=nsx-ccp wrapper.pidfile=/var/run/cloudnet/nsx-ccp.pid wr…
  • The nsx-ccp service fails to start, repeatedly logging the following error under /var/log/cloudnet/nsx-ccp.log
    <Timestamp> CCP_ERROR WrapperSimpleAppMain ccperrorlogger 94922 - [nsx@4413 comp="nsx-controller" errorCode="CCP2010150" level="CCP_ERROR" subcomp="nsx-controller"] Controller startup failed2026-07-06T05:56:00.914Z CCP_ERROR WrapperSimpleAppMain ccperrorlogger 96397 - [nsx@4413 comp="nsx-controller" errorCode="CCP2010150" level="CCP_ERROR" subcomp="nsx-controller"] Controller startup failed
  • A read failure occurred in memory, throwing the following exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.
    <Timestamp> ERROR RealizationStatusTrackerStreamListenerImpl-2-1 ProtobufSerializer 6289 Exception during deserialization!
com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.
        at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:172) ~[?:?]
        at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1388) ~[?:?]
        at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:341) ~[?:?]
        at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:815) ~[?:?]
        at com.google.protobuf.StringValue$Builder.mergeFrom(StringValue.java:407) ~[?:?]
        at com.google.protobuf.StringValue$1.parsePartialFrom(StringValue.java:558) ~[?:?]
        at com.google.protobuf.StringValue$1.parsePartialFrom(StringValue.java:550) ~[?:?]
        at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) ~[?:?]
        at vmware.nsx.ufostore.ids.model.IdsCompressedSignatureInternalProto$IdsCompressedSignatureInternalMsg.<init>(IdsCompressedSignatureInternalProto.java:284) ~[?:?]

Environment

VMware NSX 9.x

Cause

The tar file data for the default system-created IDS signature bundle (IdsCompressedSignatureInternal) became corrupted on the disk. Consequently, the CCP fails to load the record into memory, resulting in an InvalidProtocolBufferException during deserialization.

Resolution

If you believe you have encountered this issue, open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.

  1. Log in to the impacted NSX manager node via SSH using the root user account.

  2. Run the following command to collect the Corfu table data: /opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t IdsCompressedSignatureInternal > /tmp/IdsCompressedSignatureInternal.txt

  3. Generate and collect the NSX support bundle.

  4. Upload both the support bundle and the Corfu table text output file to the Broadcom support case for further analysis.

Powered by Wolken Software