NSX Edge syslog reports "Failed to collect metrics, Exception:badly formed hexadecimal UUID string" while doing the audit_log_health.remote_logging_server_error alarm check for all the rsyslog exporters.

Products

• VMware NSX

Issue/Introduction

Symptoms:
You will see the below log in NSX Edge's syslog.
 

2023-10-31T13:00:34.914Z NSX 3186 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="root" level="WARNING" s2comp="metric-collector"] Failed to collect metrics, Exception:badly formed hexadecimal UUID string, trace:Traceback (most recent call last):#012  File "/opt/vmware/nsx-netopa/lib/python/sha/core/collector/_metric_collector.py", line 157, in _collect#012    data = metric_usage()#012  File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/utils/_time_out_cache.py", line 15, in wrapper#012    record[1] = fn(*args, **kwargs)#012  File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/audit_log_health_for_remote_error_status.py", line 93, in usage#012    key = uuid.UUID(cur_logging_server[_EXPORTER_NAME])#012  File "/opt/vmware/nsx-netopa/libexec/python-3.8.3/lib64/python3.8/uuid.py", line 169, in __init__#012    raise ValueError('badly formed hexadecimal UUID string')#012ValueError: badly formed hexadecimal UUID string

Lab replication and findings :
**************************************

NSX version : 3.2.1.2.0.20541219

1. CLI configuration :

edge01> get logging-servers
Thu Nov 09 2023 UTC 06:23:27.083

You create a logging-server using Edge CLI.

edge01> set logging-server 1.1.1.1:514 proto tcp level warn
            WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.

edge01> get logging-servers
Thu Nov 09 2023 UTC 06:24:28.885
1.1.1.1:514 proto tcp level warning exporter_name 3b2439a2-58c8-40f5-9fb3-2a456db44b53

Logging server 1.1.1.1 gets configured successfully with an  exporter name as UUID string 3b2439a2-58c8-40f5-9fb3-2a456db44b53.

From /etc/rsyslog.conf file :-

$ActionQueueType LinkedList # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53
*.warning @@1.1.1.1:514;RFC5424fmt # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53

GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters

{
    "_schema": "NodeSyslogExporterPropertiesListResult",
    "_self": {
        "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters",
        "rel": "self"
    },
    "result_count": 3,
    "results": [
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7",
                "rel": "self"
            },
            "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "2.2.2.2"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/3b2439a2-58c8-40f5-9fb3-2a456db44b53",
                "rel": "self"
            },
            "exporter_name": "3b2439a2-58c8-40f5-9fb3-2a456db44b53",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "1.1.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24",
                "rel": "self"
            },
            "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24",
            "level": "INFO",
            "port": 514,
            "protocol": "UDP",
            "server": "192.168.120.200"
        }
    ]
}

2. API configuration :

POST : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters

{
  "exporter_name": "Amit-exporter",
  "facilities": ["KERN", "USER"],
  "level": "INFO",
  "msgids": ["tcpin", "tcpout"],
  "port": 514,
  "protocol": "TCP",
  "server": "5.5.5.5"
}

GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters

{
    "_schema": "NodeSyslogExporterPropertiesListResult",
    "_self": {
        "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters",
        "rel": "self"
    },
    "result_count": 4,
    "results": [
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7",
                "rel": "self"
            },
            "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "2.2.2.2"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd",
                "rel": "self"
            },
            "exporter_name": "7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "1.1.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24",
                "rel": "self"
            },
            "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24",
            "level": "INFO",
            "port": 514,
            "protocol": "UDP",
            "server": "192.168.120.200"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/Amit-exporter",
                "rel": "self"
            },
            "exporter_name": "Amit-exporter",
            "facilities": [
                "KERN",
                "USER"
            ],
            "level": "INFO",
            "msgids": [
                "tcpin",
                "tcpout"
            ],
            "port": 514,
            "protocol": "TCP",
            "server": "5.5.5.5"
        }
    ]
}

From /etc/rsyslog.conf file :-

$ActionQueueType LinkedList # nsx exporter: Amit-exporter
if $msgid == 'tcpin' or $msgid == 'tcpout' then { # nsx exporter: Amit-exporter
kern,user.info @@5.5.5.5:514;RFC5424fmt # nsx exporter: Amit-exporter
} # nsx exporter: Amit-exporter
root@edge01:~#

As we can see that if you configure exporter using API and exporter name as a string, then in /etc/rsyslog.conf, you will see nsx exporter name as string instead of UUID.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

If an exporter is configured using API POST /node/services/syslog/exporters, a non-uuid string as exporter name gets created. When NSX does a audit_log_health.remote_logging_server_error check for all the rsyslog exporters, the events would be logged as it expects an UUID instead of a string as exporter name. UUID as an exporter name is only possible if exporter configuration is made with CLI instead of an API which generates a string as exporter name.

Resolution

A mechanism for more robust handling of exporters is introduced in NSX 4.2.0 version.

Workaround:
Use CLI to configure exporter.

edge01> set logging-server 6.6.6.6:514 proto tcp level info
        WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.

NSX auto generates UUID when an exporter was configured using CLI.

$ActionQueueType LinkedList # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05
*.info @@6.6.6.6:514;RFC5424fmt # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05

Additional Information

Impact/Risks:
There is no impact.