NSX Edge syslog reports "Failed to collect metrics, Exception:badly formed hexadecimal UUID string" while doing the audit_log_health.remote_logging_server_error alarm check for all the rsyslog exporters.

Products

• VMware NSX

Issue/Introduction

Symptoms:
You will see the below log in NSX Edge's syslog.
 

2023-10-31T13:00:34.914Z NSX 3186 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="root" level="WARNING" s2comp="metric-collector"] Failed to collect metrics, Exception:badly formed hexadecimal UUID string, trace:Traceback (most recent call last):#012  File "/opt/vmware/nsx-netopa/lib/python/sha/core/collector/_metric_collector.py", line 157, in _collect#012    data = metric_usage()#012  File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/utils/_time_out_cache.py", line 15, in wrapper#012    record[1] = fn(*args, **kwargs)#012  File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/audit_log_health_for_remote_error_status.py", line 93, in usage#012    key = uuid.UUID(cur_logging_server[_EXPORTER_NAME])#012  File "/opt/vmware/nsx-netopa/libexec/python-3.8.3/lib64/python3.8/uuid.py", line 169, in __init__#012    raise ValueError('badly formed hexadecimal UUID string')#012ValueError: badly formed hexadecimal UUID string

Lab replication and findings :
**************************************

NSX version : 3.2.1.2.0.20541219

1. CLI configuration :

edge01> get logging-servers
Thu Nov 09 2023 UTC 06:23:27.083

You create a logging-server using Edge CLI.

edge01> set logging-server 10.1.1.1:514 proto tcp level warn
            WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.

edge01> get logging-servers
Thu Nov 09 2023 UTC 06:24:28.885
10.1.1.1:514 proto tcp level warning exporter_name ########-####-####-####-########4b53

Logging server 10.1.1.1 gets configured successfully with an  exporter name as UUID string ########-####-####-####-########4b53.

From /etc/rsyslog.conf file :-

$ActionQueueType LinkedList # nsx exporter: ########-####-####-####-########4b53
*.warning @@10.1.1.1:514;RFC5424fmt # nsx exporter: ########-####-####-####-########4b53

GET : https://192.168.120.1/api/v1/transport-nodes/########-####-####-####-########cbac/node/services/syslog/exporters

{
    "_schema": "NodeSyslogExporterPropertiesListResult",
    "_self": {
        "href": "/transport-nodes/########-####-####-####-########cbac/node/services/syslog/exporters",
        "rel": "self"
    },
    "result_count": 3,
    "results": [
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########92f7",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########92f7",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "10.2.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########4b53",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########4b53",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "10.1.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########3d24",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########3d24",
            "level": "INFO",
            "port": 514,
            "protocol": "UDP",
            "server": "192.168.120.200"
        }
    ]
}

2. API configuration :

POST : https://192.168.120.1/api/v1/transport-nodes/########-####-####-####-########cbac/node/services/syslog/exporters

{
  "exporter_name": "A###-exporter",
  "facilities": ["KERN", "USER"],
  "level": "INFO",
  "msgids": ["tcpin", "tcpout"],
  "port": 514,
  "protocol": "TCP",
  "server": "10.5.1.1"
}

GET : https://192.168.120.1/api/v1/transport-nodes/########-####-####-####-########cbac/node/services/syslog/exporters

{
    "_schema": "NodeSyslogExporterPropertiesListResult",
    "_self": {
        "href": "/transport-nodes/########-####-####-####-########cbac/node/services/syslog/exporters",
        "rel": "self"
    },
    "result_count": 4,
    "results": [
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########92f7",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########92f7",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "10.2.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########b2bd",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########b2bd",
            "level": "WARNING",
            "port": 514,
            "protocol": "TCP",
            "server": "10.1.1.1"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/########-####-####-####-########3d24",
                "rel": "self"
            },
            "exporter_name": "########-####-####-####-########3d24",
            "level": "INFO",
            "port": 514,
            "protocol": "UDP",
            "server": "192.168.120.200"
        },
        {
            "_schema": "NodeSyslogExporterProperties",
            "_self": {
                "href": "/node/services/syslog/exporters/A###-exporter",
                "rel": "self"
            },
            "exporter_name": "A###-exporter",
            "facilities": [
                "KERN",
                "USER"
            ],
            "level": "INFO",
            "msgids": [
                "tcpin",
                "tcpout"
            ],
            "port": 514,
            "protocol": "TCP",
            "server": "10.5.1.1"
        }
    ]
}

From /etc/rsyslog.conf file :-

$ActionQueueType LinkedList # nsx exporter: A###-exporter
if $msgid == 'tcpin' or $msgid == 'tcpout' then { # nsx exporter: A###-exporter
kern,user.info @@10.5.1.1:514;RFC5424fmt # nsx exporter: A###-exporter
} # nsx exporter: A###-exporter
root@edge01:~#

As we can see that if you configure exporter using API and exporter name as a string, then in /etc/rsyslog.conf, you will see nsx exporter name as string instead of UUID.

 

Environment

VMware NSX-T Data Center

Cause

If an exporter is configured using API POST /node/services/syslog/exporters, a non-uuid string as exporter name gets created. When NSX does a audit_log_health.remote_logging_server_error check for all the rsyslog exporters, the events would be logged as it expects an UUID instead of a string as exporter name. UUID as an exporter name is only possible if exporter configuration is made with CLI instead of an API which generates a string as exporter name.

Resolution

A mechanism for more robust handling of exporters is introduced in NSX 4.2.0 version.

Workaround:
Use CLI to configure exporter.

edge01> set logging-server 10.6.1.1:514 proto tcp level info
        WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.

NSX auto generates UUID when an exporter was configured using CLI.

$ActionQueueType LinkedList # nsx exporter: ########-####-####-####-########cf05
*.info @@10.6.1.1:514;RFC5424fmt # nsx exporter: ########-####-####-####-########cf05

Additional Information

Impact/Risks:
There is no impact.