VCSA SNMP v3 configuration failing with the error - (code com.vmware.applmgmt.err_snmp_operation_failed)
search cancel

VCSA SNMP v3 configuration failing with the error - (code com.vmware.applmgmt.err_snmp_operation_failed)

book

Article ID: 382760

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • While attempting to configure SNMP v3 on the vCenter server, users encounter the following error when executing the snmp.hash command -
    • Error in method: There is a problem with the SNMP configuration. See the log files for more details. (code com.vmware.applmgmt.err_snmp_operation_failed)
  • The SNMP set for engine ID, authentication, and privacy configuration on the VCSA appears to be defined correctly.
  • However, when specifying the file paths for the SNMP v3 user authentication and privacy passwords using the --raw_secret flag, the issue remains, and the configuration still does not work as expected.
  • The following entries are found in the /var/log/vmware/applmgmt/applmgmt.log - 

yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:root:com.vmware.appliance.version1.monitoring.snmp.hash()
yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:vmware.appliance.snmp.vmw_snmp:make_keys: loaded from config, authProto=usmHMACSHAAuthProtocol privProto=usmAESCfb128PrivProtocol, eid=de9235d25dbbf7e
yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:vmware.appliance.snmp.vmw_snmp:ap=usmHMACSHAAuthProtocol, pp=usmAESCfb128PrivProtocol
yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:vmware.appliance.snmp.vmw_snmp:running cmd: /bin/snmpd -a SHA1 -s 1*SDKx!oXq#WtNW -r -p AES128 -s Yxwv9843fhj8!3#
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:vmware.appliance.snmp.vmw_snmp:make_keys: generate hash failed: Command '['/bin/snmpd', '-a', 'SHA1', '-s', '1*SDKx!oXq#WtNW', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:vmware.appliance.snmp.vmw_snmp:make_keys: cmd stderr log was:
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:vmware.appliance.snmp.vmw_snmp:make_keys: internal error, failed: Command '['/bin/snmpd', '-a', 'SHA1', '-s', '1*SDKx!oXq#WtNW', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:Traceback (most recent call last):
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:  File "/usr/lib/applmgmt/snmp/py/vmware/appliance/snmp/vmw_snmp.py", line 1330, in make_keys
    report = sb.check_output(app_args, stdin=null_dev, stderr=errlog)
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:  File "/usr/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:  File "/usr/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:subprocess.CalledProcessError: Command '['/bin/snmpd', '-a', 'SHA1', '-s', 'Bcef1256lnp4!3@', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:
During handling of the above exception, another exception occurred:
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:Traceback (most recent call last):
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:  File "/usr/lib/applmgmt/snmp/py/vmware/appliance/snmp/vmw_snmp.py", line 1334, in make_keys
    raise Exception(str(err))
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:root:Exception: Command '['/bin/snmpd', '-a', 'SHA1', '-s', 'Bcef1256lnp4!3@', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.
yyyy-mm-ddThh:mm:ss.Z [1425]ERROR:vmware.appliance.snmp.vmw_snmp:['Traceback (most recent call last):\n', '  File "/usr/lib/applmgmt/snmp/py/vmware/appliance/snmp/vmw_snmp.py", line 1330, in make_keys\n    report =
 sb.check_output(app_args, stdin=null_dev, stderr=errlog)\n', '  File "/usr/lib/python3.7/subprocess.py", line 411, in check_output\n    **kwargs).stdout\n', '  File "/usr/lib/python3.7/subprocess.py", line 512, in run\
n    output=stdout, stderr=stderr)\n', "subprocess.CalledProcessError: Command '['/bin/snmpd', '-a', 'SHA1', '-s', 'Bcef1256lnp4!3@', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.\n",
'\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/usr/lib/applmgmt/snmp/py/vmware/appliance/snmp/vmw_snmp.py", line 1334, in make_keys\n    raise Exception(str(err))\n', "Exception: Command '['/bin/snmpd', '-a', 'SHA1', '-s', 'Bcef1256lnp4!3@', '-r', '-p', 'AES128', '-s', 'Yxwv9843fhj8!3#']' returned non-zero exit status 1.\n"]
yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:vmware.vherd.transport.xmlrpc:Trapped an exception: pint.Error('com.vmware.applmgmt.err_snmp_operation_failed', 'There is a problem with the SNMP configuration. See the log files for more details.', **{})
yyyy-mm-ddThh:mm:ss.Z [1425]DEBUG:vmware.vherd.transport.xmlrpc:Dictionary returned: {'code': 'com.vmware.applmgmt.err_snmp_operation_failed', 'message': 'There is a problem with the SNMP configuration. See the log files for more details.', 'attributes': {}, '__error__': True}

  • From the /var/log/vmware/messages, it is observed that the SNMP daemon (snmpd) encountered a failure during Log Level calibration due to an invalid engine ID string as per the below sample -

yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P2 1: BEGIN Log Level calibration
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P4 2: 3 of 4. Testing Log Levels - LOG_WARNING
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P3 3: 4 of 4. Testing Log Levels - LOG_ERR
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P2 4: END Log Level Calibration
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P3 5: hex_to_binary: invalid hex string: wrong length
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P3 6: set_engine_id: hex_to_binary(de9235d25dbbf7e) engineid conversion failed.
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[2409]: P3 7: generate_hash: snmpv3_generate_engineID unexpectedly did not return an engineid, SNMPv3 offline
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 cli: root Error in method: There is a problem with the SNMP configuration. See the log files for more details. (code com.vmware.applmgmt.err_snmp_operation_failed)
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 CROND[2494]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py >>/var/log/vmware/applmgmt/backupSchedulerCron.log 2>&1)
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 CROND[2496]: (root) CMD ( /usr/lib/applmgmt/syslog/scripts/syslog_cron.sh >/dev/null 2>&1)
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 CROND[2495]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P2 1: BEGIN Log Level calibration
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P4 2: 3 of 4. Testing Log Levels - LOG_WARNING
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P3 3: 4 of 4. Testing Log Levels - LOG_ERR
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P2 4: END Log Level Calibration
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P3 5: hex_to_binary: invalid hex string: wrong length
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P3 6: set_engine_id: hex_to_binary(de9235d25dbbf7e) engineid conversion failed.
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 snmpd[3012]: P3 7: generate_hash: snmpv3_generate_engineID unexpectedly did not return an engineid, SNMPv3 offline
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 cli: root Error in method: There is a problem with the SNMP configuration. See the log files for more details. (code com.vmware.applmgmt.err_snmp_operation_failed)
yyyy-mm-ddThh:mm:ss.Z y9900001avct301 CROND[3488]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py >>/var/log/vmware/applmgmt/backupSchedulerCron.log 2>&1)

Environment

  • VMware vCenter Server 6.x
  • VMware vCenter Server 7.x
  • VMware vCenter Server 8.x

Cause

  • The issue appears to be related to SNMPv3 configuration, specifically with key generation and engine ID conversion.
  • The error typically occurs when attempting to convert a hexadecimal string into binary format, especially when the string’s length is invalid.

Please Note - It is important to note that most systems require hexadecimal strings to have an even number of characters, as each byte is represented by two hexadecimal digits.

Resolution

  • Please verify that the engine ID adheres to the required format and length specifications. 
  • Key Requirements for Hex Strings:
    • The length must be even.
    • Only valid hexadecimal characters (0-9, a-f, A-F) should be used.
    • The length must align with the system’s expected format or specifications.
  • Additionally, ensure the SNMPv3 configuration, including authentication and privacy protocol settings, is properly validated and updated as needed.

Additional Information

Powered by Wolken Software