On-Demand Enabling or Disabling of Security Troubleshooting Metrics in the Security Services Platform (SSP)
Article ID: 390485
Updated On:
Products
Issue/Introduction
Currently, some metrics collected in SSP are disabled by default. Users may wish to enable these metrics for debugging or troubleshooting purposes.
Environment
NSX >= 4.2.2
vDefend SSP >= 5.0
Cause
Several security monitoring-related metrics are disabled by default in NSX 4.2.2 or above.
If these metrics are needed, they can be enabled on-demand for troubleshooting purposes.
It is recommended to enable them only for short durations and on specific Edges or Hosts where required.
Resolution
Here we are enumerating steps to enable a plugin (NSX ships with that plugin being disable by default). Same steps work for disabling a plugin as well.
=======================================================
Step 1: Identify the UUID/Path of the SHA plugin that you want to enable.
=======================================================
Below table contains the list of disabled Metrics and corresponding SHA Plugins
| Plugin ID | Plugin Name | Metrics |
| /infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088 | edge_fw_conn_monitor |
edge_fw_conn_sum.avg_tcp_half_open_ingress_conn edge_fw_conn_sum.avg_tcp_max_conn edge_fw_conn_sum.avg_udp_ingress_conn edge_fw_conn_sum.avg_udp_max_conn edge_fw_conn_sum.avg_icmp_ingress_conn edge_fw_conn_sum.avg_icmp_max_conn edge_fw_conn_sum.avg_others_ingress_conn edge_fw_conn_sum.avg_others_max_conn edge_fw_conn.avg_tcp_open_conn edge_fw_conn.avg_tcp_est_conn edge_fw_conn.avg_udp_est_conn edge_fw_conn.avg_icmp_est_conn edge_fw_conn.avg_others_est_conn edge_fw_per_host.avg_tx_conn_per_core edge_fw_per_host.avg_rx_conn_per_core |
| /infra/sha/pre-defined-plugins/16d2490a-e505-5f25-8259-cddefbf8040b | idps_stats_monitor |
idps.avg_event_sent_nsxi_ndr, idps.avg_event_send_failure_nsxi_ndr idps.avg_event_send_failure_nsx_manager idps.avg_event_sent_nsx_manager edge_idps_exporter.avg_event_enqueue_failure_nsxi edge_idps_exporter.avg_event_enqueue_failure_nsx_manager edge_idps_exporter.avg_critical edge_idps_exporter.avg_non_critical ege_idps_exporter.avg_total edge_idps_datapath.avg_event_callback edge_idps_datapath.avg_event_sent_count edge_idps_datapath.avg_big_event_count edge_idps_datapath.avg_event_enqueue_count edge_idps_datapath.avg_event_enqueue_failure_count edge_idps_datapath.avg_event_out_of_mem_count edge_idps_datapath.avg_socket_reconnect_count edge_idps_datapath.avg_event_send_failure_count
|
Note: Below Metrics keys may show up as available in NSX 4.2.2. However they are removed starting NSX 4.2.2 onward. So we can't enable them using below workflow.
| Plugin ID | Plugin Name | Metrics |
| /infra/sha/pre-defined-plugins/26d87226-1673-4c3c-9b56-50d85fc57479 | edge_fw_stats_monitor |
edge_fw.avg_drop_reason_3whs |
Once you have identified plugin you want to enable/disable, you can use below API to get list of all pre-defined SHA plugins and their current status:
GET https://<nsx-ip>/policy/api/v1/infra/sha/pre-defined-plugins
Sample payload:
pre-defined-plugins:
{
"results": [
{
"enabled": false, //// This field tells overall status of the plugin.
"config": {
"check_interval": 60
},
"supported_node_types": [
"NSX_EDGE"
],
"pre_req_conditions": [
"WAVE_FRONT",
"TSDB"
],
"delay_on_reboot": 300,
"resource_type": "ShaPredefinedPlugin",
"id": "63b58bc1-6c61-4414-92b5-19ef57e84088",
"display_name": "Edge firewall connection plugin",
"path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"relative_path": "63b58bc1-6c61-4414-92b5-19ef57e84088",
"parent_path": "/infra",
"remote_path": "",
"unique_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",
"realization_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739924951965,
"_last_modified_time": 1739924951965,
"_create_user": "system",
"_last_modified_user": "system",
"_revision": 0
},
{
"enabled": true,
"config": {
"check_interval": 60
},
"supported_node_types": [
"NSX_EDGE"
],
"pre_req_conditions": [
"TSDB"
],
"delay_on_reboot": 10,
"resource_type": "ShaPredefinedPlugin",
"id": "94d29bbd-0f85-427c-a226-3bbcc5291401",
"display_name": "Edge firewall connections per rule and per logical-router plugin",
"path": "/infra/sha/pre-defined-plugins/94d29bbd-0f85-427c-a226-3bbcc5291401",
"relative_path": "94d29bbd-0f85-427c-a226-3bbcc5291401",
"parent_path": "/infra",
"remote_path": "",
"unique_id": "a12151f1-7b2f-4c7f-a5e7-3cf5bb7ad53d",
"realization_id": "a12151f1-7b2f-4c7f-a5e7-3cf5bb7ad53d",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739924952024,
"_last_modified_time": 1739924952024,
"_create_user": "system",
"_last_modified_user": "system",
"_revision": 0
},
{
"enabled": true,
"config": {
"check_interval": 60
},
"supported_node_types": [
"NSX_EDGE"
],
"pre_req_conditions": [
"WAVE_FRONT",
"TSDB"
],
"delay_on_reboot": 300,
"resource_type": "ShaPredefinedPlugin",
"id": "26d87226-1673-4c3c-9b56-50d85fc57479",
"display_name": "Edge firewall stats plugin",
"path": "/infra/sha/pre-defined-plugins/26d87226-1673-4c3c-9b56-50d85fc57479",
"relative_path": "26d87226-1673-4c3c-9b56-50d85fc57479",
"parent_path": "/infra",
"remote_path": "",
"unique_id": "968af296-9eb1-4307-8ab2-786ac4ea8134",
"realization_id": "968af296-9eb1-4307-8ab2-786ac4ea8134",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739924952081,
"_last_modified_time": 1739924952081,
"_create_user": "system",
"_last_modified_user": "system",
"_revision": 0
}
.....
..... // We have omitted other plugins for brevity.
],
"result_count": 53,
"sort_by": "display_name",
"sort_ascending": true
}
=======================================================
Step 2: Identify current plugin status and confirm it's actually disabled/enabled.
=======================================================
As you can see above Plugin : 63b58bc1-6c61-4414-92b5-19ef57e84088 (Edge firewall connection plugin) is disabled by default.
GET https://<NSX-IP>/policy/api/v1/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088
Note: Check the "supported_node_types" attribute. It tells us that where this plugin can be run (On Edge Transport Nodes or Host Transport Nodes). This will be used in next step.
Edge Firewall connection plugin:
{
"enabled": false, /// Plugin is disabled by default
"config": {
"check_interval": 60
},
"supported_node_types": [
"NSX_EDGE"
],
"pre_req_conditions": [
"WAVE_FRONT",
"TSDB"
],
"delay_on_reboot": 300,
"resource_type": "ShaPredefinedPlugin",
"id": "63b58bc1-6c61-4414-92b5-19ef57e84088",
"display_name": "Edge firewall connection plugin",
"path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"relative_path": "63b58bc1-6c61-4414-92b5-19ef57e84088",
"parent_path": "/infra",
"remote_path": "",
"unique_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",
"realization_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739924951965,
"_last_modified_time": 1739924951965,
"_create_user": "system",
"_last_modified_user": "system",
"_revision": 0
}
=======================================================
Step 3: Identify Edge TNs where you want to enable this plugins.
=======================================================
We can enable a plugin on NSX_ESX or NSX_EDGE
As you can see supported_node_types for our example plugin is "NSX_EDGE", hence you can enables it on Edge Transport Nodes only.
Note: For 4.2.2 release, we allow plugin to be be enable on all Edge Transport Nodes. Support to enable/disable on selective Transport Nodes will come in future release
Check Internal Notes section for Host Side Group creation.
=======================================================
Step 4: Create a group with all Edge Transport Nodes
=======================================================
Create a Group with all Edge Transport Nodes.
PATCH : https://<NSX-IP>/policy/api/v1/infra/domains/default/groups/ALL_EDGE_TNS_GROUP
Edge Group Creation body:
{
"expression": [
{
"member_type": "TransportNode",
"key": "NodeType",
"operator": "EQUALS",
"value": "EdgeNode",
"resource_type": "Condition"
}
],
"extended_expression": [],
"reference": false,
"resource_type": "Group",
"display_name": "ALL_EDGE_TNS_GROUP",
"description": "This group contains all Edge Nodes in NSX"
}
Once above request executed, you can check in NSX UI:
=======================================================
Step 5 : Enable SHA plugin using SHA profile.
=======================================================
5A) We will cross check status of SHA plugin with ID 63b58bc1-6c61-4414-92b5-19ef57e84088 on Edge Transport Nodes : d89bbd96-ddf4-11ef-b323-005056ac915f.
As expected plugin should be Disabled.
GET https://<NSX-IP>/policy/api/v1/infra/sha/plugin-status/d89bbd96-ddf4-11ef-b323-005056ac915f
Plugin status:
{
"plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"plugin_name": "edge_fw_conn_monitor",
"status": "NORMAL",
"profile": "NAME: default-profile, ENABLE: False, CHECK_INTERVAL: 60", /// It's disabled
"detail": "Plugin is disabled.",
"node_path": "/infra/sites/default/enforcement-points/default/edge-transport-node/d89753aa-ddf4-11ef-a1bb-005056ac5faf"
},
5B) We will create a SHA profile to enable SHA plugin : 63b58bc1-6c61-4414-92b5-19ef57e84088 (step-2). We will be applying this profile on a group (ALL_EDGE_TNS_GROUP) created in step-4.
PATCH https://<NSX-IP>/policy/api/v1/infra/sha/plugin-profiles/profile1
Request:
Error rendering macro 'code': Invalid value specified for parameter 'firstline'
{
"config": {
"check_interval": 60 /// This interval tells how frequently it needs to run & collect metrics. Minimum supported value of this interval is 60 seconds.
},
"resource_type": "PredefinedPlugin",
"id": "profile1",
"display_name": "profile1",
"plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"applied_to_group_path": "/infra/domains/default/groups/ALL_EDGE_TNS_GROUP", /// Group containing Edge TNs where we are applying this Profile
"enabled": true /// Here we are enabling this plugin
}
Response :
Response body:
{
"config": {
"check_interval": 60
},
"resource_type": "PredefinedPlugin",
"id": "profile1",
"display_name": "profile1",
"path": "/infra/sha/plugin-profiles/profile1",
"relative_path": "profile1",
"parent_path": "/infra",
"remote_path": "/orgs/default/projects/default/infra/sha/plugin-profiles/profile1--nbendre-nsx",
"unique_id": "5e294a1f-3d67-4b74-8665-396efb340769",
"realization_id": "5e294a1f-3d67-4b74-8665-396efb340769",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"applied_to_group_path": "/infra/domains/default/groups/ALL_EDGE_TNS_GROUP",
"enabled": true, /// Enabled now
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739994272800,
"_last_modified_time": 1739994272800,
"_create_user": "admin",
"_last_modified_user": "admin",
"_revision": 0
}
=======================================================
Step 6: Check status of this plugin on Edge node
=======================================================
We will check status of this plugin on Edge Transport Node: d89753aa-ddf4-11ef-a1bb-005056ac5faf. It should be Enabled now.
GET https://NSX-ip/policy/api/v1/infra/sha/plugin-status/d89753aa-ddf4-11ef-a1bb-005056ac5faf
{
"plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"plugin_name": "edge_fw_conn_monitor",
"status": "NORMAL",
"profile": "NAME: profile1, ENABLE: True, CHECK_INTERVAL: 60, DESIRED_CRASH: False", // // As you can see plugin is enabled on this node. "detail": "",
"node_path": "/infra/sites/default/enforcement-points/default/edge-transport-node/d89753aa-ddf4-11ef-a1bb-005056ac5faf"
},
=======================================================
Step 7: (Optional but Recommended) Disable the plugin back.
=======================================================
User can/should disable the plugin which was enabled by following the above steps. Plugin is disabled originally to manage the scale on SSP Metrics or metrics it provides are useful for extended debugging/troubleshooting. So after the use of these default disabled plugins is done, we highly recommend disabling it
PATCH https://<NSX-IP>/policy/api/v1/infra/sha/plugin-profiles/profile1
Request:
Error rendering macro 'code': Invalid value specified for parameter 'firstline'
{
"config": {
"check_interval": 60
},
"resource_type": "PredefinedPlugin",
"id": "profile1",
"display_name": "profile1",
"path": "/infra/sha/plugin-profiles/profile1",
"relative_path": "profile1",
"parent_path": "/infra",
"remote_path": "/orgs/default/projects/default/infra/sha/plugin-profiles/profile1--nbendre-nsx",
"unique_id": "5e294a1f-3d67-4b74-8665-396efb340769",
"realization_id": "5e294a1f-3d67-4b74-8665-396efb340769",
"owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
"marked_for_delete": false,
"overridden": false,
"plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
"applied_to_group_path": "/infra/domains/default/groups/ALL_EDGE_TNS_GROUP",
"enabled": false, /// disabled again.
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1739994272800,
"_last_modified_time": 1739994272800,
"_create_user": "admin",
"_last_modified_user": "admin",
"_revision": 0
}
Feedback
