On-Demand Enabling or Disabling of Security Troubleshooting Metrics in the Security Services Platform (SSP)
search cancel

On-Demand Enabling or Disabling of Security Troubleshooting Metrics in the Security Services Platform (SSP)

book

Article ID: 390485

calendar_today

Updated On:

Products

VMware vDefend Firewall

Issue/Introduction

Currently, some metrics collected in SSP are disabled by default. Users may wish to enable these metrics for debugging or troubleshooting purposes.

Environment

NSX >= 4.2.2

vDefend SSP >= 5.0

Cause

Several security monitoring-related metrics are disabled by default in NSX 4.2.2 or above.

If these metrics are needed, they can be enabled on-demand for troubleshooting purposes.

It is recommended to enable them only for short durations and on specific Edges or Hosts where required.

Resolution

Here we are enumerating steps to enable a plugin (NSX ships with that plugin being disable by default). Same steps work for disabling a plugin as well. 

=======================================================

Step 1: Identify the UUID/Path of the SHA plugin that you want to enable. 

=======================================================

Below table contains the list of disabled Metrics and corresponding SHA Plugins

 

Plugin ID Plugin Name Metrics

/infra/sha/pre-defined-plugins/16d2490a-e505-5f25-8259-cddefbf8040b

 idps_stats_monitor 
idps.avg_event_sent_nsxi_ndr,
idps.avg_event_send_failure_nsxi_ndr
idps.avg_event_send_failure_nsx_manager
idps.avg_event_sent_nsx_manager
edge_idps_exporter.avg_event_enqueue_failure_nsxi
edge_idps_exporter.avg_event_enqueue_failure_nsx_manager
edge_idps_exporter.avg_critical
edge_idps_exporter.avg_non_critical
ege_idps_exporter.avg_total
edge_idps_datapath.avg_event_callback
edge_idps_datapath.avg_event_sent_count
edge_idps_datapath.avg_big_event_count
edge_idps_datapath.avg_event_enqueue_count
edge_idps_datapath.avg_event_enqueue_failure_count
edge_idps_datapath.avg_event_out_of_mem_count
edge_idps_datapath.avg_socket_reconnect_count
edge_idps_datapath.avg_event_send_failure_count

/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088

edge_fw_conn_monitor

 
edge_fw_conn_sum.avg_tcp_half_open_ingress_conn

edge_fw_conn_sum.avg_tcp_max_conn

edge_fw_conn_sum.avg_udp_ingress_conn

edge_fw_conn_sum.avg_udp_max_conn

edge_fw_conn_sum.avg_icmp_ingress_conn

edge_fw_conn_sum.avg_icmp_max_conn

edge_fw_conn_sum.avg_others_ingress_conn

edge_fw_conn_sum.avg_others_max_conn

edge_fw_conn.avg_tcp_open_conn

edge_fw_conn.avg_tcp_est_conn

edge_fw_conn.avg_udp_est_conn

edge_fw_conn.avg_icmp_est_conn

edge_fw_conn.avg_others_est_conn

edge_fw_per_host.avg_tx_conn_per_core

edge_fw_per_host.avg_rx_conn_per_core

 

Note: Below Metrics keys may show up as available in NSX 4.2.2. However they are removed starting NSX 4.2.2 onward. So we can't enable them using below workflow.

 

 

Plugin ID Plugin Name Metrics
/infra/sha/pre-defined-plugins/26d87226-1673-4c3c-9b56-50d85fc57479 edge_fw_stats_monitor   
edge_fw.avg_drop_reason_3whs

edge_fw.avg_drop_reason_alg
edge_fw.avg_drop_reason_bad_offset
edge_fw.avg_drop_reason_bad_timestamp
edge_fw.avg_drop_reason_congestion
edge_fw.avg_drop_reason_connection_limit
edge_fw.avg_drop_reason_drop_by_loadbalancer
edge_fw.avg_drop_reason_failed_to_copy_pkt
edge_fw.avg_drop_reason_fragment
edge_fw.avg_drop_reason_half_open_tcp_max
edge_fw.avg_drop_reason_icmp_max
edge_fw.avg_drop_reason_inactive
edge_fw.avg_drop_reason_ip_option
edge_fw.avg_drop_reason_memory

edge_fw.avg_drop_reason_nat_conn_limit
edge_fw.avg_drop_reason_nat64_no_frgm_support

edge_fw.avg_drop_reason_normalize

edge_fw.avg_drop_reason_other_max

edge_fw.avg_drop_reason_proto_cksum

edge_fw.avg_drop_reason_queued_frag

edge_fw.avg_drop_reason_redirect_iface_null

edge_fw.avg_drop_reason_rst_sent

edge_fw.avg_drop_reason_short

edge_fw.avg_drop_reason_spoofguard

edge_fw.avg_drop_reason_src_limit

edge_fw.avg_drop_reason_state_insert

edge_fw.avg_drop_reason_state_limit

edge_fw.avg_drop_reason_state_mismatch

edge_fw.avg_drop_reason_state_reuse

edge_fw.avg_drop_reason_synproxy
edge_fw.avg_drop_reason_tcp_flags
edge_fw.avg_drop_reason_tcp_seqnum
edge_fw.avg_drop_reason_translation
edge_fw.avg_drop_reason_tun_fail
edge_fw.avg_drop_reason_udp_max
edge_fw.avg_drop_reason_update_state

 

Once you have identified plugin you want to enable/disable, you can use below API to get list of all pre-defined SHA plugins and their current status: 

GET : https://<nsx-ip>/policy/api/v1/infra/sha/pre-defined-plugins

Sample payload:

pre-defined-plugins

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108 {    "results": [        {            "enabled": false,  //// This field tells overall status of the plugin.            "config": {                "check_interval": 60            },            "supported_node_types": [                "NSX_EDGE"            ],            "pre_req_conditions": [                "WAVE_FRONT",                "TSDB"            ],            "delay_on_reboot": 300,            "resource_type": "ShaPredefinedPlugin",            "id": "63b58bc1-6c61-4414-92b5-19ef57e84088",            "display_name": "Edge firewall connection plugin",            "path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",            "relative_path": "63b58bc1-6c61-4414-92b5-19ef57e84088",            "parent_path": "/infra",            "remote_path": "",            "unique_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",            "realization_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",            "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",            "marked_for_delete": false,            "overridden": false,            "_system_owned": false,            "_protection": "NOT_PROTECTED",            "_create_time": 1739924951965,            "_last_modified_time": 1739924951965,            "_create_user": "system",            "_last_modified_user": "system",            "_revision": 0        },        {            "enabled": true,            "config": {                "check_interval": 60            },            "supported_node_types": [                "NSX_EDGE"            ],            "pre_req_conditions": [                "TSDB"            ],            "delay_on_reboot": 10,            "resource_type": "ShaPredefinedPlugin",            "id": "94d29bbd-0f85-427c-a226-3bbcc5291401",            "display_name": "Edge firewall connections per rule and per logical-router plugin",            "path": "/infra/sha/pre-defined-plugins/94d29bbd-0f85-427c-a226-3bbcc5291401",            "relative_path": "94d29bbd-0f85-427c-a226-3bbcc5291401",            "parent_path": "/infra",            "remote_path": "",            "unique_id": "a12151f1-7b2f-4c7f-a5e7-3cf5bb7ad53d",            "realization_id": "a12151f1-7b2f-4c7f-a5e7-3cf5bb7ad53d",            "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",            "marked_for_delete": false,            "overridden": false,            "_system_owned": false,            "_protection": "NOT_PROTECTED",            "_create_time": 1739924952024,            "_last_modified_time": 1739924952024,            "_create_user": "system",            "_last_modified_user": "system",            "_revision": 0        },        {            "enabled": true,            "config": {                "check_interval": 60            },            "supported_node_types": [                "NSX_EDGE"            ],            "pre_req_conditions": [                "WAVE_FRONT",                "TSDB"            ],            "delay_on_reboot": 300,            "resource_type": "ShaPredefinedPlugin",            "id": "26d87226-1673-4c3c-9b56-50d85fc57479",            "display_name": "Edge firewall stats plugin",            "path": "/infra/sha/pre-defined-plugins/26d87226-1673-4c3c-9b56-50d85fc57479",            "relative_path": "26d87226-1673-4c3c-9b56-50d85fc57479",            "parent_path": "/infra",            "remote_path": "",            "unique_id": "968af296-9eb1-4307-8ab2-786ac4ea8134",            "realization_id": "968af296-9eb1-4307-8ab2-786ac4ea8134",            "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",            "marked_for_delete": false,            "overridden": false,            "_system_owned": false,            "_protection": "NOT_PROTECTED",            "_create_time": 1739924952081,            "_last_modified_time": 1739924952081,            "_create_user": "system",            "_last_modified_user": "system",            "_revision": 0        }.......... // We have omitted other plugins for brevity.             ],    "result_count": 53,    "sort_by": "display_name",    "sort_ascending": true}

 

=======================================================

Step 2: Identify current plugin stats and confirm it's disabled/enabled.

=======================================================

As you can see above Plugin: 63b58bc1-6c61-4414-92b5-19ef57e84088 (Edge firewall connection plugin) is disabled by default.

GET: https://<NSX-IP>/policy/api/v1/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088

Edge Firewall connection plugin

{

    "enabled": false,  /// <<<<<Plugin is disabled by default

    "config": {

        "check_interval": 60

    },

    "supported_node_types": [

        "NSX_EDGE"

    ],

    "pre_req_conditions": [

        "WAVE_FRONT",

        "TSDB"

    ],

    "delay_on_reboot": 300,

    "resource_type": "ShaPredefinedPlugin",

    "id": "63b58bc1-6c61-4414-92b5-19ef57e84088",

    "display_name": "Edge firewall connection plugin",

    "path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",

    "relative_path": "63b58bc1-6c61-4414-92b5-19ef57e84088",

    "parent_path": "/infra",

    "remote_path": "",

    "unique_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",

    "realization_id": "b4529cc1-79df-4fda-85bd-cfb33fb0634a",

    "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",

    "marked_for_delete": false,

    "overridden": false,

    "_system_owned": false,

    "_protection": "NOT_PROTECTED",

    "_create_time": 1739924951965,

    "_last_modified_time": 1739924951965,

    "_create_user": "system",

    "_last_modified_user": "system",

    "_revision": 0

}

 

=======================================================

Step 3: Identify TNs where you want to enable these plugins.  

=======================================================

We can enable it on ESX or NSX_EDGE

As you can see supported_node_types for our example plugin is "NSX_EDGE", hence you can enable Edge TNs only. 

Note: For NSX 4.2.2 release, we allow the plugin to be enabled on all Edge Transport Nodes. Support to enable/disable selective Transport Nodes will come in future releases.

For enabling on ESX Transport Nodes, kindly contact Broadcom Support by opening a service request.

 

=======================================================

Step 4: Create a group with TNs

=======================================================
Based on your decision create a Group with all Edge TNs or selective Edge TNs.

Create a group with all Edge Transport Nodes. 

PATCH : https://<NSX-IP>/policy/api/v1/infra/domains/default/groups/ALL_EDGE_TNS_GROUP

Edge Group Creation request body:

{

    "expression": [

        {

            "member_type": "TransportNode",

            "key": "NodeType",

            "operator": "EQUALS",

            "value": "EdgeNode",

            "resource_type": "Condition"

        }

    ],

    "extended_expression": [],

    "reference": false,

    "resource_type": "Group",

    "display_name": "ALL_EDGE_TNS_GROUP",

    "description": "This group contains all Edge Nodes in NSX"

}

 

Once the above request is executed, you can validate in NSX UI whether the group has been successfully created by navigating through "Inventory -> Groups"

 

 

=======================================================

Step 5 : Enable SHA plugin using SHA profile.

=======================================================

5A) We will cross check status of the SHA plugin: 63b58bc1-6c61-4414-92b5-19ef57e84088 on Edge Transport Node let's consider d89bbd96-ddf4-11ef-b323-005056ac915f. This transport node ID d89bbd96-ddf4-11ef-b323-005056ac915f taken here is for illustrative purposes. This can differ in your environment.

Note: To find the UUID of the Transport Node, please refer to this documentation.

As expected plugin should be Disabled.

GET https://<NSX-IP>/policy/api/v1/infra/sha/plugin-status/d89bbd96-ddf4-11ef-b323-005056ac915f

{

            "plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",

            "plugin_name": "edge_fw_conn_monitor",

            "status": "NORMAL",

            "profile": "NAME: default-profile, ENABLE: False, CHECK_INTERVAL: 60",  /// It's disabled

            "detail": "Plugin is disabled.",

            "node_path": "/infra/sites/default/enforcement-points/default/edge-transport-node/d89753aa-ddf4-11ef-a1bb-005056ac5faf"

        },

 

5B) We will create a SHA profile to enable the SHA plugin: 63b58bc1-6c61-4414-92b5-19ef57e84088 (step-2). We will be applying this profile on a group (ALL_EDGE_TNS_GROUP) created in Step-4.


PATCH https://<NSX-IP>/policy/api/v1/infra/sha/plugin-profiles/profile1

Request Body: 

{
    "config": {
        "check_interval": 60 /// This interval tells how frequently it needs to run & collect metrics. Minimum supported value of this interval is 60 seconds.
    },
    "resource_type": "PredefinedPlugin",
    "id": "profile1",
    "display_name": "profile1",
    "plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
    "applied_to_group_path": "/infra/domains/default/groups/edge1-c2-tn_group", /// Group containing Edge TNs where we are applying this Profile
    "enabled": true /// Here we are enabling this plugin
}

Response would look like below: 

{

    "config": {

        "check_interval": 60

    },

    "resource_type": "PredefinedPlugin",

    "id": "profile1",

    "display_name": "profile1",

    "path": "/infra/sha/plugin-profiles/profile1",

    "relative_path": "profile1",

    "parent_path": "/infra",

    "remote_path": "/orgs/default/projects/default/infra/sha/plugin-profiles/profile1--nbendre-nsx",

    "unique_id": "5e294a1f-3d67-4b74-8665-396efb340769",

    "realization_id": "5e294a1f-3d67-4b74-8665-396efb340769",

    "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",

    "marked_for_delete": false,

    "overridden": false,

    "plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",

    "applied_to_group_path": "/infra/domains/default/groups/edge1-c2-tn_group",

    "enabled": true, /// Enabled now

    "_system_owned": false,

    "_protection": "NOT_PROTECTED",

    "_create_time": 1739994272800,

    "_last_modified_time": 1739994272800,

    "_create_user": "admin",

    "_last_modified_user": "admin",

    "_revision": 0

}

 

=======================================================

Step 6: Check status of this plugin on Edge node 

=======================================================

We will check status of this plugin on edge node: d89753aa-ddf4-11ef-a1bb-005056ac5faf. It should be Enabled now.

GET https://NSX-ip/policy/api/v1/infra/sha/plugin-status/d89753aa-ddf4-11ef-a1bb-005056ac5faf

 

Plugin status

{

    "plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",

    "plugin_name": "edge_fw_conn_monitor",

    "status": "NORMAL",

    "profile": "NAME: profile1, ENABLE: True, CHECK_INTERVAL: 60, DESIRED_CRASH: False", // // As you can see plugin is enabled on this node.             "detail": "",

    "node_path": "/infra/sites/default/enforcement-points/default/edge-transport-node/d89753aa-ddf4-11ef-a1bb-005056ac5faf"

},

 

=======================================================

Step 7: (Optional but Recommended) Disable the plugin back.

=======================================================

Users can disable the plugin which was enabled by following the below steps.

PATCH https://<NSX-IP>/policy/api/v1/infra/sha/plugin-profiles/profile1

Request: 

{
    "config": {
        "check_interval": 60
    },
    "resource_type": "PredefinedPlugin",
    "id": "profile1",
    "display_name": "profile1",
    "path": "/infra/sha/plugin-profiles/profile1",
    "relative_path": "profile1",
    "parent_path": "/infra",
    "remote_path": "/orgs/default/projects/default/infra/sha/plugin-profiles/profile1--nbendre-nsx",
    "unique_id": "5e294a1f-3d67-4b74-8665-396efb340769",
    "realization_id": "5e294a1f-3d67-4b74-8665-396efb340769",
    "owner_id": "af183570-14f0-4fd6-bc69-367bbb8988e0",
    "marked_for_delete": false,
    "overridden": false,
    "plugin_path": "/infra/sha/pre-defined-plugins/63b58bc1-6c61-4414-92b5-19ef57e84088",
    "applied_to_group_path": "/infra/domains/default/groups/edge1-c2-tn_group",
    "enabled": false, /// disabled again.
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_create_time": 1739994272800,
    "_last_modified_time": 1739994272800,
    "_create_user": "admin",
    "_last_modified_user": "admin",
    "_revision": 0
}

 

 

Powered by Wolken Software