How to suppress/remove HTTP Response Header from an Ingress virtual server supplied with a TKGI cluster with NSX-T

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Scenario

Based on a SecOps requirement, you are required to enhance the security of your application by making sure that information about internal environments must not be exposed to external customers or even within internal teams.
 
For example, we know that Kubernetes Ingress helps us expose our application(s) to the outside world or it's a way to bring traffic to the application(s) hosted within the internal infrastructure. With an NSX-T based TKGI cluster, we get two virtual servers out of the box for the purposes of Ingress (http or https_terminated).

After administering a TKGI with an NSX-T based cluster, the application team started making use of Ingress for their application and one of the tests they run against the application, as well as the Kubernetes cluster, is to monitor the HTTP response headers when a client sends a request to the application. The tests reveal that when the client requested an API endpoint that does not exist on the Ingress resource or the Ingress resource does not have the rule to route the traffic correctly, then the following HTTP response headers are seen in the request:



As we can see from the previous screenshot, the browser (client) tried to access the application, specifically the /wear path (which we know that the application does not implement and the default backend also does not handle). If we take a closer look at the response headers, we see a field called Server that has a value of NSX LB. This might not be desirable from a security standpoint as internal details about the Load balancer are getting exposed to the end-user. The instructions section contains details on how to fix this.

Environment

Product Version: 1.7
OS: Linux

Resolution

Note: Regardless of the type of virtual server supplied by default for a TKGI cluster with NSX-T i.e. http or https_terminated, the instructions below will be sufficient to address the security concern explained previously. We have tested this with NSX-T v2.5.1.0 and v2.5.2.2.

As an overview, the solution from the following instructions lists out how to add an LB rule to a virtual server by allowing the admin user to overwrite the configurations of a virtual server. The overwrite piece is significant because, from the NSX-T Manager UI, it is not possible to add an LB rule as the UI operations do not pass the "X-Allow-Overwrite: true"  header that is required to be able to perform an overwrite operation.

The LB rule will modify the HTTP response headers so that if the key name matches "Server" and has a value of "NSX LB", then the LB will remove the header before sending out the response

Procedure

• Get the virtual server details (HTTP or HTTPS Terminated Layer 7). Make a copy of the output of the following command:

curl -v -k -u 'admin:<password>' -H "Content-Type:application/json" -X GET "https://<nsx-manager-vip-or-url>/api/v1/loadbalancer/virtual-servers/<id-of-virtual-server>"

Where

<id-of-virtual-server> - is the ID of the virtual server (Layer 7 HTTP or HTTPS Terminated).

• Create the JSON object with a rule to remove the HTTP response header (filename - update_virtual_server_with_lb_rule.json).
  • Use the output obtained from the previous command to gather the old configurations - this is imperative because we do not want to change the old configurations as that might create unseen complications
  • The following is an example of how the final JSON file will look before running the next command (some details will differ based on your environment).
  • Note: The rules array is something that we will need to add to make sure that the virtual server has the rule to remove the HTTP Response Header named Server.

{
   "virtual_server":{
      "enabled":true,
      "access_log_enabled":false,
      "ip_address":"10.213.66.211",
      "port":"80",
      "ports":[
         "80"
      ],
      "ip_protocol":"TCP",
      "application_profile_id":"4d1ff5e7-e16e-4d9b-874f-08ac26f7bae7",
      "rule_ids":[
         "f6ccd8eb-ac7c-436d-833a-078eeabd84b8"
      ],
      "resource_type":"LbVirtualServer",
      "id":"bbc5895c-137e-4256-b70e-21ad240c0681",
      "display_name":"pks-e6ca470a-c159-4287-a080-b22fccf7550e-http",
      "tags":[
         {
            "scope":"ncp/version",
            "tag":"1.2.0"
         },
         {
            "scope":"ncp/cluster",
            "tag":"pks-e6ca470a-c159-4287-a080-b22fccf7550e"
         },
         {
            "scope":"ncp/lb_vs_type",
            "tag":"http"
         },
         {
            "scope":"external_id",
            "tag":"f34320e6-12e9-5b7e-b45f-712c04f45e4d"
         },
         {
            "scope":"ext_pool_id",
            "tag":"43980caa-e48b-4020-af64-bce402d96dd7"
         }
      ],
      "_create_user":"pks-e6ca470a-c159-4287-a080-b22fccf7550e",
      "_create_time":1617732165309,
      "_last_modified_user":"pks-e6ca470a-c159-4287-a080-b22fccf7550e",
      "_last_modified_time":1617749072607,
      "_system_owned":false,
      "_protection":"REQUIRE_OVERRIDE",
      "_revision":1
   },
   "rules":[
      {
         "resource_type":"LbRule",
         "display_name":"HTTPResponseHeaderRemove",
         "description":"Rule to remove HTTP response header Server",
         "match_conditions":[
            {
               "type":"LbHttpResponseHeaderCondition",
               "header_name":"Server",
               "header_value":"NSX LB"
            }
         ],
         "match_strategy":"ALL",
         "phase":"HTTP_RESPONSE_REWRITE",
         "actions":[
            {
               "type":"LbHttpResponseHeaderDeleteAction",
               "header_name":"Server"
            }
         ]
      }
   ]
}

 

• Update a load balancer virtual server with rules. This is the command that utilizes the file and its contents from the previous steps to update the virtual server:

curl -v -k -u 'admin:<password>' -H "Content-Type:application/json" -H "X-Allow-Overwrite: true" -X PUT "https://<nsx-manager-vip-or-url>/api/v1/loadbalancer/virtual-servers/<virtual-server-id>?action=update_with_rules" -d @update_virtual_server_with_lb_rule.json

 

• After the commands from the last step complete, you can verify in the NSX-T manager UI that the rule is present on the virtual server:

 

• After this, you can test Ingress using the Chrome developer tools to verify that the "Server" HTTP response header is no longer shown. Here is an example: