Service Composer fails to push Firewall rules in VMware NSX for vSphere 6.x
search cancel

Service Composer fails to push Firewall rules in VMware NSX for vSphere 6.x

book

Article ID: 307688

calendar_today

Updated On:

Products

VMware vDefend Firewall

Issue/Introduction

This article provides information on understanding and troubleshooting Service Composer in VMware NSX for vSphere 6.x.

Service Composer Overview

Service Composer is a tool/utility that simplifies and streamlines provisioning of services to virtual machines within a software defined data center. Service Composer helps you provision and assign network and security services to applications in a virtual infrastructure. You map these services to a security group, and the services are applied to the virtual machines in the security group.

Security Group

Security Groups provide administrators a mechanism to dynamically associate and group workloads. This abstraction allows a membership definition based upon one of many vCenter constructs. An administrator has the ability to create numerous type of memberships.
You begin by creating a security group to define assets that you want to protect. Security groups may be static (including specific virtual machines) or dynamic where membership may be defined in one or more of the following ways:
  • vCenter containers (clusters, port groups, or data centers).
  • Security tags, IPset, MACset, or even other security groups. For example, you may include a criteria to add all members tagged with the specified security tag (such as AntiVirus.virusFound) to the security group.
  • Directory Groups (if NSX Manager is registered with Active Directory)
  • Regular expressions such as virtual machines with name VM1

Note: Security group membership changes constantly. For example, a virtual machine tagged with the AntiVirus.virusFound tag is moved into the Quarantine security group. When the virus is cleaned and this tag is removed from the virtual machine, it again moves out of the Quarantine security group.

For more information, see the VMware NSX Data Center for vSphere Administration Guide.

Service Composer High Level Architecture

Figure 1.0




Symptoms:
Service Composer fails to push Firewall rules in VMware NSX for vSphere 6.x

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.1.x

Cause

This issue occurs due to the following reasons:
  • Service Composer is out of sync
  • Background thread of Service Composer is still processing
  • Service Composer has yet to received a callback for a Policy change
  • There is an invalid rule on the Distributed Firewall's default section

Resolution

To resolve this issue:
  1. For more information regarding Service Composer Out of Sync issues, see Understanding and troubleshooting Service Composer in VMware NSX for vSphere 6.x (339115).

    Note: At this point onwards, no changes will be pushed until a force sync is implemented.

  2. Verify by taking a thread dump. Thread dumps are present as part of the NSX Tech Support logs.

    For more information, see Collecting diagnostic information for VMware NSX for vSphere 6.x (318901).

    After the thread dump is collected, look for a runnable thread with FirewallConfigurationSyncService in stake-traces with entries similar to:

    INFO | jvm 2 | 2015/05/27 07:43:41 | "http-nio-127.0.0.1-7441-exec-1" daemon prio=10 tid=0x000000000190a800 nid=0x17a4 runnable [0x00007f7e11ec8000]
    INFO | jvm 2 | 2015/05/27 07:43:41 | at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:58)
    INFO | jvm 2 | 2015/05/27 07:43:41 | at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1175)
    INFO | jvm 2 | 2015/05/27 07:43:41 | at com.vmware.vshield.vsm.policy.service.impl.PolicyQueryServiceImpl$AjcClosure33.run(PolicyQueryServiceImpl.java:1)
    INFO | jvm 2 | 2015/05/27 07:43:41 | at com.vmware.vshield.blueprint.firewall.provider.FirewallActionTranslationService.getLayer3FirewallRuleDtos(FirewallActionTranslationService.java:94)
    com.vmware.vshield.blueprint.firewall.provider.FirewallConfigurationSyncService.forceSync(FirewallConfigurationSyncService.java:787)
    INFO | jvm 2 | 2015/05/27 07:43:41 | - locked <0x00000006938fcab8> (a com.vmware.vshield.blueprint.firewall.provider.FirewallConfigurationSyncService)

    Where:

    • FirewallConfigurationSyncService.forceSync() indicates that the Service Composer’s force sync operation is in progress.

    • FirewallActionTranslationService.getLayer3FirewallRuleDtos() indicates that the translation of Policy to Firewall rules is in progress.

    • DefaultAutoFlushEvenListener.onAutoFlush() indicates that an hibernate auto flush is currently in progress. This is known to consume long time on setups with large numbers of Policies.
    • Further, look for blocked thread with FirewallConfigurationSyncService in stake-traces as seen similar to:

      INFO | jvm 1 | 2015/05/25 17:39:18 | java.lang.Thread.State: BLOCKED (on object monitor)
      INFO | jvm 1 | 2015/05/25 17:39:18 | at com.vmware.vshield.blueprint.firewall.provider.FirewallConfigurationSyncService.forceSync(FirewallConfigurationSyncService.java:787)
      INFO | jvm 1 | 2015/05/25 17:39:18 | - waiting to lock <0x0000000693877b68> (a com.vmware.vshield.blueprint.firewall.provider.FirewallConfigurationSyncService)


    Notes    :
    • This is a force sync thread which is waiting to acquire lock on FirewallConfigurationSyncService.
    • This indicates that some other thread already has a lock on FirewallConfigurationSyncService and this operation will have to wait until the previous operation finishes.
    • There may be multiple such threads in the waiting state.

  3. Verify if Service Composer has not received a callback. For better understanding, refer to the Architecture diagram in Figure 1.0. Here, we are referring to Thread 3 in the diagram. This thread may not get scheduled due to:

    • There was some failure in DCN thread (thread 2), due to which callback was never made.
    • All the threads in the task framework’s thread pool are busy. Service Composer will get a callback as soon as one of the thread gets available in the pool.

  4. Verify if there are Invalid Firewall rules on the Distributed Firewall default section. To do this:

    1. Get the complete Firewall configuration using REST API call:

      GET https://NSX_Manager_IP/api/4.0/firewall/globalroot-0/config

      Note: Take a note of the Section IDs which contains the invalid entries. (Service Composer with default section IDs)

    2. Get the Firewall configuration for the section IDs which has invalid entries.

      For example:

      If section ID for default is 1003:

      GET https://NSX_Manager_IP/api/4.0/firewall/globalroot-0/config/layer3sections/1003

    3. Correct the invalid entry and post the corrected section as:

      PUT https://NSX_Manager_IP/api/4.0/firewall/config/layer3sections/1003

      For example: Suppose in the default section the source VM is invalid, remove <sources> tag during re-posting of the configuration.

      <source>
      <name>gavjdw00286</name>
      <value>vm-##</value>
      <type>VirtualMachine</type>
      <isValid>true</isValid>

    4. Get the Firewall configuration for service ID belonging to Service Composer. Remove the invalid entry while re-posting the configuration for the respective section. (Refer to the PUT call from the above steps)

      For example: Suppose vNic is invalid in AppliedTo column, remove <appliedto> for invalid vNic during re-posting of the configuration:

      <appliedTo>
      <name>gavypa002891 - Network adapter 1</name>
      <value>5039218c-####-####-####7b52173c3f3e.000</value>
      <type>Vnic</type>
      <isValid>true</isValid>
      </appliedTo>

    5. Go to Service Composer page and click Synchronize Firewall Config to make service composer in Sync. For more information, see the Working with Firewall Configurations section in the VMware NSX Data Center for vSphere Upgrade Guide.


Additional Information

Powered by Wolken Software