How does the agent's Bandwidth Throttling and Speed Testing work?

Products

• IT Management Suite

Issue/Introduction

How does the agent's Bandwidth Throttling and speed testing work?

Environment

ITMS 7.x, 8.x

Resolution

Bandwidth throttling and speed tests, how does it work?

Bandwidth Throttling configuration:

Bandwidth throttling is configured in the SMP Console under 'Settings > Agents/Plugins > Symantec Management Agent > Settings > Targeted Agent Settings'. Each of the targeted Agents Settings, under “Downloads” tab, there is a section for Bandwidth/Throttling.
Note: New UI changes were introduced with SMP 8.5 Release.

Bandwidth throttling options:

1. Never throttle.
2. Enable throttling when connection speed is below <dropdown box>
   
   1. Throttle regardless of connection speed.
3. Use the server’s time for throttling settings (none configured by default).

Once bandwidth throttling has been configured, and the throttling configurations have been passed to the Altiris Agent machines, the Altiris Agent uses ICMP (ping) packets to perform the network connection speed tests. This is detailed below in the section Speed Testing Parameters. The returned speed test value is cached for a period of 6 hours.

 

Note: SMP 7.5 up to 8.1 also uses HTTP ping to test the connection. See the section Speed test changes in 7.5 and later below for more details.

When there is a scheduled package download task, the Altiris Agent either retrieves the cached connection speed value or it will initiate a new network speed test, and the results will be cached for 6 hours. Once the 6-hour cache expires, the Altiris Agent will request another speed test profile once an advertisement execution or package download begins (see Speed Testing Triggers and Speed Testing Operations below).

Throttling Rules:

When the Altiris Agent is asked to throttle, there are two throttling rules that can be configured:

1. Relative throttling—This is a user defined percentage of available bandwidth, and it is measure in 0–100 percent.
2. Absolute throttling—This is a user defined maximum allowed throughput when throttling is enabled; it is measured in bytes per second.

How the throttling process works:

The Altiris Agent:

1. Sets wait times and buffer sizes that most closely meet the target download speed of the configured relative or absolute throttle rule before starting the package download.
2. Requests a segment of data from the source.
3. Receives the data and then waits a specified period of time.
4. Monitors the download and adjusts to meet the configured throughput.
5. Requires no additional speed testing.

Speed Testing Triggers:

When the Altiris Agent is instructed to perform a speed test by its configuration policy there are four triggers that can initiate the speed test:

1. Each Software Delivery task can have a 'Download and run…' option (Advanced tab and 'Download and Execute Options') which determines the location of the file for execution, whether from a server or locally, depending on the available speed.
2. In the Altiris Agent configuration, there is an option Default minimum connection speed to run SWD Packages to test minimum throughput before execution.
3. In the Altiris Agent configuration, there is the same Download and run… option as in the Software Delivery Task, but when applied via the Altiris Agent configuration, it is a global setting which then applies to all tasks.
4. In the Altiris Agent configuration, there is download throttling when the download occurs. This method has two modes of operation: Relative throttling which is a percent of available bandwidth, and Absolute throttling which is simply a value for the throughput allowed.

Speed test operations:

Speed Tests are Directed Against Servers in the Following Manner:

1. Items 1, 3, and 4 above in Speed Testing Triggers (only with Relative throttling) are tests against the download location.
2. Item 2 above in Speed Testing Triggers is against the Notification Server, or any Item where the Altiris Agent cannot determine another server to test against will be evaluated against the Notification Server.
3. Item 4 (with Absolute throttling) does not need a speed test.

Note: Speed testing does not mean packets go on the wire every time one of these decision points is reached. Review the speed test details below under Speed Testing Parameters. Whether it is for a package download or an advertisement execution, the speed testing process is the same.

Speed testing begins with the FQDN name for the server being targeted based on the codebase and the speed is discovered and set; the NetBIOS name is not tested but it set to the same speed. If the FQDN test fails then the NetBIOS name is tested, and the speed is discovered and set.

Bandwidth Throttling when ICMP is turned off:

There are several throttle settings and parameters that need to be understood when ICMP traffic is disabled on the network. Activation of throttling by the Altiris Agent is based on the following options:

1. Throttle regardless of connection speed
2. Enable throttling when connection speed is below <dropdown box>.

If ICMP traffic is disabled on the network, the option Throttle regardless of connection speed should be selected. If the option Enable throttling when connection speed is below… is selected when ICMP traffic has been disabled, by default throttling is set to 1 KB/sec because the connection speed cannot be verified.

So, threshold settings that are 1 KB/sec or above are always throttled, and the setting of 500 bytes/sec is never throttled.

The configurable values for the throttling threshold are:

• 500 bytes/sec
• 1 KB/sec
• 2 KB/sec
• 4 KB/sec
• 10 KB/sec
• 20 KB/sec
• 50 KB/sec (default)
• 100 KB/sec
• 256 KB/sec
• 512 KB/sec
• 1 MB/sec
   

Once the throttling options are selected then the throttling limits come into effect. Again, these limits are:

1. Relative throttling — This is a user-defined percentage of available bandwidth, and it is measured in 0–100 percent.
2. Absolute throttling — This is a user-defined maximum allowed throughput when throttling is enabled; it is measured in bytes per second.

When ICMP traffic is disabled on the network there is no need to initiate speed testing and Absolute throttling is the better choice. The bandwidth limit is already known and the Altiris Agent will throttle to that limit. If Relative throttling is selected, it will still act as an absolute limit as defaults to the option of 1 KB/sec.

Since network throughput is so critical, different Altiris Agent collections should be created based on network throughput. Altiris Agent machines can then be customized based on customized connectivity configurations.

Speed Testing Parameters:

Speed testing is a result of the Altiris Agent pinging the FQDN name for a server being targeted based on the defined codebase, and the speed is discovered and cached; the NetBIOS name is not tested but it set to the same speed. If the FQDN test fails then the NetBIOS name is tested, and the speed is discovered and set.

If the codebase request is to a server that has not been profiled for a connection speed before then a speed test is initiated.

Packets used: Five 1-byte packets and then thirty 400-byte packets. (Note: Older operating systems will only use five 400-byte packets.)

Registry keys:

HKLM\SOFTWARE\Altiris\Communications\MaxServersToCheck (default 6, valid range is 1–100)

Description: Maximum number of servers to check
Values: If over 100 or under 0 then is set to 100; a value of 0 is set to 6 

HKLM\SOFTWARE\Altiris\Communications\ IP Expiry (mins) (default 360, valid range is 1–10,080)

Description: Number of minutes before connectivity to a Host is retested
Values: If over 10,080 or under 0 then it is set to10080; a value of 0 is set to 1.

HKLM\SOFTWARE\Altiris\Communications\SPEED Expiry(mins) (default 360) 

Speed test changes in SMP 7.5 through 8.1

SMP 7.5 through 8.1:

SMP version 7.5 introduces HTTP Ping to measure network speed between client and server.  NOTE: This ping test was deprecated in 8.1.

To calculate network speed client downloads 30KB page from server:

* from SMP: .../Altiris/NS/Agent/ConnectionTest.asp

* from Package Server: .../Altiris/PS/ConnectionTest.html

In case if SMP/Package Server has packages available via UNC, connection is tested using ICMP ping (detailed in the section Speed Testing Parameters).

Connection check information is also available in Symantec Management Agent logs as trace level events.

The default server (NS, PS or TS) ping interval is stored in registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Communications REG_DWORD, "Server Ping Timeout", 1800 seconds by default

This is 30 minutes, so each server will be pinged at least once each 30 minutes. Each ping is a single HEAD request.

Speed test frequency is specified by another registry entry:
REG_DWORD, "Speed Expiry (mins)", 360 minutes by default

This is 6 hours, so once in 6 hours a single ping is converted into "speed test", which is a GET request that downloads 30KB of data from the server to get rough idea is it better or worse than other servers.

Now there could be more pings when servers become available or not available. For example the server is not available and the last ping failed.
Despite that some plugin may still want to call the server via HTTP, if server call succeeds then the agent will ping the server to check that it is all right. Other case is when the server is available, someone making a web call to that server and the call fails, agent will ping the server to see if it is all right.

 

More details about network adapters, connections and server profiles are available via Symantec Management Agent Network Diagnostics.

Note: Starting SMA 8.1 RU7, Bandwidth Management has been changed. The legacy approach of throttling each connection separately is abandoned. Now all connections share a single limit of bandwidth per second.

The following is provided as reference, but those changes can’t be accessed via UI in order to modify them.

• Introduced new global memory "NAMED_SM_BANDWIDTH_INFO" which contains the bandwidth throttling description structure "AEX_BANDWIDTH_INFO".
• The new single instance NetworkPerformance class is introduced which is working with global memory and provide methods for updating it and synchronization (like BandwidthManager did before).
• HTTP sockets and UNC transfer are now used NetworkPerformance class to update the global bandwidth throttling structure.
• The Network Performance class also updates performance counters which could be monitored by Windows Performance Monitor.
• SocketBandwidthManager class is removed.

 

SMP 8.5 and later:

Starting with SMP 8.5 RU1, the Blockout and Bandwidth Throttling functionality has been improved.

Weekdays and new throttling type addition (range) to the blockout and throttling period policy cause the following changes in functionality:

1. The new throttling type "range" can be selected for a throttling period so there are 3 throttling types available: %, kb and range.
   In Targeted Agent Settings, you can configure a new throttling type Range. This throttling type regulates Symantec Management Agent traffic by keeping the agent traffinc within the specified range.
2. The throttling trigger for legacy throttling types is now applicable to each individual throttling period not the whole set of periods.
3. Both blockout and throttling period's start time can be bound to a weekday. The period can still finish the next day but cannot be longer than 24 hours.
4. The periods in the policy cannot intersect with each other anymore.
5. The policy XML format sent to the older agents remains the same.
6. The policy XML format sent to 8.5 RU1 and newer agents is different. In the event log. Every "Operation "Post"..." and similar events now contain the throttling settings used during the operation.
7. In the performance counters, "Symantec Management Agent Network Adapter Usage" can be summoned from the diagnostics UI menu.
8. "Blockouts" diagnostics UI page is now named "Blockouts and Throttling" and it shows the throttling and blockout periods configured in the policy and the currently active period.
9. The legacy throttling types "%" and "kb" are served by the old throttling engine, the "range" type is served by the new engine. The switch between throttling types is possible in the middle of the transfer.
10. "range" throttling type is available for HTTP,[s] UNC and persistent connections.
11. "%" and "kb" throttling types are available for HTTP,[s] and UNC connections only.
12. "Agent Settings" UI page on the actual client machine can display the new throttling type information when it is active

 

NOTE: The following information is provided as reference, but it is not intended to be a User Guide in how to use that functionality:

New throttling engine and algorithm

Network Adapter Monitor

The network adapter monitor is part of aexnetmon.dll and is started along with SMA service start. The purpose of the adapter monitor is gathering statistics of every network adapter in the system.

The network adapters are enumerated when the monitor starts and when machine's routing table changes. The monitor queries the adapter's statistics once a second and calculates the number of bytes that got through the adapter during the last second as well as other adapter usage counters. The statistics are queried for each adapter in the system even if SMA does not have a particular adapter currently.

The administrator has access to the information adapter monitor collects via performance counters set named "Symantec Management Agent Network Adapter". There is a menu item "Network Adapter Usage Performance Counters" in the diagnostics menu available that opens the Windows Performance Monitor window.

The network adapter monitor gathers and calculates the following statistics:

• "Bandwidth: Adapter". The maximal network adapter bandwidth in bytes per second. These include transmuting and receiving bytes.
• "Bytes/sec: Current: Adapter". The number of bytes sent and received by the adapter. This counter correlates with other system counters like "Network Interface\Bytes Total/sec" or "Network Adapter\Bytes Total/sec". There can be a slight difference between them because the system performance counters provider that calculates system counters can do that at slightly different time frames.
• "Bytes/sec: Maximal: Adapter". The maximal number of bytes send and received by the adapter per second.

All other counters shown above belong to the throttling engine.

 

New Throttling Algorithm

The main difference between the legacy throttling engine and the new engine is that the new one can regulate SMA traffic relative to the 3rd party application traffic. Using adapter monitor statistics, the new engine can estimate the 3rd party application traffic and use that in the throttling algorithms.
The currently implemented algorithm allows SMA to hold the total adapter traffic within the certain range set by the maximal and the minimal bandwidth limits depending on the 3rd party traffic.

• If 3rd party traffic does not exceed the maximal limit, then SMA traffic consumes the rest of the bandwidth up the upper limit.
• If 3rd party traffic increases up or above the upper limit, then SMA traffic drops down but not lower than the lower limit.

The reason why the lower limit exists is that SMA cannot stop sending bytes over a connection in order to connection to continue to be alive and not to be terminated by intermediate devices like routers and gateways. The operation Principe of the new throttling engine is described in the next paragraph.

New Throttling Engine

The new throttling engine consists of two parts: the engine core and the consumers.
The engine core is triggered every second by the network adapter monitor after it got the current statistics for every network adapter. The job of the core is to calculate how many bytes can every consumer "consume" the very next second. Depending on the adapter statistics gathered by the adapter monitor and the throttling algorithm Bytes consuming occurs when SMA transport that is integrated with the consumer needs to send or receive a number of bytes from the server, the transport cannot send or receive more bytes than allowed by the core.

Bandwidth Channel

There is the single-engine core and the multiple consumers exist on the machine. The consumer is created and used by the transport modules every time when a new network connection is established. The consumer connects to the engine core and allocates a bandwidth channel from the engine. There are a limited number of channels supported by the engine (256 currently), which limits how many connections can be throttled. The performance counters related to the channel statistics are:

• "Bandwidth Channels: Maximum". The maximal number of channels.
• "Bandwidth Channels: Allocated". The number of allocated channels or the number of consumers connected or the number of connections.
• "Bandwidth Channels: Active". The number of consumers that actually sent or received bytes during the previous second. The consumer becomes inactive when it does not send or receive any data. That can change every second.
   

Multiple Local IP Addresses Support

The channel is allocated based on the connection's local IP address. Every adapter can have multiple local IP addresses assigned, the adapter monitor collects that information. When consumer needs to allocate a channel, it provides the IP address to the engine core that selects the adapter assigned to that IP address. The throttling is working per adapter, i.e. there are 256 channels per each network adapter, SMA can establish multiple connections to the servers belonging to the different network subsets and the throttling will be performed for each connection separately according to the adapter the connection uses.

Agent Bandwidth

Every second the engine core calculates how many bytes can be consumed by the consumers, this number is represented by the counter "Bandwidth: Agent". It cannot be larger than the adapter's bandwidth. Currently, the engine divides the total agent bandwidth equally between all the active channels.
When the throttling is not applied the engine core continues querying adapters and calculating the agent and channel bandwidth and the consumers still continue to operate within the bandwidth.

 

Throttling Settings

The performance counters related to the throttling settings are:

"Throttling Settings: Threshold". The legacy engine setting in bytes per second that show when throttling starts, the connection speed should drop below that value for the throttling to start."Throttling Settings: Value %". The legacy engine setting in bytes per second that shows if relative throttling mode is on. The throttled connection speed is regulated relatively to the actual connection speed."Throttling Settings: Value Bytes/sec". The legacy engine setting in bytes per second that shows if absolute throttling mode is on. The throttled connection speed does not exceed the set value.

IMPORTANT LIMITATION: The new throttling engine is not used if the legacy throttling settings are configured. The administrator should turn the new settings on for the new engine to start throttling.

 

Throttling Operation Algorithm

All the algorithm internals are visible through the performance counters. The input engine receives consist of adapter's current traffic "Bytes/sec: Current: Adapter", adapter bandwidth "Bandwidth: Adapter", the channels information and the throttling settings. The rest of the performance counters calculated as below:

1. "Bytes/sec: Current: Agent". Every consumer reports how many bytes it used every second. The sum of all the bytes reported by all the consumers every second result in current SMA traffic measured in bytes per second.
2. "Bytes/sec: Average: Adapter". The momentary traffic through the adapter "Bytes/sec: Current Adapter" cannot be used by the algorithm directly since it can contain sharp peaks and downs caused by the network drivers and network modules caching. They can accumulate bytes sent by the agent but do not send anything through the adapter for a while and then send the whole chunk of bytes when the number of accumulated bytes exceeds a certain internal threshold, like MTU size or SMB buffer size. Throttling engine uses the average number of bytes sent and received through an adapter as a base for the algorithm. The bytes are averaged over 4 seconds interval currently.
3. "Bytes/sec: Current: Excess". This is the difference between the average adapter traffic and SMA traffic. This value estimates the 3rd party application traffic. Note that this value can become negative when there is no real 3rd party application traffic. This is normal and caused by the internal system caching again - not every byte sent through Windows socket by SMA every second can get through network adapter the same second, part of the sent bytes can get through the adapter the next second. The throttling engine estimates the agent bandwidth "Bandwidth: Agent" using excess by deducting it from the upper limit specified by the throttling settings.
4. "Bytes/sec: Maximal: Agent". The maximal recorded value of "Bytes/sec: Current: Agent", used for troubleshooting only.
5. "Bytes/sec: Maximal: Excess". The maximal recorded value of "Bytes/sec: Current: Excess", used for troubleshooting only.
6. "Bandwidth: Slow-down factor". Throttling engine decreases agent bandwidth by 2 percent every second in case excess of bytes consistently negative. This allows the adapter to pick up with the rate SMA sends bytes to the adapter.

 

Throttling Engine Consumers

The following connections are integrated with the new throttling engine currently.

• SMA HTTP/HTTPS transport. Every HTTP request made through standard SMA is subject to throttling. These include policy refresh, NSE sending, agent registration, package delivery, plugin requests, etc.
• SMA UNC transport. Every UNC package delivery is subject to throttling.
• Package proxy connections. Office365 updater traffic is subject to throttling.
• SSL proxy connections. Every connection made through a proxy is subject to throttling, these include:
  • Persistent connection. The whole traffic sent and received over a web-socket connection
  • Software Portal connections. All the connections made by the browsers.

A consumer can be used from either SMA service of external process running in the context of any account. That means connections made by 3rd party applications running in the context of a regular user but using SMA transport module will still be throttled.

IMPORTANT LIMITATION: SMA service should be running for throttling to operate. DS's PECTAgent should still load aexnetmon.dll and network monitor or throttling to work for PECTAgent. The network monitor starts network adapter monitor and the throttling engine internally.

 

Teamed, VPN, Wi-Fi, 4G support.

All types of network adapters are supported by the adapter monitor and the throttling engine. No filtering by adapter type can be applied currently.

 

Troubleshooting

• Warning events with "BandwidthControl" source show the operational warnings, which usually will indicate not working engine or throttling.
• Debug events with "BandwidthControl" source show the process of allocating and de-allocating of the bandwidth channels. This happens on every connection establishment.
• Events with "NetworkMonitor" source at all levels will show network adapter monitor operation events.
• Use performance counters to look if adapter monitoring and throttling engine operate normally.
• Bandwidth
  • "Bandwidth Channel: Allocated" should not be zero if there are some connections established to any server.
  • "Bandwidth Channels: Active" should not be zero when some data is being transferred over a connection.
  • "Bytes/sec: Current: Agent" should not be zero when some data is being transferred over a connection.
  • Adapter related counters should always show values when some data is being transferred in the system regardless of the throttling settings.
  • "Throttling Settings" counters should update immediately when new policy arrives with the changed settings.

 

How to enable Symantec Management Agent diagnostics

To enable SMA diagnostics use following command line:

C:\Program Files\Altiris\Altiris Agent\AexAgentUtil.exe /diags

Diagnostics window is available via right click on SMA system tray icon. Right click - Diagnostics - ...

1.  

Additional Information

See 150649 "Client bandwidth throttling speed test improvement"

Other items to consider between 7.5 and 8.1 releases:

1. Throttling does not work for UNC connections
2. HTTP and UNC use different speed tests methods. HTTP uses special 30KB page to measure the download speed. UNC uses ICMP ping to measure the speed.
3. Beginning with 8.1 version, UNC also uses the same 30KB page to measure the download speed.
4. The measured speed is not the actual speed, it's just some approximate number that can be used to compare a  few HTTP servers and a few UNC servers and select the fastest among them. UNC connections have priority while downloading, so UNC connections are usually tried first during package downloads.