App Control: Agent Upgrades Pushed by Console Are Stalled

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

• Agent Policy Status shows "Approvals out of Date"
• Computer Details shows CL version behind the Current server CL version
• Agent upgrades have stalled in an "Upgrade Waiting" or "Upgrade Scheduled" status

Environment

• App Control Server: All Supported Versions
• App Control Agent: All Supported Versions

Cause

There could be multiple causes why Agent upgrades could stall, but primarily it's related to low system resources available on the application server.

Resolution

1. Limit the number of Agents being upgraded simultaneously.
2. Reduce, or eliminate, the use of Prioritize Upgrades
3. Increase the Private Memory in IIS.
4. Remove the current Agent Upgrade or Prioritization requests:
   • From the Console: (Per Agent)
     1. Assets > Computers > relevant Computer stuck in Waiting or Scheduled > View Details (pencil icon) > right-hand menu
     2. Click Remove Agent Upgrade Request (if present).
     3. Click Remove Prioritization of Updates (if present).
   • From SQL: (All Agents)
     1. Log in to the application server as the Carbon Black Service Account.
     2. Stop the Carbon Black App Control Server and Reporter services.
     3. Launch SQL Server Management Studio and execute the following script:

        use das  
        set nocount on    
        -- --------------------------------------------------------------------------------------------  
        -- Description:   
        -- 1. Remove prioritization of upgrades; and expire selected hosts session  
        -- --------------------------------------------------------------------------------------------  
        -- Step 1: declare variables  
        	declare @cnt int;  
        	declare @max int;  
        	declare @tbl table (id int identity, host_id int)    
        			
        -- Step 2: get hosts to remove prioritization of upgrades
        	INSERT INTO @tbl (host_id)  
        	SELECT host_id FROM dbo.hostmain h WITH (NOLOCK)
        	WHERE deleted = 0 AND upgrade_state < 0 -- forced upgrade requested through the menu (0x80000000)
        
        	SELECT @max = max(id), @cnt= 1 from @tbl ;    
        			
        -- Step 3: loop; each host (and expiring the host)  
        	while (@cnt <= @max)   
        		begin    
        			declare @hostID int;    
        			select @hostID = host_id from @tbl  where id = @cnt;        
        			
        			print convert(varchar(3), @cnt) + ' host id: ' + convert(varchar(30), @hostID); 
        			
        			-- Change Upgrade State
        			UPDATE dbo.hostmain 
        			SET upgrade_state=16 --Not requested
        			WHERE host_id = @hostID
        						
        			-- Expire host    
        			EXEC dbo.ExpireHostSession @hostID;     
        			
        			-- Update counter    
        			set @cnt = @cnt + 1;       
        		end

     4. Start the Server & Reporter services.

Additional Information

• In some instances simply restarting the App Control services will allow some Agents to begin upgrading, however this may only be a temporary workaround.
• On Server 8.10.2 and Higher make sure that the AppCDownloads pool is running per this article