Rally - On-premises: Error "Service licensing failed"
search cancel

Rally - On-premises: Error "Service licensing failed"


Article ID: 132300


Updated On:


Rally On-Premise


fter uploading the license file while provisioning a new 2.0 based appliance, the following error is shown:

1 error occurred: * Service licensing failed with error: task: non-zero exit (1)


Checking the "Cluster" screen from the dashboard may show the licensing service in a down state:


There are a few causes for this error

  • Timing issue
  • Insufficient system resources
  • Missing license expiration
  • Invalid subscription ID
  • Database communication problem



Sometimes during an initial deploy there can be a timing issue between the database and licensing containers starting up. A reboot should be attempted initially to resolve this.

If the system has been operational for a period of time but both database and services VMs were restarted in tandem prior to the failure, then the services VM should be restarted again to ensure the database services have had sufficient time to become available prior to services failing due to an unavailable database.


Insufficient System Resources

Contrary to the system sizing recommendations, we have learned that 2-4 CPUs for a VM is a bit undersized and the current recommendation is to provision the VM with 6 CPUs.

Additionally, 16 GB is considered the minimum memory and the license server may benefit from an increase in system memory as well.


Check License Expiration

This issue has been observed when there is no expiration date included with the license file.

To verify the license file:

  1. Click the gear (⚙) icon in the toolbar.
  2. Click View License.

A license with a valid expiration date will show an Expiration Date line as follows:

If your license screen is missing that line, a new license will need to be issued with a valid expiration date.


Check Subscription ID

In most on-premises environments the subscription ID will be 100.  The only reason it would be different is if that environment was migrated from SAAS to on-premises and then the subscription ID would be the value that it was when the subscription was in the cloud.


Check Database Communications

This may be an issue with communication with the database and it is recommended to visit the Database Troubleshooting Guide.


Redeploy license service

If the license server does not start after checked the above solutions, it may be necessary to redeploy the license server.  This should only be performed during a period of scheduled downtime as users will be unable to access the application for a period of time.

SSH into the service VM as the ops user

Get a list of current docker services:

$ docker service ls

Note the name of the license server as shown in the sample output

ID                  NAME                                       MODE                REPLICAS            IMAGE                                                            PORTS
5o9rxhek2bt9        premkit_replicated                         replicated          1/1                 registry.replicated.com/library/premkit:v1.3.7                   
t4khaufz9k9m        repl75d5498f_admin                         replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-admin:1.0.35-2.1.0         
mmyyedmz8sne        repl75d5498f_alm                           replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0            *:30000->7001/tcp
r4xkhs5vprp4        repl75d5498f_analytics                     replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0            *:30001->8080/tcp
zuyza8xc0he8        repl75d5498f_burro                         replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-burro:1.4839.0-2.1.0       
zckml5va1jy2        repl75d5498f_licensing                     replicated          0/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-licensing:1.1.10-2.1.0     
puawm56vkjh1        repl75d5498f_proxy                         replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-proxy:1.3.8-2.1.0          *:80->80/tcp, *:443->443/tcp
e0m69pfv1z8o        repl75d5498f_smtp                          replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-smtp:1.0.2-2.1.0           
jigem7bnue1e        repl75d5498f_solr                          replicated          1/1                <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0            *:30002->8081/tcp
mz0vmb9i54bq        replicated_replicated                      replicated          1/1                 replicated/replicated:stable-2.49.0                              *:9874->9874/tcp, *:9878->9878/tcp
wpfyi0v1xsd8        replicated_replicated-operator             global              1/1                <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.49.0   
z6qhqam7vhpz        replicated_replicated-ui                   replicated          1/1                 replicated/replicated-ui:stable-2.49.0                           *:8800->8800/tcp
ptp3wuv4ekqu        retraced_retraced-api                      replicated          1/1                 registry.replicated.com/library/retraced:1.3.39                  
x1139o14xs0g        retraced_retraced-api-bootstrapper         replicated          0/1                 registry.replicated.com/library/retraced:1.3.39                  
mybc1eau7e9c        retraced_retraced-cron                     replicated          1/1                 registry.replicated.com/library/retraced:1.3.39                  
41ieyj6y23yu        retraced_retraced-display-template-setup   replicated          0/1                 registry.replicated.com/library/retraced:1.3.39                  
l7dinqc28st1        retraced_retraced-nsqd                     replicated          1/1                 registry.replicated.com/library/retraced-nsq:1.3.39              
wl3cud8g8pfg        retraced_retraced-pg-migrate               replicated          0/1                 registry.replicated.com/library/retraced:1.3.39                  
oy2n9k39kxnq        retraced_retraced-postgres                 replicated          1/1                 registry.replicated.com/library/retraced-postgres:1.3.39         
4az7ie181qi9        retraced_retraced-processor                replicated          1/1                 registry.replicated.com/library/retraced:1.3.39                  
a0oju0hkgwu4        statsd_replicated                          replicated          1/1                 registry.replicated.com/library/statsd-graphite:1.1.7-20201026   


Use that name in the following command to delete the license service

$ docker service rm repl75d5498f_licensing

The command will simply respond with the name given.

Now shut down the application with the following command:

$ replicatedctl app stop --force

At this point, you can navigate to the dashboard and click the "Start" button in the upper left tile


Additional Information

If the expiration date is invalid, the licensing log will show the following error:

2019-05-16T09:33:44.062917226Z CRITICAL >> [05/16/2019 09:33:44 AM] (license_synchronizer.py:23) - The following critical exception has occured, we are exiting the license synchronizer 'expiration_time'

2019-05-16T09:33:44.062961814Z Traceback (most recent call last):

2019-05-16T09:33:44.062973075Z File "license_synchronizer.py", line 24, in <module>

2019-05-16T09:33:44.062982682Z raise exception

2019-05-16T09:33:44.062991465Z File "license_synchronizer.py", line 19, in <module>

2019-05-16T09:33:44.063000783Z license = ReplicatedLicenseAPI.get_license()

2019-05-16T09:33:44.063009623Z File "/_build/service/license_information_service.py", line 34, in get_license

2019-05-16T09:33:44.063018801Z license = generate_license(response.json())

2019-05-16T09:33:44.063037906Z File "/_build/service/license_information_service.py", line 51, in generate_license

2019-05-16T09:33:44.063047899Z license_fields['expiration_time'])

2019-05-16T09:33:44.063056694Z KeyError: 'expiration_time'

If the subscription ID is incorrect, the licensing log will show the following error:

2019-05-30T14:50:39.724521022Z ORA-02291: integrity constraint (RALLYAPP.FK_TAG_SUB_ID) violated - parent key not found

2019-05-30T14:50:39.724763271Z Traceback (most recent call last):

2019-05-30T14:50:39.724780574Z File "license_synchronizer.py", line 24, in <module>

2019-05-30T14:50:39.724804400Z raise exception

2019-05-30T14:50:39.724815362Z File "license_synchronizer.py", line 21, in <module>

2019-05-30T14:50:39.724824664Z license_installer.install()

2019-05-30T14:50:39.724833276Z File "/_build/service/license_installer.py", line 148, in install

2019-05-30T14:50:39.724842317Z self.__finalize_transactions()

2019-05-30T14:50:39.724851028Z File "/_build/service/license_installer.py", line 139, in __finalize_transactions

2019-05-30T14:50:39.724860130Z self.connection.commit()

2019-05-30T14:50:39.724868717Z cx_Oracle.DatabaseError: ORA-02091: transaction rolled back

2019-05-30T14:50:39.724877640Z ORA-02291: integrity constraint (RALLYAPP.FK_TAG_SUB_ID) violated - parent key not found