Rally - On-premises: Error "Service licensing failed"
Article ID: 132300
Updated On:
Products
Issue/Introduction
fter uploading the license file while provisioning a new 2.0 based appliance, the following error is shown:
1 error occurred: * Service licensing failed with error: task: non-zero exit (1)
Checking the "Cluster" screen from the dashboard may show the licensing service in a down state:
Cause
There are a few causes for this error
- Timing issue
- Insufficient system resources
- Missing license expiration
- Invalid subscription ID
- Database communication problem
Resolution
Reboot
Sometimes during an initial deploy there can be a timing issue between the database and licensing containers starting up. A reboot should be attempted initially to resolve this.
If the system has been operational for a period of time but both database and services VMs were restarted in tandem prior to the failure, then the services VM should be restarted again to ensure the database services have had sufficient time to become available prior to services failing due to an unavailable database.
Insufficient System Resources
Contrary to the system sizing recommendations, we have learned that 2-4 CPUs for a VM is a bit undersized and the current recommendation is to provision the VM with 6 CPUs.
Additionally, 16 GB is considered the minimum memory and the license server may benefit from an increase in system memory as well.
Check License Expiration
This issue has been observed when there is no expiration date included with the license file.
To verify the license file:
- Click the gear (⚙) icon in the toolbar.
- Click View License.
A license with a valid expiration date will show an Expiration Date line as follows:
If your license screen is missing that line, a new license will need to be issued with a valid expiration date.
Check Subscription ID
In most on-premises environments the subscription ID will be 100. The only reason it would be different is if that environment was migrated from SAAS to on-premises and then the subscription ID would be the value that it was when the subscription was in the cloud.
Check Database Communications
This may be an issue with communication with the database and it is recommended to visit the Database Troubleshooting Guide.
Redeploy license service
If the license server does not start after checked the above solutions, it may be necessary to redeploy the license server. This should only be performed during a period of scheduled downtime as users will be unable to access the application for a period of time.
SSH into the service VM as the ops user
Get a list of current docker services:
$ docker service ls
Note the name of the license server as shown in the sample output
ID NAME MODE REPLICAS IMAGE PORTS
5o9rxhek2bt9 premkit_replicated replicated 1/1 registry.replicated.com/library/premkit:v1.3.7
t4khaufz9k9m repl75d5498f_admin replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-admin:1.0.35-2.1.0
mmyyedmz8sne repl75d5498f_alm replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0 *:30000->7001/tcp
r4xkhs5vprp4 repl75d5498f_analytics replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0 *:30001->8080/tcp
zuyza8xc0he8 repl75d5498f_burro replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-burro:1.4839.0-2.1.0
zckml5va1jy2 repl75d5498f_licensing replicated 0/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-licensing:1.1.10-2.1.0
puawm56vkjh1 repl75d5498f_proxy replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-proxy:1.3.8-2.1.0 *:80->80/tcp, *:443->443/tcp
e0m69pfv1z8o repl75d5498f_smtp replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-smtp:1.0.2-2.1.0
jigem7bnue1e repl75d5498f_solr replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:25113-2.1.0 *:30002->8081/tcp
mz0vmb9i54bq replicated_replicated replicated 1/1 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
wpfyi0v1xsd8 replicated_replicated-operator global 1/1 <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.49.0
z6qhqam7vhpz replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
ptp3wuv4ekqu retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
x1139o14xs0g retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
mybc1eau7e9c retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
41ieyj6y23yu retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
l7dinqc28st1 retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
wl3cud8g8pfg retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
oy2n9k39kxnq retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
4az7ie181qi9 retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39
a0oju0hkgwu4 statsd_replicated replicated 1/1 registry.replicated.com/library/statsd-graphite:1.1.7-20201026
Use that name in the following command to delete the license service
$ docker service rm repl75d5498f_licensing
The command will simply respond with the name given.
Now shut down the application with the following command:
$ replicatedctl app stop --force
At this point, you can navigate to the dashboard and click the "Start" button in the upper left tile
Additional Information
If the expiration date is invalid, the licensing log will show the following error:
2019-05-16T09:33:44.062917226Z CRITICAL >> [05/16/2019 09:33:44 AM] (license_synchronizer.py:23) - The following critical exception has occured, we are exiting the license synchronizer 'expiration_time'
2019-05-16T09:33:44.062961814Z Traceback (most recent call last):
2019-05-16T09:33:44.062973075Z File "license_synchronizer.py", line 24, in <module>
2019-05-16T09:33:44.062982682Z raise exception
2019-05-16T09:33:44.062991465Z File "license_synchronizer.py", line 19, in <module>
2019-05-16T09:33:44.063000783Z license = ReplicatedLicenseAPI.get_license()
2019-05-16T09:33:44.063009623Z File "/_build/service/license_information_service.py", line 34, in get_license
2019-05-16T09:33:44.063018801Z license = generate_license(response.json())
2019-05-16T09:33:44.063037906Z File "/_build/service/license_information_service.py", line 51, in generate_license
2019-05-16T09:33:44.063047899Z license_fields['expiration_time'])
2019-05-16T09:33:44.063056694Z KeyError: 'expiration_time'
If the subscription ID is incorrect, the licensing log will show the following error:
2019-05-30T14:50:39.724521022Z ORA-02291: integrity constraint (RALLYAPP.FK_TAG_SUB_ID) violated - parent key not found
2019-05-30T14:50:39.724763271Z Traceback (most recent call last):
2019-05-30T14:50:39.724780574Z File "license_synchronizer.py", line 24, in <module>
2019-05-30T14:50:39.724804400Z raise exception
2019-05-30T14:50:39.724815362Z File "license_synchronizer.py", line 21, in <module>
2019-05-30T14:50:39.724824664Z license_installer.install()
2019-05-30T14:50:39.724833276Z File "/_build/service/license_installer.py", line 148, in install
2019-05-30T14:50:39.724842317Z self.__finalize_transactions()
2019-05-30T14:50:39.724851028Z File "/_build/service/license_installer.py", line 139, in __finalize_transactions
2019-05-30T14:50:39.724860130Z self.connection.commit()
2019-05-30T14:50:39.724868717Z cx_Oracle.DatabaseError: ORA-02091: transaction rolled back
2019-05-30T14:50:39.724877640Z ORA-02291: integrity constraint (RALLYAPP.FK_TAG_SUB_ID) violated - parent key not found
Feedback
