Topic included
******************************
- Installing Restmon using the helm for http
- Installing command for NFS storage
- Configuration to use the custom Restmon port and service port
- Configuration for enabling Supportability Metric and liveness/Readiness Probes
- Installing steps for https connection
- Helm Options
- Environment Variables
- Readiness Probe Configuration
- Liveness Probe Configuration
- Node Selector
- Upgrade Helm
- helm upgrade with nfs
- helm upgrade with storage class
- Upgrade to new RESTMon (2.0 to 2.1)
- Uninstalling step
- Troubleshooting
Installing Restmon using the helm for http
Follow the below steps for deploying Restmon using helm.
NOTE
All the steps mentioned in this document have been validated on a Linux machine with HELM VERSION , This and example. Use the version as recomanded in DX APM 24.2 guide. We always prefer to use the latest version.
helm version
Version:"v3.4.1"
Step 1: Download and copy the latest helm chart , Example helm-charts-1.0.3-restmon.tgz
Step 2: Create a namespace, if new namespace is desired.
|
eg: restmon
|
Step 3:
Using helm install
|
helm install <name> helm-charts-1.0.3-restmon.tgz --set restmon.id=<id> --set restmon.storage.size=<size> --namespace <namespace>
|
|
helm install restmon1 helm-charts-1.0.3-restmon.tgz --set restmon.id=1 --set restmon.storage.size=5Gi --namespace restmon
|
**Note: another installation can be performed by passing different id like id=2
|
helm install restmon2 helm-charts-1.0.3-restmon.tgz --set restmon.id=2 --set restmon.storage.size=5Gi --namespace restmon
|
Step 4: The deployment status can be found from the command
|
helm ls -n restmon
|
Installing command for NFS storage
By default, the helm chart uses storage class "ssd" for the persistent volumes. The below attributes need to be passed for the helm install if PV using NFS is desired.
- restmon.storage.nfs_path
- restmon.storage.nfs_server
|
helm install restmon1 helm-charts-1.0.3-restmon.tgz --set restmon.id=1 --set restmon.storage.size=1Gi --set restmon.storage.nfs_path=<nfs-path> --set restmon.storage.nfs_server=<server-name> --namespace restmon
|
|
helm install restmon1 helm-charts-1.0.3-restmon.tgz --set restmon.id=1 --set restmon.storage.size=1Gi --set restmon.storage.nfs_path=/var/nfs/kubedata/test --set restmon.storage.nfs_server=<server-name> --namespace restmon
|
Note
Possible values for restmon.id : lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
Configuration to use the custom Restmon port and service port
The default protocol of http and port of 8080 is used for the pod deployment. The port can be modified as below.
|
helm install restmon1 helm-charts-1.0.5-restmon.tgz --set restmon.id=1 --set restmon.settings.restmon_port=8081
|
The service port of 8080 is used by default. The port on which the service is exposed can be modified by updating "service_port" as below.
|
helm install restmon1 helm-charts-1.0.5-restmon.tgz --set restmon.id=1 --set restmon.settings.service_port=8081
|
Configuration for enabling Supportability Metric and liveness/Readiness Probes
- Unique instance name needs to be attached to the container using the following configuration. This helps in identifying the supportability metrics of each of the instances deployed per tenant in the dashboard.
|
helm install restmon1 helm-charts-1.0.5-restmon.tgz --set restmon.id=1 --set restmon.settings.supportability_instanceName=myinstance1
|
- The default configuration available for liveness,readiness can be modified by changing any of the following attributes.
- restmon.settings.readiness_check_interval_sec
- restmon.settings.profile_queue_size_max_limit
- restmon.settings.profile_queue_size_min_limit
- restmon.settings.max_allowed_running_profiles
- restmon.settings.db_health_check_interval_sec
- restmon.settings.health_statistics_log_level
- restmon.settings.liveness_check_interval_min
- restmon.settings.unready_profile_max_allowed_retries
- restmon.settings.unready_profile_retry_time_sec
Installing steps for https connection
Step1: Create the certificate to be used (PKCS12 or JKS).
Please refer below for creating self-signed certificate
Creating key and certificate:
|
openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
|
Provide the requested details with the hostname/domain name to be used for creating the route as below.
Creating PKCS12 file from certificate and key:
Combine Key and certificate into PKCS12 bundle to be used by Restmon application as below.
|
openssl pkcs12 -inkey <key.pem> -in <certificate.pem> -export -out <restmon-certificate.p12> -name <restmon-certificate-alias>
|
Provide the desired password to generate the Keystore.
Step2: The above certificate for HTTPS connection needs to be placed in Restmon folder of extracted/unzipped helm-charts-1.0.3-restmon.tgz
Step3: Execute the below Installation command from within the extracted folder by passing the required attributes like Keystore file name, Keystore password, Keystore alias.
|
helm install restmon1 ./restmon --set restmon.id=1 --set restmon.storage.size=1Gi --set restmon.settings.restmon_protocol=https --set restmon.settings.ssl_key_store=<restmon-certificate.p12> --set restmon.settings.ssl_key_store_password=<hello> --set restmon.settings.ssl_key_store_alias=<alias1> --namespace restmon
|
Note
The values can also be updated inside the values.yaml file given in helm-charts-1.0.3-restmon.tgz
Helm Options
All the arguments used (above) in the helm installation and their explanation is as under.
| restmon.storage.size | 1Gi | Storage size for restmon |
|---|---|---|
| restmon.id | 1 | Possible values for restmon.id : lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?') |
| restmon.resources.requests.cpu | 2 | minimum CPU value to be set |
| restmon.resources.limits.cpu | 4 | maximum CPU value to be set |
| restmon.resources.requests.memory | 512Gi | minimum memory value to be set |
| restmon.resources.limits.memory | 4Gi | maximum memory value to be set |
| restmon.storage.nfs_path | /var/nfs/kubedata/testfolder | The folder path for nfs storage |
| restmon.storage.nfs_server | <Server-name> | The nfs server name |
Environment Variables
The supported environment variables are as below.
|
Environment Variable |
Supported Values /Example |
Comments |
|---|---|---|
| restmon.settings.replace_oi_attributes | true/false | If true: the OI specific environment variables will be used for replacing the existing values in restmon.config |
|
restmon.settings.jarvis_ingestion_hostname
|
nginix-route-8081-ao-doi.<host/servername> | The nginix hostname of Jarvis API ingestion endpoint used for alarms/change events ingestion. Will be used if REPLACE_OI_ATTRIBUTES=true |
|
restmon.settings.jarvis_ingestion_port
|
443 | The port on which nginix Jarvis API ingestion endpoint is exposed. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.jarvis_ingestion_protocol | http/https | The http protocol using which nginix Jarvis API ingestion endpoint is accessible. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.oi_ingestion_api_host | apmservices-gateway-ao-apm.<host/servevername> | The APM gateway ingestion endpoint used for toology/NASS metrics data ingestion. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.oi_ingestion_api_port |
443 | The port on which APM gateway ingestion endpoint is exposed. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.oi_ingestion_api_protocol | http/https | The http protocol using which APM gateway ingestion endpoint is accessible. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.oi_ingestion_tenant_token |
<tenant_token> |
The tenant/agent token to be used as authentication for APM gateway ingestion endpoint. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.tenant_id | < tenant_id> | The Cohort ID of the tenant doing the data ingestion into OI. Will be used if REPLACE_OI_ATTRIBUTES=true |
| restmon.settings.max_number_of_docs | 1000 | The number of json documents to be included in single payload for ingestion to OI. default to 1000. |
| restmon.settings.restmon_min_memory_limit | 512M | Used for -Xms while launching the application. |
| restmon.settings.restmon_max_memory_limit | 8G | Used for -Xmx while launching the application. |
| restmon.settings.log_level | info (info/debug/trace) | required log level for application logging. |
| restmon.settings.ssl_key_store_type |
JKS/PKCS12 | Type of the keystore used for https communication. |
| restmon.settings.ssl_key_store | restmon.keystore | keystore used for https communication. |
| restmon.settings.ssl_key_store_password | password | Password for the keystore used for https communication. |
| restmon.settings.ssl_key_store_alias | alias | keystore alias for the keystore used for https communication. |
| restmon.settings.user_name | admin | Restmon username to be used for basic authentication available by default. The default username used is admin. |
| restmon.settings.password | password | password to be used for basic authentication available by default. The default password used is password. |
| restmon.settings.restmon_protocol | http/https | Default: http. The protocol to be used for starting the RESTMon. |
| restmon.settings.restmon_port | 8080/8443 | Default: 8080. The port on which the application listens. |
| restmon.settings.service_port | 8080/8443 | Default: 8080. The port on which the kubernetes service is exposed. |
| restmon.settings.supportability_agentName | SuperDomain|newoiservices|opendataconnector |
The agent name against which the metrics are ingested for RESTMon connector. This not required to be modified. For more details on the metrics refer here. |
| restmon.settings.supportability_instanceName | restmon1 | Unique identifier for the Restmon instance. This has to be unique in the tenant context. |
| restmon.settings.readiness_check_interval_sec | interval in seconds(15,60,etc) |
Default: 15sec. The frequency at which the scheduler, that checks the readiness of the application to be run. The health of the application will be marked as "not ready" if the OI endpoints were not available. |
| restmon.settings.profile_queue_size_max_limit | integer | Default: 20000.
The threshold of the pending queue size. If the number of messages in the steaming queue for any of the profiles is more than the configured number, the readiness of the application will be marked as "not ready". |
| restmon.settings.profile_queue_size_min_limit | integer |
Default: 15000. The limit on the queue size against which the "not ready" application can come back to the "ready" state. If the application becomes "not ready" because of the above configuration against the queue threshold, the application can come back to the "ready" state once the number of messages comes below this configured number. |
| restmon.settings.max_allowed_running_profiles | integer |
Default: 5
|
| restmon.settings.db_health_check_interval_sec | interval in seconds(15,60,etc) |
Default: 15sec. The frequency at which the scheduler which checks db health to be run. The health of the application will be marked as "not live" if the db is down or any other issue in connecting to DB. |
| restmon.settings.health_statistics_log_level | error/info/debug/trace |
Default: ERROR. The new log file restmon_statistics.log will be available with default log level of "ERROR".
|
| restmon.settings.liveness_check_interval_min | interval in minutes(10,15,etc) |
Default: 10min. The frequency at which the liveness check scheduler to be run. This basically checks if the data ingestion for the streaming profile is active, but the processing/publishing could not be completed within the configured time. And if the processing is not completed, the liveness of the application will be disabled. The application will be back to live, as soon as the processing is completed. |
| restmon.settings.unready_profile_max_allowed_retries | integer |
Default: 10 Indicates the maximum number of times the polling profile can be retried for starting processing in case the processing could not be started because of the readiness being down ("not ready). |
| restmon.settings.unready_profile_retry_time_sec | interval in seconds(10,15,etc) |
Default: 10 Indicates the wait time for the profile retry. |
restmon.settings.show_swagger |
true/false | Default: true If false the swagger url will not be accessible. |
restmon.settings.log_file_max_size |
String | Default: 50MB The file size for each log file created. |
restmon.settings.log_file_max_history |
Integer | Default: 10 The number of log files to be created. |
Readiness Probe Configuration
The following attributes are used for enabling and configuring the readiness probe.
| restmon.readinessProbe.enable | true/false |
Default: true Used for enabling/disabling readiness probe for the pod deployment. |
| restmon.readinessProbe.initialDelaySeconds | integer in seconds |
Default: 60 The time to wait for before performing first probe by Kubernetes |
| restmon.readinessProbe.periodSeconds | integer in seconds |
Default: 30 The frequency at which the readiness probe is invoked. |
| restmon.readinessProbe.timeoutSeconds | integer in seconds |
Default: 30 Number of seconds after which the probe times out. |
| restmon.readinessProbe.successThreshold | integer | Minimum consecutive successes for the probe to be considered successful after having failed. |
| restmon.readinessProbe.failureThreshold | integer | Number of retries before marking the pod as “unready” and the data routing is stopped . |
Liveness Probe Configuration
The following attributes are used for configuring the liveness probe.
| restmon.livenessProbe.initialDelaySeconds | integer in seconds |
Default: 60 The time to wait for before performing first probe by Kubernetes |
| restmon.livenessProbe.periodSeconds | integer in seconds |
Default: 30 The frequency at which the liveness probe is invoked. |
| restmon.livenessProbe.timeoutSeconds | integer in seconds |
Default: 30 Number of seconds after which the probe times out. |
| restmon.livenessProbe.successThreshold | integer |
Default: 1 Minimum consecutive successes for the probe to be considered successful after having failed. |
| restmon.livenessProbe.failureThreshold | integer |
Default: 5 Number of retries before restarting the pod. |
Node Selector
The following attributes are used to enable node selectors to deploy restmon on specific worker node in kubernets cluster.
Assigning a label to the worker node:
|
kubectl label nodes <worker-node-1> disktype=ssd
|
Deleting a label in worker node :
|
kubectl label nodes worker-2 disktype-
|
Checking label in worker node:
|
kubectl get nodes --show-labels
|
| restmon.nodeSelector |
disktype: "ssd" |
The specified key value should be available in the Kubernetes cluster, otherwise, Kubernetes won't allow the restmon pod to deploy. Comment this entire key:value to disable node selector |
Upgrade Helm
|
helm upgrade --set <key>=<value> <name> helm-charts-1.0.0-restmon.tgz -n <namespace>
|
|
helm upgrade --set restmon.id=1 --set restmon.storage.size=1Gi --set restmon.resources.requests.cpu=2 --set restmon.resources.limits.cpu=2 --set restmon.resources.requests.memory=512Mi --set restmon.resources.limits.memory=4Gi restmon1 helm-charts-1.0.0-restmon.tgz -n restmon
|
helm upgrade with nfs
Use arguments nfs_path and nfs_server as given in the install.
eg:
|
helm upgrade --set restmon.storage.size=2Gi --set restmon.settings.log_level=DEBUG --set restmon.storage.nfs_path=/var/nfs/kubedata/test --set restmon.storage.nfs_server=<server-name> restmon1 ./restmon --namespace restmon
|
helm upgrade with storage class
eg:
|
helm upgrade --set restmon.settings.log_level=DEBUG restmon1 ./restmon -n restmon
|
Upgrade to new RESTMon (2.0 to 2.1)
Pre-Upgrade:
- Take the copy of the previously set helm values using the following command.
|
helm get values chartname -n namespace > restmon.yaml
|
Note: If any updates happened to deployment manually (not using helm), the values will not be available using the “helm get values” command.
- Deactivate polling profiles as both containers process the same data for some time during parallel execution.
- Get into the container and take the backup of config and schema folder under data folder, say backup folder.
Upgrade:
Execute the below command for upgrading RESTMon from previois version to 2.1 version.
|
helm upgrade <name> <helm chart name> -f restmon.yaml --set restmon.settings.reload_ootb_schemas=true --set restmon.imageName=<new image to be upgraded to> --namespace <namespace>
|
Example:
|
helm upgrade restmon1 restmon-2.1.23.tgz -f restmon.yaml --set restmon.settings.reload_ootb_schemas=true --set restmon.imageName=esd-oi-docker-<server-name>/opendataconnector/2.1.0/dx-restmon:2.1.0-SNAPSHOT.76 --namespace restmon
|
Once the upgrade is sucessful, the old pod will get terminate making the latest one active.
Post Upgrade:
Make sure the swagger UI is accessible and the all other endpoints are accessible.
Activate the deactivated profile (as part of Pre-Upgrade) and make sure RESTMon processs them without any issue.
Uninstalling step
|
helm uninstall <name> -n <namespace>
|
|
helm uninstall restmon1 -n restmon
|
Troubleshooting
-
To get deployment status
helm ls -n <namespace> -
To display all the pods with id for the particular namespace.
kubectl get pods -n <namespace> -
To get the pod logs.
