cannot touch /opt/SecureSpan/Gateway/node/default/var/preboot™: Permission denied/ Openshift Lab
search cancel

cannot touch /opt/SecureSpan/Gateway/node/default/var/preboot™: Permission denied/ Openshift Lab

book

Article ID: 272463

calendar_today

Updated On:

Products

CA API Gateway

Issue/Introduction

Using 10.1_CR2 tag.
Tried to add a new node in lab cluster gateway version 10.1.0 (has two nodes). Received the message below when I run the container .

Or 

Try to start a container gateway ssg 11. 

Message:

/opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/010_update_cluster_host.xml.req.bundle: Permission denied
Starting gateway in foreground
touch: cannot touch ‘/opt/SecureSpan/Gateway/node/default/var/preboot™: Permission denied

Full message

Using MySQL database
SSG_DATABASE_WAIT_TIMEOUT set to 300 seconds.
SSG_JVM_HEAP will be 2g
SSG_CLUSTER_HOST will be api-lab.corp.domain.com
SSG_GC_ARGS will be -XX:+PrintGCDetails -Xloggc:/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log
Waiting for one of the databases to come up...
Jul 26, 2023 2:14:44 PM liquibase.servicelocator
INFO: Cannot load service: liquibase.license.LicenseService: Provider liquibase.license.pro.DaticalTrueLicenseService could not be instantiated
Jul 26, 2023 2:14:57 PM liquibase.servicelocator
INFO: Cannot load service: liquibase.license.LicenseService: Provider liquibase.license.pro.DaticalTrueLicenseService could not be instantiated
####################################################
##   _     _             _ _                      ##
##  | |   (_)           (_) |                     ##
##  | |    _  __ _ _   _ _| |__   __ _ ___  ___   ##
##  | |   | |/ _` | | | | | '_ \ / _` / __|/ _ \  ##
##  | |___| | (_| | |_| | | |_) | (_| \__ \  __/  ##
##  \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___|  ##
##              | |                               ##
##              |_|                               ##
##                                                ## 
##  Get documentation at docs.liquibase.com       ##
##  Get certified courses at learn.liquibase.com  ## 
##  Free schema change activity reports at        ##
##      https://hub.liquibase.com                 ##
##                                                ##
####################################################
Starting Liquibase at 14:14:57 (version 4.5.0 #52 built at 2021-09-27 16:19+0000)
Liquibase: Update has been successful.
/opt/docker/entrypoint.sh: line 239: /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/010_update_cluster_host.xml.req.bundle: Permission denied
Starting gateway in foreground
touch: cannot touch ˜/opt/SecureSpan/Gateway/node/default/var/preboot™: Permission denied
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log instead.
[0.002s][warning][gc] -XX:+PrintGCDetails is deprecated. Will use -Xlog:gc* instead.
[0.002s][error  ][logging] Error opening log file '/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log': Permission denied
[0.002s][error  ][logging] Initialization of output 'file=/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log' using options '(null)' failed.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

used this helm:

https://github.com/CAAPIM/apim-charts/tree/stable/charts/gateway

The license is config by command line param:

--set-file "license.value=license.xml" --set "license.accept=true"

maybe it's some problem to use the image as root in openshift.

If I to try the same image from standalone docker server , its OK.

2) 10.1.0_CR3 tag , but I got the same error:

Using MySQL database
SSG_DATABASE_WAIT_TIMEOUT set to 300 seconds.
SSG_JVM_HEAP will be 2g
SSG_CLUSTER_HOST will be api-lab.corp.domain.com
SSG_GC_ARGS will be -XX:+PrintGCDetails -Xloggc:/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log
Waiting for one of the databases to come up...
Aug 10, 2023 11:12:54 AM liquibase.servicelocator
INFO: Cannot load service: liquibase.license.LicenseService: Provider liquibase.license.pro.DaticalTrueLicenseService could not be instantiated
Aug 10, 2023 11:13:06 AM liquibase.servicelocator
INFO: Cannot load service: liquibase.license.LicenseService: Provider liquibase.license.pro.DaticalTrueLicenseService could not be instantiated
####################################################
##   _     _             _ _                      ##
##  | |   (_)           (_) |                     ##
##  | |    _  __ _ _   _ _| |__   __ _ ___  ___   ##
##  | |   | |/ _` | | | | | '_ \ / _` / __|/ _ \  ##
##  | |___| | (_| | |_| | | |_) | (_| \__ \  __/  ##
##  \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___|  ##
##              | |                               ##
##              |_|                               ##
##                                                ## 
##  Get documentation at docs.liquibase.com       ##
##  Get certified courses at learn.liquibase.com  ## 
##  Free schema change activity reports at        ##
##      https://hub.liquibase.com                 ##
##                                                ##
####################################################
Starting Liquibase at 11:13:07 (version 4.5.0 #52 built at 2021-09-27 16:19+0000)
Liquibase: Update has been successful.
/opt/docker/entrypoint.sh: line 239: /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/010_update_cluster_host.xml.req.bundle: Permission denied
Starting gateway in foreground
touch: cannot touch ‘/opt/SecureSpan/Gateway/node/default/var/preboot™: Permission denied
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log instead.
[0.002s][warning][gc] -XX:+PrintGCDetails is deprecated. Will use -Xlog:gc* instead.
[0.002s][error  ][logging] Error opening log file '/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log': Permission denied
[0.002s][error  ][logging] Initialization of output 'file=/opt/SecureSpan/Gateway/node/default/var/logs/ssg_gc.log' using options '(null)' failed.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

oc get events showed :

145m        Normal    Scheduled                 pod/##-###-ssg-gateway-7ddbf4fc86-5pcgs          Successfully assigned ###-layer7/##-###-ssg-gateway-7ddbf4fc86-5pcgs to ###-###-dev-blv8z
145m        Normal    AddedInterface            pod/##-###-ssg-gateway-7ddbf4fc86-5pcgs          Add eth0 [###.##.##.###/23] from openshift-sdn
35m         Normal    Pulled                    pod/##-###-ssg-gateway-7ddbf4fc86-5pcgs          Container image "docker.io/caapim/gateway:10.1.00_CR3" already present on machine
142m        Normal    Created                   pod/##-###-ssg-gateway-7ddbf4fc86-5pcgs          Created container gateway
142m        Normal    Started                   pod/##-###-ssg-gateway-7ddbf4fc86-5pcgs          Started container gateway
44s         Warning   BackOff                   pod/l##-###-ssg-gateway-7ddbf4fc86-5pcgs          Back-off restarting failed container
170m        Normal    Pulled                    pod/##-###-ssg-gateway-7ddbf4fc86-svlcb          Container image "docker.io/caapim/gateway:10.1.00_CR3" already present on machine
150m        Warning   BackOff                   pod/##-###-ssg-gateway-7ddbf4fc86-svlcb          Back-off restarting failed container
145m        Normal    SuccessfulCreate          replicaset/l##-###--ssg-gateway-7ddbf4fc86         Created pod: ##-###-ssg-gateway-7ddbf4fc86-5pcgs
4s          Warning   FailedGetResourceMetric   horizontalpodautoscaler##-###-ssg-gateway-hpa   failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
104m        Warning   FailedGetResourceMetric   horizontalpodautoscaler/##-###-ssg-gateway-hpa   failed to get cpu utilization: did not receive metrics for any ready pods
145m        Warning   FailedToUpdateEndpoint    endpoints/##-###-ssg-gateway-management          Failed to update endpoint ###-layer7/##-###-ssg-gateway-management: 
 Operation cannot be fulfilled on endpoints "##-###-ssg-gateway-management":
 the object has been modified; please apply your changes to the latest version and try again

 

 

Environment

Release : 10.1, 11.x

Cause

Not set up pod and container security context constraint properly

heapsize low 

 from attached values_customer.yaml the heap size is less than the 50% indicated. Set heapSize at least 4g up to 6g

 

resources:
 # There are no resource limits set by default, this is a consicious choice for the user and
 # increases the chance of these running on environments with fewer resources available
 # Remove the curly braces and uncomment cpu/memory to set.
 limits:
     cpu: 400m
     memory: 8Gi
 requests:
     cpu: 200m
     memory: 8Gi

config:
 # Heap Size should be a percentage of the memory configured in resource limits
 # by default it is 50% - you should not go above 75%
 heapSize: "2g"

Resolution

Run

oc describe project <project-name>

In the output,  look the result 

openshift.io/sa.scc.supplemental-groups and openshift.io/sa.scc.uid-range: 

Exmaple: 

  openshift.io/sa.scc.supplemental-groups=1003000000/10000

  openshift.io/sa.scc.uid-range=1003000000/10000

Select user id in the rage 1003000000, 1003000000 + 10000,  group in the range 1003000000, 1003000000 + 10000

Modify the values.yaml file containerSecurityContext and podSecurityContext with user and group in the range indicated in output of openshift.io/sa.scc.supplemental-groups and openshift.io/sa.scc.uid-range: 

Use above example to modify the values.yaml as: 

containerSecurityContext:
        runAsNonRoot: true
        runAsUser: 1003009990
        capabilities:
                drop:
                - ALL
        allowPrivilegeEscalation: false
podSecurityContext:
  runAsUser: 1003009990
  runAsGroup: 1003009990   

Restart the container gateway.  This should fix the permission denied errors. 

Also,  the heapSize may need to increased.

Set the "heapSize" to 6G the deployment is up and running fine.

In addition , to resolve the health check issue, applied these configuration on Dockerfile:

1. FROM gateway:10.1.00_CR3
2. USER root
3. RUN ln -sf /usr/share/zoneinfo/usr/share/zoneinfo/America/<local time zone> /etc/localtime
4. COPY health_check.sh /opt/docker/rc.d/diagnostic/
5. RUN chmod 755 /opt/docker/rc.d/diagnostic/health_check.sh

HEALTHCHECK --interval=300s --timeout=5s --retries=1 --start-period=120s CMD /opt/docker/rc.d/diagnostic/health_check.sh || exit 1
USER ${ENTRYPOINT_UID}

5. [user@lserver gateway]$ cat health_check.sh

response=$(curl -k -s -o /dev/null -w "%{http_code}" https://localhost:9443/lbcheck)
if [ "$response" -eq "200" ]; then
   exit 0
fi
exit 1