Steps to install an Oracle database container and TDM Portal for Docker

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

What steps are needed to install the Oracle docker container, and TDM Portal for Docker? 

Environment

All supported TDM Portal for Docker versions.

Resolution

Below are the steps need to install the Oracle 21c docker image and container, and the TDM Portal 4.11.54.0 release for Docker in a support lab. Running these steps (commands are included) will get Oracle and TDM for Docker up and running in a lab/test environment.

Note:  In the TDM documentation (TDM Portal Oracle Database Container), Broadcom recommends running the Oracle docker container for testing and development purposes only. If this is intended to be used for a UAT or Production environment, Broadcom recommends using an external licensed Oracle database.

Installing and running the Oracle docker container

The Oracle database image must be built, and the Oracle container must be running, before loading and running the TDM Docker images. Just like in the Windows version of TDM Portal, the TDM gtrep repository schema is required to be created prior to installing and running the TDM Portal in Docker.

1. Downloading the files required to build the Docker container from https://github.com/oracle/docker-images/tree/main/OracleDatabase/SingleInstance/dockerfiles
   
   For installation instructions, refer to https://github.com/oracle/docker-images/blob/main/OracleDatabase/SingleInstance/README.md#how-to-build-and-run
   
   This example uses the Oracle 21.3.0 files (Oracle 21c), but any supported version can be used.
   
   Note: This example uses the following file structure in the users $HOME directory: /home/<user>/Oracle.
   
   
   1. Inside /home/<user>/Oracle place the buildContainerImage.sh file
   2. Create a subdirectory called /home/<user>/Oracle/21.3.0
   3. Download the remaining files required to build the 21.3.0 image inside this container
      • checkDBStatus.sh
      • checkSpace.sh
      • Checksum.ee
      • Checksum.se2
      • configTcps.sh
      • createDB.sh
      • createObserver.sh
      • db_inst.rsp
      • dbca.rsp.tmpl
      • Dockerfile.txt
      • Dockerfile.xe
      • installDBBinaries.sh
      • oracle-xe-21c.conf
      • relinkOracleBinary.sh
      • runOracle.sh
      • runUserScripts.sh
      • setPassword.sh
      • setupLinuxEnv.sh
      • startDB.sh
        
        
2. Build the Oracle image by running the following from the /home/<user>/Oracle directory.
   Note, if you do not have root access, you will need to run the command using sudo.
   
   ./buildContainerImage.sh -v 21.3.0 -t tdm/officialoracle-gtrep:21.3.0-xe -x
   
   
3. Run the image, create the Container, expose port 1521, and publish the port so it can be accessed by applications outside the container.
   Note: If you exposed and published the 1521 listening port to the system, you should also be able to connect using an external SQL Plus, SQL Developer tool, or another tool such as Toad, to connect and look at the gtrep table structure.
   
   docker run --name <container name> -d -p <host_ip>:<host_port>:<container_port> <image:tag>
   
   For example: docker run --name officialoracle -d -p <IP address>:1521:1521 tdm/officialoracle-gtrep:21.3.0-xe
   
   Note the Database name and SID, in this example both are XE
   
   
4. Reset the administrative password used by SYS, SYSTEM, PDBADMIN accounts by executing the setPassword.sh script inside the docker container
   
   docker exec <container name> ./setPassword.sh <newPassword>
   
   For example: docker exec officialoracle ./setPassword.sh MyPassword
   
   
5. Add the GTREP user account and Schema to the XE database. This can be done several ways. Such as using a Dockerfile, or executing a SQL script from SQL Plus in the docker container.
   
   This is an example of executing SQL Plus inside the docker container, and executing the SQL commands as shown below.
   
   1. Make sure the docker container is running. If it's not running, start it by running:
      docker start <container name>
      
      
   2. Execute SQL Plus inside the container
      docker exec -it <container name> bash
      
      
   3. At the bash prompt, log into SQL Plus using the following command:
      bash> sqlplus sys/<yourpw>@localhost:1521/XE as sysdba
      
      
   4. Create the GTREP user and grant the required privileges by running the following SQL statements:
      • alter session set "_ORACLE_SCRIPT"=true;
      • CREATE USER gtrep IDENTIFIED BY <password> DEFAULT TABLESPACE USERS;
      • GRANT CONNECT, RESOURCE TO gtrep;
      • GRANT CREATE ANY DIRECTORY, SELECT ANY TABLE, CREATE VIEW TO gtrep;
      • GRANT UNLIMITED TABLESPACE TO gtrep;
      • GRANT DBA TO gtrep;
      • GRANT CREATE ANY INDEX TO gtrep;
      • GRANT CREATE CLUSTER TO gtrep;
      • GRANT CREATE INDEXTYPE TO gtrep;
      • GRANT CREATE OPERATOR TO gtrep;
      • GRANT CREATE PROCEDURE TO gtrep;
      • GRANT CREATE SEQUENCE TO gtrep;
      • GRANT CREATE SESSION TO gtrep;
      • GRANT CREATE TABLE TO gtrep;
      • GRANT CREATE TRIGGER TO gtrep;
      • GRANT CREATE TYPE TO gtrep;
      • GRANT CREATE VIEW TO gtrep;
      • GRANT SELECT ANY DICTIONARY TO gtrep;
        
        
   5. Exit from SQL Plus
      exit
      
      
   6. Stop the container:
      docker stop <container_name>
      
      
   7. Commit the changes to the image:
      docker commit -m <message> -a "<Your Name>" <container_name> <registry>/<nameForTheChangedImage>:<tag>
      
      For Example:
      docker commit -m "Oracle 21.3.0 XE + GTREP" -a "John Doe" officialoracle tdm/officialoracle-gtrep:21.3.0-xe
      
      
   8. Optional, rename the docker container to show it includes gtrep
      docker rename <container name> <new container name>
      
      For example: docker rename officialoracle officialoracle-gtrep
      
      
   9. Start the Oracle docker container:
      docker start <container name>
      
      For example: docker start officialoracle-gtrep

Installing and running the TDM images and containers

Note: these steps are taken straight out of the README-Install.md and README-RUN.md files shipped with the TDM_Portal_docker -<version>.tgz file. For additional information, see Install TDM Portal for Docker.

1. Download the TDM_Portal_docker-<version>.tgz from the Broadcom Support Portal - My Downloads page.
   This example is using the TDM_Portal_docker-4.11.54.0.tgz file, which was downloaded from the Test Data Manager (TDM) Support Patches page. The TDM patches are comprehensive full builds of the TDM Portal, so the latest patch available on this page will be the most recent build containing all the previous fixes and enhancements from previous releases. It's highly recommended to use the latest TDM_Portal_docker-<version>.tgz release instead of the GA (General Availability) release.
   
   
2. Extract the TDM Docker images:
   tar xvzf TDM_Portal_docker-4.11.54.0.tgz
   
   
3. Load the TDM docker images from the directory where the files were extracted:
   
   • docker load -i ./TDM_images/orientdb/orientdb-2.2.37.tgz
   • docker load -i ./TDM_images/messaging/messaging-4.11.54.0.tgz
   • docker load -i ./TDM_images/action-service/action-service-4.11.54.0.tgz
   • docker load -i ./TDM_images/masking/masking-4.11.54.0.tgz
   • docker load -i ./TDM_images/tdmweb/tdmweb-4.11.54.0.tgz
   • docker pull zookeeper:3.8.0
   • docker load -i ./TDM_images/tdmtools/tdmtools-4.11.54.0.tgz
     
     
4. From inside the directory where the files were extracted, prepare the TDM environment by creating the tdm user and setting the TDM_HOME folder for the TDM Services:
   
   • ./prepareTdmUser.sh
   • source /etc/environment
   • echo $TDM_HOME
     
     
5. From inside the directory where the files were extracted, customize the docker-compose files according to your requirements.
   Note: in this example, the only file modified was the docker-compose-environment.yml, to add the gtrep database information. This yml is normally used for external Oracle database connections. Because this example exposed and published the Oracle communication port outside the docker container, so the port is listening at the system level, we can configure the TDM portal docker container to connect as if it was using an external connection. Otherwise, we would need to modify and use the docker-compose-ora.yml.
   
   For Example:
   
   • 'GTREP_DATABASE=XE'
   • 'GTREP_HOST=<Server IP Address>'
   • 'GTREP_PORT=1521'
   • 'GTREP_USER=GTREP'
   • 'GTREP_PASSWORD=<MyPassword>'
     
     
6. Copy the docker-compose files into the TDM_HOME\conf directory:
   /bin/cp ./docker-compose*.yml $TDM_HOME/conf/
   
   
7. Create docker secret for rabbitmq default password:
   
   • docker swarm init
   • echo "marmite" | docker secret create rabbitmq_default_password -
     
     
8. Copy rabbitmq configuration file to tdm conf folder:
   /bin/cp ./TDM_images/messaging/conf/rabbitmq.conf $TDM_HOME/conf/rabbitmq.conf
   
   
9. Setup SSL certificates for rabbitmq using the Broadcom self signed certificates:
   
   • /bin/cp -f ./TDM_images/messaging/certificates/cacert.pem $TDM_HOME/certificates/cacert.pem
   • /bin/cp -f ./TDM_images/messaging/certificates/cert.pem $TDM_HOME/certificates/cert.pem
   • /bin/cp -f ./TDM_images/messaging/certificates/key.pem $TDM_HOME/certificates/key.pem
     
     If you need to use your own CA certified certificates, refer to the README-RUN.md file or the TDM documentation -  Install TDM Portal for Docker.
     
     
10. Deploy and run the TDM docker stack:
    docker stack deploy --compose-file=docker-compose.yml --compose-file=docker-compose-messaging.yml --compose-file=docker-compose-masking.yml --compose-file=docker-compose-service-orchestration.yml --compose-file=docker-compose-volumes.yml --compose-file=docker-compose-environment.yml tdm-stack
    
    
11. Change directories to the TDM_HOME\logs, and check the TDMSchemaManagement.log for any errors and to ensure connection to the gtrep schema was successful. The first time the TDM Service connects, the migration script will build out the table structure for the gtrep schema. If successful, you should also see the normal TDM Portal logs in this directory.