Cannot start CORBA applications with ERROR TRACE at CsCorbaMgr.cc(1256): Exception: CORBA::TIMEOUT

Products

DX NetOps CA Spectrum

Issue/Introduction

When starting the Spectroserver, it fails with the following error as seen in the $SPECROOT/SS/VNM.OUT:

Aug 02 09:51:33 ERROR TRACE at CsCorbaMgr.cc(1256): Exception: CORBA::TIMEOUT
Minor: 1447165953
Completion Status: NO

The CORBA Namingservice shows the following error in its log

$SPECROOT/bin/VBNS/NAMINGSERVICE.OUT

ExtFactory fails!
org.omg.CORBA.INITIALIZE: Could not initialize java.security.cert.CertificateExpiredException: NotAfter: Wed Aug 02 08:03:51 EDT 2023 vmcid: 0x0 minor code: 0 completed: No
at com.borland.security.core.Init.pre_init(Init.java:672)
at com.inprise.vbroker.orb.ORB.initialize(Unknown Source)
at com.inprise.vbroker.orb.ORB.set_parameters(Unknown Source)
at org.omg.CORBA.ORB.init(ORB.java:353)
at com.inprise.vbroker.naming.ExtFactory.main(Unknown Source)

Environment

DX NetOps Fault (SPECTRUM) versions up to and including 21.2.8.

Cause

SpectroSERVER components ( LocServer, SpectroSERVER, ArchMgr) by default bind a CORBA port AND a secure CORBA port. The secure CORBA port
requires a valid certificate that by default is provided by Spectrum out of the box.

$SPECROOT/lib/SDPM/partslist/NAMINGSERVICE.idb (Corba Namingservice)
Secure: 14016
Normal: 14006

$SPECROOT/LS/.locrc (LocServer)
Secure: 14014
Normal: 14004

$SPECROOT/SS/.vnmrc (SpectroSERVER)
Secure: 14012
Normal: 14002

$SPECROOT/SS/DDM/.configrc (ArchMgr)
Secure: 14013
Normal: 14003

The certificates used to establish the secure CORBA ports expired.

Resolution

This is resolved in DX NetOps Fault (SPECTRUM) in versions 21.2.10 and higher.

In versions 21.2.8 and lower, the out-of-the-box certificate expired on Aug 2, 2023. The certificate is stored in the $SPECROOT/custom/VBNS/identities/spectrum/cert0.

$SPECROOT/custom/VBNS/identities/spectrum
command:
openssl x509 -in cert0 -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 485396356 (0x1cee8f84)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = CA, CN = spectrum-server
        Validity
            Not Before: Aug  2 12:03:51 2021 GMT
            Not After : Aug  2 12:03:51 2023 GMT
        Subject: C = US, O = CA, CN = spectrum-server
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

The solution is to either upgrade to 21.2.10 or higher (which will be EOS Dec 31, 2023) or disable secure CORBA on the SpectroSERVER:

If secure CORBA is not enabled on OneClick you can disable the CORBA Security by editing these files and change the vbroker.security.disable from false to true:

$SPECROOT/.corbarc
$SPECROOT/.jcorbarc

vbroker.security.disable=true

You will need to stop and restart processd after making this change.

As root

- Stop processd
systemctl stop processd

- Verify all processes have come down
ps -ef | grep -i spectrum

- Verify processd has removed all files from the runtime folder
cd $SPECROOT/lib/SDPM/runtime/
ls -la

(processd creates a .rtt file for each running process and these should all be removed when processd stops)

- Start processd
systemcrl start processd

Additional Information

There are two scenarios where the $SPECROOT/custom/VBNS/identities/spectrum/cert0 file can be replaced by an older version:

Restoring the $SPECROOT/custom/ directory from an older backup
Migrating the Spectrum to a new machine and copying over the $SPECROOT/custom/ directory from an older Spectrum version

To overcome it you can simply copy the $SPECROOT/bin/VBNS/identities/spectrum/cert0 file to the $SPECROOT/custom/VBNS/identities/spectrum/ directory to replace the expired Corba certificate file.

**Note**
If your SpectroSERVER is currently running (has not been stopped since before Aug 2nd) you can modify the .corbarc and .jcorbarc files as noted
above to avoid failure on restart.

1) Secure Corba
By default, the SpectroSERVERs will bind both secure and nonsecure CORBA ports. OneClick by default does not have
secure CORBA enabled. If Secure Corba is enabled it will need to be disabled as well if disabling in the steps above to
have the Spectrum processes run on the SpectroSERVER (namineservice, locserver, SpectroSERVER, ArchMgr ..etc).

2) Please also reference information pertaining to a previous issue with the Naming Service and Secure Corba:

https://knowledge.broadcom.com/external/article?articleId=238233

3) Secure Corba Certificate Dates for Various Versions

21.2.8 & earlier expired Aug 2, 2023
21.2.9 - 21.2.12 expires Jan 24, 2024
22.2.1 - 22.2.4 expires Jun 23, 2024
22.2.5 - 22.2.8 expires Dec 19, 2024
22.2.9 - 22.2.11 expires May 28, 2025