After Domain migration in our environment, unable to submit job
Article ID: 237024
Updated On:
Products
Issue/Introduction
We had domain migration in our environment to align to domain changes at organization. We noticed that post this we are unable to submit a Job though node shows online on the UVC console.
When trying to submit a job/node settings we get below error
Server cannot be reached: ServerException *
Environment
Release : 6.x
Component : DOLLAR UNIVERSE
Cause
As we reviewed the logs we see below errors.
Examples of error Messages :
From UVC log file:
#####
Agent IO Connection problem - ServerException - connect: error creating socket to node_name/port_number
#####
From unijob universe.log file:
#####
u_connect | u_gethostbyname_mf(uvms_node_name) returns error: Unknown host (gethostbyname(uvms_node_name) returns null pointer)
#####
Post domain migration it resulted in change of the DNS of nodes and this is not updated in UVMS, which still have old DNS mapping against each node. When a Job is launched, it still tried to resolve to old DNS of node which doesn't exist any more and hence result in error
Server cannot be reached: ServerException *
Validation
Note: For troubleshooting connectivity issues refer additional information section "DUAS: Communication issues between UVMS/UVC and Nodes through the network"
- UVMS server: run below command
$ping FQDN_OF_THE_NODE
#Output: request could not find host
- UVMS Server, validate the hostname of the NODES (refer additional information section for unilst command document)
$UVMS_INSTALL_DIR\app\bin\unilst NODE -full
$UVMS_INSTALL_DIR\app\bin\unilst NODE -full -node <NODE_NAME_OR_FILTER>
Resolution
Update the Dollar Universe node registration in the list of UVMS nodes. The mentioned steps need to be performed on DUAS sever
- Load node environment: $NODE_INSTALL_DIR\unienv.(bat|ksh)
- Check connectivity to UNIMS: $NODE_INSTALL_DIR\bin\unims -checkms
- Stop the node : $NODE_INSTALL_DIR\bin\unistop
- Update the node hostname: $NODE_INSTALL_DIR\bin\unims -update -host NEW_FQDN_OF_THE_NODE
- Start the node: $NODE_INSTALL_DIR\bin\unistart
Additional Information
Feedback
