Post OI 23.3 Upgrade Issues

Products

• DX Operational Intelligence

Issue/Introduction

Completed a 23.1 to 23.3 on-premise upgrade (up to HF03).  Most things work but having several issues that all seem to potentially be networking- and/or pod/service communication- related. A couple of the settings panels are returning a 404 nginx error page. Consistently getting "data fetch" errors from things like /oi/v2/... url's at the bottom right corner, and we're seeing a weird error trying to communicate to Remedy from the incident management pod. These might be all independent and/or they might be related to a common network config root cause.

Resolution

The problem is that the default value for NOTIFY_FILTER_PORT inside the pod was incorrect.  The value was in the format tcp://xxx.xxx.xxx.xxx:8090 where it should only be the port number

 

The resolution was to add an environment variable manually to the incident management deployment as follows:

 

NOTIFY_FILTER_PORT: "8090"

 

 

 As of now, it seems we do not require any further changes to the configuration.

 

 

Additional Information

NOTIFY_FILTER_PORT env variable default value is now configured within the image itself. Looks like Kubernetes is creating default env variables for each pod. Following are the examples collected from the 24.1 environment. In this case, the Kubernetes default env variable name, the env variable name that we have defined in incident management for NOTIFY_FILTER_PORT matched.

 

Because of the default env variable from Kubernetes, we got a different value than what we expected, which is causing the issue here.  

 

doi-readserver-<servername>:/opt$ env | grep -i doi | grep -i admin
DOI_ADMINUI_PORT_8080_TCP_PROTO=tcp
CPA_SERVICE_ADMIN_HOST=doi-cpa-ng
DOI_ADMINUI_URL_WITH_PROTOCOL=http://<admin_url>:8080
DOI_ADMINUI_SERVICE_HOST=<ip>
DOI_ADMINUI_SERVICE_PORT=8080
DOI_ADMINUI_PORT_8080_TCP_ADDR=<ip>
DOI_ADMINUI_PORT_8080_TCP=tcp://<ip>
DOI_ADMINUI_PORT_8080_TCP_PORT=8080
DOI_ADMIN_UI_URL=https://<adminui_url>
DOI_ADMINUI_SERVICE_PORT_ADMINUI_8080=8080
ADMINUI_ROUTE=https://<adminui_url>
DOI_ADMINUI_PORT=tcp://<ip>:8080
doi-readserver-<serverName>:/opt$ env | grep -i doi | grep -i read
HOSTNAME=doi-readserver-<servername>
DOI_READSERVER_PORT_8080_TCP_ADDR=<ip>
DOI_READSERVER_PORT=tcp://<ip>8080
DOI_READSERVER_URL=http:/<readserver_name>:8080
APM_SERVICE_NAME=<readserver_name>
DOI_READSERVER_PORT_8080_TCP=tcp://<ip>:8080
DOI_READSERVER_PORT_8080_TCP_PORT=8080
DOI_READSERVER_SERVICE_PORT_8080=8080
DOI_READSERVER_SERVICE_HOST=<ip>
DOI_READSERVER_SERVICE_PORT=8080
DOI_READSERVER_PORT_8080_TCP_PROTO=tcp
doi-readserver-557985cbc9-phk4r:/opt$ env | grep -i dxi | grep -i admin
DXI_ADMINUI_PORT=tcp://<ip>:8080
DXI_ADMINUI_PORT_8080_TCP_PORT=8080
DXI_ADMINUI_PORT_8080_TCP_PROTO=tcp
DXI_ADMINUI_SERVICE_HOST=<ip>>
DXI_ADMINUI_PORT_8080_TCP_ADDR=<ip>
DXI_ADMINUI_PORT_8080_TCP=tcp://<ip>1:8080
ESS_SERVER_SERVICE_URL=http:/<adnin_ui host>:8080
DXI_ADMINUI_SERVICE_PORT_DXI_ADMINUI_PORT_8080=8080
DXI_ADMINUI_SERVICE_PORT=8080
ESS_SERVER_URL=https://dxi-adminui.<ip>.nip.io
doi-readserver-<servername>:/opt$ env | grep -i dxi | grep -i read
DXI_READSERVER_SERVICE_PORT_DXI_READSERVER_PORT_8080=8080
DXI_READSERVER_PORT_8080_TCP=tcp://<ip>:8080
DXI_READSERVER_PORT_8080_TCP_ADDR=<ip>
DXI_READSERVER_PORT=tcp://<ip>:8080
DXI_READSERVER_PORT_8080_TCP_PORT=8080
DXI_READSERVER_PORT_8080_TCP_PROTO=tcp
DXI_READSERVER_SERVICE_PORT=8080
DXI_READSERVER_SERVICE_HOST=<ip>
READSERVER_HOST=<read_server>:8080

<readeserver>:/opt$ env | grep -i notify_filter
NOTIFY_FILTER_PORT_8090_TCP=tcp://<ip>:8090
NOTIFY_FILTER_PORT=8090
NOTIFY_FILTER_SERVICE_PORT=8090
NOTIFY_FILTER_SERVICE_HOST=<ip>
NOTIFY_FILTER_PORT_8090_TCP_ADDR=<ip>
NOTIFY_FILTER_SERVICE_PORT_8090=8090
NOTIFY_FILTER_HOST_NAME=<notify_filter>
NOTIFY_FILTER_PORT_8090_TCP_PORT=8090
NOTIFY_FILTER_PORT_8090_TCP_PROTO=tcp
NOTIFY_FILTER_REQUEST_PROTOCOL=http