Where Did My Logs and Configuration Go? — AAI 26.0.0 Post-Upgrade Orientation
Article ID: 453429
Updated On:
Products
Issue/Introduction
After upgrading to AAI 26.0.0, log files and configuration settings that used to live in familiar locations have moved.
Use this article to find where the file or setting you're looking for lives now, in both VM/traditional-installer and Kubernetes/OpenShift deployments.
New to terms like mTLS, keystore, or WAR? This article uses those terms without redefining them.
See: AAI Glossary — Terms Used in the 26.0.0 Upgrade Documentation
Environment
- AAI 26.0.0 and later.
- Applies to both VM/traditional-installer and Kubernetes/OpenShift deployments
— this matters throughout this article, especially for configuration (Part 2):
on a VM/traditional installer, log and configuration files live on disk and you can inspect them directly.
On Kubernetes/OpenShift, logs are per-pod and most configuration files are reset from the container image on every pod start
— editing or copying them directly inside a running pod is not reliable, and the real source of truth becomes your Helm values instead of a file. Know which one you're on before following either part below.
Cause
Not applicable. This is a post-upgrade orientation article covering log and configuration file locations, not a resolution to a specific reported problem.
Resolution
Prerequisites
- Shell access to your AAI installation (VM), or
kubectl/ocaccess to the correct namespace (Kubernetes/OpenShift)
— confirm withkubectl config current-context(oroc whoami) if you're not sure which cluster/namespace you're pointed at. - (Kubernetes/OpenShift only)
helm, already configured against the correct cluster, and your Helm release name/namespace (Part 2, §7 below)
— confirm withhelm version. - Beyond that, no specialized tools are required
— this article otherwise only uses directory listing and text search that ship with the OS.
Part 1: Logs
1. The log directory moved
| 24.4.x | 26.0.0 | |
|---|---|---|
| VM location | $AAI_HOME/jboss/standalone/log/ | $AAI_HOME/log/ |
| Kubernetes location | — | <per-component-volume>/<component>/log/<pod-name>/ |
If you're used to the old path from 24.4.x and find it empty or gone on 26.0.0, this is expected — every launcher (installer, container entrypoint, and service scripts alike) now points at the new location consistently, on both VM and Kubernetes.
2. A missing log file usually isn't a sign anything is broken
Log files in 26.0.0 are created only when something is first written to them — they don't exist just because the component that would write to them is running. A missing file most often means "that code path hasn't been hit yet," not "this component is broken" or "the log bundle is incomplete."
Most log files are pre-warmed (created empty at startup) specifically to avoid this confusion — but two are deliberately excluded from that pre-warming and will appear to be "missing" even on a perfectly healthy system until the first event that would write to them actually happens: the audit log and the prediction-engine log. Don't treat either of those as evidence of a problem on its own.
Self-check: on a VM, ls -la $AAI_HOME/log/ (Linux) or dir "$env:AAI_HOME\log" (Windows PowerShell) shows what currently exists. On Kubernetes, kubectl logs <pod> -n <namespace> is the only way to see a component's output if nothing has landed on a shared volume yet.
3. New log file for eEM cases: aai.eem.log
If you're troubleshooting an eEM-integration case ("login works but I can't see any data"), there's a new, separate log file specifically for AAI's own eEM-related logging, distinct from the eEM SDK's own log files (which still exist, unchanged, and only cover the SDK's side):
| File | What it covers | New in 26.0.0? |
|---|---|---|
aai.eem.log | AAI's own eEM-related logging | Yes |
eem.sdk.log | The eEM SDK's own log | No — pre-existing |
eem.network.log | The eEM SDK's own log | No — pre-existing |
eem.perf.log | The eEM SDK's own log | No — pre-existing |
Action: on any 26.0.0 eEM case, request all four — aai.eem.log plus the three pre-existing eem.*.log files. The three you're used to asking for no longer contain AAI's side of the story on their own, and you'll get an incomplete picture without aai.eem.log added in.
4. Five new services means five new log files
The new services introduced in 26.0.0 each have their own log file:
| Service | Log file |
|---|---|
| Frontend Gateway | frontendGateway.log |
| Denormalizer | denormalizer.log |
| MCP Service | mcp.log |
| AI Insights | see below — name depends on install type |
| Automation AI | see below — name depends on install type |
Two of them are named differently depending on whether you're on a containerized (Kubernetes/OpenShift) or standalone-installer deployment:
| Service | Containerized name | Standalone-installer name |
|---|---|---|
| AI Insights | ai-insights.log | aiinsights.log |
| Automation AI | automationAI.log | automationai.log |
Action: confirm which install type you're dealing with before asking for one of these two files by name — asking for the wrong spelling gets you a "no such file" response, not the log itself.
5. Two kinds of silence that prove nothing
A quiet server.log doesn't mean a clean system. Message-level filtering in 26.0.0 is broader than it used to be — certain error text (including things like specific HTTP status codes or file-lock errors) is filtered out by default. Grepping for one of those specific messages and finding nothing is expected behavior, not confirmation that the underlying condition never happened.
Alert-volume metric lines are debug-only and are absent by default. There are specific log lines that report exactly how many alerts were generated versus filtered/suppressed in a given window — extremely useful for proving whether an alert-volume drop is expected behavior or an actual problem. But these lines are logged at debug level, and nothing has been elevated to that level on a default install. If you grep for them and get nothing, that tells you the loggers aren't enabled yet — not that no filtering happened.
How to enable them. Edit aai-jboss-log4j2.yml (in the main configuration directory covered in Part 2 of this article — the same file on both VM and Kubernetes/OpenShift, since this is the one configuration file that's genuinely persistent in both environments). Add these four entries:
- name: com.termalabs.server.alert.metrics
level: DEBUG
additivity: false
AppenderRef:
- ref: JawsFile
- name: com.termalabs.server.alert.metrics.summary
level: DEBUG
additivity: false
AppenderRef:
- ref: JawsFile
- name: com.termalabs.alert.metrics
level: DEBUG
additivity: false
AppenderRef:
- ref: AlertServiceFile
- name: com.termalabs.alert.metrics.summary
level: DEBUG
additivity: false
AppenderRef:
- ref: AlertServiceFileWhere to look afterward — this is easy to get wrong: the first two entries (the "producer" side) route to jaws.log, and the last two (the "consumer" side) route to alertService.log — these are two different files, not both the same one. Reproduce the condition, then check both:
grep "[ALERT-METRIC" $AAI_LOG/jaws.log*
grep "[ALERT-METRIC" $AAI_LOG/alertService.log*No restart is required. This file is checked for changes automatically about every 30 seconds — save it and reproduce the condition shortly after.
Alternative: if your deployment has the AAI Config Tool available, its Administration → System → Logging screen can add these same four loggers through the UI instead of hand-editing the file, and takes effect just as quickly. Caution either way: if anyone later uses that same Logging screen to change something else, it rewrites this entire file from its own model — any hand-edited entry it doesn't already know about can be silently dropped. Don't leave these enabled longer than needed for the case at hand; remove the four entries (or delete them via the Config Tool) once you've collected what you need.
6. Kubernetes: the gateway's log may not be where you expect
The Frontend Gateway's log directory is controlled by one environment variable inside the pod: AAI_LOG_DIR. The chart sets this to /var/log/aai-frontend-gateway by default (matching the persistent volume mounted at that path), but if it's ever absent from the running container, the gateway's own logging config falls back to ./logs — a relative path inside the container's own ephemeral filesystem, not the persistent volume. That fallback happens silently; nothing errors out.
Why it can actually go missing: Helm does not merge YAML lists across values files — only maps. If a customer's own deployment-values override defines its own custom environment-variable list for the gateway (aai-frontend-gateway.env), even just to add one unrelated variable, it fully replaces the chart's default list rather than appending to it, which silently drops AAI_LOG_DIR along with it. This isn't a risk with AAI's own shipped reference values files — neither overrides that list — but it's a real risk in a customer's own customized values file.
Action — check what's actually set in the running pod, not just what the chart ships by default:
kubectl exec <gateway-pod> -n <namespace> -- env | grep AAI_LOG_DIRIf that comes back empty, the gateway is writing to its container-local ./logs/frontendGateway.log instead of the persistent volume. That's not a problem for getting the content, though: the gateway's own default logging config sends every event to both the console and the file with the same pattern, so kubectl logs <gateway-pod> -n <namespace> is a reliable substitute — you just won't have the historical/rotated files, only what's still in the pod's log buffer, until whoever manages the Helm values restores the missing variable.
You can also sanity-check the chart-level default before touching a live pod:
helm get values <release-name> -n <namespace> -a | grep -A6 "^aai-frontend-gateway:"An absent env: block under that key means the chart default applies and AAI_LOG_DIR is set; a present one means you need to confirm AAI_LOG_DIR is actually one of its entries.
7. Which log to ask for, by symptom
| Symptom | Ask for |
|---|---|
| Login, routing, 403s, redirect loops | frontendGateway.log, plus kubectl logs <gateway-pod> (Kubernetes/OpenShift) |
| AI answers stale, search data missing | denormalizer.log, then the search/indexing (OpenSearch) service's own log — this is not one of AAI's $AAI_HOME/log/*.log files; it's the bundled OpenSearch engine's own output, retrieved with kubectl logs <opensearch-pod> (Kubernetes/OpenShift) or from its own process's log output (VM) |
| "The assistant can't see any data" | mcp.log |
| Dashboard build fails, CSV truncated | The AI Insights log — ai-insights.log (Kubernetes/OpenShift) or aiinsights.log (VM/standalone installer); see §4 |
| LLM/chat/model failures | The Automation AI log — automationAI.log (Kubernetes/OpenShift) or automationai.log (VM/standalone installer); see §4 |
| eEM: "login works but no data" | aai.eem.log, plus eem.sdk.log, eem.network.log, eem.perf.log (see §3) |
| Report PDF/CSV, scheduled email issues | reportingService.log |
| What-if simulation, validation issues | simulationService.log |
| mTLS handshake issues (presenting/trusting/rejected) | That specific service's own log — see the per-service table in §4 |
| mTLS truststore resolution failures | jaws.log |
Part 2: Configuration
1. There are now two configuration directories (VM), or one collapsed location (Kubernetes/OpenShift)
On a VM install, settings that used to live in one place are now split across two directories, and neither is a copy of the other:
New directory: $AAI_HOME/configuration/ | Legacy directory: $AAI_HOME/jboss/standalone/configuration/ |
|---|---|
aai.properties — main runtime settings, ports, TLS paths | standalone.xml — server listener/datasource configuration (the same file also holds the JBoss Infinispan cache subsystem — there is no separate cache config file) |
aai-jboss-log4j2.yml — server + co-hosted component logging config | application.keystore, aai-server.p12, root-ca.p12 — keystores and the internal root certificate |
eiam.config — eEM/EIAM SDK settings | |
opensearch.properties — search/indexing (OpenSearch) connection settings | |
frontend-gateway-log4j2.yml, denormalizer-log4j2.yml, mcpserver-log4j2.yml, ai-insights-log4j2.yml, automation-ai-log4j2.yml — one logging config file per external service |
A JVM options file relevant to heap/GC tuning sits at the installation root — in neither of the two directories above: aai.sh.vmoptions on Linux, aai.vmoptions on Windows.
On Kubernetes/OpenShift, both directories collapse into a single shared location, organized per component. Most files there are read directly from the container image and get reset to the image's version on every pod start — a copy living on persistent storage is not reliable evidence of what's actually in effect. Configuration in this environment is shared per component (every replica of that component uses the same values), which is a different sharing model than logs, which are per pod.
2. Which file holds what, and who's allowed to edit it
Getting this wrong is the single most common cause of "I changed a setting and nothing happened":
| What you're trying to change | Where it lives | Should you hand-edit it? |
|---|---|---|
| Ports, TLS paths, main runtime settings | aai.properties | Yes — the old UI screen for this is gone (see Pre-Upgrade Checklist for AAI 26.0.0) |
| Server/co-hosted component logging | aai-jboss-log4j2.yml | No — if you use the logging-configuration screen in the admin tool, it rewrites this file completely from its own model; any setting you hand-edited that the tool doesn't know about gets silently dropped the next time you save there |
| eEM/EIAM SDK settings | eiam.config | Installer-managed |
| Search/indexing connection settings | opensearch.properties | Managed through the upgrade dialog |
| Per-external-service logging | frontend-gateway-log4j2.yml, denormalizer-log4j2.yml, mcpserver-log4j2.yml, ai-insights-log4j2.yml, or automation-ai-log4j2.yml, depending on the service | Yes, hand-edited |
3. The logging configuration file was renamed — and the old copy was deleted, not left behind
If you're used to the 24.4.x logging configuration file's old location — jboss/standalone/configuration/log4j2.yml — know that the upgrade doesn't just add a new copy at the new location (configuration/aai-jboss-log4j2.yml) — it deletes the old one outright. If documentation or a colleague tells you "it was renamed," don't go looking in jboss/standalone/configuration/ expecting a compatibility copy to still be there for reference; there isn't one, on either a VM or container build.
The eEM/EIAM settings file (eiam.config) also moved, and the line inside it that points to the logging config file is a path relative to the current working directory, not relative to eiam.config's own new location — worth knowing if that reference ever looks broken after an upgrade.
4. On upgrade, read the upgrade log to know what happened to your main settings file — don't just look at the file
This is a VM/traditional-installer-only step — there is no equivalent upgrade log on Kubernetes/OpenShift, since a Helm upgrade doesn't merge a properties file at all (§5 below).
Name and location — this trips people up: the upgrade log is install/log/upgrade.log, but that path is relative to wherever you extracted the upgrade bundle before running upgrade.sh/upgrade.bat — a temporary staging directory you chose, not $AAI_HOME. It is not copied into the running installation's own log/ directory afterward. If you've already deleted or moved that staging directory, the upgrade log is gone with it — there's no second copy to fall back on.
aai.properties goes through an automatic merge during upgrade that only adds settings you didn't already have — it never overwrites something you'd already set. Grep the upgrade log for [Properties] (or [mergeProperties] for line-by-line key detail) to see which of these outcomes actually happened:
aai.properties not found; copying shipped file.— a brand-new file was created (nothing existed before).Backed up aai.properties to aai.properties.bak.<yyyyMMdd_HHmmss>followed later byRemoved aai.properties.bak.* after successful merge.— your existing file was backed up, merged, and the backup was then deleted as a cleanup step.No prior aai.properties.bak.yyyyMMdd_HHmmss in <dir> — skipping template merge and overlay (fresh aai.properties).— the merge did not run at all — in which case, some setting a document tells you to expect may legitimately not be present, because nothing added it.
Linux:
grep -i "\[Properties\]\|\[mergeProperties\]" install/log/upgrade.logWindows (PowerShell):
Select-String -Path install\log\upgrade.log -Pattern "\[Properties\]|\[mergeProperties\]"The backup itself, while it briefly exists, is named aai.properties.bak.<yyyyMMdd_HHmmss> and lives in $AAI_HOME/configuration/ (the same directory as aai.properties itself) — but per the second bullet above, a successful merge deletes every file matching that pattern once it's done. If you want a before/after comparison, or a way back, you need to have captured the file yourself before the upgrade — the upgrade's own backup will not be there afterward to fall back on. See Pre-Upgrade Checklist for AAI 26.0.0 for the exact backup commands.
5. Kubernetes/OpenShift: editing a file inside a running pod does nothing
If a setting change appears to have no effect at all, the first question is simply "VM or Kubernetes?" — because in a container, most configuration files are the image's own defaults, copied in at pod startup and then actively removed from persistent storage so they can't be mistaken for something durable. Editing one directly inside a pod, or expecting a copy on shared storage to persist across a restart, will not work. The one exception is the logging configuration file itself, which genuinely is backed by persistent storage as a single file. For everything else, the actual, durable source of truth is your Helm release's configured values — that's what you change, and what you should be comparing against when a setting doesn't seem to be taking effect.
If you don't already know the Helm release name or namespace (common if you're new to this cluster, or don't manage it directly yourself): every AAI component's pod name starts with aai- (aai-server, aai-frontend-gateway, aai-denormalizer, aai-mcp-service, and so on), so you can locate the namespace first, then the release name:
Linux/Windows (identical kubectl/oc syntax on both):
kubectl get pods -A | grep aai-(oc get pods --all-namespaces | grep aai- on OpenShift, or add -n <namespace> to either instead of -A/--all-namespaces if you already know the namespace but not the release name.)
Once you have the namespace from the pod list above, list the Helm release(s) in it:
helm list -n <namespace>That gives you the exact <release-name> to use in every helm get values <release-name> -n <namespace> command in this article. (The two shipped reference deployment overlays install with release name aai in namespace ao-aai-platform — a reasonable first guess if kubectl get pods -A isn't available to you for permissions reasons, but confirm it rather than assuming it, since your actual install can use any release name and namespace you chose.)
If you don't have cluster-wide list permissions (kubectl get pods -A returns a permissions error) and don't already have a namespace to scope to, you'll need to ask whoever manages the cluster for the namespace and release name directly — there's no client-side workaround for that RBAC restriction.
6. Which configuration file to check, by question
| Question | Check |
|---|---|
| What port is the gateway on? Is TLS on? | aai.properties |
| Why is this component's log missing or empty? | aai-jboss-log4j2.yml |
| Why is one external service logging differently? | That service's own logging config file — see §1/§2 above for the exact filename per service |
| eEM login works but no data | eiam.config, plus aai.eem.log |
| Which search/indexing backend, and as which user? | opensearch.properties |
| Heap, GC, JVM flags | aai.sh.vmoptions (Linux) or aai.vmoptions (Windows), at the installation root |
| Listeners, datasource, keystore wiring | standalone.xml, application.keystore, aai-server.p12, root-ca.p12 — all in the legacy jboss/standalone/configuration/ directory (unchanged from 24.4.x) |
| Anything at all on Kubernetes/OpenShift | Your Helm values — not a file inside a pod |
7. Quick self-check commands
Confirm which two configuration locations actually exist on a VM, and whether the main properties file was merged or skipped during upgrade:
Linux:
ls -la $AAI_HOME/configuration $AAI_HOME/jboss/standalone/configuration
grep -i "properties" <your-upgrade-log-file>Windows (PowerShell):
Get-ChildItem "$env:AAI_HOME\configuration","$env:AAI_HOME\jboss\standalone\configuration"
Select-String -Path <your-upgrade-log-file> -Pattern "properties"On Kubernetes/OpenShift, confirm what your release is actually configured with — this is the file to compare against, not anything inside a pod:
helm get values <your-release-name> -n <your-namespace>Additional Information
- What to Know Before Upgrading to AAI 26.0.0 — broader background on what changed architecturally in 26.0.0.
- Pre-Upgrade Checklist for AAI 26.0.0 — the pre-upgrade configuration backup steps referenced in Part 2, §4 above
- To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on the respective region.
Feedback
