NSX Malware Prevention - Malware Verdict Shows as UNKNOWN
search cancel

NSX Malware Prevention - Malware Verdict Shows as UNKNOWN

book

Article ID: 327763

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

To provide workaround for malware verdicts showing "UNKNOWN"

Symptoms:
  • Files submitted for malware analysis as part of NSX Malware Prevention feature are unable to be analyzed, resulting in errors and "unknown" analysis results, and in some cases, no results at all.
  • If an operator SSH into an SVM that has malware analysis provisioned and runs "docker ps", one (or some) of the containers are not in a 'healthy' status and are not running.
root@svm-cf64862d-97d9-4ea4-ba9d-f456603def2e:/var/log# docker ps
CONTAINER ID   IMAGE                                           COMMAND                  CREATED        STATUS                             PORTS                      NAMES
4826f618102d   nsx-security-monitor:current                    "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         service_security_monitor
c873b5fecd6f   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-api-backend_1
e43fc5c3d7b5   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-analysis-completion_1
36bb6a648d3f   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Restarting (1) About an hour ago       <-----                nsx-lastline-rapid_analyst-sdk-malscape-completion_1
49a685a3a42a   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-analysis-completion_2
fadb6f8009df   avbd-scan:48-2b204f3e.bionic                    "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_avbd_1
d1781450fd5f   ullar:44-f2db2f2d.bionic                        "/entrypoint.sh"         23 hours ago   Up About an hour (healthy)                                    nsx-lastline-rapid_ullar_1
bd77d4b91685   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-api-streaming_1
55ccc9bc0b2b   analyst-sdk-reverse-proxy:423-57b4a2bb.bionic   "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-reverse-proxy_1
755d04900b9c   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-health-checker_1
2480ef3868ff   memcached:1.6.21                                "memcached --memory-…"   23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_memcached_1
90434209c07c   rabbitmq:3.12.4                                 "docker-entrypoint.s…"   23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_rabbitmq_1
a000f094c82d   analyst-sdk:423-1a6c642a.bionic                 "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)                                         nsx-lastline-rapid_analyst-sdk-submission-rate-limiter_1
16351273f7d0   analyst-sdk-nginx:423-57b4a2bb.bionic           "/entrypoint.sh"         23 hours ago   Up 23 hours (healthy)              127.0.0.1:9090->8000/tcp   nsx-lastline-rapid_analyst-sdk-api_1
  • For any container not in "healthy" status, inspect the container for additional details with the command: 
    • "docker inspect --format "{{ json .State }}" <container name>"
    • eg:  "docker inspect --format "{{ json .State }}" nsx-lastline-rapid_analyst-sdk-malscape-completion_1". 
 
  • You will see an output similar to:
  
{
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 1,
    "Error": "failed to create task for container: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/36bb6a648d3f4e30ccc5a903c10893bdf39b0ece00e77eacc8419ab694c78f8a: no space left on device: unknown",
    "StartedAt": "2024-02-13T05:52:35.501103738Z",
    "FinishedAt": "2024-02-13T05:52:36.058135495Z",
    "Health": {
         "Status": "unhealthy",
         "FailingStreak": 0,
    }
}

 
  • If the state shows Status "exited", and "Restarting" is false then the service will need to be restarted to restart this container.


Environment

VMware NSX-T Data Center

Cause

  • For a period of time there was no available disk space on the SVM.
  • This will prevent the Malware Prevention service from running correctly, however when free space finally becomes available the Malware Prevention service in most cases should recover.
  • If the Malware Prevention service does not recover automatically, it may require the workaround described in this article.

Resolution

Perform workaround specified in workaround section.

Workaround:
  • The anti-malware analysis service on the SVM must be restarted:
    • SSH into the SVM appliance and issue the command: "service nsx-lastline-rapid restart"
    • This will restart the service and malware analysis should continue correctly after this.
    • Validate by issuing the command "docker ps" and validating that all containers have started.


Additional Information

Impact/Risks:
NSX Malware Prevention is not able to analyze new files for the affected SVM

Powered by Wolken Software