VMware VirtualCenter Server service crashes and fails to start with the vpxd.log error: panic NOT_REACHED
search cancel

VMware VirtualCenter Server service crashes and fails to start with the vpxd.log error: panic NOT_REACHED

book

Article ID: 305797

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides a workaround for the VirtualCenter Server service startup problem.

Symptoms:
  • The VirtualCenter Server service fails to start with event ID 1000.
  • Re-installation does not solve the issue.
  • ODBC connection is OK and uses SQL server authentication.
  • The vpxd.log contains entries similar to:

    [14:25:06.220 04812 error 'App'] Found group with invalid name 001 - color services - BLACK / WHITE. Removing /vpx/group/#v2555/
    [14:25:06.220 04812 warning 'Libs'] ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdInvtGroup.cpp:309
    [14:25:06.220 04812 error 'App'] [VpxVmomi] Double register of key: Datacenters
    [14:25:12.533 04812 info 'App'] CoreDump: Writing minidump
    [14:25:15.455 04812 panic 'App']

    Panic: NOT_REACHED d:/build/ob/bora-345043/bora/vpx/common/vpxVmomi.cpp:903

    Backtrace:
    backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
    backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFileWriter
    backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace
    backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
    backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
    backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
    backtrace[06] rip 00000001402e5793 (no symbol)
    backtrace[07] rip 0000000140b40dd9 (no symbol)
    backtrace[08] rip 0000000074004830 tanhf
    backtrace[09] rip 0000000073fd3b3c _is_exception_typeof
    backtrace[10] rip 0000000076e0f001 RtlRestoreContext
    backtrace[11] rip 000000014031bc88 (no symbol)
    backtrace[12] rip 000000013f9b9537 (no symbol)
    backtrace[13] rip 000000013f9bcbc5 (no symbol)
    backtrace[14] rip 000000013f9bdb4c (no symbol)
    backtrace[15] rip 000000013f905662 (no symbol)
    backtrace[16] rip 000000013f900ab8 (no symbol)
    backtrace[17] rip 000000013f8fbb4f (no symbol)
    backtrace[18] rip 000007feff0ba82d RegisterServiceCtrlHandlerExA
    backtrace[19] rip 0000000076cbf56d BaseThreadInitThunk
    backtrace[20] rip 0000000076df2cc1 RtlUserThreadStart

    [14:25:15.455 04812 panic 'App'] NOT_REACHED d:/build/ob/bora-345043/bora/vpx/common/vpxVmomi.cpp:903

    Or eventually :

    [14:26:10.626 01044 error 'App'] Found group with invalid name My generic folder name . Removing /vpx/group/#v2590/
    [14:26:10.626 01044 warning 'Libs'] ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdInvtGroup.cpp:309
    [14:26:10.626 01044 error 'App'] [VpxVmomi] Double register of key: Datacenters
    [14:26:17.095 01044 info 'App'] CoreDump: Writing minidump
    [14:26:19.564 01044 panic 'App']

    Panic: NOT_REACHED d:/build/ob/bora-345043/bora/vpx/common/vpxVmomi.cpp:903

    Backtrace:
    backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
    backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFileWriter
    backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace
    backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
    backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
    backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
    backtrace[06] rip 0000000140275793 (no symbol)
    backtrace[07] rip 0000000140ad0dd9 (no symbol)
    backtrace[08] rip 00000000740b4830 tanhf
    backtrace[09] rip 0000000074083b3c _is_exception_typeof
    backtrace[10] rip 0000000076e0f001 RtlRestoreContext
    backtrace[11] rip 00000001402abc88 (no symbol)
    backtrace[12] rip 000000013f949537 (no symbol)
    backtrace[13] rip 000000013f94cbc5 (no symbol)
    backtrace[14] rip 000000013f94db4c (no symbol)
    backtrace[15] rip 000000013f895662 (no symbol)
    backtrace[16] rip 000000013f890ab8 (no symbol)
    backtrace[17] rip 000000013f88bce6 (no symbol)
    backtrace[18] rip 000000013f88c0c4 (no symbol)
    backtrace[19] rip 000000013f88adf2 (no symbol)
    backtrace[20] rip 0000000076cbf56d BaseThreadInitThunk
    backtrace[21] rip 0000000076df2cc1 RtlUserThreadStart

    [14:26:19.564 01044 panic 'App'] NOT_REACHED d:/build/ob/bora-345043/bora/vpx/common/vpxVmomi.cpp:903

    Note: For more information on log locations, see Location of vCenter Server log files(1021804).


Environment

VMware vCenter Server 4.1.x

Resolution

The issue occurs due to the presence of special characters or space in the end of the folder name, this might have been authorized by vCenter Chargeback.

Note: The parameter of the SQL query SELECT and UPDATE need to be adapted to your case and environment. If you are unsure, contact your DBA.

To work around the issue :
  1. Open command prompt and start vCenter Server with this command:

    vpxd.exe -s ,

    Note: The vpxd.exe should be located in Program Files\Vmware\Infrastructure\Virtual Center
  2. Open Microsoft SQL Server Management Studio and connect to the vCenter Server database.

    Note: Ensure you have a backup of you database before continuing further. Inaccuracies when executing the query could cause damage to the database.

  3. Right-click on the database and select New Query.
  4. Use the logs from vpxd to find the culprit entity ID:

    For example:

    - in "[yyyy-mm-dd 14:25:06.220 04812 error 'App'] Found group with invalid name 001 - color services - BLACK / WHITE. Removing /vpx/group/#v2555/"
    - in "[yyyy-mm-dd 14:26:10.626 01044 error 'App'] Found group with invalid name My generic folder name . Removing /vpx/group/#v2590/"

    In this example, the culprit IDs are 2555 and 2590 respectively.

  5. Perform these steps on the query window:

    1. First ensure you are working on the right entity by executing :

      SELECT * FROM VPX_ENTITY WHERE ID = 'XXXX'

      Where the XXXX represents the culprit ID.

      In this example, the culprit ID is 2555:

      SELECT * FROM VPX_ENTITY WHERE ID = '2555'
    2. Update the name of the entity using this command.

      UPDATE VPX_ENTITY SET NAME = 'YYYY' WHERE ID = 'XXXX'

      Where the XXXX represents the culprit ID and the YYYY represents the folder name.

      In this example, the culprit ID is 2555 and 001 - color services - BLACK - WHITE is the folder name:

      UPDATE VPX_ENTITY SET NAME = '001 - color services - BLACK - WHITE' WHERE ID = '2555'

  6. Repeat step 5 on other culprit IDs.

    In this example, the other culprit ID is 2590:

    SELECT * FROM VPX_ENTITY WHERE ID = '2590'
    UPDATE VPX_ENTITY SET NAME = 'My generic folder name' WHERE ID = '2590'

    Note: You can re-run the SELECT query to ensure that the name has been updated.

  7. Start the VirtualCenter Server service. If this fails again with the same error, apply the same procedure with updated ID and NAME.


Additional Information


Location of vCenter Server log files

Powered by Wolken Software