Gen 8.6 Toolset automation "No such interface" 0x80004002 errors

Products

• Gen - Workstation Toolset
• Gen

Issue/Introduction

We're having an issue attempting to use the Gen 8.6 Toolset Automation via COM/OLE in standalone (non-Gen) projects. 
FAIL1 - Windows 10 (21H2)
FAIL2 - Windows 11 (22H2)
WORK1 - WIndows Server 2019 (1809)  

On 1 machine (WORK1), elementary VB.NET code and PowerShell scripts work and work consistently.  The VB.NET code works in the VS IDE and also standalone.

On 2 machines (FAIL1 and FAIL2), the VB.NET code copied from the working machine fails, and the PowerShell scripts are intermittently successful - sometimes they attach to the running Toolset, sometimes they start a new one in the background.

The powershell solution isn't viable as it's part of a larger product offering.

WORK1 and FAIL1 have the same PTF levels - WKS86200 + up to RTN86224, TSN86224, CCN86203, RTJ86211, RTA86205, CSN86214, BTN86210.

FAIL2 has the PTF level of WKS86200 + WKS86300.

The error received in the VB.Net code looks like:
System.InvalidCastException
  HResult=0x80004002
  Message=Unable to cast COM object of type 'System.__ComObject' to interface type 'Coolgen.Toolset'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{87726FE3-F382-11D3-80BD-00C04F2EF99D}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
  Source=ToolsetTest
  StackTrace:
   at ToolsetTest.Module1.Main() in D:\Local\ToolsetAutomation\VSProj\ToolsetTest\Module1.vb:line 6

Steps to reproduce from Visual Studio:

• Create a New Console App (.NET Framework), targeting Visual Basic (VB)
• Right click project => Properties
• Select References Tab
• Click Add...
• Choose Browse...
• Navigate to "%Gen86%\Gen" folder
• Choose TOOLSET.exe
• Add Test Code
• Run it
• Failure on New Coolgen.Toolset line ...

The VB.Net code is:
*****
Sub Main()

    Console.WriteLine("Initializing COM object")
    Try
      Dim toolset = New Coolgen.Toolset()

      Console.WriteLine("Getting model count")
      Dim count = toolset.Models.Count
      Console.WriteLine("Model count: " + count.ToString())
      If count > 0 Then
        Dim m = toolset.Models.Item(1)
        Dim aw = m.ActiveWindow
        Dim caption = aw.Caption
        Console.WriteLine("Open window is ... '" + caption + "'")
      End If
      toolset = Nothing
    Catch e As System.InvalidCastException
      Console.WriteLine(e)
    End Try
    Console.WriteLine("Done!")

End Sub
*****

The PowerShell script (run on PowerShell 7.2.7 on all 3 boxes) is:
*****
function logIt( [string]$value ) {
  $dt = get-date -f "yyyy-MM-dd HH:mm:ss.fff"
  Write-Host "$dt $value"
}
logIt "Creating toolset object"
$toolset = New-Object -ComObject COOLgen.Toolset
logIt "COM methods"
$toolset | GM
logIt "Is Toolset visible?"
$toolset.Visible() # Returns true
logIt "How many models?"
$toolset.Models().Count() # Returns 1
logIt "Caption of the active window?"
$toolset.Models().Item(1).ActiveWindow().Caption() # Returns 'AB_CALLING_AB_IN_ZLIB - Action Diagram'
logIt "Cleaning up COM object"
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($toolset)
logIt "Done!"
*****

Environment

Release : 8.6
Gen 8.6 Toolset automation.

Cause

Gen L1 Support and Engineering could not recreate the problem.
It was subsequently found that the registry keys for the Toolset automation/OLE/COM had been corrupted in some way i.e. some keys were completely missing while others had missing values.
For example on the failing machines for the error message IID 87726FE3-F382-11D3-80BD-00C04F2EF99D this key had missing (Default) value for ProxyStubClsid32 when it should match the (Default) value for ProxyStubClsid i.e. the {00020424-0000-0000-C000-000000000046} below was missing:
===
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\Interface\{87726FE3-F382-11D3-80BD-00C04F2EF99D}\ProxyStubClsid
    (Default)    REG_SZ    {00020424-0000-0000-C000-000000000046}

HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\Interface\{87726FE3-F382-11D3-80BD-00C04F2EF99D}\ProxyStubClsid32
    (Default)    REG_SZ    {00020424-0000-0000-C000-000000000046}
===
After Gen L1 Support removed the above (Default) value for ProxyStubClsid32 they could also recreate the same error.

Resolution

Gen Engineering confirmed that it is only the base Gen 8.6 install that adds/updates registry keys for the Toolset automation/OLE/COM and PTFs do not do not update registry updates. Customer also confirmed they are using the standard Gen installer for the base install and not a custom version.
Engineering also have no suggestions on how the keys could have become partially corrupted.
The presence of antivirus software was considered a possible factor but the same Windows Defender is present on the working machine as one of the failing machines, so that is inconclusive.
To fix the problem either performing registry repairs or a Gen 8.6 uninstall/install would seem to be the only options. Unless a Gen 8.6 uninstall/install causes the same problem then the root cause may just have to be put down to transient environment problems.

The root cause could not be found and after comparing registry keys on working v failing machines in detail, registry key repairs were performed with a PowerShell script.

Additional Information

Microsoft Developer Network - Archived Forums 441-460 > Visual Studio Tools for Office (VSTO)