If an invalid SQL instance name is specified in the 'Server name' field in the sqlserver probe connection setup, the connection test reports that the connection is OK, while it should actually fail.
(If the port number is not provided, it will fail)
The issue is not on the probe but is with Microsoft SQL server itself. The sqlserver probe uses the sqlserver related API's to connecto to the server. The MSSQL allows the connection even if the instance name is not correct.
This can be proved by using MS SQL Server Management Studio and opening a connection to the same database with a wrong instance:
in MS SQL Management Studio:
Connect to DB instance: