While creating a new project from Template, it is possible to filter by Template = 'All' or Template = 'No', showing projects that are not templates.
This is working as intended.
However, it is possible to limit the projects displayed to Templates only:
The resulting portlet will only allow to filter by Templates and even clicking on Show All button will still filter by Template = 'Yes'.
This change only applies to the user that makes the changes. If you want to apply to all users, you will need to do the same in the Administration > Objects > Project > Views > Project Templates / List Filter [Fields] and apply the same steps.
After this change is made it will be necessary to Publish, as anyone with a customized views will override the published version. Please note that Publishing will not only publish the Project Template List view, but all Project views for all users.