When setting the maximum number of tokens per client  as documented in :

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-management-oauth-toolkit/4-3/installation-workflow/configure-authentication/token-configuration.html

-  Set the Maximum Token Count section 

We still  can request more than 5 tokens while the "max_oauth_token_count" set to 5 and "max_oauth_token_behaviour" set to error

Under which circumstances is this count applied? Is it per OTK session, per issued token?

The max token count applies to the combination resource owner and  client credentials . The Client credentials grant type does not require a resource owner as it acts on its own behalf meaning to get a token you only need to provide client credentials knowing the ID and the Secret.

The max token value does only for work for the combination resource owner/ client credentials . and there fore does not work with the client credential authentication flow.