A SCIM user that was provisioned cannot be removed even it was removed from the provisioned group.
4.x, 5.x
This is an expected behavior. The SCIM user has a grace period of 30 days after it is removed from provisioning in IDP side (it is not actually deleted, but disabled). After 30 days, the Admin will be able to delete the user. In these 30 days the user could be restored by adding it to provisioning again.