This issue occurs if you are building your perspective schema and changing the ref_ids with each post to the API.
You'll most likely see these duplicated empty groups added to the bottom of the perspective grouping list. They will have the same group name as the original group that was created for that perspective.
To fix this issue, you should be able to call your schema ahead of time to get the existing group ref_ids, which you will use to continue updating/deleting the desired group.