The VMware SGVA 3D driver for Windows Vista or later guest operating systems in the VMware Tools version that comes with Workstation 11 limits the screen resolution to 2560x1920 in the virtual machine.
If the host supports resolutions higher than 2560x1920, you cannot configure the screen resolution of the virtual machine to those higher resolutions.