If you're running Windows 2008 Server Core on your VMWare ESX system you also need to install VMWare tools as this upgrades drivers etc.

Here's how I did it:

1. Log in to your Windows 2008 Server Core virtual machine using the administrator account
2. From your VI client right click the virtual machine and choose: "Install/Upgrade VMWare tools". This will mount the VMWare tools CD in your virtual machine.
3. From the command line in your VM switch to the CD-drive (D: or what ever drive letter the CD is mounted as).
4. Run the following command: msiexec /i "d:\VMware Tools.msi" (Remember to substitute d: with correct drive letter)
5. Click "Next" to start installation
6. Choose "Typical" and click "Next"
7. Click "Install" to start installation
8. Click "OK" when the warning about old Internet Explorer appears
9. Click "Finish" to finish installation
10. Click "Yes" to reboot

It's important not to choose any other installation type than "Typical" as the installation will fail....

Another way to mount the VMWare Tools CD is to edit settings for the VM. Under "CD/DVD drive 1", click on "Connected" in "Device Status" and click on "Datastore ISO file" in "Device Type". Browse to local datastore/tools-isoimages and choose the "Windows.iso" file as this contains VMWare tools.

Update:

On vSphere (ESX v4.0) the 32 bit version of the tools are named: VMware Tools.msi 64 bit version of the tools are named: VMware Tools64.msi