Measuring the Infrastructure Efficiency of Virtual Environments
Top 20 VMware Performance
Metrics You Should Care About
Category: VMware Virtualization
- ESXi 5.x host gets disconnected from vCenter Server
- Cannot connect the ESXi 5.x host to vCenter Server
- Cannot connect to ESXi 5.x host using the vSphere Client
- The hostd.log file indicates that the host is out of space:
VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device
VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device - Running the vdf -h command shows that there is sufficient space in the root file system
- The vmkernel.log file indicates that it is out of inodes:
cpu4:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.
cpu5:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full. - The output of stat -f / indicates that inodes are low
Cause
This issue is caused by the hardware monitoring service (sfcdb) that populates the /var/run/sfcb directory with over 5000 files.
Resolution
To resolve this issue, you must free up inodes.
To free up inodes:
- Connect to the ESXi host using SSH.
- Run this command to stop the sfcbd service:
/etc/init.d/sfcbd-watchdog stop
- Manually delete the files in /var/run/sfcb directory to free inodes.
To remove these files, run this command:
# cd /var/run/sfcb
# rm [0-2]*
# rm [3-6]*
# rm [7-9]*
# rm [a-c]*
# rm [d-f]* - Run this command to restart the sfcbd service:
/etc/init.d/sfcbd-watchdog start
Source: VMware KB
This a GUI that allows you to Backup ESXi 5.0 or ESXi 5.1 hosts to a destination of your choice on a local drive. It also allows you to restore ESXi Host configuration bundle backups taken, to other Hosts.
The main improvement introduces a function in the script which backs the GUI to check that ESX hosts are valid before attempting to backup or restore these.
Very cool tool and useful:
vCenter server 5.x running on Windows system:
Port 80 – vCenter Server requires port 80 for direct HTTP connections. Then the port 80 redirects requests to HTTPS port 443. Of course you can directly access through the HTTPs, but this redirection can be helpfull if accessing in clear through http only.
Side Note: Make sure that you don’t conflict with IIS or other webservers, which might also use port 80. You can use NETSTAT with the “–abo” switch to determine which ports are used. Or you can also use Process Explorer tool provided by Windows Sysinternals http://technet.microsoft.com/en-us/sysinternals/bb896653
Port 389 – This is the LDAP port number for the Directory Services (DS) for the vCenter Server group. If there is another service, you might be wanting to change that. The LDAP service can run different ports. Here is the range: 1025 – 65535. The vCenter Server system needs to bind to port 389, even if you are not joining this vCenter Server instance to a Linked Mode group.
Port 443 – This is the default port where vCenter Server listens. If not already, open the firewall port. This port (443) is also used for monitoring the data tranfer from SDK clients. You might want to use other port for https. In this case use https://<ip-of-your-vcenter-server>:port-number. Example : https://10.10.7.23:444
Port 636 – This port is used in case you’re using vCenter Server in Linked Mode, where this is the SSL port of the local instance. In case you’re conflicting with another port, you might want to change the other port to different one. If another service is running on this port, it might be preferable to remove it or change its port to a different port.
Port 902 – This is the default vCenter server port. vCenter server uses this port to send data to managed hosts, which also uses this port to send heartbeat (through UDP).
Port 903 – Used for diplay the VMs console. It must be opened between the vSphere Client and the hosts.
Port 8080 – This port is used for the VMware VirtualCenter Management Web Services (windows service).
Port 8443 – This port is used for the VMware VirtualCenter Management Web Services.(on SSL).
Port 60099 – Web Service change service notification port
Port 6501 – Auto Deploy Service
Port 6502 – Auto Deploy management
Port 7444 – vCenter Single Sign On HTTPS
Port 7005 – vCenter Single Sign On Base shutdown port
Port 7080 – vCenter Single Sign On HTTP port
Port 7009 – vCenter Single Sign On AJP port. For more information, see the documentation page onConfiguring VMware Tomcat Server Settings in vCenter Server 5.1
Port 9443 – vSphere Web Client HTTPS
Port 9090 – vSphere Web Client HTTP
Port 10080 – vCenter Inventory Service HTTP
Port 10443 – vCenter Inventory Service HTTPS
Port 10111 – vCenter Inventory Service Management
Port 10109 – vCenter Inventory Service Linked Mode Communication
More information at kb.vmware
The original ESXi system provided by VMware is made up of multiple software packages (currently 60 for ESXi 5.0 and 65 for ESXi 5.1) called VIBs (VMware Installation Bundles). There is e.g. one large VIB for the base system (esx-base), one for the VMware Tools (tools-light), and most of the remaining VIBs are hardware device drivers (e.g. net-e1000 or sata-ahci).
With an ESXi patch one or multiple of these VIBs are updated (or in very rare cases added). An example: The latest patch for ESXi 5.0 (ESXi-5.0.0-201209001 of Sep 2012) updates the VIBsesx-base, tools-light and misc-drivers.
ESXi patches are provided by VMware in the format of so-called Offline Bundles (in this case also called Patch Bundles) in ZIP format.
Now the most important point to understand is that these Patch Bundles do not only include the patched/updated VIBs, but all VIBs that make up an ESXi system, and in fact the latest version of them all.
Read more about this at v-front.de
- Download vCenter 5.1.0a ISO and mount it inside vCenter. If you splitted the components on several servers, use the ISO on the corrisponding server
- first of all, SSO. Open a command line, move into the folder “Single Sign On” of the DVD, and run:
1VMware-SSO-Server.exe/S/v" /L*v \"%temp%\vim-sso-msi.log\" /qn"You can then open %temp% folder in Windows Explorer and monitor the automatic upgrade from here. You will find the log file and a temporary sub-directory. When that directory will disappear, the installation will be completed. Check the log to be sure the upgrade was completed successfully.
- After SSO, proceed with Inventory Service. You can run the installer from the Autorun interface of the DVD. It will find out you already have Inventory Service in place and offer you to upgrade. Obviously choose YES, and wait for the upgrade to finish. There will be no interactive screen to deal with.
- Third component, Web Client. I usually like to install it even before vCenter, since it’s the only graphical interface to SSO, so if something is not working you could use it to manage SSO. As before, it will automatically discover the previous version and will upgrade it.
- Fourth, vCenter Server. Even here, it will automatically discover the previous version and will upgrade it
- Last one, Update Manager, if you have installed it.
A new version of the vSphere vCenter Server has been released to address some of the issues encountered with various scenarios of deploying and operating the 5.1.0 version. This release is a full version and you can upgrade from 4.0-5.1GA to 5.1.0A
vCenter Server™ Appliance 5.1.0a | 25 Oct 2012 | Build 880472
What’s New
This release of VMware vCenter Server 5.1.0a includes a number of bug fixes that have been documented in theResolved Issues section.
Upgrades for This Release
For instructions about upgrading vCenter Server and ESX/ESXi hosts, see the vSphere Upgrade documentation.
Resolved Issues
- This release resolves issues in the following subject areas.
Upgrade and Installation
- vCenter Server Services fail on startup after upgrade to vCenter Server 5.1
After you upgrade to vCenter Server 5.1, the vCenter Server services fail to start up and generate vpxd dumps. This issue occurs if old certificates are removed during upgrade of vCenter Server or if you attempt to perform a fresh install of vCenter Server with an already upgraded vCenter Server 5.1 database.This issue is resolved in this release. If you have encountered this issue in the past, upgrading to vCenter Server 5.1.0a will resolve the issue.
- Users unable to log in to vCenter Server after upgrading to vCenter Server 5.1
After upgrading to vCenter Server 5.1, you might be unable to log in with the user or group administrator privileges provided by the vCenter Server installer during the upgrade process. This issue can occur if vCenter Server finds a vCenter Server Administrator user or group that can be authenticated against vCenter Single Sign On while searching the vCenter Server database.This issue is resolved in this release.
Invalid solution certificate. Certificate already expired.error message occurs when upgrading to vCenter Server 5.1 and registering the vCenter Server instance to vCenter Single Sign On
The vCenter Server installer does not allow upgrade if the certificate is invalid or already expired. To regenerate expired SSL certificates, see KB 1009092: Regenerating expired SSL certificates.This issue is resolved in this release.
- vCenter Server takes an unusually long time to start and the vSphere Client might time out
When a large number of permissions are assigned to objects in the vCenter Server inventory, the vCenter Server service does not start as quickly as expected as vCenter Server verifies that the users and groups exist in the identity source. Also, the connection to the vSphere Client might time out when you log in with Windows session credentials.The following messages appear in the vCenter Server logs while the service is starting:
[SSO] [SsoAdminFacadeImpl] [FindGroup]
[UserDirectorySso] GetUserInfo (DOMAIN\ *USER OR GROUP*, true) res: DOMAIN\ *USER OR GROUP*
[UserDirectorySso] NormalizeUserName (DOMAIN\ *USER OR GROUP*, false) re: DOMAIN\ *USER OR GROUP*This issue is resolved in this release.
- Upgrading to vCenter Server 5.1 might fail with error 29107 even though the service or solution user is already registered
Attempts to upgrade vCenter Server 5.1 might fail with the following error message:Error 29107. The service or solution user is already registered. Check VM_ssoreg.log in system, temporary folder for details.
This issue is resolved in this release.
- Installing or upgrading vCenter Server 5.1 fails with the error:
Could not contact Lookup Service
vCenter Simple Install might fail with the following error if the Fully Qualified Domain Name (FQDN) contains the string port during the installation:Error 29102 . Could not contact Lookup Service. Please check VM_ssoreg.log in system temporary folder for details.
This issue is resolved in this release.
- Installing vCenter Single Sign On fails with the error:
Unable to create database users: Password validation failed
When you attempt to install vCenter Single Sign On using an existing database or the bundled SQL Express database, the installation fails with the following error:Error 29114. Cannot connect to DB
This issue is resolved in this release.
vCenter Server and vSphere Client
- Unable to log in to the vSphere Web Client due to incorrect proxy settings
Slow network speeds caused by incorrect proxy settings might result in delay during the interaction between the vSphere Web Client, domain controller, and external vCenter Single Sign On. You might encounter the following issues due to this delay:- Attempts to log into the vSphere Web Client fail with the following error:The vSphere Web Client cannot connect to the vCenter Single Sign On server.
- A delay in excess of 20 minutes to list all domain users
- A delay in excess of 20 minutes to load the whole vCenter Server inventory after domain use login
This issue is resolved with this release.
vMotion and Storage vMotion
- vCenter Server Services fail on startup after upgrade to vCenter Server 5.1
- Unable to access the cross-host Storage vMotion feature from the vSphere Web Client with an Essentials Plus license
If you start the migration wizard for a powered on virtual machine with an Essentials Plus license, the Change both host and datastore option in the migration wizard is disabled, and the following error message is displayed:Storage vMotion is not licensed on this host.
To perform this migration without a license, power off the virtual machine.This issue is resolved in this release
Source: blogs.vmware.com
Usually the patching of VMware vSphere Cluster is done through vSphere Update Manager (VUM), but for this you need vCenter. In case you’re managing single ESXi host without vCenter, you might be interested to know how to patch the ESXi host from the command line.
No need to say that I’m supposing that you don’t use vCenter in purpose, and so that you want to patch your ESXi 5 standalone host or that you want to patch VMware ESXi 5 Free version without VUM.
Otherwise it’s much more convenient using VUM which can automate the patching process by using vMotion to move the running VMs to other hosts inside of the cluster, putting the host which you want to patch into maintenance mode and do the restart.
The patching of VMware ESXi 5 is quite straightforward process, but there is one condition: SSH access
The SSH access can be easily enabled by using the VMware vSphere client through the GUI.
How to patch ESXi 5:
Step 0 – Enabling SSH access to the VMware ESXi 5 host
Yes, there is step 0 actually. How to do it? Simple:
– Through vSphere client click on the VMware ESXi 5 server > Confiure > Security Profile (services)
– SSh > Options >
Step 1 – Stage the ESXi 5 patches to a Datastore accessible by the VMware ESXi 5 host
Via the vSphere Datastore browser upload the patch which you have downloaded from VMware Patch Website. This website lists all VMware ESXi Patches.
During the copy operation which can take some time, you have plenty of time to gracefully shut down all your VMs which runs at the moment on your host.
I have created a sub-folder for that purpose.
Step 2 – Connect via SSH by using putty for example
Putty is the most known SSH client.
Run this command and wait, because the decompression and installation take some time……
esxcli software vib install -d /vmfs/volumes/<YOUR VOLUME>/<NAME OF PATCH FILE>.zip
In my case, I executed this command:
The process will probably take some time.
You should see this screen confirming the installation and requiring the reboot of the host. You can click to enlarge.
Step 3 – Reboot your VMware ESXi 5 host
Now when you reboot, the system will probably complain that the maintenance mode hasn’t been enabled….
And you should see the version number change on your VMware ESXi 5 host.
Source: vladan.fr
VMware vSphere 4.x did introduced virtual hardware v. 7 and VMs with 256 Gigs of RAM. Now with VMware vSphere 5 it is possible to configure VMs with 1TB of RAM.. Those VMs are rather Monster VMs. Together with up to 32 virtual CPUs this is quite huge.
Another new feature in the new Virtual hardware v 8 is the 3D graphics for Windows Aero support and run some basics 3D applications.. Cool.
Here are all the new features introduced in Virtual hardware v. 8:
– 32-way virtual SMP. ESXi 5.0 supports virtual machines with up to 32 virtual CPUs,which lets you run larger CPU-intensive workloads on the VMware ESXi platform.
– 1TB virtual machine RAM. You can assign up to 1TB of RAM to ESXi 5.0 virtual machines.
– Non hardware accelerated 3D graphics for Windows Aero support. ESXi 5.0 supports 3D graphics to run Windows Aero and Basic 3D applications in virtual machines.
– USB 3.0 device support. ESXi 5.0 features support for USB 3.0 devices in virtual machines with Linux guest operating systems. USB 3.0 devices attached to the client computer running the vSphere Web Client or the vSphere Client can be connected to a virtual machine and accessed within it. USB 3.0 devices connected to the ESXi host are not supported at this time.
– UEFI virtual BIOS. Virtual machines running on ESXi 5.0 can boot from and use the Unified Extended Firmware Interface (UEFI).
The upgrade process is pretty straight forward. Upgrade your VM tools first, and then upgrade the Virtual Hardware to version 8. Note that there is no way to downgrade back, so If you’re not sure, make a snapshot before right-clicking that VM…. -:).
You can see the upgrade process on the screenshot below:
And as you can see, after the upgrade the memory limit was lifted…to 1 Tb..
Source: vladan.fr
kb4you - A Virtualization World 