Category: Virtualization

Measuring the Infrastructure Efficiency of Virtual Environments

Top 20 VMware Performance
Metrics You Should Care About

  • ESXi 5.x host gets disconnected from vCenter Server
  • Cannot connect the ESXi 5.x host to vCenter Server
  • Cannot connect to ESXi 5.x host using the vSphere Client 
  • The hostd.log file indicates that the host is out of space:

    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device
    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device

  • Running the vdf -h command shows that there is sufficient space in the root file system
  • The vmkernel.log file indicates that it is out of inodes:

    cpu4:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.
    cpu5:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.

  • The output of stat -f / indicates that inodes are low


This issue is caused by the hardware monitoring service (sfcdb) that populates the /var/run/sfcb directory with over 5000 files.



To resolve this issue, you must free up inodes.
To free up inodes:
  1. Connect to the ESXi host using SSH.
  2. Run this command to stop the sfcbd service:

    /etc/init.d/sfcbd-watchdog stop

  3. Manually delete the files in /var/run/sfcb directory to free inodes.

    To remove these files, run this command:

    # cd /var/run/sfcb
    # rm [0-2]*
    # rm [3-6]*
    # rm [7-9]*
    # rm [a-c]*
    # rm [d-f]*

  4. Run this command to restart the sfcbd service:

    /etc/init.d/sfcbd-watchdog start


Source: VMware KB

This a GUI that allows you to Backup ESXi 5.0 or ESXi 5.1 hosts to a destination of your choice on a local drive. It also allows you to restore ESXi Host configuration bundle backups taken, to other Hosts.

The main improvement introduces a function in the script which backs the GUI to check that ESX hosts are valid before attempting to backup or restore these.

Very cool tool and useful:

ESXi Host Backup & Restore GUI Utility

vCenter server 5.x running on Windows system:

Port 80 – vCenter Server requires port 80 for direct HTTP connections. Then the port 80 redirects requests to HTTPS port 443. Of course you can directly access through the HTTPs, but this redirection can be helpfull if accessing in clear through http only.

Side Note: Make sure that you don’t conflict with IIS or other webservers, which might also use port 80. You can use NETSTAT with the “–abo” switch to determine which ports are used. Or you can also use Process Explorer tool provided by Windows Sysinternals

Port 389 – This is the LDAP port number for the Directory Services (DS) for the vCenter Server group. If there is another service, you might be wanting to change that. The LDAP service can run different ports. Here is the range:  1025 – 65535.  The vCenter Server system needs to bind to port 389, even if you are not joining this vCenter Server instance to a Linked Mode group.

Port 443 – This is the default port where vCenter Server listens. If not already, open the firewall port. This port (443) is also used for monitoring the data tranfer from SDK clients. You might want to use other port for https. In this case use https://<ip-of-your-vcenter-server&gt;:port-number. Example :

Port 636 – This port is used in case you’re using vCenter Server in Linked Mode, where this is the SSL port of the local instance. In case you’re conflicting with another port, you might want to change the other port to different one. If another service is running on this port, it might be preferable to remove it or change its port to a different port.

Port 902 – This is the default vCenter server port. vCenter server uses this port to send data to managed hosts, which also uses this port to send heartbeat (through UDP).

Port 903 – Used for diplay the VMs console. It must be opened between the vSphere Client and the hosts.

Port 8080 – This port is used for the VMware VirtualCenter Management Web Services (windows service).

Port 8443 – This port is used for the VMware VirtualCenter Management Web Services.(on SSL).

Port 60099 – Web Service change service notification port
Port 6501 – Auto Deploy Service
Port 6502 – Auto Deploy management
Port 7444 – vCenter Single Sign On HTTPS
Port 7005 – vCenter Single Sign On Base shutdown port
Port 7080 – vCenter Single Sign On HTTP port
Port 7009 –  vCenter Single Sign On AJP port. For more information, see the documentation page onConfiguring VMware Tomcat Server Settings in vCenter Server 5.1
Port 9443 – vSphere Web Client HTTPS
Port 9090 – vSphere Web Client HTTP
Port 10080 – vCenter Inventory Service HTTP
Port 10443 – vCenter Inventory Service HTTPS
Port 10111 – vCenter Inventory Service Management
Port 10109 – vCenter Inventory Service Linked Mode Communication


More information at kb.vmware




The original ESXi system provided by VMware is made up of multiple software packages (currently 60 for ESXi 5.0 and 65 for ESXi 5.1) called VIBs (VMware Installation Bundles). There is e.g. one large VIB for the base system (esx-base), one for the VMware Tools (tools-light), and most of the remaining VIBs are hardware device drivers (e.g. net-e1000 or sata-ahci).
With an ESXi patch one or multiple of these VIBs are updated (or in very rare cases added). An example: The latest patch for ESXi 5.0 (ESXi-5.0.0-201209001 of Sep 2012) updates the VIBsesx-basetools-light and misc-drivers.

ESXi patches are provided by VMware in the format of so-called Offline Bundles (in this case also called Patch Bundles) in ZIP format.
Now the most important point to understand is that these Patch Bundles do not only include the patched/updated VIBs, but all VIBs that make up an ESXi system, and in fact the latest version of them all.

Read more about this at

  1. Download vCenter 5.1.0a ISO and mount it inside vCenter. If you splitted the components on several servers, use the ISO on the corrisponding server
  2. first of all, SSO. Open a command line, move into the folder “Single Sign On” of the DVD, and run:
    1 VMware-SSO-Server.exe /S /v" /L*v \"%temp%\vim-sso-msi.log\" /qn"

    You can then open %temp% folder in Windows Explorer and monitor the automatic upgrade from here. You will find the log file and a temporary sub-directory. When that directory will disappear, the installation will be completed. Check the log to be sure the upgrade was completed successfully.

  3. After SSO, proceed with Inventory Service. You can run the installer from the Autorun interface of the DVD. It will find out you already have Inventory Service in place and offer you to upgrade. Obviously choose YES, and wait for the upgrade to finish. There will be no interactive screen to deal with.
  4. Third component, Web Client. I usually like to install it even before vCenter, since it’s the only graphical interface to SSO, so if something is not working you could use it to manage SSO. As before, it will automatically discover the previous version and will upgrade it.
  5. Fourth, vCenter Server. Even here, it will automatically discover the previous version and will upgrade it
  6. Last one, Update Manager, if you have installed it.


A new version of the vSphere vCenter Server has been released to address some of the issues encountered with various scenarios of deploying and operating the 5.1.0 version. This release is a full version and you can upgrade from 4.0-5.1GA to 5.1.0A


vCenter Server™ Appliance 5.1.0a | 25 Oct 2012 | Build 880472

What’s New

This release of VMware vCenter Server 5.1.0a includes a number of bug fixes that have been documented in theResolved Issues section.

Upgrades for This Release

For instructions about upgrading vCenter Server and ESX/ESXi hosts, see the vSphere Upgrade documentation.

Resolved Issues

  • This release resolves issues in the following subject areas.

    Upgrade and Installation

    • vCenter Server Services fail on startup after upgrade to vCenter Server 5.1
      After you upgrade to vCenter Server 5.1, the vCenter Server services fail to start up and generate vpxd dumps. This issue occurs if old certificates are removed during upgrade of vCenter Server or if you attempt to perform a fresh install of vCenter Server with an already upgraded vCenter Server 5.1 database.

      This issue is resolved in this release. If you have encountered this issue in the past, upgrading to vCenter Server 5.1.0a will resolve the issue.

    • Users unable to log in to vCenter Server after upgrading to vCenter Server 5.1
      After upgrading to vCenter Server 5.1, you might be unable to log in with the user or group administrator privileges provided by the vCenter Server installer during the upgrade process. This issue can occur if vCenter Server finds a vCenter Server Administrator user or group that can be authenticated against vCenter Single Sign On while searching the vCenter Server database.

      This issue is resolved in this release.

    • Invalid solution certificate. Certificate already expired. error message occurs when upgrading to vCenter Server 5.1 and registering the vCenter Server instance to vCenter Single Sign On 
      The vCenter Server installer does not allow upgrade if the certificate is invalid or already expired. To regenerate expired SSL certificates, see KB 1009092: Regenerating expired SSL certificates.

      This issue is resolved in this release.

    • vCenter Server takes an unusually long time to start and the vSphere Client might time out
      When a large number of permissions are assigned to objects in the vCenter Server inventory, the vCenter Server service does not start as quickly as expected as vCenter Server verifies that the users and groups exist in the identity source. Also, the connection to the vSphere Client might time out when you log in with Windows session credentials.

      The following messages appear in the vCenter Server logs while the service is starting:

      [SSO] [SsoAdminFacadeImpl] [FindGroup]
      [UserDirectorySso] GetUserInfo (DOMAIN\ *USER OR GROUP*, true) res: DOMAIN\ *USER OR GROUP*
      [UserDirectorySso] NormalizeUserName (DOMAIN\ *USER OR GROUP*, false) re: DOMAIN\ *USER OR GROUP*

      This issue is resolved in this release.

    • Upgrading to vCenter Server 5.1 might fail with error 29107 even though the service or solution user is already registered
      Attempts to upgrade vCenter Server 5.1 might fail with the following error message:

      Error 29107. The service or solution user is already registered. Check VM_ssoreg.log in system, temporary folder for details.

      This issue is resolved in this release.

    • Installing or upgrading vCenter Server 5.1 fails with the error: Could not contact Lookup Service
      vCenter Simple Install might fail with the following error if the Fully Qualified Domain Name (FQDN) contains the string port during the installation:

      Error 29102 . Could not contact Lookup Service. Please check VM_ssoreg.log in system temporary folder for details.

      This issue is resolved in this release.

    • Installing vCenter Single Sign On fails with the error: Unable to create database users: Password validation failed
      When you attempt to install vCenter Single Sign On using an existing database or the bundled SQL Express database, the installation fails with the following error:

      Error 29114. Cannot connect to DB

      This issue is resolved in this release.

    vCenter Server and vSphere Client

    • Unable to log in to the vSphere Web Client due to incorrect proxy settings
      Slow network speeds caused by incorrect proxy settings might result in delay during the interaction between the vSphere Web Client, domain controller, and external vCenter Single Sign On. You might encounter the following issues due to this delay:

      • Attempts to log into the vSphere Web Client fail with the following error:The vSphere Web Client cannot connect to the vCenter Single Sign On server.
      • A delay in excess of 20 minutes to list all domain users
      • A delay in excess of 20 minutes to load the whole vCenter Server inventory after domain use login

      This issue is resolved with this release.

    vMotion and Storage vMotion

  • Unable to access the cross-host Storage vMotion feature from the vSphere Web Client with an Essentials Plus license
    If you start the migration wizard for a powered on virtual machine with an Essentials Plus license, the Change both host and datastore option in the migration wizard is disabled, and the following error message is displayed:

    Storage vMotion is not licensed on this host.
    To perform this migration without a license, power off the virtual machine.

    This issue is resolved in this release




Usually the patching of VMware vSphere Cluster is done through vSphere Update Manager (VUM), but for this you need vCenter. In case you’re managing single ESXi host without vCenter, you might be interested to know how to patch the ESXi host from the command line.

No need to say that I’m supposing that you don’t use vCenter in purpose, and so that you want to patch your ESXi 5 standalone host or that you want to patch VMware ESXi 5 Free version without VUM.

Otherwise it’s much more convenient using VUM which can automate the patching process by using vMotion to move the running VMs to other hosts inside of the cluster, putting the host which you want to patch into maintenance mode and do the restart.

The patching of VMware ESXi 5 is quite straightforward process, but there is one condition: SSH access

The SSH access can be easily enabled by using the VMware vSphere client through the GUI.

How to patch ESXi 5:

Step 0 – Enabling SSH access to the VMware ESXi 5 host

Yes, there is step 0 actually. How to do it? Simple:

– Through vSphere client click on the VMware ESXi 5 server > Confiure > Security Profile (services)
–  SSh > Options >

How to patch ESXi 5

Step 1 – Stage the ESXi 5 patches to a Datastore accessible by the VMware ESXi 5 host

Via the vSphere Datastore browser upload the patch which you have downloaded from VMware Patch Website.  This website lists all VMware ESXi Patches.

During the copy operation which can take some time, you have plenty of time to gracefully shut down all your VMs which runs at the moment on your host.

How to patch VMware ESXi 5 without vCenter and Update Manger

I have created a sub-folder for that purpose.

Step 2 – Connect via SSH by using putty for example

Putty is the most known SSH client.

Run this command and wait, because the decompression and installation take some time……

esxcli software vib install -d /vmfs/volumes/<YOUR VOLUME>/<NAME OF PATCH FILE>.zip

In my case, I executed this command:

How to patch VMware ESXi 5 without vCenter Update Manager

The process will probably take some time.

You should see this screen confirming the installation and requiring the reboot of the host. You can click to enlarge.

How to patch VMware ESXi 5 without vCenter Update Manager

Step 3 – Reboot your VMware ESXi 5 host

Now when you reboot, the system will probably complain that the maintenance mode hasn’t been enabled….

And you should see the version number change on your VMware ESXi 5 host.

How to patch VMware ESXi 5 without vCenter Update Manager




Registering a virtual machine with the hypervisor is a handy tool that can be used when moving VMs between hosts, changing storage locations or even manual creation of virtual machines. Hyper-V 3 in Windows Server 2012 has some new advanced features for registering a virtual machine and options for importing them into Hyper-V.

To get started, launch Hyper-V Manager, select your host in the left node, right-click and select Import Virtual Machine. The Import Virtual Machine wizard launches and you are presented with the “before you begin” welcome screen; click the Next button to continue.

The next screen (Fig. 1) will prompt you to specify the current location of the VM files that you want to import. Enter the correct value and click Next to continue. You will then be prompted to select the virtual machine you wish to import from the location you specified earlier. This is relevant in case that location has more than one virtual machine stored on it. Select your VM and click Next to continue.

Locate Folder.
Figure 1. Locate Folder. (Click image to view larger version.)

Up next is the import type (Fig. 2) and that is essentially the heart of the import process.

Choose Import Types
Figure 2. Choose Import Types. (Click image to view larger version.)

You should make your selection after you have understood what each option does as follows:

Register — This option assumes that the VM and its files are in the right place and that you simply need to register the VM’s unique ID with Hyper-V. This could come in handy if you bring up a new Hyper-V host and for whatever reason or circumstance you present the storage to Hyper-V and you need to manually register the VMs on that storage. The unique VM ID is preserved and does not change in this option.

Restore — This option is handy if you have your VM files stored on a file share somewhere or maybe even on a removable disk and you need to register the VM’s unique ID with Hyper-V, but also need to move the files from the current storage location into a more appropriate location. This option will copy the files and will register the VM, all the while preserving the VM unique ID.

Copy — This can come in handy when you are manually copying the VM files for the purpose of creating a new VM. Essentially you are using the “template” approach, except you are doing it manually. In this case using the Copy command will generate a new unique ID for the VM you are creating out of the copy process.


Source: virtualizationreview

%d bloggers like this: