How to make ESXi crash

While looking at a VMware KBTV video, I saw a cool trick to make your ESXi host crash (which can be good for testing purposes):

From SSH, type: vsish -e set /reliability/crashMe/Panic 1

Apparently, I'm telling nothing new: Seaching for that string on Google, I found the links below for more info:

http://www.seancrookston.com/2012/01/09/forcing-a-kernel-dump-on-a-vsphere-host-the-purple-screen-of-death/

http://www.ntpro.nl/blog/archives/1388-Lets-create-some-Kernel-Panic-using-vsish.html

Unable to remove ESXi host from vCenter

I needed to remove an ESXi host from vCenter that I wanted to re-use, but I was unable to. After disconnecting the server, the "Remove" button was greyed out:

 Some searching gave the tip to remove it via PowerCLI (Remove-VMHost), but that did not work either:

It turned out the ESXi host could not be removed because it was part of a cluster.

Solution:

Move the ESXi host entry (which is disconnected) to the top of your vCenter tree, out of any cluster. Then re-run the Remove-VMHost command (most likely the Remove function will be visible in the GUI as well, but I just pressed up and enter in my PowerCLI screen, and it started to remove)

HP Servers disconnecting

We come across an issue lately, with several types of HP servers that have QLogic/NetXen NC375i networkcards in them. They disconnect, causing a disruption of service. You can imagine that having an NFS mount or iSCSI target with that happening is less than desirable and has caused Windows clusters to fail over and ESX/ESXi hosts to go crazy. This problem is solved by rebooting the host. This issue is very much OS independent!

In windows eventlog you may see things like:
DEVICE: HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter #4
PROBLEM: Tx path is hung. The device is being reset.

In ESX you see things in /var/log/vmkernel like:
Jul 31 21:02:12 server01 vmkernel: 165:01:40:09.914 cpu19:4295)<5>nx_nic[vmnic8]: Device is DOWN. Fail count[8]
Jul 31 21:02:12 server01 vmkernel: 165:01:40:09.915 cpu19:4295)<3>nx_nic[vmnic8]: Firmware hang detected. Severity code=0 Peg number=2 Error code=1 Return address=0


HP has brought out an advisory saying that indeed there are problems:

Network Adapters and Affected Firmware Versions

Network Adapter	Affected Firmware Versions
CN1000Q Dual Port Converged Network Adapter	EARLIER than firmware version 4.8.22
NC375i Integrated Quad Port Multifunction Gigabit Server Adapter	EARLIER than firmware version 4.0.585
NC375T PCI Express Quad Port Gigabit Server Adapter	EARLIER than firmware version 4.0.585
NC522m Dual Port Flex -10 10GbE Multifunction BL-c Adapter	EARLIER than firmware version 4.0.585
NC522SFP Dual Port 10GbE Server Adapter	EARLIER than firmware version 4.0.585
NC523SFP 10Gb 2-port Server Adapter	EARLIER than firmware version 4.9.81

The NC375i adapter is integrated on the following servers and storage systems:

• ProLiant DL370 G6 Server
• ProLiant DL580 G7 Server
• ProLiant DL585 G7 Server
• ProLiant DL980 G7 Server
• HP Business Data Warehouse Appliance
• StorageWorks D2D4312 Backup System
• StorageWorks D2D4324 Backup System

Servers manufactured after 1 april 2012 are not affected by this, but check the firmware level if you suffer from this issue. An older interface may still have this issue in your newer machine.

How to check the firmware version:

Windows:
Go to the HP network utilities, and click on the network interface you are having issues with, and click Properties. The Information tab will show the Boot Code, which is the firmware version:

Alternatively, you can run the update tool, and it will tell you which version you are currently running as well.

Linux:

Type "modinfo netxen_nic" and look for the firmware line.

[user@server-01 ~]$ modinfo netxen_nic | grep firmware

firmware: phanfw-4.0.579.bin   <--------  version 4.0.579, so needs an update

ESX/ESXi:
VMware have released a KB article to get the firmware and driver version, available here.

Resolution:

The resolution is to update the firmware of the network cards. The advisory lists the latest drivers and firmware. For Windows and Linux, there are proper update tools, but unfortunately for VMware, no firmware update utility is given, and the Linux firmware utility does not work.

On ESX/ESXi you have to make use of a Linux LiveCD and boot from it (ESX-server in Maintenance mode and reboot). In our case we used Novell SLES11 CD (free ISO download, registering necessary) as the Rescue-CD for RHEL5 gave several errors running the firmware update-utility. Perhaps a OpenSUSE, Fedora, Ubuntu or other distro LiveCD can be used as well, but we haven't tested those.

Many thanks go to my colleague Sven for the info :-)

Getting an overview of patches for your ESX hosts

A customer asked for an overview of which patches exactly were needed for his ESX hosts. He wanted to review the patches before they were applied with Update Manager. Unfortunately, you can't get the list from the gui in a nice way, so some PowerCLI goodness was needed.

First off, you need to have the Update Manager cmdlets installed to be able to use the Get-Compliance cmdlet in the code.

The following script will go over each host, and output the severity, patch ID, release date, additional info (a link to the kb article) and a short update to what the patch is for.

It will look something like this:

HostSecurity,ESX410-201204402-SG,"4/26/2012 10:00:00 AM","For more information, see http://kb.vmware.com/kb/2014988.","Updates libxml2"

Now for the code itself:

ForEach ($HostToCheck in Get-VMHost){

$Details = Get-Compliance $HostToCheck -Detailed| Select -ExpandProperty NotCompliantPatches| Select @{N="Hostname";E={$HostToCheck}}, Severity, IdByVendor, ReleaseDate, Description, Name

$ComplianceResult += $Details

}

$ComplianceResult | Export-CSV -Path c:\NeededPatches.CSV -NoType

I had an issue where I needed to put a machine in maintenance mode, but some VM's did not want to vMotion off of the host, because the VM's were busy doing a VMWare Tools installation. These machines needed a reboot to finish the installation, but my customer did not want to do that just yet.

I tried to do an "End VMWare Tools Install" via the menu, but this was greeted by the message "The operation is not allowed in the current state."

At first I thought I needed to wait for my customer to reboot the machines at his convenience, but a colleague of mine knew the solution:

• Enable Remote Tech Support for the ESXi host where the machine is located (from Configuration > Security profile)

• Connect via SSH with root account

• Run the command: usr/bin/vim-cmd vmsvc/getallvms

• Take the VMid of the VM with the hanging install and then use the second command to cancel it: /usr/bin/vim-cmd vmsvc/tools.cancelinstall vmid

• Start an interactive VM tools installation and manually run it from the mounted CD

Hey presto, the VM's are able to be vMotioned off, and hosts are put in maintenance mode.
 
 Thanks Paul ;-)

Fixing slow network drive access in Windows 7

I have a few Windows 7 devices, and all of them have rather slow network drive access. You click on a share, and you see the green bar just hanging, and it's all rather annoying. Some quick searching around led me to a Windows 7 forum, and the fix was simple:

Go to the Internet Explorer options and go to Tools ->Internet Options -> Connections -> LAN Settings -> Automatic Configurations: Make sure that "Automatically detect settings" is unchecked.

Close Internet Explorer, and that is it..

Speeding up PowerCLI

As you may have seen, I like PowerCLI. Only thing is, it's not the fastest to start up, and running the first cmdlet takes some time. After that, it goes fine.

The first tip from the VMware PowerCLI Blog is to go to:

Windows > Control Panel > Internet Options > Advanced > Security > Check for publisher's certificate revocation and uncheck that box (also helps the SQL Server Management Studio start up faster if it is not connected to the internet).

Unfortunately, this did not help me so much with running the first cmdlet. So I was mighty glad that I came across this article from vNugglets to speed up PowerCLI 5. This led to another VMware PowerCLI Blog article with all the steps for earlier versions.

Basically, the issue is that due to the fact that the .Net framework compiles the underlying code on first use, so you can precompile things to speed up first run:

Open up an administrative command prompt, and run the following commands:

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService41.XmlSerializers, Version=4.1.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService40.XmlSerializers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService25.XmlSerializers, Version=2.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

If you are running on 64-bit OS, you need to run the following as well:

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService41.XmlSerializers, Version=4.1.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService40.XmlSerializers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService25.XmlSerializers, Version=2.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

This needs to be done once, and works for all users and applications.

I did a test on one setup, using the test  on the PowerCLI blog which is:

Measure-Command { Get-VMHost } | fl TotalSeconds

This test went from 24.4 seconds to 4.6 seconds! More than 5 times as fast!! Again, thanks go to vNugglets!

How much space are the VM's using?

Quick script to give me an overview of which VM has how much diskspace in use, and also how much is free.

Get-VM -name MYVM-* | Where { $_.PowerState -eq "PoweredOn" } |

Get-VMGuest |

Select VmName -ExpandProperty Disks | Select VmName, Path, Capacity, @{ N="PercFree"; E={ [math]::Round( ( 100 * ( $_.FreeSpace / $_.Capacity ) ),0 ) } }

| Out-File c:\temp\myvmspace.txt

Note that this is of only the VM's that start with "MYVM-". Change this according to whatever you want ( or remove the -name to just get all vm's, sizes and free space).

List the portgroups for the VM's

As part of a makeshift DR solution, one of my requirements is to have a list of vm's that shows which networkcard is in which portgroup. Powershell to the rescue!

Get-VM | Get-NetworkAdapter | Select-Object @{N="VM";E={$_.Parent.Name}},@{N="NIC";E={$_.Name}},@{N="Network";E={$_.NetworkName}}| Export-Csv vms_in_networks.csv

Hey presto, you have a list that shows which nic goes where.

Middle mouse button emulation

My new Dell XPS laptop is a sweet machine, but the Synaptics drivers didn't have the ability to emulate a middle mouse button functionality by clicking the left and right button on the touchpad.

However, if you add a new key to the registry, then you get that functionality. Copy the following into a .reg file:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Synaptics\SynTP\TouchPadPS2]
"HasBothButtonFeature"=dword:00000001

Log off and back on, and you should have the middle mouse button functionality again. I really don't get why Synaptics keep removing/hiding this feature (My HP work laptop had the same issue fixed it there too).