Citrix Xenapp Best Practices

A while ago I talked about Citrix Xenapp on VMWare, but now I came across best practices, both from VMWare as well as Citrix. So without further ado:


http://www.vmware.com/files/pdf/solutions/vmware-citrix-xenapp-best-practices-EN.pdf

http://support.citrix.com/article/CTX129761

Both of them say about the same I said in my first post, with some more details.

Invalid configuration for device ’0′ when enabling a NIC in vSphere

Suddenly a VM was unreachable on all interfaces. Upon investigation, I saw that none of the NIC's were set to connected:







When I tried to enable them again I got Invalid configuration for device ’0′:

Some googling led to a simple solution, which is to restart the management agents:

service mgmt-vmware restart

and

service vmware-vpxa restart

Good to know. Thanks go to the cupfighters for investigating it deeper and leading me to the answer.

Disable Symantec AV via scripting

A small script to quickly disable Symantec Antivirus 11 (Symantec Endpoint Protection) for those moments that it needs to be disabled quickly:

Open up an administrative commandprompt and type the following commands:

sc config "Symantec AntiVirus" start= disabled
sc config ccEvtMgr start= disabled
sc config SmcService start= disabled

net stop "Symantec AntiVirus"
net stop ccEvtMgr

taskkill /IM smc.exe /F

Symantec Antivirus is now completely disabled

NFS advanced settings for ESX/ESXi

Netapp has this wonderful Best Practices document available online, which is well worth the read. It used to include the CLI commands but this has been transferred to a new document.

As part of the best practices of an ESX/ESXi installation , you need to change some settings for NFS. I keep forgetting which ones they are, so this a reminder to myself (and anyone that reads this blog ;-) )

The easiest thing to do, is to start up the Remote Tech Support (SSH) service in the security profile, SSH to the host, and copy/paste this into the SSH window:

/usr/sbin/esxcfg-advcfg -s 30 /Net/TcpipHeapSize 

/usr/sbin/esxcfg-advcfg -s 120 /Net/TcpipHeapMax 

/usr/sbin/esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures 

/usr/sbin/esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency 

/usr/sbin/esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout 

/usr/sbin/esxcfg-advcfg -s 64 /NFS/MaxVolumes

Note that the last setting says MaxVolumes: 64. The default is set to 8, which means that the maximum number of NFS volumes is 8 by default. Setting the maximum to 64 works for ESX 4.x, but used to be 32 for ESX 3.x. ESX 5.x can even go to 128.

 If you want to follow what VMWare says instead of the Netapp Best Practices, you can set the TCPIP Heap Size to 32 in ESX4/5(see here).

Update for 5.1:

/usr/sbin/esxcfg-advcfg -s 32 /Net/TcpipHeapSize
/usr/sbin/esxcfg-advcfg -s 128 /Net/TcpipHeapMax
/usr/sbin/esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
/usr/sbin/esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
/usr/sbin/esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout
/usr/sbin/esxcfg-advcfg -s 256 /NFS/MaxVolumes

Which NTP server has been configured (and used) for Windows 2008R2

Quick one to remember:

Which NTP server has been configured:

w32tm /query /configuration

Which NTP server is actually being used for the last poll:

w32tm /query /status

Activating Windows servers to KMS server

For people who use a Key Management Server (KMS), you don't need to fill in a serial number and activate with Windows 7/2008/2008R2 directly, but you use a separate server for this.

On the server you need to activate, open an administrative command prompt, and type:
slmgr.vbs /skms <kmsserver> (the part between brackets is the kms server in your organisation)

It will pop up a message saying the KMS server has been set correctly:

After this, type: slmgr.vbs /ato

If all goes well, it activates the server:

You may encounter an error:

 

 
In this case, check the following:

• Can you ping the kms server?
• Can you telnet to port 1688 on the kms server?
• Is the time and date set correctly on the server you want to activate?

 I had the last one today, and forgot to check the date. The time was set correctly, but the date was 2 months off(doh!).

No Remote Desktop client access licenses available

That's what a colleague of mine got when trying to connect to a terminal server. I had seen this message a long time before, and found a solution back then, but every time I get this message, I need to go through my extensive mailbox and search for the answer, so I thought I'd post it here:

Here’s how you fix the error message:

-    Open up regedit on your machine (Start->Run->Type: regedit.exe)
-    Go to HKEY_LOCAL_MACHINE\Software\Microsoft\MSLicensing\Store
-    Click the LICENSE00x folder (probably LICENSE000)
-    Click File->Export, and export the LICENSE00x folder to a registry file
-    Now delete the LICENSE00x folder.
-    Try to log on again.

So far it worked every time for me and my colleagues.

Powershell selection stuff..

Something to remember for myself about selecting:


get-vm | gm as an example, shows all methods and properties you can do with get-vm.

get-vm | where {$_.PowerState -eq "PoweredOn" -and $_.Name -ne "PietjePuk" }

Shows all VM's that are powered on but not that one VM that is called "PietjePuk".
$_.Powerstate is one of the properties of the get-vm statement.

This is how you can select VM's based on different properties.

So get-vm | where {$_.MemoryMB -eq "4096"} shows you all vm's that have 4GB of memory.

-eq = equals
-ne = does not equal
-and = a secondary variable to select.

Active Route gets removed on Windows Server 2008 offline Cluster IP Address

An interface on a Windows 2008R2 cluster was no longer reachable. It was a management interface, and the machine was still up, but of course the server came up in the monitoring as not reachable. There was a persistent route present on the server, but somehow this route got removed from the active routing table.

Some searching by one of my colleagues turned up this article: http://blogs.technet.com/b/networking/archive/2009/05/21/active-route-gets-removed-on-windows-server-2008-offline-cluster-ip-address.aspx

Apparently when adding persistent routes on cluster nodes you need to make sure that you specify the interface as the destination rather than the next hop IP address. An example:

Adding a route as usual, open a command prompt: route add -p 10.10.0.0 mask 255.255.0.0 10.10.10.1

This would route all traffic to 10.10.x.x addresses via 10.10.10.1, and the only interface on the host (e.g. with host IP address 10.10.10.20) within that network range would use that route. However, this route could dissapear from the active routing table *despite having a persistent route* because the next hop specified is part (or could be part) of a clustered network.

With Windows 2008 (and R2) you need to add it a little bit differently for cluster nodes, according to the Technetblog:

• Open up the Network and Sharing Center
• Click Change adapater settings
• Look at the interface you want the route for and note which the "Device Name" of the interface (see screenshot). As an example, the first interface in the screenshot is the one that needs the route. It has device name "Intel(R) PRO/1000 MT Network Connection #3"

• Open up the command prompt as administrator
• Type route print, and look at the first lines that show the interfaces:

As you see the Intel interface with #3 at the end has a number in front of it: 21 in this case. This is the Interface Number

• Now type the route you want, e.g.: route add -p 10.10.0.0 mask 255.255.0.0 0.0.0.0 if 21

The trick is to add "0.0.0.0 if <interface number>" instead of the next hop address. Now, the route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline.

Updating VMWare Tools on all powered on Windows VM's

Quick PowerCLI line to update the VMWare Tools all Windows VM's that are powered on without rebooting them:

get-vm | where {$_.PowerState -eq "PoweredOn" -and $_.Guest.OSFullName -match "Microsoft Windows*" } | Update-Tools -NoReboot

Oh how I love PowerCLI...

There are errors during the remediation operation

"There are errors during the remediation operation" . I stared blankly at the screen. Why was this happening?

I was trying to update an older vSphere 4.0 setup with update manager. The server was already in maintenance mode, but after clicking "Remediate" the error message came up almost immediately. A retry did not help. Quick googling came up with the following:

http://www.vmware.com/support/vsphere4/doc/vsp_vum_40_rel_notes.html

Host Patch and Upgrade Remediation Might Fail
Host patch and upgrade remediation might fail with the message There are errors during the remediation operation if an inaccessible virtual machine exists on the host. The reason for this failure might be that the virtual machine files reside on a disconnected network storage.
Workaround: Connect the disconnected network storage or remove the inaccessible virtual machine from the vSphere inventory.


I checked the host, and indeed there was an inaccessible VM! It was a no longer used one, so I removed it from the inventory, and updating went fine after that.

Where would we be without Google, right? ;-)

vm (invalid) and greyed out

A number of vm's in a vSphere setup were showing VMWare Tools not running. Checking the vm's themselves showed that it was running. I tried reinstalling VMWare Tools, but when I clicked the "Install/Upgrade VMWare Tools" the vm turned gray, and was appended with "(invalid)"

Some googling showed up this: http://communities.vmware.com/message/861038

•  Log on to the host where the vm is running on
• type: "service vmware-vpxa restart"
• type: "service mgmt-vmware restart"

The host and vm's that were running on there became disconnected in the interface for a few seconds, but after that came back, and the vm's VMWare Tools became OK. It turned out the vm's didn't need updating anyway.

Configuring ILO from the commandline

I just got an interesting question from a colleague. He wanted to know if it was possible to configure ILO from the command line. A quick google search came with an interesting link:

http://briandesmond.com/blog/read-write-ilo-details-from-the-command-line/

So the quick answer is "yes". You do need to have the HP Proliant Support Pack installed, but when you have it, it's under C:\Program Files\HP\hponcfg. There is also a gui version, in that directory, to make things even easier: hponcfg_gui.exe

Want to do it for Linux? Possible too!

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=397991&prodTypeId=18964&prodSeriesId=397989&swLang=8&taskId=135&swEnvOID=2026

Look for the "HP Lights-Out Online Configuration Utility for Linux"on that page.

You're welcome.

Cloned VM's and WSUS

 When you clone Windows machines, you may run into the issue where WSUS updates are not working. Even though sysprep has been run, the wsus authorization appears to not have been reset with it. I use this script below to reset the Windows Update to a fresh state.

net stop wuauserv
del c:\windows\WindowsUpdate.log
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v AccountDomainSid /f
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v PingID /f
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v SusClientId /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v LastWaitTimeout /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v DetectionStartTime /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v NextDetectionTime /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v AUState /f
net start wuauserv
wuauclt /resetauthorization /detectnow
pause

Paste this into a batchfile, run it, and wait a short while, and then the updates should start again. Note that this deletes the WindowsUpdate.log file. If you want to do troubleshooting of previous WSUS errors, either delete that line, or copy the file to a safe place. I delete it so I get a fresh view of what is happening with the Windows Updates.

Setting 2008R2 NTP from the commandline

There I was, happily configuring a new Windows 2008R2 server, when I wanted to set the NTP hosts.

net time /setsntp:"ntpserver1,ntpserver2"

Error! It appears the command has changed since Windows 2008. It turns out to be the following now:


W32tm /config /syncfromflags:manual /manualpeerlist:ntpserver1,ntpserver2

Oh well, gotta remember that one.

Doing work on remote Windows servers not in a domain

Having to manage multiple servers from multiple customers that are not in a domain, you sometimes need to get creative when you need to script batchfiles.

Luckily, Microsoft has something that worked with Windows NT4 and still works with Windows 2008R2, provided there are no firewalls blocking you.

From the command prompt of your administrative pc, type:

net use \\hostnametomanage\ipc$ /user:adminuser password

If you get a "Command completed successfully" back, you are in luck. With psexec, available from the excellent sysinternals pstools suite you can now do everything on the remote server, as if it were done locally.

With that one command and psexec you can use batchscripting to do almost anything you want.

Using Perfmon for SQL Performance Tuning

A while ago I found this older but still fantastic blog post about SQL performance tuning:

http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/

Excellent video in there, well worth watching. Yes, the video is from 2006, but even now, it is just as relevant.

A little after I saw the blog post, I got the chance to put this in practice too; A customer suspected his SQL cluster wasn't performing too well. I added the counters from the video to the perfmon counter log, and kept the perfmon running.

Then our customer came back after a half a year; he found the SQL performance had degraded even more. We had a nice list of performance statistics now that went deeper than just CPU and memory usage. Now, I got the number of User Connections, and saw that it went up 4 times.

The perfmon counters Brent Ozar was talking about:

• These are listed OBJECT first, then COUNTER
• Memory – Available MBytes
• Paging File – % Usage
• Physical Disk – Avg. Disk sec/Read
• Physical Disk – Avg. Disk sec/Write
• Physical Disk – Disk Reads/sec
• Physical Disk – Disk Writes/sec
• Processor – % Processor Time
• SQLServer: Buffer Manager – Buffer cache hit ratio
• SQLServer: Buffer Manager – Page life expectancy
• SQLServer: General Statistics – User Connections
• SQLServer: Memory Manager – Memory Grants Pending
• SQLServer: SQL Statistics – Batch Requests/sec
• SQLServer: SQL Statistics – Compilations/sec
• SQLServer: SQL Statistics – Recompilations/sec
• System – Processor Queue Length

To find out what they all mean, see the video:

http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/

You shouldn't virtualize *everything*

VMWare says you can virtualize everything, but that doesn't mean you should. One of those things you actually shouldn't (dare I say can't) virtualize, is NTP. If someone asks you to virtualize the NTP server, here are two nice links to give you why that's a bad idea:

http://support.ntp.org/bin/view/Support/KnownOsIssues

9.2.2. Xen, VMware, and Other Virtual Machine Implementations

NTP was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy. No known virtual machine is capable of meeting these requirements.

Run NTP on the base OS of the machine, and then have your various guest OSes take advantage of the good clock that is created on the system. Even that may not be enough, as there may be additional tools or kernel options that you need to enable so that virtual machine clients can adequately synchronize their virtual clocks to the physical system clock.

http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf

Page  18:

Using NTP in Linux and Other Guests

The Network Time Protocol is usable in a virtual machine with proper configuration of the NTP daemon.

The following points are important:

Do not configure the virtual machine to synchronize to its own (virtual) hardware clock, not even as a fallback with a high stratum number. Some sample ntpd.conf files contain a section specifying the local clock as a potential time server, often marked with the comment “undisciplined local clock.” Delete any such server specification from your ntpd.conf file

Continued: Storage vMotion via PowerCLI

I spoke before about Storage vMotion via PowerCLI, and today I got to put it into practice. It works, *but* there's a small thing: It doesn't do thin provisioning on the fly, as far as I can tell. So if you had a thick VM, and you wanted to make it thin, normally via the interface you get an option to do thin provisioning during the Storage vMotion. The Move-VM statement has no such feature yet (I think, at least I couldn't find it). Some searching around told me that the awesome LucD posted a function on poshcode to get the thin provisioning into Move-VM (sort of). I haven't tested it out yet, but it seems to be what I needed.


Oh and for my own reference: If I want to move all vm's from "cluster1" to the New-Storage datastore:

Get-Cluster cluster1 | Get-VM | Move-VM -Datastore(Get-Datastore "New-Storage")

Citrix XenApp on VMWare

A few years ago, there was a lot of discussion regarding Citrix XenApp/Terminal Server running on VMWare. Our personal experience was that it wasn't that good, and with anything above a few (4-5) concurrent users performed terrible when doing day to day tasks. We even tried the tweaks that were going around (stuff like only use 1 vcpu on your Citrix server, and various other tweaks going around). Our general feeling was to not do it anymore, and left it at that.

However, time goes on, and technology moves forward. But the articles remained online, and no one really talked about the advancements that have been made. Then, on VMWorld 2009 there was a session about running XenApp using vSphere. If you create a (free) account on vmworld.com, you can watch it for free

Basically, a few best practices are given:

1. Use the newest CPU's in your hosts (Nehalem architecture or higher). So if you are still using older hosts (like HP G5 series servers) then think again. The newer CPU's alone would give about 30% better performance (according to the video, ymmv)
2. Use vSphere, mainly because it supports MMU virtualization which gives a good performance boost for Xenapp (but if you are still on ESX3.5, you *really* should be thinking of upgrading anyway, instead of thinking of virtualizing your Xenapp servers)
3. The usercount on a VM will never be the same as on a physical server. The idea is to be running multiple smaller VM's and thereby getting more users in total per physical box.
4. Don't use p2v'd systems. It's much better to start with a clean OS, but if you must, remove old hardware, hardware management agents, and unused OS features (wallpaper, menu animations, systemtray animating things such as network indicators and system clock. Of course there are other tweaks you always need to do for any terminal server environment.The sweetspot of the Xenapp VM's is often  2vCPU's and 4GB RAM. More vCPU's will usually give less performance.
5. Old blogposts are no longer valid, so tweaks like "disable page sharing and memory ballooning" are no longer necessary.
6. Use realistic tests. It's no good if your environment performs well in synthetic tests if the applications that the users will use are not performing up to par.

The video also has someone from eBay giving a talk about their environment, and they came to 6 to 8 users per VM, each with 2vCPU's and 4GB RAM, running 10 VM's on one DL380G6. Like I said, the video is free to watch, and highly informative.

Removing old hardware in Windows VM's

If you P2V (physical to virtual) a system, or even when you create a VM on your VMWare workstation and move that to an ESX host (V2V, virtual to virtual), old hardware is "left behind" in Windows. The goal of a VM is to make it use as little resources as possible, so it is best you clean that up. Secondly, if the hardware in question is an old network interface, the IP address on that interface still exists, so Windows starts moaning there's a duplicate IP address.

There's a relatively simple way of fixing that: First, open up a command prompt and type

set devmgr_show_nonpresent_devices=1

Next, type: devmgmt.msc

The device manager will start up. In there, go to the menu "View" and click on "Show hidden devices". Now all sorts of previously hidden hardware will show up. Right mouseclick on the old hardware you want to remove (the icons that are of a lighter color than the existing hardware).

Now in Windows 2008/2008R2, if you do the same with a command prompt, you will get nothing. Fix that issue by running the command prompt as an administrator.

More than one RDP session per user on Windows 2008

Just saw an awesome post on a Dutch site that I noticed before, but never really took the time to investigate further:

It appears Windows 2008 (and noticed the same thing with 2008 R2) by default only accepts one RDP session per user. So when administrator logs in, he could potentially kick out another person who uses the administrator account. 

So Rudolf Kleijwegt posted the following command:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fSingleSessionPerUser /t REG_DWORD /d 0x0 /f

AWESOME tip, and I'll definitely add this to our buildserver.

vCenter: Don't forget the SQL Server Agent!

So this is one I learned the hard way: If you use SQL server for your vCenter installation, don't forget to start the SQL Server agent automatically!

vCenter installs rollup jobs which it needs to process the statistics to make it need less granular so it takes up less space.

These are called rollup jobs, and are controlled by the statistics in the vCenter server settings. I hear you ask: So what happens if I don't start SQL Server agent automatically?

Well.. Nothing, at first. But after a week, the statistics don't get rolled up, so the first table in the SQL database that is used for the statistics (VPX_HIST_STAT1) keeps growing and growing, and the data won't be processed into the next table, called VPX_HIST_STAT2, nor the next, etc. There are 4 tables. The last 3 tables will have no data, so when you look at the day, week, month or year statistics, you'll have no statistics. Starting up the SQL Server Agent quickly will give SQL time to roll up the data, but wait too long and you have too much data to process. There is a way to delete the data, described in this article, but you will lose all statistics.

So save yourself some trouble: If you are installing SQL Server for vCenter, don't forget the SQL Server Agent

TCPDump of NFS traffic that is using jumbo frames

I had an issue a while ago where I needed to troubleshoot poor NFS performance, so VMWare support asked me to do a tcpdump of the traffic. I was using jumbo frames and the performance was poor after about 5 minutes of not using the NFS.

On the vswitch that has the NFS VMKernel (you ARE using a dedicated vswitch for NFS, right?) create a new service console.

Use an unused IP address from the range that you use on your storage LAN. If you are using a different vlan, don't forget to set that too. You will be using the vswif that is created with this previous action to do the tcpdump. Go to the properties of the vSwitch and set promiscuous mode to "Accept".

Now to do a tcpdump of your traffic.

tcpdump -i vswif2 -w ./dump_1 -s 256 (-s 256 is a buffer of 256MB I believe)

You will get a lovely SMALL file.. Not what you want. This is because vswif2 (the newly created vswif) is not using jumbo frames. To get vswif2 on jumbo frames type the following in the command prompt:

ip link set dev vswif2 mtu 9000
service network restart vswif2

Now do the tcpdump again, and watch your harddrive fill up with data. Stop tcpdump with CTRL-C (and don't wait too long, because it fills up quickly).

VMWare data traffic with jumbo frames

Set the jumbo frames on vSwitch1 (which I am using for NFS). Log on to the console of the ESX host, and type:

esxcfg-vswitch -m 9000 vSwitch1

# remove the old vmkernel for storage which is on MTU 1500

esxcfg-vmknic -d "VMkernel - Storage"

# add the vmkernel back for storage again with MTU  9000 (vmkernel has ip 192.168.1.2)

esxcfg-vmknic -a -i 192.168.1.2 -n 255.255.255.0 -m 9000 "VMkernel - Storage"

Now the vSwitch1 and NFS vmkernel are using MTU 9000, and if your physical switch and SAN are configured for jumbo frames too, you should have a speed increase.

Pushing Host Profiles via PowerCLI.

Nice.. I had to push a host profile to a whole number of hosts, but they were all running a lot of VM's. Anyone who's seen vSphere knows that via the interface, you need to set a server in maintenance mode, apply the host profile, then take it out of maintenance mode. This is fine for one or two machines, but by the time you get to the 5th or 6th server, you get a bit bored, especially when you know you've got 20 to do.. So in comes Powershell (I should say PowerCLI) again, to save me some time.. I thought.

First things first: Figure out how to get a host profile:

$MyHostProfile = Get-VMHostProfile -Name "coolprofile"

Fail. Fail? Yes, fail. Somehow I run into a wall, with PowerCLI telling me that my profile can't be found. Some googling tells me that there's a bug in PowerCLI, but there's a workaround:

Get-VMHostProfile -Entity *

Woohoo! It shows me a hostprofile that was already pushed to the client before. So my statement would now become:

$MyHostProfile = Get-VMHostProfile -Entity * 

Note that this workaround would only work with one host profile. I'll figure out how to get a specific hostprofile implemented some day, but I've got one, so I got lucky (this time).

Now to apply a host profile to an esxhost:

Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false

Damn, that was easy. Especially since the first part was so difficult..OK, now now to get a host in maintenance mode: 

Set-VMHost -VMHost $esxhost -State maintenance 

Great, that works! Now how do I get it out? 

Set-VMHost -VMHost $esxhost -State connected 


Cool, works too! Now I have to put the whole thing together:

$hosts = "esx1","esx2","esx5","esx6"
$MyHostProfile = Get-VMHostProfile -Entity *
foreach ($esxhost in $hosts) {
Set-VMHost -VMHost $esxhost -State maintenance
Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false
Set-VMHost -VMHost $esxhost -State connected
}


As you can see, I used an array of hosts in this case (the first line of code. Didn't want all hosts in my case) but I could have changed that to all hosts by just doing a get-vmhost.

Running this now puts each host in maintenance mode, applies the host profile, and takes it out again.. Now imagine that for 20, 30 or even 100 hosts...... Yes, I *like* PowerCLI...

Storage VMotion via PowerCLI

I keep running into occasions where I need to storage vmotion a lot of VM's to different locations. I looked at a solution some time ago, but that was an old version of PowerCLI. Turns out nowadays it's really easy:

get-vm "MyVM"| move-vm -datastore(Get-Datastore "New-Storage")

This simply moves the MyVM virtual machine to the New-Storage datastore.. Remove the "MyVM", and suddenly ALL vm's move to the New-Storage datastore: SWEET. Gotta love PowerCLI, and especially since I've got a job coming up where I need to be moving a whole bunch of vm's to new storage....

Migrating/upgrading a 32bit vCenter 4.0 installation to a 64bit 4.1 installation

With the coming of vSphere 4.1, vCenter now has a prerequisite of needing a 64bit Windows operating system. Many customers that upgraded from ESX3.5 to vSphere were using a 32bit Windows installation, but also many vSphere installations were built using 32bit Windows. vCenter 4.0 even needed 32bit DSN's so it appeared it was easier to just install vCenter using a 32bit installation. To make things worse, many vCenter implementations have SQL Server installed on the same server as vCenter itself. This article will be about the upgrade process of upgrading windows 2003 32bit with SQL 2005 32 bit to a 64bit environment, and migrating the data.

During testing, this upgrade was quite difficult because of little snags, so it is very wise to do a dry run of this using a P2V’d system and testing the upgrade thoroughly before you do this on a live system. You wouldn't want to reinstall your OS and find out your migration data is useless.

Needed

-          Windows 2003/8/8R2 x64 Standard

-          SQL Server 2005/8 x64 Standard (2008R2 is NOT supported yet)

-          VMWare vSphere 4.1 vCenter installation media

-          External source, such as an external harddisk or fileshare.

Basic steps

1. Make sure all data is safely backed up, and you have all the information off of your 32bit installation
2. Stop virtual center service, update manager service
3. Run the backup.bat from VMWare's datamigration tool
4. Copy the datamigration complete with data to an external source
5. Stop SQL Server service and SQL Server agent, copy the ESX and Update Manager database to the external source
6. Wipe server and install server with 64 bit OS, using the same name and IP as the previous one
7. Install Microsoft SQL Server
8. Copy vCenter and Updatemanager database to the newly installed server
9. Set permissions on the databases, and on msdb
10. Set compatibility mode of the db to SQL 2005 if your SQL Server came from SQL 2000.
11. Create the ODBC links
12. Copy the datamigration directory back to the server
13. Make sure you have the vCenter installation media on dvd or on the server in a directory as well
14. Run install.bat from the datamigration directory
15. Answer the installation questions from the installer

Detailed Steps

So you want to migrate a 32bit system to a 64bit system. This means you will be doing a complete re-installation of the operating system. I will assume only the database (in my tests SQL2005) is installed and standard users have been created, and no extra software has been installed.
 
Once the first VMWare services are stopped, management through vCenter will be impossible, but the VM’s will keep on running. Services like HA and DRS will be unavailable then, but VM’s should not notice anything of this. 

Collect information and backup the system
Note the following:
-    IP address(es)/subnetmask(s)
-    Hostname
-    Workgroup (if the machine is part of a domain, take note of that as well)
-    Routes
-    Host file (c:\windows\system32\drivers\etc)
-    Usernames (and passwords, if you have them)
-    Possible created groups
 
For SQL look at the following things:
-    Which databases are configured
-    Where are the databases stored
-    Which Service Pack is the database currently
-    Any extra settings made in SQL, such as maintenance and backup plans.
-    The ODBC DSN’s used to connect the database to the Virtual Center.
 
Of course, if you have installed extra software, make sure you have copied any settings and data from that application. Finally, make sure you have a working backup, or pull one of the disks of the raid 1 set. This will ensure you can easily go back to the 32bit environment.
 
Stop vCenter services
In computer management, stop the following services
-    VMware VirtualCenter Server
-    VMware VirtualCenter Management Webservices (will be stopped along with the first service)
-    VMware Update Manager Service
 
Datamigration
Fortunately, VMWare has made a datamigration tool to help with the migration from 32bit to 64 bit. On the installation media, there is a directory called datamigration. In that directory is a zipfile. Extract that zipfile to your harddisk, or to the external harddisk, as this extracted directory will also hold the data from the migration. I have tried to run the tool from a network share, but it wouldn't work properly, so my advice is to not do that.
 
Open a command prompt, change to the datamigration directory and type “backup.bat”. The script will prompt you if the data needs to be backed up. Type Y and press enter to continue.
 
New files and directories will be created in the datamigration directory. The “data” directory will hold the configuration data from vCenter, Update Manager and vCenter Orchestrator (directories will be created vc, vum and vco). There will also be a "log" directory to show the results of the migration.
 
When the backup is done, check the vc and vum directories for data (and if vco has been used, check if there is data there too). The vc directory should have a vc_ssl with keys in them, and also a vc_data file, which is kind of small (in my test environment of 4 servers, it was 24kb).  The vum will have data there too, as well as update files downloaded from the VMWare site. This will be quite big (several GB).
 
Copy the data
Copy the entire datamigration directory to the external source (if you haven’t extracted it to the external source beforehand).
 
Stop SQL services and copy the databases
Stop the following services for SQL:
-    SQL Server
-    SQL Server Agent
You can now copy the MDF and LDF files belonging to vCenter and the Update Manager from the server to the external source. To speed up the copy, you could shrink the database before you copy the files. My test environment database shrank from 2GB to 200MB.
 
“Nuke and repave”
Reinstall the server with the 64bit operating system, using the same name and same IP information.  There will be issues with vCenter if other IP information is used. Patch the system as you would normally patch a system. Create routes, possible hosts file entries, users and groups. Re-install a SQL 64bit edition, and install the same service pack as the 32bit edition. You can go from SQL 2005 to SQL 2008 without needing extra configuration. I like to configure SQL so that it installs the databases to another disk from the SQL installation.

Copy databases back and configure SQL
Copy the MDF and LDF to the newly installed server, in the directory where the other databases are stored.  In the SQL Server Management Studio, attach the databases.

Make sure vCenter user has been assigned “db_owner” for both the vCenter and Update Manager database, as well as the MSDB database. This is necessary to be able to upgrade the database during the vCenter installation.

Another important thing is that the compatibility level be set to SQL 2005 (90). If the SQL instance you came from used to be SQL 2000 (e.g. when you have an old vCenter that has been upgraded from VI3) it is likely that the compatibility level is set to SQL 2000. Not setting this properly will make the upgrade fail!  

Configure other settings as needed, such as re-implement maintenance plans, and set the recovery model to Simple again. Do the same for the Update manager database if you use that.

Recreate ODBC connections
To recreate the database connections you will need DSN’s. Go to the Data Sources (ODBC) tool in Control Panel -> Administrative Tools and add a system DSN for the vCenter database and Update Manager. If you plan to migrate update manager as well, you need to use a 32 bit DSN for that. Run c:\windows\syswow64\odbcad32.exe to get the 32 bit version, and add a system DSN there.

With me so far? Good, then the fun part can start: Getting vCenter back up and running.

Installation/Upgrade of vCenter server 4.1
Copy the datamigration directory back to the server, and for speed sake, copy/extract the vCenter installation directory to the local harddrive. In any case you need to have this available when the datamigration installer is started.
 

Open a command prompt and change to the datamigration directory. If you are using Windows 2008, open a command prompt by "run as administrator". Type “install” to start. The installer script will check to see if the vCenter data is available, and tell you if all data is available or not. If it is available, it will use these settings during installation. It will ask for the directory of the installation media (e.g. D:\vim410).
After this it immediately checks for the Upgrade Manager upgrade data and ask for the installation media again. This is the same directory as before.

As a sidenote: I have had three or four attempts of testing this in a test environment, and a the first times the installer said there was no upgrade data available. I needed to run the backup script on the source again. That is why it is important to check for the existence of files during the backup and to test this upgrade before you try this on a live environment.

After the question for the Upgrade Manager has started, the installer for vCenter 4.1 will start.  Select the vCenter DSN and answer “yes” to the question if the database should be upgraded. At one moment, the question will come if the host agents can be upgraded automatically. I answer no.
In vCenter the ESX hosts will be in a disconnected state, but the VM information will be seen, and you should be able to see the performance data of the VM’s. I select no, because you can still fall back to the old installation until the complete installation finishes (remember that disk you took from the raid 1 set, or the backup of the previous installation? The ESX hosts can then still be managed by vCenter 4).

After the installation, the installation for Upgrade Manager will immediately start. Follow the instructions as they appear on screen.

Install the updated Virtual Infrastructure Client from the vpx directory in the installation media (vi-client.exe) and then connect to the vCenter. The hosts are in a disconnected state, but you can simply rightclick the hosts, and click "Connect". The agent will update, and HA will be re-enabled.


This should be all you need to complete a successful migration.

Sources
Technodrone
VMWare

Note to self: Make some screenshots next time...