This is a quick one, as I’m in a training course at the moment, but this message from VMware just hit my inbox about a serious problem with Update 2. Full details can be found here. The text of the message is below:

“Dear VMware Customers,
Please find the latest update about the product expiration issue. From this point on, we’ll provide an update every two hours. Thanks.

Problem:
An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:
• VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2
• Reports of problems with ESX 3.5 U1 with the following 3.5 Update 2 patch applied.
        1. ESX350-200806201-UG
• No other VMware products are affected.

What has been done?
• Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
• VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
• A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs. A new static page has been published at http://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.
• The phone system has been updated to advise customers of the problem
• Vmware partners have been notified of the issue.

Workarounds:
1. Do not install ESX 3.5 U2 if it has been downloaded from VMware’s website or elsewhere prior to August 12, 2008.
2. Set the host time to a date prior to August 12, 2008. This workaround has a number of very serious side affects that could impact product environments. Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken. These include, but are not limited to database servers, mail servers, & domain administration systems.
Next Steps:
VMware to notify customers who have downloaded this version and provide an update every two hours.
Resolution:
VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today. The target timeframe is 6pm, August 12, 2008 PST.

FAQ:
1. What would this express patch do?

More information will be provided in subsequent communication updates.
2. Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?

Yes. We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week. We will provide an ETA for the update patch bundles subsequently. NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA. They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.
3. Why does VMware plan to reissue the upgrade media before the patch bundles? That is a wrong priority call!

This is not a matter of priority. Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.
4. Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.
5. Does this issue affect VC 2.5 Update 2?

No.
6. What is VMware doing to make sure that the problem won’t happen again?

We are making improvements on all fronts. The product team had endeavored to deliver a release with support customers deem important. But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused. We have identified where the holes are and they will be addressed to restore customers’ confidence. ”

I’m thinking that next week when I’m back in town might be a little busy with this one …

**Update 1** - 13/08/2008 - 14:15 (AEST)

VMware have released an express update for this issue. Please look here for more information concerning the patch and its implementation. You can use VUM or run esxupdate on the affected hosts to install the patch. Which is nicer than frigging about with times on your ESX nodes. I haven’t had time to run the patch, but I’m sure I’ll hear about it from my customers if it is problematic :)

**Update 2** - 13/08/2008 - 15:30 (AEST)

I  may have spoken too soon regarding express patch deployment - according to the latest missive from VMware, the express patches have only been tested using esxupdate, not VUM.

“There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5 Update 2. They are specifically targeted for customers who have installed or fully upgraded to ESX/ESXi 3.5 Update 2 or who have applied the ESX350-200806201-UG patch to ESX/ESXi 3.5 or ESX/ESX 3.5 Update 1 hosts. For customers who haven’t done either, these express patches should not be applied.
To be noted is that these patches have been validated to work with esxupdate. However, testing with the VMware Update Manager is still under way. In subsequent communications, we will provide confirmation whether the patches work with VMware Update Manger or if a re-spin is required.
We are currently testing an option to apply the patch without requiring VMotion or VM power-off and re-power-on at the point of patch application. To immediately refresh vmx on the VM, one can VMotion off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs can be powered off before the patches are applied and powered back on afterwards.”

You having fun yet?

**Update 3** - 15/08/2008 - 09:30 (AEST)

Apparently VMware have now released updated install media for ESX 3.5 Update 2 and ESX3i, including upgrade files, etc. So go to http://www.vmware.com/download and get yourself some new media.

Sometimes you won’t have access to VMware’s Update Manager. But that doesn’t mean you shouldn’t be patching your hosts. Start by visiting the VMware website for the latest contents file and the latest binary patches. I don’t recommend you rely on the patches you downloaded 3 months ago, as VMware will invariably have superceded a number of patches since then. I also don’t recommend attempting to mount a DVD-R of burnt patches on a CD-ROM drive, as this will also fail. You then need to scp or otherwise copy everything to somewhere like /ESXpatch. I don’t know whether it is still the case, but running things like patches and installation routines from VMFS-2 used to be a problem. Whether that’s really an issue with VMFS-3, I’m not so sure. You will also need the latest copy of “ESX-3.5.0-contents.zip”. This can be found here.

Unzip ESX-3.5.0-contents.zip in the root of /ESXpatch. This will give you contents.xml and contents.xml.sig in the root of the directory. The xml file points to the current patch bundles and their default location. Kind of like what I used to do with my esxupdate script, but way more Web 2.0, because it’s all xml and stuff.

You then need to unzip all of the patch bundles that you’ve downloaded from the VMware ESX patch repository. If you used the datastore browser to upload the zip files to a VMFS volume, do the following:

mv /vmfs/volumes/datastorename/patches/ /ESXpatch/

Of course, if you had a working copy of Veeam’s FastSCP on your Vista laptop, you wouldn’t need to do that step. But what do I know?

cd /ESXpatch

unzip ‘*.zip’

This will extract all of the zip files ready to use.

Delete the zip files

rm -f *.zip

I recommend you enter maintenance mode before you do decide to do your patching, as this method won’t automatically evacuate guests on the host to other nodes in the DRS cluster. To enter maintenance mode from the Service console, run

vimsh -n -e /hostsvc/maintenance_mode_enter

Obviously you’ll need to exit Maintenance Mode once the patches and reboots are all done.

run esxupdate update from /ESXpatch and away you go …

I also recommend reading the ESX Server 3 Patch Management Guide. This is a very useful document for understanding just what is going on when you’ve patching your environment and gives some useful suggestions for ways of doing things. Most of the above could be put in a simple shell script, with the only manual process being downloading the patch files.

I just heard that “VMware ESXi will be available for download at no cost as of the week of July 28, 2008″. I think this might make things a little more interesting in the marketplace.

Sorry for the break in posts. A lot has been happening but not a lot that would mean much to my loyal readership of 3. I’d like to post some more information about what I’ve been doing with Backup Exec at a client site recently but I still need to get a few more facts from various people before I launch into that particular tirade. Instead, I’ve been meaning to post some stuff on Henry Rollins for a while now, but have forgotten every time I’ve had a moment to do it. So today is Henry Rollins Appreciation Day. You won’t find that on any calender, I just made it up. I first came across Henry when I saw the clip for “Low Self Opinion” when I was in high school - and I was hooked. I think Henry is important and prolific and to celebrate I made a mix-tape on CD of some of my favourite tracks that he has contributed to. There’s only a bit there, and he’s done a lot of stuff, including books, CDs, spoken-word tours, acting, radio and a TV show. Read some more about Henry here. I’m currently reading “Smile, You’re Travelling”, which chronicles his travels during 1997 - 1998. You should check out Rollins Archive for a complete archive of the radio work he’s been doing. And here’s my favourite clip from his Rollins Band days:

And without further ado, here’s my Henry Rollins mix-tape:

Neon - Rollins Band
Right Here, Too Much - Rollins Band
Do It - Rollins Band
Just Like You - Rollins Band
Rise Above - Black Flag
The End of Something - Rollins Band
Hard - Rollins Band
Burned Beyond Recognition - Rollins Band
Modern Man - Black Flag
Black Coffee - Black Flag
Slip It In - Black Flag
I Can’t Get Behind That - (with William Shatner)
Civilized - Rollins Band
Shine - Rollins Band
Turned Out - Rollins Band

I thoroughly recommend checking out his spoken word DVDs and try and get a ticket to the “ageing alternative icon” show if he’s ever in your part of the world. And he will be.

I spent a good part of the long weekend in Sydney doing a CX300 - CX3-20 upgrade. The process itself was reasonably seamless, although my heart skipped a beat when the utility partition wouldn’t boot. Apparently this happens frequently enough that EMC wrote a Service Note about it, along the lines of, “try it again a few times and it should work”. And it did, and I was able to continue with the conversion. You’ll be happy to know that no data was lost, and thus no service partners were harmed.

I stayed at a hotel nearby. I didn’t have high hopes as I’d stayed there before and it was comfortable, but not amazing. So when I walked into my room I was pleased to see this:

Hooray, a slightly nicer TV than I normally get when travelling. The cool thing about this setup, however, was the AV panel they’d installed in the wall:

You could plug in composite video, vga or hdmi. It even had a usb jack to use as a charger if you needed. I thought it was kind of neat. The fact that they had set 4:3 content to pan and scan meant that the Broncos game I watched on Friday night looked like they had a lot of thicker than usual players, but the DVDs I played off my laptop looked just fine at 1360 * 768.

I had lunch with Viktor Guard the other day and he told me about Mouthguard’s upcoming European tour. They are playing a farewell show at the Jubilee on Saturday 12 July. Get yourself along to see one of Brisbane’s best hardcore / street punk bands. Be sure you get there in time to see Provoke as well.

As I mentioned previously, I had the opportunity last week to use VMware Converter 3.0.3. It worked a charm on a slightly odd P2V. Odd in the sense that the customer ripped 2 hard disks out of a HP Blade BL20P and put them in a pizzabox server, booted it up, logged me in and said go fo it. As the release notes promised, you can now “convert individual volumes on a single physical disk from the source physical machine to separate and independent virtual disks across different datastores”. This would have made things a lot easier at quite a few sites where the physicals used mirrored disks with volumes that needed to go to different datastores. And it seems quicker too. The whole P2V only took less than 2 hours, including mucking about in device manager and testing connectivity.

I noticed this morning that Converter 3.0.3 (build 89816) is now available for download. I will have the good (!) fortune of using it next week, so I’ll post if I have any issues or otherwise. The release notes can be found here.

Jesse was mildly annoyed about being stuck on “low-end” CLARiiON projects recently. Well, I’m lucky enough to have been scraping a little lower down the barrel and have now deployed my first AX4-5. It’s branded as a CLARiiON, but just like its predecessors (the AX100 and AX150), it’s a little, er, different to your average CX or CX3. For a good overview of the tech specs and howtos, etc, have a look at the support site and click on “Learn”. I have only had the opportunity to initialise the array and haven’t loaded it up yet. The Navisphere Express interface is different to full Navisphere, but you can upgrade that if need be.

There are a few things to look out for when you configure one of these arrays. Firstly, it runs a variation of FLARE (Release 23 for those keeping score at home), and the first 4 disks (as opposed to the first 5 in the CX and CX3) are used for this code and other array features. This means that, when you configure a RAID Group (or Disk Pool), using these disks, you’ll lose approximately 17GB per spindle. It’s not as bad as the 33GB on the CX3, but it can still pooch your storage calculations. Especially when the type of customer deploying this array has little exposure to storage anyway, and might think that 1 TB equals 1024GB. Secondly, the hot spare needs to be on a disk other than a FLARE disk. This is nothing new, but again, these kind of deployments aren’t always done with these kind of constraints in mind. I have a bad feeling I’ll be doing a deployment shortly on a 4-disk SATA shelf that was sold without any hot spare at all. Thirdly, the only way you can assign LUNs (Virtual Disks) to different SPs is via Disk Pool membership. The manual suggests that you make at least 2 disk pools to balance the load across the SPs. Sounds good, except when you’re working on a 6-disk (total) deployment, one of which is already a hot-spare. Unless I did something nasty with 2 RAID 1/0 pools (I can’t afford the capacity penalty), I couldn’t do this. Watch out for storage designs that only look at capacity. Finally, if you’re getting it with SAS disks, make sure the “AX4-5 Expansion Pack” is ordered, as this allows you to actually use the SAS disks, rather than just look at them. A colleague of mine had to wait while this was shipped before he could finish a deployment. The job I did recently had the enabler pre-installed (I guess they rectified that issue). By the way, it looks like it’s manufactured by Foxconn (judging by the big foxconn label on the box and this news item). If nothing else, Foxconn’s press releases are hilarious.

Oh yeah, and we’re throwing a 4-node vi3 cluster and 18 guests at this little puppy. I can’t wait to watch it melt.