Solution to a VMware license problem

I'd been having a problem with our VirtualCenter installation: I'd removed an ESX server from the inventory, then tried to add it back. This operation would fail with an error message of "There are not enough licenses to perform the operation", and an event would show up reading "Not enough CPU licenses".

Now, we have plenty of VC agent licenses, (especially since I'd just removed that same server from the inventory) so I opened a call with VMware. After making the mistake of calling it a license problem (which bounced me to a FlexLM-only support group that couldn't bounce me back -- but they did validate the license file I was using, and verify that yes, we are really licensed) I was able to talk to a moderately useful representative.

We walked through the log collection process, gathered a bunch of data, discovered a corrupt VM in the inventory (removed it), gathered more logs, and I went home for the day. The next morning, the ESX server added with no problems. So I closed the case.

Now, over the weekend, we had one of our ESX servers die. I got paged and was told (third-hand: the user reported to ops who reported to another SE who told me) that something was wrong with $otherserver. Oh well, I logged in and could tell what they were complaining about -- $server was unresponsive. Unfortunately, I hadn't turned on HA on that cluster, so it didn't fix itself automatically, and I wasn't able to migrate the VMs to the other host (the VMs are on shared disk) So I deleted $server, added the VMs to the inventory via $otherserver, booted the VMs, and went on with my thanksgiving.

Today, when I booted $server (power was off, and I didn't have the DRAC configured, also the KVM was unplugged -- I think this was the original problem) and tried to add it back to the inventory, *POOF* same "There are not enough licenses to perform the operation". So do I open another mostly-useless support call? No! I'll fix it myself this time.

`strings -10 /usr/lib/vmware/vpx/vpxa | grep / | more` eventually found the config file /etc/vmware/vpxa.cfg. I "service vmware-vpxa stop" and then mv'd that config file to a backup, and added the server back in.

And the fscking thing worked. Grr. The newly-created vpxa.cfg file is exactly identical to the old one too.

--Joe

Learn something new

Every once in a while I pick up a new trick... Here's solaris's answer to "what if a file has weird whitespace in its name". GNU userland has "find -print0" and xargs0, since you can't have a \0 as part of the filename, so it's safe to use as a delimiter.

find [ ... ] -exec cmd {} +

I had to use this when I was searching and cataloging (and checksumming) files from various previous hard drives. I had transferred all the data over to a ZFS pool (with compression) from a couple of Windows installs, and needed a good way to walk through "/tank/hdc/Documents and Settings/" and "/tank/hdc/Program Files" nicely. And out (of google) pops something that I've missed for years.

Not that I would have really expected to look for this gem in the man page, since I already knew how to use find. But there it was. I guess it's a documented interface now.

--Joe

(P.S. with multibyte characters in file names, is it really safe to assume that \0 will no longer occur?)

wishful feature: zfs splitfs

Example:
I have a tank/data filesystem, with my important "stuff" in it, including /tank/data/oracle and /tank/data/webcontent. This is a production system, so I can't shut down to move the data around. I need to quota off the web content so it doesn't run Oracle out of space.

So what I'd like to do is...
zfs splitfs tank/data/webcontent
zfs set quota=5g tank/data/webcontent

Conceptually, it seems simple enough. Just create the appropriate new zfs filesystem entries in the pool with its root inode pointing at an existing directory. No data copying necessary.

Unfortunately, I think it would not work because there may be open files on the new (-ly partitioned) filesystem, so the (fsid,inode) pair on those open files would have to be changed to be (newid,inode) on all processes. Atomically. As part of the update to the zpool metadata. Or else the kernel would have to be able to realize that the same inode is referenced by two different filesystems. :(

--Joe

Installing SunCluster

We've bought into Sun Cluster (AKA Java Availability Suite), and it's my job now to install it. I have 3x SunFire V490s, 32G memory each, 4 physical CPUs (8 cores), each has a quad-gig card (plus its 2 onboard nics) and a dual-port SAN card.

Solaris 10 update 4 (08/07). Sun Cluster 3.2.

Notes on the installation:

So far, it's been pretty straightforward except when I was trying to create the cluster. When it rebooted the first node, it never noticed that the node had rebooted in cluster mode.

This is because the rpc/bind (portmap) service is set to only allow local connections out of the box. A quick "svccfg network/rpc/bind setprop config/local_only=false" (on all nodes) and cluster is now up and running.

Bug opened.

--Joe

Installing OpenSolaris b63 on VMware Workstation 5

I've got the Big Disk Server (an x4500) and since it's going to be an iSCSI target, I have to install it for now with OpenSolaris post-build-54. b54 is where the iscsitgt code got its putback.

Anyway, I've installed b63 on that monster, but before I can get everything working, I have a week-long "vacation" to learn EMC Control Center administration. So what should I do while I'm free? I'll work on setting up the rest of the administrative nicities that I need for the BDS.

Since I already have VMware Workstation on my new laptop, I'll install a b63 box, give it a couple of virtual disks for the ZFS layer, and see what code I can crank out while I'm gone.

It's never that easy, though.

Bug1: recent builds of OpenSolaris (specifically the mpt driver) cause VMware to crash. So it's IDE disks for me.
Bug2: the default X config makes the screen resolution bigger than my laptop's LCD, so I have to scroll through. Since I prefer a text-based install, I'd rather turn off X entirely.
Bug3: There's almost no documentation on how to get it to do that. There's the old "nowin" command line option (still in this version according to the docs) but I can't figure out how to pass that to grub. And the menu I'm given has 3 options (Install, Add drivers, or Shell) rather than the 7 the documentation shows.
Bug4: I don't want to install the whole distribution. The damn thing beeps if I haven't selected things correctly. Even though I have my laptop muted. And the VMware audio disconnected. And a headphone plug in the jack. How the fsck is it getting the beep through?

Bug3's workaround is to use the "Solaris Express" menu option in grub, rather than "Solaris Express Developer Edition". Grr.

--Joe