My Media project

I have had a digital camera for about 5 years now, and we've accumulated a lot of pictures from it. The biggest problem we've run into is that we have a tendency to not ever print any of the pics, nor really to organize them, or really do anything else with them. In the face of my fourth hard drive "data event", I'm looking for The Better Way to deal with the volume of digital media I'm saving.

What am I trying to do:
1. Never lose an original piece of digital media (photo, video, CD, .m4p, etc)
2. Have my digital media accessible so we can use it, in reasonable format (web-thumbnail, .mp3, etc)
3. Not keep a buncha copies of everything
4. Do it automatically as much as possible

So far what I have is about 350GB of potentially-original pictures and video on our home computer and on a set of DVDs. (m4p's are not included yet, and CDs are still on the shelf). I've got Apache sharing them out via WebDAV onto our local lan. I'd like to turn on SSL client authentication (distribute self-signed certs to our laptops) and be able to mount that drive up from any net connection. But there's a couple of minor issues (I'll talk about them later). I'm seeing the limits of my Unix-oriented thought processes when I'm running on a Windows XP Home platform.

For example, #3 above is easily accomplished -- "find . | xargs md5sum | sort" piped to a short awk script... if the md5sums match, diff the files, if they're identical, remove the second, and hardlink a second "copy" in its place. But Windows doesn't have find, xargs, sort, md5sum, awk, diff, or ln. (without cygwin)

#2 I think could be solved with a tool like Subversion... Check in the original media (take a CD for example...) titleoftrack1.wav and tag that version "primary". Then check it out, oggenc it, "svn rename .wav .ogg" and tag that version ogg-hq. Similarly, encode a mp3-hq, mp3-lq, etc. (I only have a 2GB ipod so I like 64kbps vbr mp3's there, but can afford higher-quality on my laptop or home stereo)

#1... I burn two copies of the original to DVD+R, md5sum the original (store the md5sums on the DVD too) and compare them after writing. I need 2 verified copies. One should be offsite, the other in a storage container onsite. (offsite will be implemented someday)

And that just leaves 4. How the **** do I automate all this?

--Joe

Simplified ESX patching

Here's what I've set up for our VMware ESX servers...

I have a space that's accessible via HTTP (snippet from httpd.conf)

<Directory /usslsbds001/esxpatches>
        Options +Indexes
        Order allow,deny
        Allow from all
</Directory>

Alias /esxpatches/ "/usslsbds001/esxpatches/"

In there, I have directories corresponding to dates VMware has released patches (that I'm interested in)

# pwd
/usslsbds001/esxpatches
# ls -l
total 33
drwxr-xr-x   5 root     root           5 Dec  6 15:24 20071115
drwxr-xr-x   8 root     root           8 Dec  6 15:25 20071130
drwxr-xr-x   3 root     root          13 Dec 12 14:13 latest
drwxr-xr-x   2 root     root          14 Dec 12 14:14 packed

packed has the downloaded tgz files. $YYYYMMDD has the extracted patches for that date, and latest has the unpacked directory of 3.0.2 update 1, and symlinks ESX-1234567 -> ../YYYYMMDD/ESX-1234567. When a patch is superceded, I `chmod 0` it, and remove its link from latest.

I also have a scriptwriter that generates a set of esxupdate commands:

# cat ../latest/make-install.sh
#!/bin/sh
# generate an "install" file for the ESX patches in the current directory

DS=`date +%Y%m%d%H%M`

ls | grep -v install | while read patch ; do
echo "esxupdate -n -r http://`uname -n`/esxpatches/latest/$patch update" >> install.$DS
done

rm -f install && ln -s install.$DS install

All of this rolls together on the ESX service console by simply doing (make sure HTTP client is open in the firewall)

GET http://thestorageplace/esxpatches/latest/install | sh

and rebooting...

--Joe

Straddling the firewall with Zones

Our zonehosts have multiple NICs, on multiple subnets. This means that they have multiple default routes defined, so non-local packets are passed to those default routers in a round-robin fashion. In the past, this has not been a problem, because these default routers are actually just routers.

However now, I am creating a set of zonehosts that will be straddling a firewall. And like any good firewall, they will drop packets that are coming in on the "wrong" interface. So here's what I had to do to make this work:

Here's the config for this example:
ce0 (192.168.1.10/24) -> fw interface 192.168.1.1
ce1 (192.168.2.10/24) -> fw interface 192.168.2.1

On the global zone, edit /etc/ipf/ipf.conf to add the following rules for each interface
block out quick on ce0 to ce1:192.168.2.1 from 192.168.2.0/24 to any
block out quick on ce1 to ce0:192.168.1.1 from 192.168.1.0/24 to any

Now all the packets are put on their correct interface.

The only remaining question is "how does this deal with IPMP and link failures". That's something for this afternoon's research.

--Joe

Solution to a VMware license problem

I'd been having a problem with our VirtualCenter installation: I'd removed an ESX server from the inventory, then tried to add it back. This operation would fail with an error message of "There are not enough licenses to perform the operation", and an event would show up reading "Not enough CPU licenses".

Now, we have plenty of VC agent licenses, (especially since I'd just removed that same server from the inventory) so I opened a call with VMware. After making the mistake of calling it a license problem (which bounced me to a FlexLM-only support group that couldn't bounce me back -- but they did validate the license file I was using, and verify that yes, we are really licensed) I was able to talk to a moderately useful representative.

We walked through the log collection process, gathered a bunch of data, discovered a corrupt VM in the inventory (removed it), gathered more logs, and I went home for the day. The next morning, the ESX server added with no problems. So I closed the case.

Now, over the weekend, we had one of our ESX servers die. I got paged and was told (third-hand: the user reported to ops who reported to another SE who told me) that something was wrong with $otherserver. Oh well, I logged in and could tell what they were complaining about -- $server was unresponsive. Unfortunately, I hadn't turned on HA on that cluster, so it didn't fix itself automatically, and I wasn't able to migrate the VMs to the other host (the VMs are on shared disk) So I deleted $server, added the VMs to the inventory via $otherserver, booted the VMs, and went on with my thanksgiving.

Today, when I booted $server (power was off, and I didn't have the DRAC configured, also the KVM was unplugged -- I think this was the original problem) and tried to add it back to the inventory, *POOF* same "There are not enough licenses to perform the operation". So do I open another mostly-useless support call? No! I'll fix it myself this time.

`strings -10 /usr/lib/vmware/vpx/vpxa | grep / | more` eventually found the config file /etc/vmware/vpxa.cfg. I "service vmware-vpxa stop" and then mv'd that config file to a backup, and added the server back in.

And the fscking thing worked. Grr. The newly-created vpxa.cfg file is exactly identical to the old one too.

--Joe

Learn something new

Every once in a while I pick up a new trick... Here's solaris's answer to "what if a file has weird whitespace in its name". GNU userland has "find -print0" and xargs0, since you can't have a \0 as part of the filename, so it's safe to use as a delimiter.

find [ ... ] -exec cmd {} +

I had to use this when I was searching and cataloging (and checksumming) files from various previous hard drives. I had transferred all the data over to a ZFS pool (with compression) from a couple of Windows installs, and needed a good way to walk through "/tank/hdc/Documents and Settings/" and "/tank/hdc/Program Files" nicely. And out (of google) pops something that I've missed for years.

Not that I would have really expected to look for this gem in the man page, since I already knew how to use find. But there it was. I guess it's a documented interface now.

--Joe

(P.S. with multibyte characters in file names, is it really safe to assume that \0 will no longer occur?)