Moving RHEL to ESX

We have a cloud application that we have purchased for in-house deployment. It's a long story that I won't share, but we have been seeing performance problems with it, and have ended up with a (physical) server in our datacenter that they have configured just like their cloud resources, so we can compare performance on our (virtual) system against the way they run things.

To make sure we capture the most data we can about this system (and to demonstrate that it either is or is not the virtualization layer causing slowness) we've been tasked with copying the physical server into a similarly-configured virtual machine.

Unfortunately, VMware Converter apparently can't do Linux. So we had to use the more standard drive-image-transfer toolkit of dd and netcat. But even after the image was transferred, the kernel would crash because it couldn't find the root disk.

This is to be expected, but only Google knows how to solve it. If you know the right keywords. Good screenshots of the process are at http://virtualaleph.blogspot.com/2007/05/virtualize-linux-server-with-vmware.html but it's modprobe.conf not modules.conf in RHEL5.

So here's the steps I took:

Boot from the (correct) RHEL install CD #1 with "linux rescue". Note that it has to be the correct RHEL version and architecture. Since the appliance was running 5.2 x64 edition, my 5.1 x86 cd didn't work, and I had to download a different CD1.

Skip the networking config since it won't help, search for RHEL installations, and continue to the root shell (in read-write mode)

chroot /mnt/sysimage and edit /etc/modprobe.conf. Change the eth0 module alias to pcnet32 (you can remove the eth1 alias if you don't have a second nic in your VM) and change the scsi_hostadapter alias to BusLogic. (again, you can remove other aliases if you want)

Then copy /boot/initrd-.img initrd--phys.img as a backup, and build a new initrd file with the new devices: mkinitrd -v -f /boot/initrd-.img

If that works, you should be able to boot the VM and have it come up cleanly.



Reflections on x4500+ZFS+NFS+ESX

I was asked about my thoughts on running ESX over NFS to a ZFS backend. For posterity, here they are:

x4500+ZFS+NFS+ESX is a quite functional stack. There are a few gotchas that I've run into:

First, the ESX "storage delegate" functionality doesn't. This is supposed to change the EUID that the ESX server sends with it writes. Well, it does for most of the requests, but not for things like creating the VM's swap file. So you pretty much have to export your NFS shares with root=vmkernel.ip.address

We have many ESX servers, so keeping the sharenfs= parameters got unweildy. I ended up putting them in a text file in the NFS share for easy editing, and when I have to add/change an ESX server, I edit the file and zfs set `cat zfs.shareprops` /pool/path/to/share

NFS is much better than iSCSI. At least in the version I did iSCSI testing, all of the ZFS volumes presented from OpenSolaris were recognized by ESX as being the same disk. This meant that I had a dozen paths to the same vmfs datastore, some 100GB, some 500GB, etc. This Was Bad. NFS made it better.

NFS also gives you a couple of other benefits: On NFS datastores, the vmdk files are by default thin-provisioned. This means that if you give your VM a 5TB vmdk, and don't use more than 10GB, it takes up 10GB of capacity on the physical disks. It's also much better understood by troubleshooting tools (wireshark) so it's easier to find problems like the storage delegate issue above. Also, it's a first-class citizen from Sun. NFS serving has been in Solaris since 1994, and isn't broken by the latest Nevada builds. Sun takes NFS seriously.

The downside of NFS is that ESX makes all its requests O_SYNC. This is good for ESX but bad for ZFS. Your nvram cards should help a lot. I ended up with a different solution: The business agreed that these are not Tier-1 VMs, and they're not on Tier-1 storage. So I've turned off all ZFS sync guarentees with /etc/system:

* zil_disable turns off all syncronous writes to ZFS filesystems. Any FSYNC,
* O_SYNC, D_SYNC, or sync NFS requests are services and reported completed
* as soon as they've been transferred to main memory, without waiting for
* them to be on stable storage. THIS BREAKS THE SAFETY SEMANTICS AND CAN
* CAUSE DATA LOSS! (clients have moved on thinking the data was safely written
* but it wasn't)
* However, in our case, we can afford to lose this data. For DEV/Test systems
* rollback to the latest (hourly) snapshot is considered acceptable.
set zfs:zil_disable=1

As the comment says, this would be a bad thing. But I know that the vmdk files are crash-consistant every hour and that's OK to the users. If they lose an hour of work, it's annoying but worth the cheaper storage.

Finally, and most importantly:

MAKE SURE YOUR POOL IS CONFIGURED FOR YOUR WORKLOAD. Vms are effectively a random-read and random-write workload. There is no sequential access of the vmdk files except when you're cloning a VM. So you have to understand the read and write characteristics of your zfs pool. RAID-Z and RAID-Z2 always read and write a full RAID stripe every time. This means it has to read from all of the disks in the pool to return a single byte of data to the ESX host. Mirrored pools, on the other hand, read from a single disk, and if the checksum is correct, passes it back to the ESX host. So in my case, I can have 44 simultaneous read requests from the ESX servers being serviced at the same time (44 disks in the pool) and/or 22 simultaneous writes (each write is written to two disks). Basically RAID-Z[2] is bad for random workloads, but mirroring is expensive.

With this in mind, performance on the thumper is excellent. We can easily saturate the onboard 1Gbps network link with NFS traffic, I've got link aggregation and can easily saturate the combined 2Gbps link. I haven't seen what happens with 4 uplinks, but I'd expect that the network will still be the slowest part of the chain. Doing basic I/O benchmarks on the thumper, I can get 1GBps out of the disks. Yes, that's 1GB per second.


How to grow an iSCSI-presented zvol in 3 easy steps

Well, ok, it's not quite 3 easy steps.

A couple of things that don't work: iscsitadm modify target -z . This only works if the iscsi target's backing store is a regular file, which in the case of a zvol, it is not.

The easy bit: Make the zvol bigger:
zfs set volsize=200G tank/iscsi/thevol

Now we have to hack around in the iscsi parameters file: Locate the /etc/iscsi/tgt//params.# file that corresponds to the right target and lun and change the <size> parameter to be the new (in hex) size of the bigger volume in 512-byte blocks. Or in other words,
zfs get -Hp volsize tank/iscsi/thevol | perl -lane 'printf("%llx", $F[2]/512)'

Once that's done, apparently you have to bounce the iscsitgtd to get it to reread the params file.

Then on to the initiator...

format c3tAREALLYLONGSTRINGOFDIGITSFORTHEDISKGUIDd0s0 and changing the parameters won't work, since I'm using EFI labels and it says very strongly
partition> label
Unable to get current partition map.
Cannot label disk while it has mounted partitions.

So I have to go in the other way. While I'm in format, print out the current partition table, and make note of the Last Sectors for the slices. Also, run prtvtoc against the disk to get any other useful bits.

Then I can make the actual partition changes with fmthard:

At first, just copy in the line(s) for the slices you already have, but move slice 8 to the end of the disk:
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 34 251641754 251641787 /zones/mars/data
8 11 00 419413982 16384 419430365

Then (check it in format to make sure the disk is still healthy) change the Last sector and sector count for the real partition. (Last is s8's first -1, and the sector count is s8's first -34)

Then it's a simple growfs -M /zones/mars/data /dev/rdsk/c3tAREALLYLONGSTRINGOFDIGITSFORTHEDISKGUIDd0s0



Goodbye to a problem indicator

We have a new Sun T5220 with the new Niagara-II chip in it. I don't know if it's 8-core/8-thread or 4-core/16-thread but it shows up in Solaris as 16 CPUs.

I'm running a cpu-intensive process (bzip) and prstat is only showing it as using 1.2% of the CPU. On another older system, the same bzip takes up 25% of the 4-proc box.

On the old system, it's clear that bzip is a single-CPU bottleneck (because it's single-threaded). On the new one, well running full-out but only taking 1% doesn't look like much.



Forcing kernel dump on Opensolaris x86

Just in case you were caught off guard by the documentation that points to using "rip::c" in mdb, that apparently doesn't work on Opensolaris, at least on build 70 that we have installed on our x4500s.

But on the bright side, I found a mention at wikia (http://opensolaris.wikia.com/wiki/Miscellaneous_FAQ) that says I should use $<systemdump. And that works.


Firewall + IPMP + IPF + Zones = lost packets


There's no intelligence in IPF to know where to send packets when IPMP detects a link failure.

So, even though the zone will get all the packets destined for it, the global kernel will pick an interface based on its global route table, pass the packet down the wrong interface, ipf will bounce it to the FAILED interface (below the ipmp layer, apparently) and the packet is never delivered.

So I need something that will notice when an IPMP failure occurs, and rewrites the IPF rules to respond to the networking topology change. Look for my PF_ROUTE socket watcher program to be posted shortly.


Media server v0.1a

To solve the automation problem, I'll use Linux. Since I don't have any spare computers laying around any more (I'll turn in my geek card), I've gotta set it up in a VMware instance. Luckily, VMware server is free, and I have disk space here.

Two disks... 8GB for OS, and a 32GB "data" disk to start with. Both sparsely-allocated, cuz I don't have that much disk space.

I know debian better than any other distro. So, a base install, no tasksel, update and upgrade, then installed ssh, apache2, subversion, subversion-tools, and sudo. So far I'm using 515MB. And it can't do anything yet.

Someday I'd like to switch it over to Solaris so I can run the data under zfs, but for now, I like the convenience of aptitude.

In the mean time, it's LVM I guess... I pvcreate'd /dev/sdb, vgcreated dataVG /dev/sdb, and lvcreated dataLV in it. Then mke2fs -j /dev/mapper/dataVG/dataLV and tuned it to want a fsck every 365 mounts or 730d. Mounted it as /data, and I think I'm ready to start developing.

First some data, though... I grabbed about 350MB of data from various picture CDs, copied it to /data/media/$cdname, and copied them around again to make sure there were duplicates in the data...

More to come...


Getting code/screens to show up nicely

After a knowledgebase search, I found Adam Flater's blog entry http://adamflater.blogspot.com/2007/02/on-source-code.html describing a nice-looking way to do code on blogger.

I've added to my template the following style definition:
pre.code {
overflow: scroll;

and now I can wrap the code with <pre class="code"> and have it look nice.

Thanks, Adam!


What's going onto the disk?

On our thumpers, `iostat -nxz 5` gives a good picture of what's happening on each of the disks, and `zpool iostat 5` gives a good overall picture of how fast things are going at the moment, but neither of these break the picture down into "Who's writing?"

A Solution:
# fsstat -T u `zfs list -H -o mountpoint -t filesystem` 5
0 0 0 0 0 0 0 0 0 0 0 /uscisbds001
0 0 0 106 0 0 0 0 0 53 217K /uscisbds001/esx3
0 0 0 0 0 0 0 0 0 0 0 /uscisbds001/esxpatches
0 0 0 2 0 0 0 0 0 0 0 /uscisbds001/isos
0 0 0 0 0 0 0 0 0 0 0 /uscisbds001/nfs
0 0 0 0 0 0 0 0 0 0 0 /uscisbds001/saperisrv1
0 0 0 0 0 0 0 0 0 0 0 /uscisbds001/saperisrv2
0 0 0 2 0 0 0 0 0 0 0 /uscisbds001/templates
0 0 0 1.56K 0 0 0 0 0 654 15.8M /uscisbds001/temprestore

I can see that somebody's writing a buncha data to temprestore, and a little bit is happening in the esx3 directory.



Fun console error messages

For you long-time Solaris console message debuggers out there, try this one. It's an old (Solaris 8) www server. This is what's scrolling on the serial console.

r ,R qAeStCdQB:o k0 x603,0 8F R U : 0 x 0
s b u s @V1nfo,:0S/mQiLsG C , i s p @ 2 , 1 0 0 0 0 /esidl@N5m,e0: (ws d 5 )[:

a E rerso re :fHorrw rC ormom
@WvReIlG: sRue@tfr,y/aLbCliesi u o e 0 8 > , A C Q xE,rFrUo r0 0L
B, 0R0e/qdu@e,s tse5d:
0 l o c kR:q e1t1d3B9o7k8 8285w i e 1 ) E r r L v lE rerroarl
V e n d o r : rSoy mlbci:o3s9 4 8
e d r y b o S eSrriaa lu bNru m b e r :[
dw S n9e[K yg
ea eSEernos
4 K e y :S :H0a8r d<weadrreu iEurercodr
<) ASSQC :x ,0FxR8U4 x(
CvAeNnNd:o/rs busn@ifq,u/eQ Gc,ospd2e, 0000x/8d45>0)(,s 5A:S
)Q : 0 xr0o, oF RoUm:n :0wxi0e
5 / Q L G Ce,uissep @l2c,k1 05060808/ s d @ 5 , 1 ( srdo6 5l)c::
r r o r feonro :CSommimsa n d : w r i t e ( 1 0 ) e i l N m b r r [ gE
lr o r LeesveeKly: HRrewtrr yraob
r 9 8 2 4E s u @ f 0 Q GE,rsr@o,r0 0B/ldo5c1k :s 67)38:2040 R : 0 0
e u s e l c :S1e0r2i3a2l N u m b e r : r o wl c : 1 0 293[

r eSdern syeb oK e y : H a r d w a r e ESrrrao ru
e 8 4 (S<nveeKnyd oard auen irqou
x c o d eA C0 xx844(>v)n,o rAuSiCuQ :o e00x80>,, FSRQU :x ,0FxU0

pAANRNN:I/NbGs:1 ,//sLbGu,ss@@12f1,000/sQ@L,G Cs,5i:s
@ 2 , 1 0r0o0 0o/ sodm@n5:, 0r t((s0d)5 ) :
ErrorroLrv lf:oRrt yCboem
m a n d :e urseea dl(c1:03)9 0 8 e E r r lEcr:r3o9r0 8L
bv e l : VFnadtra ly
o B l oASc:k0:x 42(5v1n6o7 5n2q0ec kS:r a2l5N1m6e7:5 2w0 9
oe 0V8e>n,dAoCr::0 0,SFyUm bxi
sA N N : / b s 1 , / L C , s @ , 00 0 s @ , s 6 )
v S e r iraol oN uommbne: rr:t ( 0 w 9 [E rgr
e n d o re:u sSey mlbci:o5s3 1 7 2 E r r B o c : 5 3 1 1
: SVenroi: aylb oN u m b e r : w S9r[a um
wS e n9s e
oK e y : eHsa red:wHarrwer Error
0W0RxI8G4 >s)u,@ fA0SQCGQ,:s @0,x000,/ dF5R1U(:d 50:xq0u0e Rc:o0d0e

0 / s dR@q5e,t0d B(oskd 59)4:21 f , 0 / Q L GECr,riLsvpl@ 2e,r1a0l0
: w r iVtneo(:1 0y)b o ESrrrao rN mLee:v ewl : 9R t
er y a b lSen
E B l o cAkC: x342(9v5n4o0 8n0q e c d x 4 ) S Q x , F U x
rArNoNr: TBelAorcak :d i3e2 9s5r4t0r8i0g
:W R I G s u @ f 0 Q G , sS@e,r0i0a/ld @N,u msb5e:r c o 6 6
i w r o o 9o[m ng:
e r r o rR)e n s e K e yE:r rHLavrld wearrael
dt dABSoCk: 605x5824 ( < v e n d o rr ou nlicq:u3e3 0c9o
be 0 x 8e4d>r) ,y bAoS C Q : 0 x 0 , F R U : S0rxa0
@ 2 , 1A0C0 0x04/(svdn@o5 ,n1q e(csdd 6x54)):
fx ,E rRrUo rx
l E r r f rE rormonr: wLieev1e)l : R e t r yEarbrlLev
E s t e dR qBeltodcBko:k 17705373462 7 2 E r r B l c : 7 0 0 3
r r o r eBdlro cykb:o 1 7 5 7 4 2 7 2
V eSnrdao ru:b rS y mwb i o9s
S : 0S8e4r(ivanlo Nnuqmebcedr :x 4 ) wS Q x , F9U[ x
NS:e/nbsse1 ,K/eLyC:i pH2a1r0d0w/adr5e0 (Edr)r
4 > ) , eAuSsCtQd:B o0kx 08,8 6F0R U : 0 x 0 irqou ee ec:oRdtey b0lx
u s @ 1Vfn,o0:/SQmLiGsC , i s p @ 2 , 1 0 0 0 0 / sSdr@a5l,N0m e(:s dw5 ) :9
: E r r oSrn efKoyr aCdoamemEarnod
l w r i tSe:(0180 )< e d r u i u c d x 4 ) , AESrQr oxr, FLUe vxe
G:A RRIeGt rsyua@bfl0eQ
i,is @V,e0n0d/odr5:1 (Sdy5m:b
o s E r r f r C m a d r t E r r LSeeer:iRatly bNeu
m e r :e u s ew l c : 6 49 [
s a r e VEnror:oSrm Kreryr:B oHka r4d
o 0 , FARCU :0 80 x<0eSaCeQE:r r0ernad ourb ru n i q u e9 c
Q WnAqReNcIdN Gx:4 )/ sSbQu sx@,1FfU, 0x/
5LAGNCN,:i/sbps@12,,/1Q0G0,0s0@/,s0d0@/5d,511 ((ds5d:6
E) :
r rEfrrrComra df orrt (C0o m m a n d : wrroi teee : R t y b e
e4u s e l c :E2r4r4o2r L e v e l : REertrr ylacb:l2e4
e q u e sRteeuds eB llocc:k2:1 5180 6 6 4 0 E r o l c : 2 5 6 8 8 E r reodrr Bylbooc k : 1 0 6 6 4 0
S n e K y ar w r r o i o
[ S : 0 8 <Seedrriuailue cNdu mxb4e)r :S Q : 0w0 R U x9
: A N
E/s uS@efn0sQeG ,Kse@y,:0 0H/adr5d1w(adr5e:
rr r o rE
f rACSmCa:d 0rxt8(40 ( < v e n d o Err ruLnvilq ueer acloed
e 0 x 8R4q>k)t,d BAocS:C7Q3:6 80 x 0 , F R U : E0rxr0B
N G : /Vsnbou:sS@m1ifs, 0 / Q L G C , i s p @ 2 , 1S0r0a0 0N/msed:@ 5w, 1 (96 d
n5 ) :
o o m m a nSd::0 xw4r(ivtneo( 1n0q)e c d 0 8 > , A C : 0 0 F UE rxr
0rA RLIeGv e/lb:s 1R,e/tLrCyiasb@l,e0
e/ dR5e1q(ude5s:t
r B lRoqcekt:d B7o3k4 422895444 1 ) E r r L v l e rEarlr
V n o : S m i s SSrear iuablr N u m b e[r w S n e e9:[H r w
er oS
Sn s e KSe:y0:x 4H(avrndow anrqee cEdr rxo4r)
A S CEQr:o 0fxr0C,m aFdR Ur:t e01x)0 e@ 500x(8d4)>
W A R N IrNrGr:L v/ls bRutsy@b1ef
, 0 / Q LeGuCs,ei slpc@:23,91008080 0 / s d @85 , 1 E(rsrdB6o5c)::3

E r r o re dfro ry bCoo m m a n d : w r i t e ( 1S0r)a u b r t E rSrnoerK yL eavdeale: rRoe
0r y a b lSe:
kW R I G s u @ f 0 Q G , s @ , 0E0r/rdo5r1 (Bdl5o:c6:0080
a: 2 6 4r4o4 6o0r8C
d: WrAtR(N0I N G : T h e rAor reaey: Rdtryibvee
r i s eruesteu rlnci:n7g7 ra8n4 E r r o r e d IE/rOo, lwci:t7h7 7e8r
n o 5 ,e dorn yMbood u l e 1 , L u n 0 , s eecitloNrm e1:1 3w9 7 898 8

I f , 0 /AQCL GxC4,(ivsnpo@ 2n,q1u0 0o0e00/8s>d,@A5C,:00 0( sRd:50)0:
1G Esrur@ofr0 QfGo,rs @C,o0m0m0asn@d5:1 (rde5a)d
) E r r f r C m a d r t (E0r r o r L e v eElr:r LRvelt reyraabll
m V n o :ESrmriosr B l o c k : 3 2 9 5 3 3e2i8lr r B o k 5 6 8 8
,W R I G9 [s ugs 8 <> ,NAuCm:b0e0r :R U : 0w0
a/ LSCeinps2e1 0K0esy@:, 1H(adr5d:w
Cr e E rrroo rf
: o d e R0qxe8s4e> )l,o kA S6C3Q0:8 0 x 0 , F R U :E r0rxB0l
G : / sebdurs:@S1mfi,s0 / Q L G C , i s p @ 2 , 1 0e0i0l0N/msed:@ 5 , 1 ([se 5 ) :
5WrRoIrN :L/ebvse1l,:0 QRGe,tsr@y,a0b0l/ed 4 ) S Q x 0 R : 0 0E
e1 (Rde5q:u
r r o r eBulsoec kl:c :7 362824928 2 4 E ro e e l e r a l
r V n o : S m i s e iSleNrmiea:l wN u m9b e
r: Swn e K y a9d[a e E
x S e n sAeC Kxe4y(:<v nHoa rndqweacrde xE4r)r oSrQ
, ARS:C0:0
00AxN8N4: /(b x 8 4 > )r,o AoS CoQm:n :0wxi0e,1 )F R U : 0 x 0r
/ s b uesue@s1ef ,l0c/:Q5L3G1C1, i s p @ 2 , 1 0 0 0r0o/ Bsod: d 5 ) :V2(
e s e : H r wErr rroor m a nSdr:a wurbirt e ( 1 0 )[
uL e v e lA:S :R0e8t r<yeadbrluei
rW3R6I3G5 0/5b9s21 , / L C i p 2 1 0 0 s @ 5 1 ( d 5 :E
r o r Brloo cfkr:C m3a6d3 5w0i5e9120
V e n d oErr:r LSvylm bRitoysb e
e u s e l c : 4 4 9 8 r ESrerrriBaolk :N4u4m9b8e
: we d o : S m9i[s g
U S e rAiCa lx 4N(uvmnboe rn:q e c dw x 4 ) S9Q[ x ,
e xS
:nAsNeN :K/ebys:1 ,H/aLrCdiwpa2r1e0 0Esr@r,o rs

, A SRCqQe:t d0Bxo0k, 1F9R8U8: 0 x 0 )u n i q u e cEordreL v0lx 8a4a>
g @ 1 f ,V0n/oQ:LSGyCb,oi s p @ 2 , 1 0 0 0 0 / s d @ 5S,r1a (usbdr6 5 ) :
l w r i tAeC( 1x04)( v n o u i u o e 0 8 > , AECr:r0o0r RL:e0v0e
w: R e tersy aeb:lHer
s t e dA SB:l oxc4k(:v n2o5 8n0q5e2c4d8 x 4 ) S Q 0 0 R : 0 x
) A NENr:r/obrs 1B,l/oLcCki:p 22150800s5@2,4 8s
c S eRrqieatld BNoukm b6e4r6:4 w 9E[r r B L v e l e r a l
K e y : eHdarr dywbaor e E r r o r
A S C :S r0ax 8u4b r( < v e n d9o r
)u n i q uees ceo:dHer w0rx 8r4o>
,, A S CSQ::0 80 x<0e,d rFuRiUu: o0ex008
sAWCA:R0N0I NRG:: x/
dbAuNsN@:1/fb,s01/fQ0LQGGC,,si@s,p0@02/,d1@0,0 0s05/:s
i@ 5 , 1 r(os do6 5o)m:n
r r y a bVlneo1t0d)B l c : 2 1 5 8 8 E r rEorrr BLoekv:e2l1:6 8R8e
u c k : A7C1 7x347(9v8n4ooerE rBrlc k : S7r1a7 3u7b9r8 4 w 9 [ g
W R I G h r a r i e s r t r i g n ESrerrdi a/l, wNtu mrbne r,: n M d6l , L n 09 [e t r

SAeNnNs:e/ bKse1y,:/ LHCairpd2w1a0r0es @E,r rso5r:

4 A S Cr:o 0ox 8C4m a(d< vretn(d0o r u n i q u eE rcroLdvel 0axa8
> ) , AeSuCsQe: l0cx:03,9 0F8R U : 0 x 0
b WrAoR NlIcN:G3:9 0/8s
u s @ 1 fe,d0r/ QyLbGoC , i s p @ 2 , 1 0 0 0 0 / sSdr@a5 ,u1b r( s d 6 5 )[:g
tWeRvIeGl :h Rreat rryvaeb lser e c d x 4 ) S Q 0 0E rRr:o0r0
r iRgeaq ureoset e/d, wBtl orcnk :, 2n5M1d6l5 8,8L8n 0 e t r 7 7 86

8 A N N : / b sE1r,r/oLrC iBpl2o1c0k0:s d@2,5 1s665)8:8

rSoe nos eo mKne:yw:i e(H0a r d w a r e EErrrroLrv
:e rAaSlC
0 x 0 ,e dFrR SUm:i s0 x 0 5u4n7i q u e c o d e r0ox 8l4o>k) ,1 5A4S7C
e W A R N I N G : STrhael Aurber:a yw d r9i v
2WeR I1G, sLuu@nf, /1L,C ispe2c1t0o0rs @1,6 0s8d254)3: 0o0n RM:o0d0u

WrAoR NoIrN Go:m n/:swbiues1@)1 f , 0 / Q L G Cr,oi sepe@:2R,t1y0b0e0
0 / s d @ 5e,u1s e( sldc6:57)3:6
w r i t ee(d1r0 )y b i s E r r oerr aL euvberl : F a [a l
: R eSqnueeKsyt eadd aBrl orcok
o 5 0 3 3A1S7:1028 < e d r n q e c d x 4 ) S QE rxr,oFrU Bxl
icAkN:I G5 0s3u3@1f701Q2L
ip 2V1e0n0d/odr5:1 (Sdy5m:b
b e r :R q e twd B o k 096[0 E r r LSveer:iRatly bNlu
a S e n s eE rKre yB:o kH:a1r6d4w
br e E rerdorr
do A S C : 0 x 8 4 ( < v eSnrdao ru burn i q u e [c o
n , F RAUC: 008x40(aSdCaQe:E r0rx
ioW AnRqNeIcNdG :x 4T)h eS QA rxr,aFyU dxr
vAeNrN :i/sb sr@eft0uQrGn,isn@g, 0a0n/ dE5r1r(odr5e:d
I / O , rwoi toh oemrnr:nwoi e51,) o n M o d uElreo 1e,e :LRutny b1e,
s : c teours e6 4l
G2W4A R N I N G : / s bruos @l1cf:,704/2Q4L
mC , i s pV@n2o,:1S0m0i0s0 / s d @ 5 , 1 ( s d 6 5 )e:i
oe :E rwr o r9 f
1r C o memsa ned::H rwwrri treo(
0 ) A C x 4 ( v n o u i uE roreo0r8 >L,eAvCe:l0:0 FRa:t0axl

e r r r f r C m a d r tE(r0r)o r B l o c k :r o4 9e4e4:9R2t8y0b

V e neduosre: lScy:m2b4i4o0s E r r B o c : 2 4 4 0
e d rS eyrmiiasl N u m b e r : w e i l N9m[e :
n 9S e
o r K eSyn:e KHya raddwaarr er rErr
xWeR I0NxG8 4s>u)@,f 0AQSGC,Qs:@ ,00x00/,d 5F1R(Ud:5 :0uRe: 0c0o
M o d uVlneo :1S,m iLou n 1 , s e c t o r 2 4S2r8a3 4u5b6r 56,3 0o8n
r K e yS:n eHKayr:dHwrawrre EErrrr
tW0RxI8G4 >h) ,r aA SrCvQr:i 0exu0n,n FnREUr:r d0Ix/0,0e Rc:o0d0e
,WiRsIpG@ 2s,u1@0f000/0L/Csids@@5,,010 /(ds5d16(5d)5::

E rrroo ro foomrn :C ormimea1n)d : w r i t e E r r L v l a t l
l e q e tEdrBrookr 6L8e4v3e l : R e t r y aEbrlreB
ec kR:e q6u8e4s2t
d B l oVcnko:: S6m4i o S r a u b r : Ewr r o9rB
, oAcNkN:: /6b4s
/ LVCeinpd2o1r0:0 sS@y,m bsi5o:s
e r : R q ewt d B o k 92[5 3 8 rSoe revila le rNaulm
a S e n s e rKoe yl:c :H3a9r3d2w
ir e EVrnroo:rS
0 d e 0Sxn8e4 >e): ,a dAaSeCEQr:o ernad ourb ru n i q u e[ gc
x 0 , FARCU :x 40(xv0n
/niWqAeRcNdI NxG4:) /SsQb uxs,@F1Uf ,x0
dQALNGNGC ,sius@pf@02Q,G1,0s0@0,00/0s/dd@551,(0d 5(:s
o 5 ) :
oE rormonr: wfioer( 0C o m m a n d :E rwrrLivtle (a1a0l)
e u s e l ck 3 2 8E4r r o r L e v e lr:o Rlect:r7y2a9b2l

R eeqnuoe:sStmeids B l o c k : 4 8 3 8 3 6 8 0e r a u b r 9
8 E rerso re :BHlrowcrk :r o4
x3 8 3 6 8S0C
W R I G : T e A r a r v r i e uSnenr inaElr oNeu m/b,ewrt: r n w, o o use91[ ug
dWyR:I GH asrud@wfa0rQeG ,Esr@r,o0r0
05 0A SsC5::
x 8 4 (r<ov>L)e,e :AFStClQ
: 0 0e,u sFeR Ul:c :03x300
: L G C ,Vinsop:@S2m,i1s0 0 0 0 / s d @ 5 , 0 ( s de5i)l:N
l 0 ) A C : 0 8 < e d r n q eEcrdr oxr4 )L eSvQe lx:, FFUa txa
N NR:eTqeuAersat erdv rBil oecukn:n 3n2E9r5r4d0 8/0, w t r n , o o c e 1 u n 0 Eertrro1r1 4B7l6o
C :A N3N2:9/5b4s018,0/
oi pV2e,n0d0o/rd:5 1S(ydm5b:i
s r o o o m n : w i e 1 ) E r r LSvelr iearly bNeu
m b r :R q e twd B o k 790[3 3g6
a W A R N I NrGo: lTch:e7 0A0r3r
y d r ievderr yibso r e t u r n i n g a n E rSrroar eudm eI:/ Ow, w9iht
: 1 , ALCu nx 41(,v nsoe cutiour o6e30984>2,4A
0 0 0 /Esrdr@f5r,C1m a(ds de6d51)): ,di5s0p(@d2),
E r rroor efeo:rR eCroamlmea
n d : weruistee (l1c0:)2 1 7 2 e r oE rlrco:r2 1L7e2v
Sl : F aetdarl:
i o c k :A C7:10783 7<9e8d4rr oErr rBk :S r7a1 7u3b7e9:8 4 [ g
W R I G : T e A r y d r v r i e u n n n E r oSee r/i,awlt Nrurmob5e rn:M d2l w, u , s9c[o 7
dWnRsIeG :K/ebys:1 ,H/aLrGd,wsa@r2e, 0E0r/rdo5r1
S5 :A
A S C QR:q e0txe0 ,l cF:R2U4:4 20 x 0 q u e c o dreo 0exe8:4 >a)a,
r a y dVrniovre:rS mibso r e t u r n i n g a n E r reoirleNdu bIr/ O , i9t h
Q , L uAnS :10,8 s<eecdtoo rn q2e1c4d0 6x343)6
GA:N N/:sTbeuAsr@y1dfi,e0 /sQrLtGrCi,gias pr@o2e, 1/0,0w0t0 /rsnd @,5o,n1M d(lsod,6L5u) :,
o 2E8r0r
AfNoNr: /Csoum@mfa0nQdG:, sw@r,i0t0e/(d1501)( d 5 :
r o o oEmrnr:owrr tL(e0v e l : F a t arlo
e eR:eFqtule
s t e d eBulsoec kl:c :215514675 8 8 8 E r r B o k E r r oVrn oBrl oymciks: 2 5 1 6 5 8 8 8
W R I A C x 4 ( vSneor inaqle cNdu mxb84e>r,:A C : 0w0 R : 0 09
5 SAernysdrev rKie yr:t rHiagrad wraoree /E,rwrto rr
u nAMSdCl: 10 xu8 4, s(c<ov e7n7d2o2r
,nAiNIqNu:e/ bcso1d,e0 /0LxC8i4p>2)1,0 0A/SdC@Q,: sd05x:0
m F R U :r o0 xo0
nW:AwRiNeI1N)G : T h e A rrroa ye vder:iFvtelr
i s r eetuusren ilnogk o3n3 3E4r r o r e d I / Or,o wlict:h7 3e6r3r8n
5 , oVnn oM:oSdmuilse 1 , L u n 1 , s e cetiolrN m6e4:
r 9 E
WeRtIrGy ahb lrea < e d o n q e c d x 4E)r rSoCr: 0L0e vRe:l0:0
r vRreiq ueeusntiegda BrlrorcdkI:O 7i6h0e3r0n3 3,6o n M d l , L n 1 dWoRrI GB:l/obcsk1:, /7L6C0,3s0@3,3060
r5 1V(edn5d:o
B S e r iRaqle tNduBmobke r5:0 2 8 w 9 [ r ro e e l a a
e y : VHnaor:dSwmairse E r r o r
u A S C : S0rxa8l4N m(e<:v ewn d o9r
Sn i q u eS ecso dee: H0rxd8a4e> )r,o rA
dC Q : 0SxC0 ,x 4F(RvUn:o 0nxq0e
@ WxA4R)N ISNQG :x ,/FsUb uxs
d 5 ) :EQ L/GbCs,1i,s/pL@C2i,p120100000s/@s,d @(5d,50:
o Eormraodr rfto r C o m m a n d : rroe aede(:1F0t)l
e u e t d B o k 1 6E4r r o r L e v e l : rRoe tlrcy:a1b6l4e

R e qVuneos:tSemdi sB l o c k : 2 5 1 6 7 5 2 0S r a u b e : w 9
0 6 7 5 2 0Sso re :BHlaodcake:E r2r5
8 V<eenddrouri:u>o,s S Q x , F U x
A N N : T e A r a r v r i e uSnenr inaElr rNduImOb eirh:e r o 5w n M d l s9,[ ug
dWeRyI:N :H/abrsd1w,a0rQeG ,Esr@r,o0r0
05 1A(SdC5::
4 0 x 0 ,e uFsRtUe: l0cx:0 wiiqeu1e) c o d e 0 x 8r4o>r)L,v lA SaCaQ
m C , i sVpn@o2:,S1m0i0s0 0 / s d @ 5 , 1 ( s d 6 5S)r:a
oe :E rwr o r9 f
b A C x 4 ( v n o n q eE roreo0r8 >L,eAvCe:l0:0 ,R eFtUr yxa
: TReeAqruyedsitee ds rBtlroicgka: r6o4e / O , w t r n , o o u e 0WoRcIkG: s6u4@ r3r1o2r6
r : R qwe t d B o k9 [6 4 6 e 1 ) SEerrriLavle :NFutmlb
e S e n s e KEeryr:B oHka r6d4w6a8r
r E r r oVre
< F R U :A C0 xx04S CaQd:a e0Exr0r,e n d oSrr au nuibqru e c o d[e



My Media project

I have had a digital camera for about 5 years now, and we've accumulated a lot of pictures from it. The biggest problem we've run into is that we have a tendency to not ever print any of the pics, nor really to organize them, or really do anything else with them. In the face of my fourth hard drive "data event", I'm looking for The Better Way to deal with the volume of digital media I'm saving.

What am I trying to do:
1. Never lose an original piece of digital media (photo, video, CD, .m4p, etc)
2. Have my digital media accessible so we can use it, in reasonable format (web-thumbnail, .mp3, etc)
3. Not keep a buncha copies of everything
4. Do it automatically as much as possible

So far what I have is about 350GB of potentially-original pictures and video on our home computer and on a set of DVDs. (m4p's are not included yet, and CDs are still on the shelf). I've got Apache sharing them out via WebDAV onto our local lan. I'd like to turn on SSL client authentication (distribute self-signed certs to our laptops) and be able to mount that drive up from any net connection. But there's a couple of minor issues (I'll talk about them later). I'm seeing the limits of my Unix-oriented thought processes when I'm running on a Windows XP Home platform.

For example, #3 above is easily accomplished -- "find . | xargs md5sum | sort" piped to a short awk script... if the md5sums match, diff the files, if they're identical, remove the second, and hardlink a second "copy" in its place. But Windows doesn't have find, xargs, sort, md5sum, awk, diff, or ln. (without cygwin)

#2 I think could be solved with a tool like Subversion... Check in the original media (take a CD for example...) titleoftrack1.wav and tag that version "primary". Then check it out, oggenc it, "svn rename .wav .ogg" and tag that version ogg-hq. Similarly, encode a mp3-hq, mp3-lq, etc. (I only have a 2GB ipod so I like 64kbps vbr mp3's there, but can afford higher-quality on my laptop or home stereo)

#1... I burn two copies of the original to DVD+R, md5sum the original (store the md5sums on the DVD too) and compare them after writing. I need 2 verified copies. One should be offsite, the other in a storage container onsite. (offsite will be implemented someday)

And that just leaves 4. How the **** do I automate all this?