Photo Archiving

This is in response to BenR's post at http://www.cuddletech.com/blog/pivot/entry.php?id=1016 which I can't seem to get past his comment-spam filter.

As a fellow father and sys/storage admin, I have similar questions. Have you made the jump to video already? A MiniDV tape at LP (90 mins) quality -- a little less than DVD quality but with worse compression, eats up 15GB of disk space when I dump the AVI stream. Not to mention the gigabytes of SD and CF cards from the camera.

I'm confident in my 3-tier archiving scheme: An active in-the-house full-quality copy on simple disk, a "thumbnail" (screen-resolution or compressed video) version on S3, and two copies of the original format on DVD - one onsite and one offsite.

I expect to have to move from DVD media periodically, but I can put that off until the higher-capacity disk wars play out. Every file on the DVDs are md5sum'd, and i know I can use ddrescue to pull data blocks off either wafer, if S3 and my home drive die, assuming the scratch doesn't hit both disks in the same place. It'd be nice to have an automatic system to track which file is on what DVD, but I haven't implemented such an HSM yet.

I'm enough of a pack rat to keep a DVD drive and probably a computer that can read it essentially forever, and if not, there's always eBay.

The biggest problem I face is not deleting all of the content from a card (or tape) before popping it back into the camera and adding more. So when I copy a media into the "system" I might have other duplicate copies of the pictures. I'd love to be able to deduplicate those and store only one copy (and links to it). And even better would be a content-aware dedup that could tell that x.jpg is the same picture as Y.raw... (and that song_64kvbr.mp3 can be derived from song.flac)

But I haven't put that together yet, either.



VMware View 3.0 and proxies

Oops, I haven't blogged the first part of this story. Oh well, maybe later. In brief, we have VMware VDM to satisfy das corporate security. It was working for people on our LAN and on the corporate network, and I got it to work from the internet (but requiring a valid smartcard (SSL User Certificates) before letting a user in). This was a cool project I'll have to document here some time.

Well, time moves on and VMware View Manager 3.0 (nee VDM 3.0) was released and implemented in this environment.

The first problem we noticed started when a home user upgraded their View client to 3.0 as they were prompted on the login page. This was when the smartcard authentication from the internet stopped working. A little investigation (watching network traffic, decrypting with Wireshark, etc) and I found that while the old client would send an HTTPS post command just like IE, the new client didn't send the user SSL certificate. But since VMware never supported this sort of setup, I just worked through it (another cool solution I'll have to post later). A little bit of rearchitecture, and I was able to still protect enough of the View environment to make me feel secure and to convince the security people that it was sufficient.

Now, I've got a similar error from the corporate network. Same message: Connection to View server could not be established". But WTF? this is on the LAN, there shouldn't be a proxy problem. IE works just fine*, but View can't connect.

That is to say IE worked fine with the proxy, but the proxy requires user authentication, which is cached for the browser session, and I didn't think of that until later.

So fire up Wireshark again, and once again, the first couple of View CONNECT :443 requests from IE happily sent the Proxy-Authorization: header, but the last one tried to do a CONNECT without that header, and was tossed back a Squid Authentication Required 407.

Ah, that's a relatively easy one to fix, if only I could get the proxy admin to turn of authentication (nope, that's verbotten) or do the same sort of magic as I did on the outside firewall deployment (eww, that'd be messy) or maybe bypass the proxy for this? I mean, they're on the LAN. Luckily VMware apparently thought of this and implemented an undocumented registry key: HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\ProxyBypass that contains a MultiSZ list of names or IPs for View to connect directly to instead of using the proxy.

Did I mention that all of this new behavior is undocumented? And that what I'd been doing in the first place was both unsupported and completely WORKING?

I'd guess that the new View client switched from a standard MS HttpRequest method to something they threw together without the nice functionality that IE bundles into its method. Oh well. It's working again now.