iPhone backup that wasn't

Spent a bit of time this weekend trying to help my cousin copy everything off of her iPhone 6, so that it can be wiped and transferred to another person.  But since I'm stubborn and contrary, I didn't want to do it in iTunes (plus I didn't have her password)...

Enter libimobiledevice, and idevicebackup, referenced by a nice post from Santoku Linux https://santoku-linux.com/howto/mobile-forensics/howto-create-a-logical-backup-of-an-ios-device-using-libimobiledevice-on-santoku-linux/ .  But since I'm stubborn and contrary, the CD image wouldn't install in VirtualBox on my laptop.  Sigh.  Ok, build from source.  Vagrant ubuntu, then clone libimobiledevice and just about all the other pieces of their github, run idevicebackup and... well, this iPhone is too new, try using idevicebackup2.  Cool, backing up, ##### marks dancing across the screen in little boxes, stuff being written unencrypted to disk, excellent.

Verify that the backup is readable and not encrypted, sweeet.  Now what did I get in my 2.5GB of disk space?  I still have no idea.

idevicebackup has an "unback" command that should convert the plist and SQLite files into a meaningful folder structure, which is said to work on version of IOS > 6.  Unfortunately, the feature on the phone (which needs to be plugged in) that they were using to read and translate the backup was broken between IOS 10.3.0 and 10.3.1 (this fact was, and still is undocumented in the toolset, except for the closed github issue).  So unback is not available.

End state: 2.5GB of possibly-useful-in-the-future data, and a wiped iPhone.   Oh well, that's life on the bleeding edge of tech.

Scanner characterization (free) to correct the colors in scanned pictures

I'm working to get my non-digital life in order by scanning in the large underbed box of photos that I've accumulated, but I've noticed that the color isn't quite right on the scanned images. The scanner "autocorrect colors" checkbox doesn't seem to help. I figure the best way to deal with it is to scan the pictures without any scanner-based color correction, and then apply a proper color modification to the resulting image. But the challenge is in 1) convincing the HP "easy scanning" junk to just give me the bits, and in 2) mapping the colors that the scanner sees to what's on the print.

Now, I could spend $60+ on a standardized color card, and use an expensive program to generate a color profile that could be applied to make the correction. But come on, it's just software. Instead, I am sending a color card I generated to the local Walgreens.com in-store pickup, and I'll use that to characterize the scanner. There's a toolset called Argyll that seems to do what I want, but it's not exactly the clearest documentation for someone who doesn't do digital image workflow for a living. But here's what I've figured out so far:

There's a zip of Windows executables. They seem to run on my Win7 laptop.

First, we generate a "target".

targen -v -d 2 target

This gives us a "target.ti1" (that's a one, btw) file.

Second, we turn this ti1 file into a TIF image (and at the same time we make a .cht map of the image that the tool will later use to recognize the image)

printtarg -i SS -v -a .4 -t 300 -p 4x6 -s -m 10 target

This gives us target.tif, and target.cht (and whatever target.ti2 is)

Third, since Walgreens only deals in .jpg files, I convert it to a 100% quality jpeg via GIMP. And I end up with a 336k file to have printed.


And now I'm waiting to pick up that picture.

Before I found the Argyll software, I had grand plans of figuring out the formulas to do the transformation myself in gimp. I made my own blocks of color, got them printed, and discovered that the transformations required to map the resulting RGB values to their originals was, well, let's just say complicated. Probably there is an easier formula in some other color metric (HSV or CMYK or something) but that's a lot of work to figure out.

My next attempt was with Argyll, but I thought the hexagonal color pattern was nicer than the color bars that come out of the default TIF. Unfortunately, Argyll won't create a CHT file to recognize the hexagons. So that was another $0.20 wasted. Oh well.

More to come.

--Joe

DVDs with VLC

Mostly to install this into my NVRAM, but also because there seems to be a shortage of quick instructions for this.

Using VLC Media Player (Windows, version 1.1.2 in my case)
Ctrl-S (streaming)
Disc tab, No Menus, adjust title/chapter if needed
_S_tream
Next
Add a destination file and optionally "Display Locally"
I added a custom profile, MPEG-PS, Video and audio codec "Keep Original"
Stream

This apparently builds a stream output string of ":sout=#file{dst=d:\\junk.ps} :no-sout-rtp-sap :no-sout-standard-sap :sout-keep"

Then rename the .ps to .mpeg or .mpg or whatever.

This runs faster than DVD-time (at least on my system) and doesn't appear to lose any quality, presumably since it doesn't re-code anything.

--Joe

Photo Archiving

This is in response to BenR's post at http://www.cuddletech.com/blog/pivot/entry.php?id=1016 which I can't seem to get past his comment-spam filter.

---

As a fellow father and sys/storage admin, I have similar questions. Have you made the jump to video already? A MiniDV tape at LP (90 mins) quality -- a little less than DVD quality but with worse compression, eats up 15GB of disk space when I dump the AVI stream. Not to mention the gigabytes of SD and CF cards from the camera.

I'm confident in my 3-tier archiving scheme: An active in-the-house full-quality copy on simple disk, a "thumbnail" (screen-resolution or compressed video) version on S3, and two copies of the original format on DVD - one onsite and one offsite.

I expect to have to move from DVD media periodically, but I can put that off until the higher-capacity disk wars play out. Every file on the DVDs are md5sum'd, and i know I can use ddrescue to pull data blocks off either wafer, if S3 and my home drive die, assuming the scratch doesn't hit both disks in the same place. It'd be nice to have an automatic system to track which file is on what DVD, but I haven't implemented such an HSM yet.

I'm enough of a pack rat to keep a DVD drive and probably a computer that can read it essentially forever, and if not, there's always eBay.

The biggest problem I face is not deleting all of the content from a card (or tape) before popping it back into the camera and adding more. So when I copy a media into the "system" I might have other duplicate copies of the pictures. I'd love to be able to deduplicate those and store only one copy (and links to it). And even better would be a content-aware dedup that could tell that x.jpg is the same picture as Y.raw... (and that song_64kvbr.mp3 can be derived from song.flac)

But I haven't put that together yet, either.

--Joe

Media server v0.1a

To solve the automation problem, I'll use Linux. Since I don't have any spare computers laying around any more (I'll turn in my geek card), I've gotta set it up in a VMware instance. Luckily, VMware server is free, and I have disk space here.

Two disks... 8GB for OS, and a 32GB "data" disk to start with. Both sparsely-allocated, cuz I don't have that much disk space.

I know debian better than any other distro. So, a base install, no tasksel, update and upgrade, then installed ssh, apache2, subversion, subversion-tools, and sudo. So far I'm using 515MB. And it can't do anything yet.

Someday I'd like to switch it over to Solaris so I can run the data under zfs, but for now, I like the convenience of aptitude.

In the mean time, it's LVM I guess... I pvcreate'd /dev/sdb, vgcreated dataVG /dev/sdb, and lvcreated dataLV in it. Then mke2fs -j /dev/mapper/dataVG/dataLV and tuned it to want a fsck every 365 mounts or 730d. Mounted it as /data, and I think I'm ready to start developing.

First some data, though... I grabbed about 350MB of data from various picture CDs, copied it to /data/media/$cdname, and copied them around again to make sure there were duplicates in the data...

More to come...

My Media project

I have had a digital camera for about 5 years now, and we've accumulated a lot of pictures from it. The biggest problem we've run into is that we have a tendency to not ever print any of the pics, nor really to organize them, or really do anything else with them. In the face of my fourth hard drive "data event", I'm looking for The Better Way to deal with the volume of digital media I'm saving.

What am I trying to do:
1. Never lose an original piece of digital media (photo, video, CD, .m4p, etc)
2. Have my digital media accessible so we can use it, in reasonable format (web-thumbnail, .mp3, etc)
3. Not keep a buncha copies of everything
4. Do it automatically as much as possible

So far what I have is about 350GB of potentially-original pictures and video on our home computer and on a set of DVDs. (m4p's are not included yet, and CDs are still on the shelf). I've got Apache sharing them out via WebDAV onto our local lan. I'd like to turn on SSL client authentication (distribute self-signed certs to our laptops) and be able to mount that drive up from any net connection. But there's a couple of minor issues (I'll talk about them later). I'm seeing the limits of my Unix-oriented thought processes when I'm running on a Windows XP Home platform.

For example, #3 above is easily accomplished -- "find . | xargs md5sum | sort" piped to a short awk script... if the md5sums match, diff the files, if they're identical, remove the second, and hardlink a second "copy" in its place. But Windows doesn't have find, xargs, sort, md5sum, awk, diff, or ln. (without cygwin)

#2 I think could be solved with a tool like Subversion... Check in the original media (take a CD for example...) titleoftrack1.wav and tag that version "primary". Then check it out, oggenc it, "svn rename .wav .ogg" and tag that version ogg-hq. Similarly, encode a mp3-hq, mp3-lq, etc. (I only have a 2GB ipod so I like 64kbps vbr mp3's there, but can afford higher-quality on my laptop or home stereo)

#1... I burn two copies of the original to DVD+R, md5sum the original (store the md5sums on the DVD too) and compare them after writing. I need 2 verified copies. One should be offsite, the other in a storage container onsite. (offsite will be implemented someday)

And that just leaves 4. How the **** do I automate all this?

--Joe