Listening ports

One of our many applications wouldn't start, with an obscure message that had nothing to do with the underlying problem (nsrexecd "Cannot start portmapper", to be specific and to make sure this is googleable for the next person)

It turns out that another process had been randomly assigned the ports that Networker had to listen on, to an outgoing TCP connection. Which, of course, meant that Networker couldn't bind to those ports to LISTEN. This is the first time this has happened. But it's a potential time bomb for any service that listens on specific ports. Such as Oracle, Weblogic, SAP, etc.

Linux controls what ports are randomly assigned using two sysctl's, ip_local_port_range and ip_local_reserved_ports. Unfortunately, the Oracle installer prerequisite check requires that ip_local_port_range be set wrong (1024-65500, which includes their own listener port) so we have to work with the other one, ip_local_reserved_ports. It's a "comma-separated list of ranges", so for us, I picked an excessive range for our big 3 applications- Oracle (1520 - 1530), SAP (3200 - 3699), and Networker (7937 - 8065).

sysctl net.ipv4.ip_local_reserved_ports=1520-1530,3200-3699,7937-8065

--Joe

Scanner characterization (free) to correct the colors in scanned pictures

I'm working to get my non-digital life in order by scanning in the large underbed box of photos that I've accumulated, but I've noticed that the color isn't quite right on the scanned images. The scanner "autocorrect colors" checkbox doesn't seem to help. I figure the best way to deal with it is to scan the pictures without any scanner-based color correction, and then apply a proper color modification to the resulting image. But the challenge is in 1) convincing the HP "easy scanning" junk to just give me the bits, and in 2) mapping the colors that the scanner sees to what's on the print.

Now, I could spend $60+ on a standardized color card, and use an expensive program to generate a color profile that could be applied to make the correction. But come on, it's just software. Instead, I am sending a color card I generated to the local Walgreens.com in-store pickup, and I'll use that to characterize the scanner. There's a toolset called Argyll that seems to do what I want, but it's not exactly the clearest documentation for someone who doesn't do digital image workflow for a living. But here's what I've figured out so far:

There's a zip of Windows executables. They seem to run on my Win7 laptop.

First, we generate a "target".

targen -v -d 2 target

This gives us a "target.ti1" (that's a one, btw) file.

Second, we turn this ti1 file into a TIF image (and at the same time we make a .cht map of the image that the tool will later use to recognize the image)

printtarg -i SS -v -a .4 -t 300 -p 4x6 -s -m 10 target

This gives us target.tif, and target.cht (and whatever target.ti2 is)

Third, since Walgreens only deals in .jpg files, I convert it to a 100% quality jpeg via GIMP. And I end up with a 336k file to have printed.


And now I'm waiting to pick up that picture.

Before I found the Argyll software, I had grand plans of figuring out the formulas to do the transformation myself in gimp. I made my own blocks of color, got them printed, and discovered that the transformations required to map the resulting RGB values to their originals was, well, let's just say complicated. Probably there is an easier formula in some other color metric (HSV or CMYK or something) but that's a lot of work to figure out.

My next attempt was with Argyll, but I thought the hexagonal color pattern was nicer than the color bars that come out of the default TIF. Unfortunately, Argyll won't create a CHT file to recognize the hexagons. So that was another $0.20 wasted. Oh well.

More to come.

--Joe

Link aggregation in a cross-platform environment

Everybody in the world knows that LACP (802.1ad) is the standard for Link Aggregation and Control, right? Well, not exactly.

We have VMware ESX and Solaris servers connected to our Cisco edge switches. Sounds good, right? We'd like to bond the multiple gig-E NICs into a multi-GB aggregate. Sounds good, right? Well, it's not so easy.

ESX doesn't support true 802.1ad aggregation. They fake it with their vSwitch NIC teaming properties. They do the same thing as L3 LACP (hash of the source and destination IPs) but don't call it that. Fortunately, they use the same hash algorithm as Cisco, so we can work with it.

On the cisco side, we add the interfaces to a channel-group with mode "on". This uses the default-for-the-switch port-channel load-balance setting, which we had to set to src-dst-ip.

Unfortunately, since that setting is a global switch option and is not set on a per-port-channel level, this means that our Solaris boxes (who speak LACP properly) can't use Layer-4 (hash of source and dest IPs and ports) balancing. This sucks, because our Solaris boxes are the heavy-network-hitters (backup servers) that could really use the extra bandwidth provided by spreading the multiple TCP connections across multiple links.

I'm not sure who to blame here, VMware for not doing LACP, or Cisco for not allowing multiple loadbalancing methods on different port channel groups.

--Joe

Oh yeah, the rest of EMC World

The last days of EMC world were fairly uneventful. I was called in on a couple of work problems, which made it hard to concentrate on the talks. But from what I could tell, they were all either high-level "cloud is king" or very introductory sessions, so I didn't really get much out of them.

I did have a nice seafood dinner at the Rio after the conference closed out, and a quite forgettable plane ride home.

Now, back to the real world.

--Joe

Cleaning up View Composer VMs

We've had frequent issues where our VMware View desktops will get into a state of Provisioning Error (missing) with a popup box that a "Virtual Machine with Input Specification already exists"

This symptom is described pretty well in http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1008658, but here's some more info:

At least in the version of Composer 4.5 that I'm running, the sviconfig command doesn't know the RemoveSviClone that they reference in the KB. So it's the manual way for me.

This seems to happen if the Composer database bits get out of sync with what's in the ADAM database that View uses (Can we please pick ONE database).

This weekend's problems came when the Oracle DB that supports our VirtualCenter, View Composer, and Update Manager environments had a corrupted file. I had to roll back to a previous Oracle state, which naturally meant that it wasn't quite the same as ADAM.

The manual cleanup (besides being MSSQL-specific in table names and interface reference) requires a significant amount of C&P to run through in SQL/Plus. So I declared an Oracle procedure that, given a VM name, cleans up the data automatically:

create or replace procedure cleanup_clone
 ( p_vmname in varchar )
as
begin
 delete from SVI_VM_NAME where NAME = p_vmname;
 delete from SVI_COMPUTER_NAME where NAME = p_vmname;
 delete from SVI_SC_PDISK_INFO where PARENT_ID in
  (select id from SVI_SIM_CLONE where VM_NAME = p_vmname);
 delete from SVI_SC_BASE_DISK_KEYS where PARENT_ID in
  (select id from SVI_SIM_CLONE where VM_NAME = p_vmname);
 delete from SVI_SIM_CLONE where VM_NAME = p_vmname;

 commit;

end cleanup_clone;

With this in place, I can "execute cleanup_clone('uscimposer-99');" at the SQL/Plus prompt (having logged in as the Composer user) and it nicely wipes out the input specification for that VM, and a new one can be provisioned. The only other manual step then, is to remove the provisioning-error'd VM from the View Admin interface.

--Joe

EMC World 2011

I've made it through the 2nd day of EMC world, and am starting on the third. Tuesday brought some interesting talks on Networker and enterprise apps performance tuning (specifically MSSQL).

But the driving theme of the conference has me a bit confused. "IT As A Service" sounds great, and we keep hearing about how ITAAS can deliver benefits through standardization (aka service catalog)

At least in my experience, though, there's a problem- The service catalog is never "good". That is to say, it's either incomplete (sorry, we don't have MySQL in the catalog), or overly restrictive (pick a different DB platform for your LAMP app), or forces the business into shadow-IT operations (run your own d*** database). And in the case of business-driven tool selection, this is a problem.

The service catalog as I see it will cover maybe 90% of the requirements, and every process/function will need a slightly different 10%. In order to deliver to those processes, ITAAS has to deal with those 1-off "oh yeah, MySQL had to be installed in /usr/local instead of the standard /apps/mysql-version to make this OOTB app work" kind of gotchas that plague sysadmins.

And, of course, technology moves ahead faster than the service book. In particular, marketing to business decision makers moves a helluva lot faster. Think about iPhone/tablet/Android adoption- IT has had to completely rethink what kind of device a user will be coming from-- It's not a corporate-owned laptop running an image-deployed copy of Windows XP with IE 6, it's now the iPad the CEO bought for his daughter.

So how does ITAAS respond to these shifting sands? That's the brazilian-dollar question. Do we chase the business's tail and add too many poorly-supported products to our service catalog? Do we lock the business into the properly-blessed old way of doing things, and out of the innovation that drives us?

--Joe

EMC World 2011

I'm here at EMC world 2011, taking advantage of their "Bloggers Lounge" where they have better WiFi and more comfortable chairs.

So far, the conference is unremarkable-- the first keynote could be summarized as "Cloud, blah, blah, lots of data, blah, new products, blah, blah, blah" Nothing particularly groundbreaking.

But still, being the first travel I've been on in almost 5 years, I'm looking forward to it. Lots of topics that can help my quest for Infrastructure Strategy.