Enterprise SA

Listening ports

2011-08-24T09:30:00.004-04:00

One of our many applications wouldn't start, with an obscure message that had nothing to do with the underlying problem (nsrexecd "Cannot start portmapper", to be specific and to make sure this is googleable for the next person)

It turns out that another process had been randomly assigned the ports that Networker had to listen on, to an outgoing TCP connection. Which, of course, meant that Networker couldn't bind to those ports to LISTEN. This is the first time this has happened. But it's a potential time bomb for any service that listens on specific ports. Such as Oracle, Weblogic, SAP, etc.

Linux controls what ports are randomly assigned using two sysctl's, ip_local_port_range and ip_local_reserved_ports. Unfortunately, the Oracle installer prerequisite check requires that ip_local_port_range be set wrong (1024-65500, which includes their own listener port) so we have to work with the other one, ip_local_reserved_ports. It's a "comma-separated list of ranges", so for us, I picked an excessive range for our big 3 applications- Oracle (1520 - 1530), SAP (3200 - 3699), and Networker (7937 - 8065).

sysctl net.ipv4.ip_local_reserved_ports=1520-1530,3200-3699,7937-8065

--Joe

Scanner characterization (free) to correct the colors in scanned pictures

2011-07-21T22:04:00.006-04:00

I'm working to get my non-digital life in order by scanning in the large underbed box of photos that I've accumulated, but I've noticed that the color isn't quite right on the scanned images. The scanner "autocorrect colors" checkbox doesn't seem to help. I figure the best way to deal with it is to scan the pictures without any scanner-based color correction, and then apply a proper color modification to the resulting image. But the challenge is in 1) convincing the HP "easy scanning" junk to just give me the bits, and in 2) mapping the colors that the scanner sees to what's on the print.

Now, I could spend $60+ on a standardized color card, and use an expensive program to generate a color profile that could be applied to make the correction. But come on, it's just software. Instead, I am sending a color card I generated to the local Walgreens.com in-store pickup, and I'll use that to characterize the scanner. There's a toolset called Argyll that seems to do what I want, but it's not exactly the clearest documentation for someone who doesn't do digital image workflow for a living. But here's what I've figured out so far:

There's a zip of Windows executables. They seem to run on my Win7 laptop.

First, we generate a "target".

targen -v -d 2 target

This gives us a "target.ti1" (that's a one, btw) file.

Second, we turn this ti1 file into a TIF image (and at the same time we make a .cht map of the image that the tool will later use to recognize the image)

printtarg -i SS -v -a .4 -t 300 -p 4x6 -s -m 10 target

This gives us target.tif, and target.cht (and whatever target.ti2 is)

Third, since Walgreens only deals in .jpg files, I convert it to a 100% quality jpeg via GIMP. And I end up with a 336k file to have printed.

And now I'm waiting to pick up that picture.

Before I found the Argyll software, I had grand plans of figuring out the formulas to do the transformation myself in gimp. I made my own blocks of color, got them printed, and discovered that the transformations required to map the resulting RGB values to their originals was, well, let's just say complicated. Probably there is an easier formula in some other color metric (HSV or CMYK or something) but that's a lot of work to figure out.

My next attempt was with Argyll, but I thought the hexagonal color pattern was nicer than the color bars that come out of the default TIF. Unfortunately, Argyll won't create a CHT file to recognize the hexagons. So that was another $0.20 wasted. Oh well.

More to come.

--Joe

Link aggregation in a cross-platform environment

2011-06-13T11:03:00.002-04:00

Everybody in the world knows that LACP (802.1ad) is the standard for Link Aggregation and Control, right? Well, not exactly.

We have VMware ESX and Solaris servers connected to our Cisco edge switches. Sounds good, right? We'd like to bond the multiple gig-E NICs into a multi-GB aggregate. Sounds good, right? Well, it's not so easy.

ESX doesn't support true 802.1ad aggregation. They fake it with their vSwitch NIC teaming properties. They do the same thing as L3 LACP (hash of the source and destination IPs) but don't call it that. Fortunately, they use the same hash algorithm as Cisco, so we can work with it.

On the cisco side, we add the interfaces to a channel-group with mode "on". This uses the default-for-the-switch port-channel load-balance setting, which we had to set to src-dst-ip.

Unfortunately, since that setting is a global switch option and is not set on a per-port-channel level, this means that our Solaris boxes (who speak LACP properly) can't use Layer-4 (hash of source and dest IPs and ports) balancing. This sucks, because our Solaris boxes are the heavy-network-hitters (backup servers) that could really use the extra bandwidth provided by spreading the multiple TCP connections across multiple links.

I'm not sure who to blame here, VMware for not doing LACP, or Cisco for not allowing multiple loadbalancing methods on different port channel groups.

--Joe

Oh yeah, the rest of EMC World

2011-05-31T10:11:00.002-04:00

The last days of EMC world were fairly uneventful. I was called in on a couple of work problems, which made it hard to concentrate on the talks. But from what I could tell, they were all either high-level "cloud is king" or very introductory sessions, so I didn't really get much out of them.

I did have a nice seafood dinner at the Rio after the conference closed out, and a quite forgettable plane ride home.

Now, back to the real world.

--Joe

Cleaning up View Composer VMs

2011-05-31T09:58:00.004-04:00

We've had frequent issues where our VMware View desktops will get into a state of Provisioning Error (missing) with a popup box that a "Virtual Machine with Input Specification already exists"

This symptom is described pretty well in http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1008658, but here's some more info:

At least in the version of Composer 4.5 that I'm running, the sviconfig command doesn't know the RemoveSviClone that they reference in the KB. So it's the manual way for me.

This seems to happen if the Composer database bits get out of sync with what's in the ADAM database that View uses (Can we please pick ONE database).

This weekend's problems came when the Oracle DB that supports our VirtualCenter, View Composer, and Update Manager environments had a corrupted file. I had to roll back to a previous Oracle state, which naturally meant that it wasn't quite the same as ADAM.

The manual cleanup (besides being MSSQL-specific in table names and interface reference) requires a significant amount of C&P to run through in SQL/Plus. So I declared an Oracle procedure that, given a VM name, cleans up the data automatically:


create or replace procedure cleanup_clone
 ( p_vmname in varchar )
as
begin
 delete from SVI_VM_NAME where NAME = p_vmname;
 delete from SVI_COMPUTER_NAME where NAME = p_vmname;
 delete from SVI_SC_PDISK_INFO where PARENT_ID in
  (select id from SVI_SIM_CLONE where VM_NAME = p_vmname);
 delete from SVI_SC_BASE_DISK_KEYS where PARENT_ID in
  (select id from SVI_SIM_CLONE where VM_NAME = p_vmname);
 delete from SVI_SIM_CLONE where VM_NAME = p_vmname;

 commit;

end cleanup_clone;

With this in place, I can "execute cleanup_clone('uscimposer-99');" at the SQL/Plus prompt (having logged in as the Composer user) and it nicely wipes out the input specification for that VM, and a new one can be provisioned. The only other manual step then, is to remove the provisioning-error'd VM from the View Admin interface.

--Joe

EMC World 2011

2011-05-11T12:25:00.004-04:00

I've made it through the 2nd day of EMC world, and am starting on the third. Tuesday brought some interesting talks on Networker and enterprise apps performance tuning (specifically MSSQL).

But the driving theme of the conference has me a bit confused. "IT As A Service" sounds great, and we keep hearing about how ITAAS can deliver benefits through standardization (aka service catalog)

At least in my experience, though, there's a problem- The service catalog is never "good". That is to say, it's either incomplete (sorry, we don't have MySQL in the catalog), or overly restrictive (pick a different DB platform for your LAMP app), or forces the business into shadow-IT operations (run your own d*** database). And in the case of business-driven tool selection, this is a problem.

The service catalog as I see it will cover maybe 90% of the requirements, and every process/function will need a slightly different 10%. In order to deliver to those processes, ITAAS has to deal with those 1-off "oh yeah, MySQL had to be installed in /usr/local instead of the standard /apps/mysql-version to make this OOTB app work" kind of gotchas that plague sysadmins.

And, of course, technology moves ahead faster than the service book. In particular, marketing to business decision makers moves a helluva lot faster. Think about iPhone/tablet/Android adoption- IT has had to completely rethink what kind of device a user will be coming from-- It's not a corporate-owned laptop running an image-deployed copy of Windows XP with IE 6, it's now the iPad the CEO bought for his daughter.

So how does ITAAS respond to these shifting sands? That's the brazilian-dollar question. Do we chase the business's tail and add too many poorly-supported products to our service catalog? Do we lock the business into the properly-blessed old way of doing things, and out of the innovation that drives us?

--Joe

EMC World 2011

2011-05-09T13:04:00.003-04:00

I'm here at EMC world 2011, taking advantage of their "Bloggers Lounge" where they have better WiFi and more comfortable chairs.

So far, the conference is unremarkable-- the first keynote could be summarized as "Cloud, blah, blah, lots of data, blah, new products, blah, blah, blah" Nothing particularly groundbreaking.

But still, being the first travel I've been on in almost 5 years, I'm looking forward to it. Lots of topics that can help my quest for Infrastructure Strategy.

Global networks

2010-11-16T10:23:00.003-05:00

At work, each network segment is classified with a color based on its access list. We have brown, green, red, purple, blue, orange, and firewalls all over the place. This has inspired...

The Network Connection

Why are there so many damn network colors,
And routers and firewalls?
Show them on pow'r point, but only to man'gers
And they will approve the change.
So we've been told and some choose to believe it
I know that I'll wait and see
Someday they'll make it
My Network Connection,
The cables, the VLANs, and me...

Who said that every server needs IP
A link to one thousand base T
Somebody thought of that, approving the purchase
Except for the cabling
A WAN so amazing, with OC-3's blazing
Sometimes the pings make it through.
Someday they'll make it,
A global connection,
The cables, the VLANS, and me...

All of us scanned by Nessus,
We know that it's probably magic

Have you been half-routed? Dropped by Rule Zero?
Lost packets silently
This is annoying, quit making changes
On Friday at end of the day.
Some of us do work, even on weekends, we need the network to flow.
Someday I'll have it
A working connection,
The cables, the VLANS and me...

--Joe

Building stand-alone Collectd plugin - Part 2

2010-11-08T15:29:00.008-05:00

Actually, this wasn't as difficult as I was expecting.

The only real challenge was in the fact that the ESX Guest SDK libraries are only distributed as a shared object, which means my plugin had to dlopen() the required library, rather than being able to link it in statically. Luckily, I was able to cannibalize some of the example guest SDK code for this.

Here's the basic idea:

// Function to get how much CPU time we've gotten
VMGuestLibError (*GuestLib_GetCpuUsedMs)(VMGuestLibHandle handle, uint64 *cpuUsedMs);

//In the plugin_init function, I dlopen("libvmGuestLib.so") and assign the function
GuestLib_GetCpuUsedMs = dlsym(dlHandle, "VMGuestLib_GetCpuUsedMs");
//And open the Guestlib handle.  Each plugin_read loop, I
// GuestLib_UpdateInfo(glHandle); and then can get the latest data.
glError = GuestLib_GetCpuUsedMs(glHandle, &(CpuUsedMs));
values[0].counter = CpuUsedMs;
// and plugin_dispatch_values().

Then I copied the resulting .libs/esx_guest* and the libvmGuestLib.so into the ~collectd/lib/collectd/ directory (where it looks for plugin SOs) and fired it up. I also had to add entries to the types.db for my data sources.

From here, I get cool graphs for my Linux VMs like these.

I'll put this code up on my personal site when I get a chance, and contribute these documentation to the collectd project.

--Joe

Building stand-alone Collectd plugin

2010-11-03T13:09:00.006-04:00

I'm working on building a plugin for the collectd data collection system (www.collectd.org) that will gather stats on our ESX RedHat VMs through the VMware Guest API.

Here's what I've found so far...

unpack collectd, configure and make it.
In a separate directory, create the source files for the new plugin.

(minimal plugin code:

#include "collectd.h"
#include "common.h"
#include "plugin.h"

static int my_read(void) {
        value_t values[1];
        value_list_t vl = VALUE_LIST_INIT;

        values[0].counter = 0;

        vl.values = values;
        vl.values_len = 1;
        sstrncpy (vl.host, hostname_g, sizeof (vl.host));
        sstrncpy (vl.plugin, "test1", sizeof (vl.plugin));
        sstrncpy (vl.type, "counter", sizeof (vl.type));

        plugin_dispatch_values (&vl);

        return 0;
}

void module_register(void) {
       plugin_register_read ("test1", my_read);
}

Then I use libtool to build the .o:
$ libtool --mode=compile gcc -DHAVE_CONFIG_H -I ../collectd-4.10.1/src -Wall -Werror -g -O2 -MT test1.lo -MD -MP -MF test1.Tpo -c -o test1.lo test1.c
and link it:
$ libtool --tag=CC --mode=link gcc -Wall -Werror -g -O2 -module -avoid-version -o test1.la -rpath /apps/collectd-4.10.1/lib/collectd -lpthread -ldl test1.lo

This generates the files in ./libs/test1.* which I copy into the $prefix/lib/collectd/ directory and enable it in my config.

So much for part 1... Up next, getting actual data.

--Joe

DVDs with VLC

2010-08-06T23:47:00.002-04:00

Mostly to install this into my NVRAM, but also because there seems to be a shortage of quick instructions for this.

Using VLC Media Player (Windows, version 1.1.2 in my case)
Ctrl-S (streaming)
Disc tab, No Menus, adjust title/chapter if needed
_S_tream
Next
Add a destination file and optionally "Display Locally"
I added a custom profile, MPEG-PS, Video and audio codec "Keep Original"
Stream

This apparently builds a stream output string of ":sout=#file{dst=d:\\junk.ps} :no-sout-rtp-sap :no-sout-standard-sap :sout-keep"

Then rename the .ps to .mpeg or .mpg or whatever.

This runs faster than DVD-time (at least on my system) and doesn't appear to lose any quality, presumably since it doesn't re-code anything.

--Joe

Windows NLB clustering and SIDs

2010-07-09T15:24:00.002-04:00

A colleague has been working to set up a MS NLB cluster for a set of .NET machines. As is our standard practice, these are created as ESX VMs, and for convenience, we create them based on our standard template. Then the VMware guest customization process runs, and we have a VM we can turn over to the application team.

The problem in this case (the reason I'm involved in a Windows issue) is that NLB wasn't starting. There were various false-starts with configuration items randomly disappearing (why is only one of the NICs in the selection box on this system?).

Somehow, it was suggested that maybe the reason NLB wouldn't start was because the SID of the two VMs was the same. Of course, no, since we run the guest customization which does a NewSID(1m). That'd be impossible.

But it turns out that the SIDs were the same, popping the machines out of the domain and newsid'ing them resolved the issue. Whod'a thunk?

On further reflection, the system's SID is probably the best option for a locally-unique identifier to use to map the loadbalancing traffic via NLB. There has to be some way for all the cluster members to agree on who cares about which packets, so why not use the SID as part of the hash function? Makes perfect sense, since SIDs are, of course, unique.

--Joe

Is today the last $weekday of the month?

2010-03-29T13:22:00.007-04:00

Classic problem in sysadmin: How to run a script on the last Sunday (or whatever day) of the month. This snippet is correct for all days, even in leap years between 1904 and 2096, inclusive. It messes up on February 22 each century, except when the year is divisible by 400 (works in 1600, 2000, 2400, etc. but breaks in 2100, 2200, 2300) But I'll be retired by then...

My answer, in Korn Shell:


# Day of the week to run on:
DOW=Sunday

OLDLC_TIME=$LC_TIME
export LC_TIME=C
# Requires /usr/bin/ksh to run correctly.
# I couldn't be bothered to get the expr quoting and backquoting working in /bin/sh
case `date +%b` in
  Jan|Mar|May|Jul|Aug|Oct|Dec) D=31;;
  Apr|Jun|Sep|Nov) D=30;;
  Feb) D=$(( 28 + ( $(date +%Y) %4 == 0 ));;
  # FIXME: Y2.1K bug here.
esac

if [[ `date +%A` == "$DOW" ]] && [[ $(( $(date +%e) + 7 -gt $D ]]
# then next Sunday is after the end of this month, so today's the last Sunday of this month
then
 ....
fi

LC_TIME=$OLDLC_TIME

IT Reporting

2009-11-09T15:24:00.003-05:00

A recent "Ask Slashdot" asked what information a sysadmin should take to an executive. Here's what I think. I've picked this up from a variety of sources, including a very-skilled manager.

--------------------------
There are three key things that executives want to hear:

1) What has the department done in the past? The core of this point is to get to the question "Does the past justify continued investment?" and its correlary "We've sunk so much money into IT, what have we gotten from it?" This is where usage statistics (website hits, business transaction data, dollars-per-downtime and Nines, return on cost-saving measures, etc) are presented. This should be in high-level terms with drill-down slides available, but only presented on request. Focus on the trends of service delivery vs. IT budget and/or headcount.

2) What is the department doing now? Here we focus on what is happening with their current business. This is where a primary element of capacity planning comes in: The Headroom Metric. How much additional user load can we support on our current systems and network, before the service is degraded? In concrete terms, ignoring everything except CPU, if you're delivering 100 pages per second, and using 40% of the server's CPU, you have a headroom of 150 additional pp/s. By extrapolating this to the business need - say the marketing department has launched 5 campaigns this year, the current systems may be able to support 10, but should not be expected to support 20 without additional investment. Note that this headroom metric must look at the end-to-end utilization, like disk, memory, network, and most importantly administration effort in order to be accurate.

3) What will the department do in the future? What are the business-focused projects that the department is working on? How will the investment in these projects result in money coming into or staying in the business? What is the Return on Capital, Return on Investment?

As far as timing, there should be at least an annual "full report" on the state of IT. Depending on the dynamics of the business, quarterly updates should be sufficient, unless something changes significantly. And depending on the team and scope of the projects. You don't want to face this with a "we haven't done anything since the last report" status. But it's also important to reconnect with the executives regularly so that they don't forget about what you're doing, and also so that you can react and change to meet their changing business plans.

The most important thing we in IT can do is to be aligned to the business. This means focusing on the things that matter: delivering the product or service in exchange for money. Everything else is overhead. And the better your IT department is at aligning itself, the better you look when an outsourcer tries to talk your executives into cutting everything except the "core competancies".

--Joe

Idle curiosity about iLOM

2009-09-29T10:16:00.003-04:00

Why does the service processor on our brand new Sun T5240 server have a SPARC 885 processor, and run Linux? Why not (Open)Solaris?

Kinda ironic that Sun boots its latest servers with Linux.

Maybe it's the fact that it has 32MB of flash to work with, and only 128MB of ram. But that should be enough to run Solaris.

--Joe

FSF Windows 7 sins

2009-08-27T15:48:00.001-04:00

I don't normally post political messages here, but this one's important, I think.

The Free Software Foundation has posted 7 Windows 7 "sins" at http://windows7sins.org/, and I think they left out what in my mind is the most important issue. It's sorta covered in "Corrupting Education" and "Lock-In", but not really:

With Windows 7 (and Office 2003 before that, and Vista before that, and XP before that, and Windows 9x/W2K before that) users will have to retire/obsolete all of their existing training in the Windows user interface in favor of the newest cosmetic decisions Microsoft has made for its products.

I don't argue that there aren't significant productivity benefits to the current Windows shell (vs. Program Manager in NT and 3.x) or in the improvements from '95 to XP. I haven't seen much of Vista's Aero, or the new Windows 7 UI, and I'm sure all of the changes have been run by major interface testers.

But when I switched from Office 2000 to Office 2003, I had a rather steep learning curve to deal with the "Ribbon" UI. Even though I taught Office 97 to Computers 101 users in grad school (and was able to take that through to O2K) I was lost with the new "Where the h*** did the menu go" interface. (Ok, If I were an Excel developer, would I consider search&replace General (Home) thing, or a Data thing. It used to be in the Edit menu... )

But I relearned. And I was able to relearn because as I was growing up, the UI changed dramatically (from Write on my Apple ][+ to PC/Word Perfect to WPfW to vim/TeX and on to MS Office*) But for someone who's used to and has memorized the keystrokes/mouse clicks to insert a text box, this is a whole new ballgame.

When I was applying for jobs after college for example, one of the companies asked that I take an "aptitude test" which included things like typing speed and accuracy, formatting documents, generating mail merges etc. This computer-based test was graded on if you click the right menu option first. If you picked "Edit" instead of "Tools" (or if you right-clicked and chose "Format") you got the question wrong. Not that this was a good test, but it's typical for the industry. And the answers completely changed when 2K7 came out.

Of course, in my line of work, we're more concerned about the OS than about the Office apps. So it's things like the changes in networking that annoy me about Vista. Wow, the way I set up a dialup connection has changed. Hmm, I wonder what happens if I right-click here... etc. So I have to learn a whole new way to fix things that go wrong. Not to mention that Vista Home is quite different interface-wise than Vista Business.

And I'd expect that the various Windows 7 editions will look different too. After all, would the wizard that helps gramma connect to the wireless internet at Starbucks be the best way for IT professionals to diagnose an 802.1x authentication problem? If I learn how to do it with my home PC, will that apply to the real business world?

--Joe

Netapp - Waster of space

2009-08-19T09:32:00.002-04:00

We have a Netapp that we use to provide Tier-2 LUNS to our SAN. It was price-competitive on raw disk space, but I didn't realize at the time just how much overhead this appliance had.

The obvious overhead is RAID-DP and hot spare drives. Easily calculated. 1 HS per 30 drives of each size. DP is 2 drives per plex, so that's 6 wasted drives out of the 28 in two shelves, leaving 22 * 266GB drives usable = 5.7TB.

I'd heard that space is reserved for OS and bad-block overhead (about 10%) so that brings us down to 5.2TB usable.

Well, the web interface shows the aggregate as 4.66TB. So that's 600GB I haven't accounted for. But still, 4.66 TB is a good amount of space.

From the aggregate, we create a flexvol (note that this places 20% by default as inaccessible snap reserve space). On the flexvol, we create LUNs and present them to our servers. And here's where the space consumption is nasty:

By default, if you create a 1TB lun, OnTAP reserves 1TB of disk blocks in the volume. That's nice, and exactly what I'd expect. Although in practice, we use thin provisioning (lun create -o noreserve) for most of our LUNs

What I didn't expect going in was that the first time you create a snapshot, OnTAP would reserve ANOTHER 1TB for that LUN. And interestingly enough, that 1TB is never touched until there's no other space in the volume.

Ok, That ensures that even if you overwrite the ENTIRE lun after you take a snapshot. But it reduces the usable size of LUN-allocation to 2.33TB. And if you have multiple snapshots, those don't seem to go into the snap reserve, but rather are in addition to the 2*LUNsize that is already allocated.

So out of a raw disk capacity of (28*266) 7.2 TB (which is quoted as 28*300GB disks = 8.2TB) we get just over 2TB of space that can be used for holding actual system data.

Wow.

Now, there are non-default settings that can change that, but they're only available at the CLI, not the web interface:

# snap reserve 0 - this will set the snap reserve from 20% to 0%, which is recommended for volumes that hold only LUNs.
# vol options fractional_reserve ## - This changes the % of LUNsize that is reserved when a LUN snapshot is taken.

It is not entirely clear what happens to a LUN when its delta becomes larger than the fractional_reserve. Some documentation says it may take the LUN offline, but I would hope that only would happen if there's no remaining space in the volume (like what happens with snapshot overflow in traditional NAS usages). But it's not clear.

As far as I can tell, the current best practice is to set the snap reserve to the amount of change you expect in the volume, and set the fractional_reserve to the amount of change you expect in the LUN. And to set up either volume auto-grow and/or snapshot auto-delete to make sure you have free space when things get full.

On the gripping hand, the default options make sure that you have to buy a lot of disks to get the storage you need.

--Joe

SCSI disk identifiers

2009-07-13T20:02:00.003-04:00

Whoever it was that thought they'd be cute and put the VT100 "clear screen" character string as part of their disk identifier, I want to buy you a drink.

The probe-scsi-all output wasn't nice.

Hemlock. Your choice of flavors.

--Joe

Discovering R for performance analysis

2009-04-03T12:29:00.002-04:00

I've seen references in various conferences and performance blogs about the "R" statistical analysis package, and how it can be used to data mine system performance data. I'm going to learn it.

Fun.

Firewall project

2009-03-06T15:18:00.003-05:00

A big consumer of my time this week (and last week) is building a pilot implementation of a new internet-facing DMZ. Well, that's understating the requirements a bit. Corporate requires a special "reverse proxy" system to be sitting in the internet-facing parts, so we have to make some major changes anyway, but I wasn't happy with just having a DMZ out there, it needs to be reliable. Preferably more reliable than our internet feed. But we have more than 1 datacenter, with more than 1 internet provider, why not take advantage of that?

Basically, the goal is to have a single IP address (for www.dom.ain) that is internet-routed through both datacenter ISPs, and have Linux do some magic so that packets can come or go through whichever pipe. Apparently, there are companies that make such magic happen for lots of $$$ but in this economy, they aren't an option. And since Linux is free (and my time is already paid for) here's a chance to save the company money. That's what I sold to management anyway.

It should be simple enough: advertise that magic netblock out both pipes, put a Linux router on the link as the gateway for that block, NAT the magic.xxx address of www to the internal IP address of the apache server, and toss out of state packets over to its peer so that the firewalls between this box and the apache server wouldn't see them.

In ascii:


Internet --- Linux ---- FW --+-- LAN --- apache
              ^-v            |
Internet --- Linux ---- FW --+

(We've assumed that the WAN is important enough internally that if it's down, our external site is going to have problems anyway. Which is true, unfortunately. WAN outages between our 2 main datacenters tend to break everything even for local users.)

So far I've gotten 3/4 of the packet-handling stuff working for a single system using just iptables. nat PREROUTING DNAT rewrites the magic.xxx to apache's address, POSTROUTING MASQUERADE gives apache something routable to return the packets to, and I can see the entries in the /proc/net/ip_conntrack file. Unfortunately, I can't seem to find how nat is supposed to de-masquerade the packets back according to the state that caused them.

I have a packet coming in from 10.0.05 (client) -> 192.168.1.13 (www) (magic block is 192.168.1/24). It leaves my box as 192.168.5.182 (lx-int) -> 192.168.6.13 (www-web0). www-web0 gets the SYN, and sends its SYN+ACK back 192.168.6.13 -> 192.168.5.182. I see those packets on the wire, and it's what I'd expect.

What I don't see is a way to take that SYN+ACK, look up in the connection tracking table for the original client and rewrite it to be 192.168.1.13 -> 10.0.0.5.

--Joe

Photo Archiving

2009-02-19T09:34:00.002-05:00

This is in response to BenR's post at http://www.cuddletech.com/blog/pivot/entry.php?id=1016 which I can't seem to get past his comment-spam filter.

As a fellow father and sys/storage admin, I have similar questions. Have you made the jump to video already? A MiniDV tape at LP (90 mins) quality -- a little less than DVD quality but with worse compression, eats up 15GB of disk space when I dump the AVI stream. Not to mention the gigabytes of SD and CF cards from the camera.

I'm confident in my 3-tier archiving scheme: An active in-the-house full-quality copy on simple disk, a "thumbnail" (screen-resolution or compressed video) version on S3, and two copies of the original format on DVD - one onsite and one offsite.

I expect to have to move from DVD media periodically, but I can put that off until the higher-capacity disk wars play out. Every file on the DVDs are md5sum'd, and i know I can use ddrescue to pull data blocks off either wafer, if S3 and my home drive die, assuming the scratch doesn't hit both disks in the same place. It'd be nice to have an automatic system to track which file is on what DVD, but I haven't implemented such an HSM yet.

I'm enough of a pack rat to keep a DVD drive and probably a computer that can read it essentially forever, and if not, there's always eBay.

The biggest problem I face is not deleting all of the content from a card (or tape) before popping it back into the camera and adding more. So when I copy a media into the "system" I might have other duplicate copies of the pictures. I'd love to be able to deduplicate those and store only one copy (and links to it). And even better would be a content-aware dedup that could tell that x.jpg is the same picture as Y.raw... (and that song_64kvbr.mp3 can be derived from song.flac)

But I haven't put that together yet, either.

--Joe

VMware View 3.0 and proxies

2009-02-18T17:02:00.003-05:00

Oops, I haven't blogged the first part of this story. Oh well, maybe later. In brief, we have VMware VDM to satisfy das corporate security. It was working for people on our LAN and on the corporate network, and I got it to work from the internet (but requiring a valid smartcard (SSL User Certificates) before letting a user in). This was a cool project I'll have to document here some time.

Well, time moves on and VMware View Manager 3.0 (nee VDM 3.0) was released and implemented in this environment.

The first problem we noticed started when a home user upgraded their View client to 3.0 as they were prompted on the login page. This was when the smartcard authentication from the internet stopped working. A little investigation (watching network traffic, decrypting with Wireshark, etc) and I found that while the old client would send an HTTPS post command just like IE, the new client didn't send the user SSL certificate. But since VMware never supported this sort of setup, I just worked through it (another cool solution I'll have to post later). A little bit of rearchitecture, and I was able to still protect enough of the View environment to make me feel secure and to convince the security people that it was sufficient.

Now, I've got a similar error from the corporate network. Same message: Connection to View server could not be established". But WTF? this is on the LAN, there shouldn't be a proxy problem. IE works just fine*, but View can't connect.

That is to say IE worked fine with the proxy, but the proxy requires user authentication, which is cached for the browser session, and I didn't think of that until later.

So fire up Wireshark again, and once again, the first couple of View CONNECT :443 requests from IE happily sent the Proxy-Authorization: header, but the last one tried to do a CONNECT without that header, and was tossed back a Squid Authentication Required 407.

Ah, that's a relatively easy one to fix, if only I could get the proxy admin to turn of authentication (nope, that's verbotten) or do the same sort of magic as I did on the outside firewall deployment (eww, that'd be messy) or maybe bypass the proxy for this? I mean, they're on the LAN. Luckily VMware apparently thought of this and implemented an undocumented registry key: HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\ProxyBypass that contains a MultiSZ list of names or IPs for View to connect directly to instead of using the proxy.

Did I mention that all of this new behavior is undocumented? And that what I'd been doing in the first place was both unsupported and completely WORKING?

I'd guess that the new View client switched from a standard MS HttpRequest method to something they threw together without the nice functionality that IE bundles into its method. Oh well. It's working again now.

--Joe

Moving RHEL to ESX

2008-09-23T16:47:00.004-04:00

We have a cloud application that we have purchased for in-house deployment. It's a long story that I won't share, but we have been seeing performance problems with it, and have ended up with a (physical) server in our datacenter that they have configured just like their cloud resources, so we can compare performance on our (virtual) system against the way they run things.

To make sure we capture the most data we can about this system (and to demonstrate that it either is or is not the virtualization layer causing slowness) we've been tasked with copying the physical server into a similarly-configured virtual machine.

Unfortunately, VMware Converter apparently can't do Linux. So we had to use the more standard drive-image-transfer toolkit of dd and netcat. But even after the image was transferred, the kernel would crash because it couldn't find the root disk.

This is to be expected, but only Google knows how to solve it. If you know the right keywords. Good screenshots of the process are at http://virtualaleph.blogspot.com/2007/05/virtualize-linux-server-with-vmware.html but it's modprobe.conf not modules.conf in RHEL5.

So here's the steps I took:

Boot from the (correct) RHEL install CD #1 with "linux rescue". Note that it has to be the correct RHEL version and architecture. Since the appliance was running 5.2 x64 edition, my 5.1 x86 cd didn't work, and I had to download a different CD1.

Skip the networking config since it won't help, search for RHEL installations, and continue to the root shell (in read-write mode)

chroot /mnt/sysimage and edit /etc/modprobe.conf. Change the eth0 module alias to pcnet32 (you can remove the eth1 alias if you don't have a second nic in your VM) and change the scsi_hostadapter alias to BusLogic. (again, you can remove other aliases if you want)

Then copy /boot/initrd-.img initrd--phys.img as a backup, and build a new initrd file with the new devices: mkinitrd -v -f /boot/initrd-.img

If that works, you should be able to boot the VM and have it come up cleanly.

--Joe

Reflections on x4500+ZFS+NFS+ESX

2008-09-17T09:27:00.002-04:00

I was asked about my thoughts on running ESX over NFS to a ZFS backend. For posterity, here they are:

x4500+ZFS+NFS+ESX is a quite functional stack. There are a few gotchas that I've run into:

First, the ESX "storage delegate" functionality doesn't. This is supposed to change the EUID that the ESX server sends with it writes. Well, it does for most of the requests, but not for things like creating the VM's swap file. So you pretty much have to export your NFS shares with root=vmkernel.ip.address

We have many ESX servers, so keeping the sharenfs= parameters got unweildy. I ended up putting them in a text file in the NFS share for easy editing, and when I have to add/change an ESX server, I edit the file and zfs set `cat zfs.shareprops` /pool/path/to/share

NFS is much better than iSCSI. At least in the version I did iSCSI testing, all of the ZFS volumes presented from OpenSolaris were recognized by ESX as being the same disk. This meant that I had a dozen paths to the same vmfs datastore, some 100GB, some 500GB, etc. This Was Bad. NFS made it better.

NFS also gives you a couple of other benefits: On NFS datastores, the vmdk files are by default thin-provisioned. This means that if you give your VM a 5TB vmdk, and don't use more than 10GB, it takes up 10GB of capacity on the physical disks. It's also much better understood by troubleshooting tools (wireshark) so it's easier to find problems like the storage delegate issue above. Also, it's a first-class citizen from Sun. NFS serving has been in Solaris since 1994, and isn't broken by the latest Nevada builds. Sun takes NFS seriously.

The downside of NFS is that ESX makes all its requests O_SYNC. This is good for ESX but bad for ZFS. Your nvram cards should help a lot. I ended up with a different solution: The business agreed that these are not Tier-1 VMs, and they're not on Tier-1 storage. So I've turned off all ZFS sync guarentees with /etc/system:


* zil_disable turns off all syncronous writes to ZFS filesystems.  Any FSYNC,
* O_SYNC, D_SYNC, or sync NFS requests are services and reported completed
* as soon as they've been transferred to main memory, without waiting for
* them to be on stable storage.  THIS BREAKS THE SAFETY SEMANTICS AND CAN
* CAUSE DATA LOSS! (clients have moved on thinking the data was safely written
* but it wasn't)
* However, in our case, we can afford to lose this data.  For DEV/Test systems
* rollback to the latest (hourly) snapshot is considered acceptable.
set zfs:zil_disable=1

As the comment says, this would be a bad thing. But I know that the vmdk files are crash-consistant every hour and that's OK to the users. If they lose an hour of work, it's annoying but worth the cheaper storage.

Finally, and most importantly:

MAKE SURE YOUR POOL IS CONFIGURED FOR YOUR WORKLOAD. Vms are effectively a random-read and random-write workload. There is no sequential access of the vmdk files except when you're cloning a VM. So you have to understand the read and write characteristics of your zfs pool. RAID-Z and RAID-Z2 always read and write a full RAID stripe every time. This means it has to read from all of the disks in the pool to return a single byte of data to the ESX host. Mirrored pools, on the other hand, read from a single disk, and if the checksum is correct, passes it back to the ESX host. So in my case, I can have 44 simultaneous read requests from the ESX servers being serviced at the same time (44 disks in the pool) and/or 22 simultaneous writes (each write is written to two disks). Basically RAID-Z[2] is bad for random workloads, but mirroring is expensive.

With this in mind, performance on the thumper is excellent. We can easily saturate the onboard 1Gbps network link with NFS traffic, I've got link aggregation and can easily saturate the combined 2Gbps link. I haven't seen what happens with 4 uplinks, but I'd expect that the network will still be the slowest part of the chain. Doing basic I/O benchmarks on the thumper, I can get 1GBps out of the disks. Yes, that's 1GB per second.

How to grow an iSCSI-presented zvol in 3 easy steps

2008-07-28T11:30:00.002-04:00

Well, ok, it's not quite 3 easy steps.

A couple of things that don't work: iscsitadm modify target -z . This only works if the iscsi target's backing store is a regular file, which in the case of a zvol, it is not.

The easy bit: Make the zvol bigger:
zfs set volsize=200G tank/iscsi/thevol

Now we have to hack around in the iscsi parameters file: Locate the /etc/iscsi/tgt//params.# file that corresponds to the right target and lun and change the <size> parameter to be the new (in hex) size of the bigger volume in 512-byte blocks. Or in other words,

zfs get -Hp volsize tank/iscsi/thevol | perl -lane 'printf("%llx", $F[2]/512)'

Once that's done, apparently you have to bounce the iscsitgtd to get it to reread the params file.

Then on to the initiator...

format c3tAREALLYLONGSTRINGOFDIGITSFORTHEDISKGUIDd0s0 and changing the parameters won't work, since I'm using EFI labels and it says very strongly

partition> label
Unable to get current partition map.
Cannot label disk while it has mounted partitions.

So I have to go in the other way. While I'm in format, print out the current partition table, and make note of the Last Sectors for the slices. Also, run prtvtoc against the disk to get any other useful bits.

Then I can make the actual partition changes with fmthard:

fmthard -s - /dev/rdsk/c3tAREALLYLONGSTRINGOFDIGITSFORTHEDISKGUIDd0s0

At first, just copy in the line(s) for the slices you already have, but move slice 8 to the end of the disk:

*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00         34 251641754 251641787   /zones/mars/data
       8     11    00  419413982     16384 419430365

Then (check it in format to make sure the disk is still healthy) change the Last sector and sector count for the real partition. (Last is s8's first -1, and the sector count is s8's first -34)

Then it's a simple growfs -M /zones/mars/data /dev/rdsk/c3tAREALLYLONGSTRINGOFDIGITSFORTHEDISKGUIDd0s0

--Joe

Goodbye to a problem indicator

2008-06-26T17:18:00.002-04:00

We have a new Sun T5220 with the new Niagara-II chip in it. I don't know if it's 8-core/8-thread or 4-core/16-thread but it shows up in Solaris as 16 CPUs.

I'm running a cpu-intensive process (bzip) and prstat is only showing it as using 1.2% of the CPU. On another older system, the same bzip takes up 25% of the 4-proc box.

On the old system, it's clear that bzip is a single-CPU bottleneck (because it's single-threaded). On the new one, well running full-out but only taking 1% doesn't look like much.

--Joe

Forcing kernel dump on Opensolaris x86

2008-06-22T09:48:00.003-04:00

Just in case you were caught off guard by the documentation that points to using "rip::c" in mdb, that apparently doesn't work on Opensolaris, at least on build 70 that we have installed on our x4500s.

But on the bright side, I found a mention at wikia (http://opensolaris.wikia.com/wiki/Miscellaneous_FAQ) that says I should use $<systemdump. And that works.

Firewall + IPMP + IPF + Zones = lost packets

2008-01-16T16:10:00.000-05:00

Blarg!

There's no intelligence in IPF to know where to send packets when IPMP detects a link failure.

So, even though the zone will get all the packets destined for it, the global kernel will pick an interface based on its global route table, pass the packet down the wrong interface, ipf will bounce it to the FAILED interface (below the ipmp layer, apparently) and the packet is never delivered.

So I need something that will notice when an IPMP failure occurs, and rewrites the IPF rules to respond to the networking topology change. Look for my PF_ROUTE socket watcher program to be posted shortly.

--Joe

Media server v0.1a

2008-01-16T11:45:00.000-05:00

To solve the automation problem, I'll use Linux. Since I don't have any spare computers laying around any more (I'll turn in my geek card), I've gotta set it up in a VMware instance. Luckily, VMware server is free, and I have disk space here.

Two disks... 8GB for OS, and a 32GB "data" disk to start with. Both sparsely-allocated, cuz I don't have that much disk space.

I know debian better than any other distro. So, a base install, no tasksel, update and upgrade, then installed ssh, apache2, subversion, subversion-tools, and sudo. So far I'm using 515MB. And it can't do anything yet.

Someday I'd like to switch it over to Solaris so I can run the data under zfs, but for now, I like the convenience of aptitude.

In the mean time, it's LVM I guess... I pvcreate'd /dev/sdb, vgcreated dataVG /dev/sdb, and lvcreated dataLV in it. Then mke2fs -j /dev/mapper/dataVG/dataLV and tuned it to want a fsck every 365 mounts or 730d. Mounted it as /data, and I think I'm ready to start developing.

First some data, though... I grabbed about 350MB of data from various picture CDs, copied it to /data/media/$cdname, and copied them around again to make sure there were duplicates in the data...

More to come...

Getting code/screens to show up nicely

2008-01-15T10:43:00.001-05:00

After a knowledgebase search, I found Adam Flater's blog entry http://adamflater.blogspot.com/2007/02/on-source-code.html describing a nice-looking way to do code on blogger.

I've added to my template the following style definition:
pre.code {
overflow: scroll;
}

and now I can wrap the code with <pre class="code"> and have it look nice.

Thanks, Adam!

--Joe

What's going onto the disk?

2008-01-15T10:28:00.001-05:00

On our thumpers, `iostat -nxz 5` gives a good picture of what's happening on each of the disks, and `zpool iostat 5` gives a good overall picture of how fast things are going at the moment, but neither of these break the picture down into "Who's writing?"

A Solution:

# fsstat -T u `zfs list -H -o mountpoint -t filesystem` 5
1200435752
    0     0     0     0     0      0     0     0     0     0     0 /uscisbds001
    0     0     0   106     0      0     0     0     0    53  217K /uscisbds001/esx3
    0     0     0     0     0      0     0     0     0     0     0 /uscisbds001/esxpatches
    0     0     0     2     0      0     0     0     0     0     0 /uscisbds001/isos
    0     0     0     0     0      0     0     0     0     0     0 /uscisbds001/nfs
    0     0     0     0     0      0     0     0     0     0     0 /uscisbds001/saperisrv1
    0     0     0     0     0      0     0     0     0     0     0 /uscisbds001/saperisrv2
    0     0     0     2     0      0     0     0     0     0     0 /uscisbds001/templates
    0     0     0 1.56K     0      0     0     0     0   654 15.8M /uscisbds001/temprestore

I can see that somebody's writing a buncha data to temprestore, and a little bit is happening in the esx3 directory.

--Joe

Fun console error messages

2008-01-09T14:45:00.000-05:00

For you long-time Solaris console message debuggers out there, try this one. It's an old (Solaris 8) www server. This is what's scrolling on the serial console.


r ,R qAeStCdQB:o k0 x603,0 8F R U :   0 x 0
s b u s @V1nfo,:0S/mQiLsG C , i s p @ 2 , 1 0 0 0 0 /esidl@N5m,e0:  (ws d 5 )[:

a        E rerso re :fHorrw rC ormom
@WvReIlG:  sRue@tfr,y/aLbCliesi u   o e 0 8 > , A C Q   xE,rFrUo r0 0L
B,      0R0e/qdu@e,s tse5d:
0 l o c kR:q e1t1d3B9o7k8 8285w i e 1 )               E r r L v lE rerroarl
         V e n d o r :  rSoy mlbci:o3s9 4 8
          e d r   y b o                          S eSrriaa lu bNru m b e r :[
dw        S n9e[K yg
ea      eSEernos
4  K e y :S :H0a8r d<weadrreu iEurercodr
<)       ASSQC :x ,0FxR8U4  x(
CvAeNnNd:o/rs busn@ifq,u/eQ Gc,ospd2e, 0000x/8d45>0)(,s 5A:S
)Q :   0 xr0o,  oF RoUm:n :0wxi0e
5 / Q L G Ce,uissep @l2c,k1 05060808/ s d @ 5 , 1   ( srdo6 5l)c::
E8
 r r o r   feonro :CSommimsa n d :   w r i t e ( 1 0 )    e i l N m b r         r [ gE
lr o r   LeesveeKly:  HRrewtrr yraob
 e
r 9 8 2 4E s u @ f 0 Q GE,rsr@o,r0 0B/ldo5c1k :s 67)38:2040   R : 0 0
           e u s e   l c :S1e0r2i3a2l   N u m b e r :  r o  wl c : 1 0 293[

r       eSdern syeb oK e y :   H a r d w a r e   ESrrrao ru
e 8 4   (S<nveeKnyd oard auen irqou
x  c o d eA C0 xx844(>v)n,o rAuSiCuQ :o e00x80>,,  FSRQU :x ,0FxU0

pAANRNN:I/NbGs:1 ,//sLbGu,ss@@12f1,000/sQ@L,G Cs,5i:s
 @ 2 , 1 0r0o0 0o/ sodm@n5:, 0r t((s0d)5 ) :
         ErrorroLrv lf:oRrt yCboem
                                  m a n d :e urseea dl(c1:03)9 0 8              e    E r r   lEcr:r3o9r0 8L
bv e l :  VFnadtra ly
o   B l oASc:k0:x 42(5v1n6o7 5n2q0ec kS:r a2l5N1m6e7:5 2w0    9
oe      0V8e>n,dAoCr::0 0,SFyUm bxi
 sA N N : / b s 1 , / L C , s @ , 00 0 s @ ,   s 6 )
v  S e r iraol  oN uommbne: rr:t ( 0  w           9 [E rgr
  e n d o re:u sSey mlbci:o5s3 1 7 2                  E r r B o c : 5 3 1 1
:        SVenroi: aylb oN u m b e r :       w          S9r[a   um
        wS e n9s e
oK e y :  eHsa red:wHarrwer  Error
 r
0W0RxI8G4 >s)u,@ fA0SQCGQ,:s @0,x000,/ dF5R1U(:d 50:xq0u0e  Rc:o0d0e

  0 / s dR@q5e,t0d B(oskd 59)4:21 f , 0 / Q L GECr,riLsvpl@ 2e,r1a0l0
:   w r iVtneo(:1 0y)b o                            ESrrrao rN mLee:v ewl :  9R t
er y a b lSen
E B l o cAkC:  x342(9v5n4o0 8n0q e c d   x 4 )   S Q   x , F U   x
 rArNoNr: TBelAorcak :d i3e2 9s5r4t0r8i0g
:W R I G   s u @ f 0 Q G , sS@e,r0i0a/ld @N,u msb5e:r c o   6 6
i      w  r o   o  9o[m ng:
e r r o rR)e n s e   K e yE:r rHLavrld wearrael
dt      dABSoCk:  605x5824   ( < v e n d o rr ou nlicq:u3e3 0c9o
be   0 x 8e4d>r) ,y bAoS C Q :   0 x 0 ,   F R U :  S0rxa0
  @ 2 , 1A0C0 0x04/(svdn@o5 ,n1q e(csdd 6x54)):
fx      ,E rRrUo rx
l        E r r f rE rormonr: wLieev1e)l :   R e t r yEarbrlLev
E s t e dR qBeltodcBko:k  17705373462 7 2              E r r B l c : 7 0 0 3
 r r o r  eBdlro cykb:o  1 7 5 7 4 2 7 2
         V eSnrdao ru:b rS y mwb i o9s
           S : 0S8e4r(ivanlo  Nnuqmebcedr :x 4 )  wS Q   x , F9U[  x
A
        NS:e/nbsse1 ,K/eLyC:i pH2a1r0d0w/adr5e0 (Edr)r
  4 > ) ,  eAuSsCtQd:B o0kx 08,8 6F0R U :   0 x 0  irqou ee ec:oRdtey b0lx
  u s @ 1Vfn,o0:/SQmLiGsC , i s p @ 2 , 1 0 0 0 0 / sSdr@a5l,N0m e(:s dw5 ) :9
:        E r r oSrn efKoyr  aCdoamemEarnod
l  w r i tSe:(0180 )< e d r u i u   c d   x 4 ) , AESrQr oxr, FLUe vxe
G:A RRIeGt rsyua@bfl0eQ
i,is    @V,e0n0d/odr5:1 (Sdy5m:b
 o s      E r r f r C m a d   r t                     E r r LSeeer:iRatly bNeu
                                                                              m   e r :e u s ew  l c : 6 49 [
s a r e  VEnror:oSrm Kreryr:B oHka r4d
o 0 ,   FARCU :0 80 x<0eSaCeQE:r r0ernad ourb ru n i q u e9  c
Q WnAqReNcIdN Gx:4 )/ sSbQu sx@,1FfU, 0x/
5LAGNCN,:i/sbps@12,,/1Q0G0,0s0@/,s0d0@/5d,511 ((ds5d:6
E) :
 r      rEfrrrComra df orrt (C0o m m a n d :   wrroi teee : R t y b e
                                                                               e4u s e   l c :E2r4r4o2r   L e v e l :   REertrr ylacb:l2e4
R2
 e q u e sRteeuds eB llocc:k2:1 5180 6 6  4 0            E r o   l c : 2 5 6 8 8    E r  reodrr  Bylbooc k :   1 0 6 6 4 0
         S n e K y   ar w r   r o i o
[         S : 0 8   <Seedrriuailue cNdu mxb4e)r :S Q : 0w0   R U   x9
: A N
E/s     uS@efn0sQeG ,Kse@y,:0 0H/adr5d1w(adr5e:
rr  r o rE
 f      rACSmCa:d  0rxt8(40  ( < v e n d o Err ruLnvilq ueer acloed
                                                                   e   0 x 8R4q>k)t,d BAocS:C7Q3:6 80 x 0 ,   F R U :  E0rxr0B
N G :   /Vsnbou:sS@m1ifs, 0 / Q L G C , i s p @ 2 , 1S0r0a0 0N/msed:@ 5w, 1   (96 d
n5 ) :
o o m m a nSd::0 xw4r(ivtneo( 1n0q)e c d   0 8 > , A C : 0 0   F UE rxr
0rA RLIeGv e/lb:s 1R,e/tLrCyiasb@l,e0
e/      dR5e1q(ude5s:t
  r   B lRoqcekt:d B7o3k4 422895444 1 )               E r r L v l   e rEarlr
         V n o : S m i s                             SSrear iuablr  N u m b e[r       w  S n e   e9:[H r w
er      oS
Sn s e   KSe:y0:x 4H(avrndow anrqee cEdr rxo4r)
    A S CEQr:o  0fxr0C,m aFdR Ur:t e01x)0 e@ 500x(8d4)>
  W A R N IrNrGr:L v/ls bRutsy@b1ef
                                   , 0 / Q LeGuCs,ei slpc@:23,91008080 0 / s d @85 , 1  E(rsrdB6o5c)::3

        E r r o re dfro ry bCoo m m a n d :   w r i t e ( 1S0r)a   u b r        t     E rSrnoerK yL eavdeale:  rRoe
0r y a b lSe:
kW R I G   s u @ f 0 Q G , s @ , 0E0r/rdo5r1 (Bdl5o:c6:0080
a:   2 6 4r4o4 6o0r8C
 d: WrAtR(N0I N G :   T h e  rAor reaey: Rdtryibvee
                                                   r   i s  eruesteu rlnci:n7g7 ra8n4  E r r o r e d   IE/rOo,  lwci:t7h7 7e8r
 n o   5 ,e dorn  yMbood u l e   1 ,   L u n   0 ,   s eecitloNrm e1:1 3w9 7 898 8

I f , 0 /AQCL GxC4,(ivsnpo@ 2n,q1u0 0o0e00/8s>d,@A5C,:00 0( sRd:50)0:
1G       Esrur@ofr0 QfGo,rs @C,o0m0m0asn@d5:1 (rde5a)d
   )        E r r f r C m a d   r t (E0r r o r   L e v eElr:r LRvelt reyraabll
m        V n o :ESrmriosr   B l o c k :   3 2 9 5 3 3e2i8lr r B o k   5 6 8 8
,W R I G9 [s ugs 8   <> ,NAuCm:b0e0r :R U : 0w0
a/      LSCeinps2e1 0K0esy@:, 1H(adr5d:w
Cr e   E rrroo rf
: o d e  R0qxe8s4e> )l,o kA S6C3Q0:8  0 x 0 ,   F R U :E r0rxB0l
N W6A3R8N4I
 G :   / sebdurs:@S1mfi,s0 / Q L G C , i s p @ 2 , 1 0e0i0l0N/msed:@ 5 , 1   ([se 5 ) :
5WrRoIrN :L/ebvse1l,:0 QRGe,tsr@y,a0b0l/ed 4 )   S Q   x 0   R : 0 0E
e1      (Rde5q:u
  r r o r  eBulsoec kl:c :7 362824928 2 4             E ro   e e l   e r a l
r        V n o : S m i s                              e iSleNrmiea:l  wN u m9b e
r:       Swn e K y   a9d[a e E
x        S e n sAeC  Kxe4y(:<v nHoa rndqweacrde  xE4r)r oSrQ
 ,       ARS:C0:0
00AxN8N4: /(b x 8 4 > )r,o  AoS CoQm:n :0wxi0e,1 )F  R U :   0 x 0r
 eWeA:RRNtIyNbGe:
                   / s b uesue@s1ef ,l0c/:Q5L3G1C1, i s p @ 2 , 1 0 0 0r0o/ Bsod: d 5 ) :V2(
           e s   e : H r wErr rroor m a nSdr:a  wurbirt e ( 1 0 )[
uL e v e lA:S :R0e8t r<yeadbrluei
rW3R6I3G5 0/5b9s21 , / L C i p 2 1 0 0 s @ 5 1 ( d 5 :E
 r o r   Brloo cfkr:C m3a6d3 5w0i5e9120
         V e n d oErr:r LSvylm bRitoysb e
                                                   e u s e   l c : 4 4 9 8      r            ESrerrriBaolk :N4u4m9b8e
 :       we d o : S m9i[s  g
U   S e rAiCa lx 4N(uvmnboe rn:q e c dw  x 4 )   S9Q[  x ,
e       xS
:nAsNeN :K/ebys:1 ,H/aLrCdiwpa2r1e0 0Esr@r,o rs

  ,   A SRCqQe:t d0Bxo0k,  1F9R8U8:   0 x 0  )u n i q u e   cEordreL v0lx 8a4a>
g @ 1 f ,V0n/oQ:LSGyCb,oi s p @ 2 , 1 0 0 0 0 / s d @ 5S,r1a  (usbdr6 5 ) :
l w r i tAeC( 1x04)( v n o   u i u   o e 0 8 > , AECr:r0o0r  RL:e0v0e
w:   R e tersy aeb:lHer
  s t e dA SB:l oxc4k(:v n2o5 8n0q5e2c4d8  x 4 )   S Q   0 0   R : 0 x
) A NENr:r/obrs 1B,l/oLcCki:p 22150800s5@2,4 8s
c     S eRrqieatld BNoukm b6e4r6:4      w           9E[r r B L v e l   e r a l
  K e y :  eHdarr dywbaor e   E r r o r
         A S C :S r0ax 8u4b r( < v e n d9o r
)u n i q uees  ceo:dHer w0rx 8r4o>
,,   A S CSQ::0 80 x<0e,d rFuRiUu:  o0ex008
sAWCA:R0N0I NRG::  x/
dbAuNsN@:1/fb,s01/fQ0LQGGC,,si@s,p0@02/,d1@0,0 0s05/:s
i@ 5 , 1  r(os do6 5o)m:n
r r y a bVlneo1t0d)B l c : 2 1 5 8 8              E r rEorrr BLoekv:e2l1:6 8R8e
u c k :  A7C1 7x347(9v8n4ooerE rBrlc k :  S7r1a7 3u7b9r8 4  w       9 [ g
 W R I G   h   r a   r i e   s r t r i g   n ESrerrdi a/l, wNtu mrbne r,:  n M d6l   , L n 09 [e t r


        SAeNnNs:e/ bKse1y,:/ LHCairpd2w1a0r0es @E,r rso5r:

4        A S Cr:o  0ox 8C4m a(d< vretn(d0o r   u n i q u eE rcroLdvel  0axa8
 > ) ,   AeSuCsQe:  l0cx:03,9 0F8R U :   0 x 0
b WrAoR NlIcN:G3:9 0/8s
 u s @ 1 fe,d0r/ QyLbGoC , i s p @ 2 , 1 0 0 0 0 / sSdr@a5 ,u1b r( s d 6 5 )[:g
tWeRvIeGl :h  Rreat rryvaeb lser e c d   x 4 )   S Q   0 0E rRr:o0r0
 r      iRgeaq ureoset e/d, wBtl orcnk :,  2n5M1d6l5 8,8L8n 0   e t r 7 7 86

8  A N N : / b sE1r,r/oLrC iBpl2o1c0k0:s d@2,5 1s665)8:8

        rSoe nos eo mKne:yw:i e(H0a r d w a r e   EErrrroLrv
:e      rAaSlC
    0 x 0 ,e dFrR SUm:i s0 x 0 5u4n7i q u e   c o d  e  r0ox 8l4o>k) ,1 5A4S7C
e W A R N I N G :  STrhael  Aurber:a yw  d r9i v
2WeR I1G,  sLuu@nf, /1L,C ispe2c1t0o0rs @1,6 0s8d254)3: 0o0n  RM:o0d0u


  WrAoR NoIrN Go:m n/:swbiues1@)1 f , 0 / Q L G Cr,oi sepe@:2R,t1y0b0e0
                                                                       0 / s d @ 5e,u1s e( sldc6:57)3:6
  w r i t ee(d1r0 )y b i s                        E r r oerr aL euvberl :   F a  [a l
:        R eSqnueeKsyt eadd aBrl orcok
o  5 0 3 3A1S7:1028   < e d r   n q e c d   x 4 )   S QE rxr,oFrU  Bxl
icAkN:I G5 0s3u3@1f701Q2L
ip      2V1e0n0d/odr5:1 (Sdy5m:b
  b e r :R q e twd B o k   096[0                      E r r LSveer:iRatly bNlu
a        S e n s eE rKre yB:o kH:a1r6d4w
br e   E rerdorr
do       A S C :   0 x 8 4   ( < v eSnrdao ru burn i q u e  [c o
n ,   F RAUC:  008x40(aSdCaQe:E r0rx
ioW AnRqNeIcNdG :x 4T)h eS QA rxr,aFyU  dxr
 vAeNrN :i/sb sr@eft0uQrGn,isn@g, 0a0n/ dE5r1r(odr5e:d
 I / O ,  rwoi toh  oemrnr:nwoi e51,)  o n   M o d uElreo  1e,e :LRutny b1e,
                                                                              s : c teours e6 4l
47
G2W4A R N I N G :   / s bruos @l1cf:,704/2Q4L
mC , i s pV@n2o,:1S0m0i0s0 / s d @ 5 , 1   ( s d 6 5 )e:i
oe      :E rwr o r9  f
1r   C o memsa ned::H rwwrri treo(
 0 )      A C   x 4 ( v n o   u i uE roreo0r8 >L,eAvCe:l0:0  FRa:t0axl

e          r r r f r C m a d   r tE(r0r)o r   B l o c k :r o4 9e4e4:9R2t8y0b

         V e neduosre:  lScy:m2b4i4o0s                  E r r B o c : 2 4 4 0
          e d rS eyrmiiasl   N u m b e r :       w    e i l N9m[e :
n       9S e
  o r  K eSyn:e KHya raddwaarr er rErr
xWeR I0NxG8 4s>u)@,f 0AQSGC,Qs:@ ,00x00/,d 5F1R(Ud:5 :0uRe: 0c0o
 0
  M o d uVlneo :1S,m iLou n   1 ,   s e c t o r   2 4S2r8a3 4u5b6r  56,3 0o8n
  r K e yS:n eHKayr:dHwrawrre  EErrrr
tW0RxI8G4 >h) ,r aA SrCvQr:i  0exu0n,n  FnREUr:r d0Ix/0,0e  Rc:o0d0e
,WiRsIpG@ 2s,u1@0f000/0L/Csids@@5,,010 /(ds5d16(5d)5::

         E rrroo ro  foomrn :C ormimea1n)d :   w r i t e  E r r L v l   a t l
                                                                                l      e q e tEdrBrookr  6L8e4v3e l :   R e t r y aEbrlreB
ec      kR:e q6u8e4s2t
 d   B l oVcnko:: S6m4i o                             S r a   u b r :  Ewr r o9rB
 l
, oAcNkN:: /6b4s
 /      LVCeinpd2o1r0:0 sS@y,m bsi5o:s
  e r :  R q ewt d B o k  92[5 3 8                    rSoe revila le rNaulm
a        S e n s e  rKoe yl:c :H3a9r3d2w
ir e    EVrnroo:rS
0 d e   0Sxn8e4 >e): ,a dAaSeCEQr:o ernad ourb ru n i q u e[ gc
 x 0 ,   FARCU :x 40(xv0n
/niWqAeRcNdI NxG4:)  /SsQb uxs,@F1Uf ,x0
dQALNGNGC ,sius@pf@02Q,G1,0s0@0,00/0s/dd@551,(0d 5(:s
o 5 ) :
        oE rormonr: wfioer( 0C o m m a  n d :E rwrrLivtle (a1a0l)
                                                                          e u s e   l ck   3 2 8E4r r o r   L e v e lr:o  Rlect:r7y2a9b2l

         R eeqnuoe:sStmeids  B l o c k :   4 8 3 8 3 6 8 0e r a   u b r           9
8      E rerso re :BHlrowcrk :r o4
x3 8 3 6 8S0C
 W R I G : T e A r a   r v r i   e uSnenr inaElr oNeu m/b,ewrt:  r n  w, o   o use91[  ug
dWyR:I GH asrud@wfa0rQeG ,Esr@r,o0r0
05      0A SsC5::
 x 8 4   (r<ov>L)e,e :AFStClQ
                                                                          :   0   0e,u sFeR Ul:c :03x300
: L G C ,Vinsop:@S2m,i1s0 0 0 0 / s d @ 5 , 0   ( s de5i)l:N
l 0 )    A C : 0 8   < e d r   n q eEcrdr oxr4 )L eSvQe lx:, FFUa txa
A
 N      NR:eTqeuAersat erdv rBil oecukn:n  3n2E9r5r4d0 8/0, w t   r n   , o   o c e 1   u n 0  Eertrro1r1 4B7l6o
 k
C :A N3N2:9/5b4s018,0/
oi      pV2e,n0d0o/rd:5 1S(ydm5b:i
 s        r o   o   o m n : w i e 1 )               E r r LSvelr iearly bNeu
                                                                            m b   r :R q e twd B o k   790[3 3g6
a   W A R N I NrGo:  lTch:e7 0A0r3r
 y   d r ievderr  yibso  r e t u r n i n g   a n   E rSrroar eudm eI:/ Ow,   w9iht
:   1 ,  ALCu nx 41(,v nsoe cutiour  o6e30984>2,4A
00
  0 0 0 /Esrdr@f5r,C1m a(ds de6d51)): ,di5s0p(@d2),
         E r rroor  efeo:rR eCroamlmea
                                      n d :   weruistee (l1c0:)2 1 7 2          e          r oE rlrco:r2 1L7e2v
Sl :   F aetdarl:
i o c k :A C7:10783 7<9e8d4rr oErr rBk :S r7a1 7u3b7e9:8 4        [ g
 W R I G : T e A r y d r v r i   e u n n   n E r oSee r/i,awlt  Nrurmob5e rn:M d2l  w,   u   , s9c[o   7
dWnRsIeG :K/ebys:1 ,H/aLrGd,wsa@r2e, 0E0r/rdo5r1
S5      :A
  A S C QR:q e0txe0 ,l cF:R2U4:4 20 x 0  q u e   c o dreo  0exe8:4 >a)a,
r a y   dVrniovre:rS mibso  r e t u r n i n g   a n   E r reoirleNdu bIr/ O ,     i9t h
Q ,   L uAnS :10,8  s<eecdtoo rn q2e1c4d0 6x343)6
x
N,WFAUR NxI
 GA:N N/:sTbeuAsr@y1dfi,e0 /sQrLtGrCi,gias pr@o2e, 1/0,0w0t0 /rsnd @,5o,n1M d(lsod,6L5u) :,
o       2E8r0r
 r
   AfNoNr: /Csoum@mfa0nQdG:, sw@r,i0t0e/(d1501)( d 5 :
          r o   o   oEmrnr:owrr tL(e0v e l :   F a t arlo
 e      eR:eFqtule
                  s t e d  eBulsoec kl:c :215514675 8 8 8            E r r B o k  E r r oVrn oBrl oymciks:   2 5 1 6 5 8 8 8
 W R I   A C   x 4 ( vSneor inaqle cNdu mxb84e>r,:A C : 0w0   R : 0 09
 5       SAernysdrev rKie yr:t rHiagrad wraoree  /E,rwrto rr
u       nAMSdCl:  10 xu8 4, s(c<ov e7n7d2o2r
,nAiNIqNu:e/ bcso1d,e0 /0LxC8i4p>2)1,0 0A/SdC@Q,:  sd05x:0
m  F R U :r o0 xo0
 nW:AwRiNeI1N)G :    T h e   A rrroa ye vder:iFvtelr
                                                      i s   r eetuusren ilnogk  o3n3 3E4r r o r e d   I / Or,o  wlict:h7 3e6r3r8n
   5 ,   oVnn oM:oSdmuilse   1 ,   L u n   1 ,   s e cetiolrN m6e4:
r     9  E
 WeRtIrGy ahb lrea   < e d o   n q e c d   x 4E)r rSoCr: 0L0e vRe:l0:0
 r      vRreiq ueeusntiegda  BrlrorcdkI:O  7i6h0e3r0n3 3,6o n M d l   , L n 1   dWoRrI GB:l/obcsk1:, /7L6C0,3s0@3,3060
r5      1V(edn5d:o
B S e r iRaqle tNduBmobke r5:0 2 8  w           9 [  r ro   e e l   a a
  e y :  VHnaor:dSwmairse   E r r o r
u        A S C :  S0rxa8l4N m(e<:v ewn d o9r
Sn i q u eS ecso dee: H0rxd8a4e> )r,o rA
dC Q :   0SxC0 ,x 4F(RvUn:o  0nxq0e
@ WxA4R)N ISNQG :x ,/FsUb uxs
 1
  d 5 ) :EQ L/GbCs,1i,s/pL@C2i,p120100000s/@s,d @(5d,50:
 o       Eormraodr  rfto r   C o m m a n d :  rroe aede(:1F0t)l
                                                                         e u e t d B o k   1 6E4r r o r   L e v e l :  rRoe tlrcy:a1b6l4e

         R e qVuneos:tSemdi sB l o c k :   2 5 1 6 7 5 2 0S r a   u b e :   w     9
0 6 7 5 2 0Sso re :BHlaodcake:E r2r5
 8       V<eenddrouri:u>o,s  S Q   x , F U   x
  A N N : T e A r a   r v r i   e uSnenr inaElr rNduImOb eirh:e r o 5w  n M d l s9,[  ug
dWeRyI:N :H/abrsd1w,a0rQeG ,Esr@r,o0r0
05      1A(SdC5::
4   0 x 0 ,e uFsRtUe:  l0cx:0 wiiqeu1e)  c o d e   0 x 8r4o>r)L,v lA SaCaQ
m C , i sVpn@o2:,S1m0i0s0 0 / s d @ 5 , 1   ( s d 6 5S)r:a
oe      :E rwr o r9  f
b        A C   x 4 ( v n o   n q eE roreo0r8 >L,eAvCe:l0:0 ,R eFtUr yxa
NlAeN
 :      TReeAqruyedsitee ds rBtlroicgka:  r6o4e   / O , w t   r n   , o   o u e 0WoRcIkG:  s6u4@ r3r1o2r6
  r :    R qwe t d B o k9 [6 4 6 e 1 )              SEerrriLavle :NFutmlb
e        S e n s e   KEeryr:B oHka r6d4w6a8r
r  E r r oVre
< F R U :A C0 xx04S CaQd:a e0Exr0r,e n d oSrr au nuibqru e   c o d[e

--Joe

My Media project

2008-01-06T11:23:00.000-05:00

I have had a digital camera for about 5 years now, and we've accumulated a lot of pictures from it. The biggest problem we've run into is that we have a tendency to not ever print any of the pics, nor really to organize them, or really do anything else with them. In the face of my fourth hard drive "data event", I'm looking for The Better Way to deal with the volume of digital media I'm saving.

What am I trying to do:
1. Never lose an original piece of digital media (photo, video, CD, .m4p, etc)
2. Have my digital media accessible so we can use it, in reasonable format (web-thumbnail, .mp3, etc)
3. Not keep a buncha copies of everything
4. Do it automatically as much as possible

So far what I have is about 350GB of potentially-original pictures and video on our home computer and on a set of DVDs. (m4p's are not included yet, and CDs are still on the shelf). I've got Apache sharing them out via WebDAV onto our local lan. I'd like to turn on SSL client authentication (distribute self-signed certs to our laptops) and be able to mount that drive up from any net connection. But there's a couple of minor issues (I'll talk about them later). I'm seeing the limits of my Unix-oriented thought processes when I'm running on a Windows XP Home platform.

For example, #3 above is easily accomplished -- "find . | xargs md5sum | sort" piped to a short awk script... if the md5sums match, diff the files, if they're identical, remove the second, and hardlink a second "copy" in its place. But Windows doesn't have find, xargs, sort, md5sum, awk, diff, or ln. (without cygwin)

#2 I think could be solved with a tool like Subversion... Check in the original media (take a CD for example...) titleoftrack1.wav and tag that version "primary". Then check it out, oggenc it, "svn rename .wav .ogg" and tag that version ogg-hq. Similarly, encode a mp3-hq, mp3-lq, etc. (I only have a 2GB ipod so I like 64kbps vbr mp3's there, but can afford higher-quality on my laptop or home stereo)

#1... I burn two copies of the original to DVD+R, md5sum the original (store the md5sums on the DVD too) and compare them after writing. I need 2 verified copies. One should be offsite, the other in a storage container onsite. (offsite will be implemented someday)

And that just leaves 4. How the **** do I automate all this?

--Joe

Simplified ESX patching

2007-12-12T15:11:00.000-05:00

Here's what I've set up for our VMware ESX servers...

I have a space that's accessible via HTTP (snippet from httpd.conf)

<Directory /usslsbds001/esxpatches>
        Options +Indexes
        Order allow,deny
        Allow from all
</Directory>

Alias /esxpatches/ "/usslsbds001/esxpatches/"

In there, I have directories corresponding to dates VMware has released patches (that I'm interested in)

# pwd
/usslsbds001/esxpatches
# ls -l
total 33
drwxr-xr-x   5 root     root           5 Dec  6 15:24 20071115
drwxr-xr-x   8 root     root           8 Dec  6 15:25 20071130
drwxr-xr-x   3 root     root          13 Dec 12 14:13 latest
drwxr-xr-x   2 root     root          14 Dec 12 14:14 packed

packed has the downloaded tgz files. $YYYYMMDD has the extracted patches for that date, and latest has the unpacked directory of 3.0.2 update 1, and symlinks ESX-1234567 -> ../YYYYMMDD/ESX-1234567. When a patch is superceded, I `chmod 0` it, and remove its link from latest.

I also have a scriptwriter that generates a set of esxupdate commands:

# cat ../latest/make-install.sh
#!/bin/sh
# generate an "install" file for the ESX patches in the current directory

DS=`date +%Y%m%d%H%M`

ls | grep -v install | while read patch ; do
echo "esxupdate -n -r http://`uname -n`/esxpatches/latest/$patch update" >> install.$DS
done

rm -f install && ln -s install.$DS install

All of this rolls together on the ESX service console by simply doing (make sure HTTP client is open in the firewall)

GET http://thestorageplace/esxpatches/latest/install | sh

and rebooting...

--Joe

Straddling the firewall with Zones

2007-12-05T11:49:00.000-05:00

Our zonehosts have multiple NICs, on multiple subnets. This means that they have multiple default routes defined, so non-local packets are passed to those default routers in a round-robin fashion. In the past, this has not been a problem, because these default routers are actually just routers.

However now, I am creating a set of zonehosts that will be straddling a firewall. And like any good firewall, they will drop packets that are coming in on the "wrong" interface. So here's what I had to do to make this work:

Here's the config for this example:
ce0 (192.168.1.10/24) -> fw interface 192.168.1.1
ce1 (192.168.2.10/24) -> fw interface 192.168.2.1

On the global zone, edit /etc/ipf/ipf.conf to add the following rules for each interface
block out quick on ce0 to ce1:192.168.2.1 from 192.168.2.0/24 to any
block out quick on ce1 to ce0:192.168.1.1 from 192.168.1.0/24 to any

Now all the packets are put on their correct interface.

The only remaining question is "how does this deal with IPMP and link failures". That's something for this afternoon's research.

--Joe

Solution to a VMware license problem

2007-11-26T13:52:00.000-05:00

I'd been having a problem with our VirtualCenter installation: I'd removed an ESX server from the inventory, then tried to add it back. This operation would fail with an error message of "There are not enough licenses to perform the operation", and an event would show up reading "Not enough CPU licenses".

Now, we have plenty of VC agent licenses, (especially since I'd just removed that same server from the inventory) so I opened a call with VMware. After making the mistake of calling it a license problem (which bounced me to a FlexLM-only support group that couldn't bounce me back -- but they did validate the license file I was using, and verify that yes, we are really licensed) I was able to talk to a moderately useful representative.

We walked through the log collection process, gathered a bunch of data, discovered a corrupt VM in the inventory (removed it), gathered more logs, and I went home for the day. The next morning, the ESX server added with no problems. So I closed the case.

Now, over the weekend, we had one of our ESX servers die. I got paged and was told (third-hand: the user reported to ops who reported to another SE who told me) that something was wrong with $otherserver. Oh well, I logged in and could tell what they were complaining about -- $server was unresponsive. Unfortunately, I hadn't turned on HA on that cluster, so it didn't fix itself automatically, and I wasn't able to migrate the VMs to the other host (the VMs are on shared disk) So I deleted $server, added the VMs to the inventory via $otherserver, booted the VMs, and went on with my thanksgiving.

Today, when I booted $server (power was off, and I didn't have the DRAC configured, also the KVM was unplugged -- I think this was the original problem) and tried to add it back to the inventory, *POOF* same "There are not enough licenses to perform the operation". So do I open another mostly-useless support call? No! I'll fix it myself this time.

`strings -10 /usr/lib/vmware/vpx/vpxa | grep / | more` eventually found the config file /etc/vmware/vpxa.cfg. I "service vmware-vpxa stop" and then mv'd that config file to a backup, and added the server back in.

And the fscking thing worked. Grr. The newly-created vpxa.cfg file is exactly identical to the old one too.

--Joe

Learn something new

2007-10-04T11:45:00.000-04:00

Every once in a while I pick up a new trick... Here's solaris's answer to "what if a file has weird whitespace in its name". GNU userland has "find -print0" and xargs0, since you can't have a \0 as part of the filename, so it's safe to use as a delimiter.

find [ ... ] -exec cmd {} +

I had to use this when I was searching and cataloging (and checksumming) files from various previous hard drives. I had transferred all the data over to a ZFS pool (with compression) from a couple of Windows installs, and needed a good way to walk through "/tank/hdc/Documents and Settings/" and "/tank/hdc/Program Files" nicely. And out (of google) pops something that I've missed for years.

Not that I would have really expected to look for this gem in the man page, since I already knew how to use find. But there it was. I guess it's a documented interface now.

--Joe

(P.S. with multibyte characters in file names, is it really safe to assume that \0 will no longer occur?)

wishful feature: zfs splitfs

2007-09-18T10:08:00.000-04:00

Example:
I have a tank/data filesystem, with my important "stuff" in it, including /tank/data/oracle and /tank/data/webcontent. This is a production system, so I can't shut down to move the data around. I need to quota off the web content so it doesn't run Oracle out of space.

So what I'd like to do is...
zfs splitfs tank/data/webcontent
zfs set quota=5g tank/data/webcontent

Conceptually, it seems simple enough. Just create the appropriate new zfs filesystem entries in the pool with its root inode pointing at an existing directory. No data copying necessary.

Unfortunately, I think it would not work because there may be open files on the new (-ly partitioned) filesystem, so the (fsid,inode) pair on those open files would have to be changed to be (newid,inode) on all processes. Atomically. As part of the update to the zpool metadata. Or else the kernel would have to be able to realize that the same inode is referenced by two different filesystems. :(

--Joe

Installing SunCluster

2007-09-17T10:01:00.001-04:00

We've bought into Sun Cluster (AKA Java Availability Suite), and it's my job now to install it. I have 3x SunFire V490s, 32G memory each, 4 physical CPUs (8 cores), each has a quad-gig card (plus its 2 onboard nics) and a dual-port SAN card.

Solaris 10 update 4 (08/07). Sun Cluster 3.2.

Notes on the installation:

So far, it's been pretty straightforward except when I was trying to create the cluster. When it rebooted the first node, it never noticed that the node had rebooted in cluster mode.

This is because the rpc/bind (portmap) service is set to only allow local connections out of the box. A quick "svccfg network/rpc/bind setprop config/local_only=false" (on all nodes) and cluster is now up and running.

Bug opened.

--Joe

Installing OpenSolaris b63 on VMware Workstation 5

2007-07-27T13:59:00.000-04:00

I've got the Big Disk Server (an x4500) and since it's going to be an iSCSI target, I have to install it for now with OpenSolaris post-build-54. b54 is where the iscsitgt code got its putback.

Anyway, I've installed b63 on that monster, but before I can get everything working, I have a week-long "vacation" to learn EMC Control Center administration. So what should I do while I'm free? I'll work on setting up the rest of the administrative nicities that I need for the BDS.

Since I already have VMware Workstation on my new laptop, I'll install a b63 box, give it a couple of virtual disks for the ZFS layer, and see what code I can crank out while I'm gone.

It's never that easy, though.

Bug1: recent builds of OpenSolaris (specifically the mpt driver) cause VMware to crash. So it's IDE disks for me.
Bug2: the default X config makes the screen resolution bigger than my laptop's LCD, so I have to scroll through. Since I prefer a text-based install, I'd rather turn off X entirely.
Bug3: There's almost no documentation on how to get it to do that. There's the old "nowin" command line option (still in this version according to the docs) but I can't figure out how to pass that to grub. And the menu I'm given has 3 options (Install, Add drivers, or Shell) rather than the 7 the documentation shows.
Bug4: I don't want to install the whole distribution. The damn thing beeps if I haven't selected things correctly. Even though I have my laptop muted. And the VMware audio disconnected. And a headphone plug in the jack. How the fsck is it getting the beep through?

Bug3's workaround is to use the "Solaris Express" menu option in grub, rather than "Solaris Express Developer Edition". Grr.

--Joe

NDMP tape restores

2007-06-16T11:06:00.000-04:00

(Background: last week, "they" decreed that users should clear up unused disk space. Being a technology company, at least one user decided to write a script to clean up all his unused files, and ran this script on /net, or something. Anyway, there are now three important areas of the file shares that have no content any more. It's kinda interesting to note that all three of these areas had "test" as a component of the directory path.

The environment: Netapp filers, no snapshots of this space, monthly full backup controlled by EMC Networker via NDMP to TAN-attached SDLT600s

In the past, these sorts of problems would be handled by either Ops or my group, depending on the year, and where the (give us something to do) vs. (do it right) pendulum is swinging. Currently, it'a pointing at Ops. Except that their documentation is incomplete, so I have to get involved throughout, but on the bright side, they'll watch the tapes spin overnight. Assuming the restores go well.

Naturally, the restores aren't going well, otherwise I wouldn't be blogging about them.

My test restores (grab 1 file off tape) worked. The first restore worked using the nwrecover GUI. It was able to pull 200GB off tape and put it back onto the "autotest" share in about 24 hours.

$COWORKER's test restores (grab a couple of files off tape) didn't. They failed with an error of "NDMP Service Log: Only one path for each src and dst can be specified." Restore #2 (2GB of web content) broke with the same error message. Restore #3 (1MB of user scripts) failed also.

Well, ok, the error message reads like Networker's putting something wierd in the NDMP protocol.

A dig-in with Ethereal should help, and maybe I can figure out what inputs it needs to get the right outputs. Or not. Ethereal has some understanding of NDMP, but doesn't seem to be willing to splice back together the multi-packet NDMP requests and display them in a way that makes sense to me. Oh well.

But I know that NDMP is simply a Command & Control (C&C) protocol, the actual on-tape format is whatever native format the data server uses. In the case of Netapp, it's compatible with ufsdump on solaris. All I have to do is position the tape to the correct file, and pull the data over onto my Solaris backup server. Since I have shared tape drives, this'll be easy.

mminfo -q ssid=123456789 -r volume,mediafile,mediarec
sjimm 0.100.0 slot 16 drive 1
mt -f /dev/rmt/ciqsjb01_1 fsf 3
ufsrestore -ivfb /dev/rmt/ciqsjb01_1 60

browse to the data, and restore it. This works well for restore #2. The data comes back to the backup server, and it's ready to be copied off.

The backup for restore #3, on the other hand, spans 2 tapes. This makes things much more complicated. As I mentioned, NDMP is purely C&C. When a tape gets full, NDMP simply pauses the writes until the NDMP tape server has the next one ready, then resumes the writes. There's no feedback to the dump process that the tape has been changed, so dump considers it to be a single-volume backup. And inbetween the "unload the tape" and the "next tape is ready" steps, Networker naturally puts "load the next tape" (makes sense) and "write Networker label on the tape" (which adds file marks to the tape, which I have to skip before passing the next block to ufsrestore)

So how do I fake out ufsrestore to use the 6th file on tape 1, then when that runs out of data (rather than abort with an i/o error) wait until I load the next tape, then seek forward 3 files, and continue reading? Something like "(dd if=/dev/rmt/thefirsttape ; dd if=/dev/rmt/thesecondtape) | ufsrestore -ivfb - 60" should work, except that I can't tie up both tape drives for that long, and I don't trust Ops not to break things. I need it to switch tapes in the drive.

But this doesn't work, and I don't know why. mt gets an i/o error on the 2nd tape.

( # Have the first tape positioned correctly
dd if=/dev/rmt/ciqsjb01_1 bs=61440
mt -f /dev/rmt/ciqsjb01_1 offl
sjimm 0.100.0 drive 1 slot 22
sjimm 0.100.0 slot 23 drive 1
mt -f /dev/rmt/ciqsjb01_1 fsf 3
dd if=/dev/rmt/ciqsjb01_1 bs=61440 ) | ufsrestore -ivfb - 60

This should work, right?

In the end, I've opened a call with EMC. This is apparently a bug between Networker <7.2.2 and OnTap version >=7.2.2, and it's fixed in the latest version of Networker. But in the mean time, a full-saveset recover will work, and I have that running now.

--Joe

Fun's over!!!

2007-05-29T17:18:00.001-04:00

Looks like my fun with filesystems is over... http://sunsolve.sun.com/search/document.do?assetkey=1-26-102899-1 describes the same sort of output as http://enterprise-sa.blogspot.com/2006/11/fun-with-filesystems.html

Oh well.

--Joe

The Good-Enough trap

2007-04-28T20:32:00.000-04:00

There's a danger out there... It grows in little alcoves and cubicles, where a Group needs a piece of software to fill a particular need.

Smart People create The Solution. Sometimes there is an explicit description of the Requirements, which does not go beyond a handful of users. With or without Requirements, the Smart People select and begin work on The Platform. This selection is based on many criteria: how familiar it is to them (or the learning curve), wanting to add "$Platform programming" to the resume, I just read an article on how $Platform makes $task easy, etc. Usually not on the list is "ability to scale" or "backup support" or "algorithmic efficiency" or "we have supported hardware to run it on" or "plays nicely with other applications"

But The Solution is created and used by the Group. And it works. And the Group is more productive because of it. So naturally, since other groups want to be more productive, they want to be Users of The Solution too. And as the Users grow in number and timezone-diversity, the limitations of The Platform become more apparent. At least to those of us on the back end. Faults, Inefficiencies, Downtimes, Management headaches, these are usually hidden from the Users (or at least aren't visible enough often enough to generate real complaints) Eventually, Leadership recognizes the value of The Solution (or at least, they recognize the value of the increased productivity), and The Solution becomes an integral part of the business.

At this point, we are put into a difficult position. Limitations of The Platform, unscalable User management tools, hardware choices, etc. mean that The Solution needs to be upgraded, improved, or otherwise replaced. But naturally, work on The Solution really doesn't fall into the area of expertise of the Group any more (or at best, fixing The Solution doesn't generate billable hours), and the Smart People who developed it in the first place have either left the company, or are too busy to reimplement it. So it comes down to IT having to choose between: 1) Support and Maintain the Unmaintainable, or 2) Replace it with "the IT way", and deal with the costs of development as well as retraining the Users (and the political cost of insulting the Smart People's Solution) Usually we're stuck with (1).

--Joe

Ask the right question, and you have the answer

2007-03-20T11:46:00.000-04:00

I really hate when I'm sending an email to ask a highly-technical question, and in the process of formulating the question find the answer.

We're looking for the Next Big Disk Array to replace the Previous Big Disk Array, which has lately been showing its age in the performance arena. This is the Big Disk Array that we use as a Networker adv_file device, where we write the Big Database backups.

There's lots of people who sell BDAs. I can pretty much characterise the product options as:

Proprietary (usually web-based) interface that doesn't integrate with any other management tool (that's another rant to be ranted someday)
Proprietary ASIC on a controller board (possibly redundant Active/Active, or Active/Passive)
Some number of 1,2, or 4Gb Fibre and/or 1G iSCSI ports
Cache memory, usually up to 2GB
as many disks as will fit in that number of rack units

And it takes a 3-page PDF to marketspeak that. Anyway, from a performance standpoint, the only two numbers ever referenced are the uplink speed (!look! we have 4Gb fibre) and maximum throughput (which is never explicitly defined).

Max throughput, I generally assume, means "read whatever the optimal block size is out of cache, and imagine that the whole array is that fast" (cf/ peak transfer rates from consumer disk drives). Unless the unit supports expansion units, in which case it's "get as many expansion units as we can install, stripe a single disk group across all of them, and then report the aggregate throughput from that"

Neither is particularly helpful for me to figure out if we can write "database_backup.tar" onto the array fast enough. But I digress.

The question I was trying to ask is:

Where does it make sense to perform I/O reordering, redundancy, and cacheing:

On the array's controller card (which is a custom ASIC with 2GB of cache) -or-
In the Solaris I/O stack (including ZFS) on a server with 8GB of RAM and knowledge of the application's I/O pattern and years of performance optimization

In addition, this is not an exclusive-or: the Solaris layer is still going to be optimizing its I/O pattern, possibly with wrong assumptions about the performance and parallelism of the LUN. Or even worse, our PBDA couldn't act as a single big LUN, so the solaris layer is queueing 3 I/Os in parallel to what it thinks are 3 differnet disks, but in fact must be serialized by the controller with a long seek in between. This is clearly not optimal.

(Which reminds me... the custom ASIC has virtually no ability to actually measure or tune any performance of the system. There is no concept of exposing performance or profiling data, and there's no way to determine that these seeks are really causing the slowness. On the solaris side, OTOH, there' s things like seeksize.d that can help figure out why the fscking thing is so slow)

Just framing the question has taken me from 60/40 in favor of JBOD to about 95% in favor of it.

Fighting Rogue

2007-03-12T09:56:00.000-04:00

I'm about >< close to going "rogue sysadmin".

What does that mean? I'm very close to just saying "screw it" when it comes to any sort of collaborative decision making about technlology, and I'm just going to implement what I think is best. That's what makes me a "senior technical lead", right? That I know best? Or that I can at least make a decision of what's best for my team without having to get buy-in from a half-dozen other managers whose groups have wildly different and conflicting goals?

Why? I'm very angry about several projects that have been stalled waiting for other groups to buy-in on a framework that will solve everyone's problems. The missing functionality (monitoring) has very much come to the front over the past 30 days or so, since the big SAP upgrade. And especially with Saturday's DST patching.

--Joe

One helluva worklist

2007-01-09T09:12:00.000-05:00

This is my first work-tuesday since november. (last tuesday counts as a work-monday since monday was a holiday) Last night, I realized I had forgotten how draining this work stuff is. I'm glad I have a supportive wife at home.

Met with $PHB yesterday from 3:00-4:50 for an hour-long meeting. Discussed what's happened since I was last in the office...

The company is going with ScaleFarce for CRM. This was a surprising turnaround given the (admittedly fourth-hand) account of the negative reception of this deal the first time. And in order to make it easy to integrate "their" systems with "ours", we're going to implement MS BizTalk as a inter-middleware layer.

Did I mention we have no experience (as a company) with MS BizTalk? And that my group is expected to deliver a production-quality landscape (including DEV and TEST systems) by early march?

Also, $COWORKER{"leftcoast"} is moving up the coast a couple of hours to work in a sales office (rather than in the building with the datacenter) And since he wants to be on a management track (he even has an MBA), he's going to be given lots of responsibility. $PHB wants to virtually split his org between "Systems Infrastructure" and "Application Infrastructure" (but since there's not enough people, everybody gets an SI hat and an AI hat)

He wants me to (long-term) sit as technical advisor/architect atop both teams. I'm probably going to be saddled with (short-term) technical team lead responsibilities over the SI hats. I think I agree with the long-term plan, but I'm not so sure about being a "team lead". I definitely don't want the administrative headache of being a real manager (with a box on the org chart), like salary, reviews, budgeting, HR issues, etc. not to mention the endless "IT Manager Meeting"s

After discussing history, we went on to list the team's major projects (as we see them) for 2006. The list filled the whiteboard. It's going to be an interesting year.

Catching up on sudokus:
1/2:
132548769
478296153
695173842
913425678
784619235
256837491
567981324
321754986
849362517
1/3:
354629718
621857394
879314526
186732945
297485163
435961872
713546289
948273651
562198437

--Joe

TOIL

2007-01-08T09:34:00.000-05:00

Time Off In Lieu (although in this case, it's more appropriately In Labor)

I'm back to work today for what feels like the first time since before thanksgiving. That makes today one hell of a monday.

I was supposed to be back last tuesday, but my son went to the hospital that night, so I was out the rest of the week (he's fine, it was a virus that caused a high fever, but no other impact)

But that doesn't mean nothing's been happening around here since then. There was a flurry of activity the second weekend in December when I migrated the production SAP database to a new symmetrix, with a new LUN layout.

Then even more fun when I was trying to reformat parts of that new symmetrix to support our BCVs, which caused the production database to lose I/Os. That was real fun, let me tell you. (maybe in another post)

We had an unusually quiet (for me) end-of-year, $PHB sent out the holiday on-call schedule, and I was not on it. (I still got called once during my vacation, but only once)

Last week, I managed to copy the production SAP database over to the sandbox server without missing too much time at home (it was the afternoon we got back from the hospital, and I was able to get it done while he was sleeping), so the SAP team can run through another "trial" upgrade in prep for our upgrade in Feb.

Otherwise, it's been very quiet.

Oh yeah, and the networking team has moved out of my row of cubicles, over to the other side of the basement. So no more shouting over the cube wall "teh Intarweb's broke".

I'm already a week behind in my new year's resolutions:
1. 1400x1050
2. blog the "sudoku of the day"
3. Clean up my cubicle
4. Get my home computer working right (get it to stop locking up when I have the USB wireless adapter connected)
5. Install the copy of Adobe Premiere Elements SWMBO got me for christmas and learn to use it (by finishing the video of my cousin's wedding, and the kids videos)

Also on my todo list... need to set up the virtualization lab at the end of the hall. I've got 2 shiny new Dell 490s with dual dual-core (and I think HT and VT) "workstations" that I need to get working :)

So the sudoku for 1/1 reads:
162758934
784936215
539421867
295164378
613875429
847392651
378619542
956247183
421583796

--Joe

Legacy and transition

2006-11-21T22:48:00.000-05:00

I hate trying to transition work to other people. I'm at the hopital right now helping SWMBO have our 2nd baby. So I'll be off for a while.

So I'm leaving unfinished several projects... the SAP upgrade sandbox systems, the BEA monitoring project, the Oracle installation & monitoring project, the whole EMC upgrade, the cluster implementation, as well as supporting the treasury project, the hyperion upgrade, the webfocus upgrade... not to mention the usual stuff. Much of it is in the critical path for our big SAP upgrade (4.5 to 6.0) in February.

And I guess I'm just not comfortable that I can successfully hand these projects over to the rest of my team.

Previously, I have interpreted this as a lack of communication on my part-- I haven't taken the time over the past 2 months (not like this wasn't a planned leave) to make sure that the rest of the team has the knowledge to keep these projects moving. Now, I'm not so sure that I could have done anything differently.

The members of the team that are skilled to take up any of these projects are vastly overcommitted (not all of these projects are just mine -- I just advise and consult on some of them) and I don't think I can help the remaining team learn what they would need to learn in order to make meaningful contributions to these projects (for example, they're windows administrators, and this is a solaris problem... it doesn't help if I basically use them as a speech-to-commandline interpreter)

Trouble is, I'm the most technically-skilled unix guy on the team, so I get in the critical path of so many projects. But am I realistically supposed to be able to transfer knowledge about ongoing problems where I'm also new to them?

Oh well, this post took a long time to come out, and lots of stuff has happened since then. The question still remains, though: How am I supposed to get everything done, including training a backup, when the whole team (me and all potential backups) are overcommitted?

--Joe

Fun with Filesystems

2006-11-16T10:04:00.000-05:00

I think there's a race condition in Solaris... we had a filesystem get full with Oracle archivelogs, so I removed them, then checked to see what effect that had:


# rm D*_60[012345]?_*.dbf
# df -h .
Filesystem             size   used  avail capacity  Mounted on
/oracle/D01/saparch    5.9G 16384E   6.4G 301049486643838% /oracle/D01/saparch

A moment later, it was happy:


# df -h .
Filesystem             size   used  avail capacity  Mounted on
/oracle/D01/saparch    5.9G   257M   5.6G     5%    /oracle/D01/saparch

This is not the first time I've noticed some wierdness with removing data on S10. Last time, I wiped out a copy of our big oracle database, (rm -rf sapdata*/*) which only took a few seconds, but to unmount the filesystem took over 8 hours.

--Joe

Write something

2006-10-17T10:07:00.000-04:00

It's been over a month since I last posted. It's not like I haven't been dealing with lots of enterprise-SA type material, just that I've been too busy to even breathe, much less distill my thoughts into something for this site. But since I'm sick right now, I sorta have a little bit of time on my hands...

Some of the recent topics that are worth discussing (probably in their own posts, or several posts)...

Thoughts from the monitoring meeting (discussions about what we need for enterprise monitoring, but not all related to monitoring): false buy vs. build dichotomy; fundamental architectural difference between BB-style and SNMP trap... (no explicit "OK" status) ; Industry combination of Monitoring tools with Management tools; the myth of Agentless monitoring; SNMP support on Windows (SNMP Informant)
An Infrastructures.org mailing list post Message-ID: <20060818174228.B26037@so.lanier.com>
The usefulness of professional services and consultancy in enterprise application deployment: experiences with CA, EMC, and Hyperion
Why the hell can't I keep my desk clean?
I miss going to conferences: VMworld is on now, LISA is in December. I'm expecting a new baby about halfway between, and there's no way I can go out of town for a week.
I hate being sick. Daytime TV sucks even with satellite and a DVR. If I'd known I was going to be sick this long, I should have joined NetFlix.

Not a bad topic list... now, discuss amongst myself.

--Joe

The Ultimate P2V

2006-08-25T11:15:00.000-04:00

There's been a lot of talk about the "Blue Pill" trick where a hypothetical virus would use the new x86 virtualization features (VT or pacifica) to move a running OS under a hypervisor (where the virus would run undetectably) It would be very interesting to extend this into a positive technology...

Imagine a program that uses Blue Pill to move the OS under a hypervisor. That's fine, but the OS is still coupled to the physical devices (network cards, disks, etc). Now have the hypervisor generate a virtual (hotplug) PCI bus and attach it to the running OS. And have it hotplug a vmnic and an emulated scsi controller. The OS notices the new redundant paths to the disks (standard multipathing software) and fails over all the network connections onto the virtual card. Then the hypervisor virtually unplugs the real PCI bus, and we're left with a completely virtualized (i.e. VMotion-able) machine. Without a downtime.

That would be really cool.

This would require:

A bluepill-compatible hypervisor that can create virtual hotplug PCI buses, and that can transport running VMs across physical machines
An OS that supports PCI hotplug, dynamic disk multipathing, and transparent network failover
All the disks on the physical system being on a SAN or otherwise multihosted

--Joe

DamnDamnDamnDamn

2006-07-18T16:35:00.000-04:00

The hard drive in my work laptop is in the process of dying. That is to say, it has died (bluescreen: kernel inpage error) but has occasionally spun up enough to boot Windows.

Just long enough for the backup software to load and start a backup, not long enough for the backup to finish.

On the bright side, Support has sent me a new drive, and it's an 80GB: a 20GB upgrade from what I had. So I should have enough space now for some of the virtual machines I've been meaning to create.

Unfortunately, I still haven't finished installing my software on the new image (so far going on 4 hours of work). The only reason I have email is because OWA actually works through Firefox on Linux. Whoda thunk?

--Joe

Enterprise Monitoring

2006-06-22T14:09:00.000-04:00

In the grand quest for the "One True Ring^W^W^WSilver Bullet^W^WIntegrated Solution", this week's goal is to reduce the number of tools we're using for enterprise monitoring.

Currently we have 5 major players:

Microsoft Operations Manager
Big Brother
MRTG
Cesura
Custom-written "check" scripts

MOM has "Management support" (and, therefore $$$), Big Brother has a rich history of success (and is free), MRTG is tightly integrated with the way Networking does their stuff, Cesura has gone out of business (but they had some really cool demo technology), and of course, nobody really knows what those scripts do.

On the bright side, there's this Hobbit project I've been following for a while, which looks like a better Big Brother than BB...

On the really bright side, I've not been tasked with getting all this crap together.

I just get called on to get it working because $COWORKER[0] doesn't know Solaris at all (production enterprise is Solaris) but he's the MOM wizard, and $COWORKER[1] needs to learn more about our environment (relative new guy), and needs some visibility in the larger organization. I just happen to be the only expert in the monitoring world, just like everywhere else.

So because there's money for MOM, we're looking to see if there's any way to get non-Windows platforms to work with this Microsoft solution. As it happens, there are several third-party addons (management pack extensions) that purport to "monitor" non-Windows clients. Also the Windows guys love MOM because it has links to MSKB articles about how to tune Exchange servers when there's a low memory alert, for example.

The first extension we tried (from eXc Software) sucked. Not so much that the eXc software sucked, but OOTB, it monitors 4 items: Total CPU usage (alert if CPU is >10% busy), CPU usage by process (alert if a single process is eating more than 10% of the CPU), disk free space, and swap space usage. And that's it. Anything more and we have to write our own JScript (or VB) test that runs on the MOM server, leverages their "clientless" (aka telnet) interface to gather status on the server, and the parses the output to create a MOM/WMI event. And then maintain that code. Not exactly what we had in mind.

But eXc also has an SNMP extension agent to monitor Solaris via SNMP, so we'll try that too. A few clickety-clicks later, I've configured the basic SNMP service that's installed with the community names and it's running on our test box. Except that the software is exclusively trap-driven. And the Solaris side doesn't have any (readily apparent) way to throw the traps. Basically the eXc stack is just the Solaris trap MIBs pre-configured.

Well, if we're going down the SNMP route, let's see what MOM can do on its own. After all, it says it can monitor via SNMP. One KB article later, (and lowering the monitoring standards significantly) and I have our linux-based Digi CM console server happily SNMP-trapping into MOM. Ok, it was a lot more than just the KB article, there was also some registry editing, MIB compiling, MIB editing so it would be acceptable to the MS SMI compiler, interpreting the help page for the MS SMI compiler, some minor VB scripting and finally, turning a checkbox on on the Digi. And we still only have SNMP traps. No queries, no performance trending, no performance alerts. Also, no MIB translation (so you have to be able to recognize that 1.3.6.1.4.1.332.10.14.14.0.2 means "authentication failure", which I'm sure we'll get good at in no time at all)

So back to the drawing board... there's 2 other extension packs for MOM that we're going to try out... one from here in Cincinnati (version 1.0 was released last week) and one that appears to be a whole management infrastructure that surrounds and integrates MOM (and happens to do non-Windows clients too)

The really unfortunate thing is (as I mentioned above) there's this Hobbit project, which would leverage our existing Big Brother clients and successes, and looks like it would be fairly straightforward to implement and has a reasonably sane, extensible architecture (but it isn't MOM -- the Windows guys really like MOM)

So I ask myself "what would it take to make Hobbit work with MOM?" (at least as well as the SNMP integration or the other products did)

Hobbit's backend consists of passing messages along "channels". Messages such as "serverX is down" and channels such as "status" or "page" (or "data"), passed via IPC to worker modules. It should just be a Small Matter Of Programming to create a worker module that would accept "stachg" (status change) and/or "data" channels, massage them into something like WBEM events, and toss them across to the WMI receiver on the MOM server. I mean heck, if VB can massage SNMP traps into WBEM, surely it can't be that hard. There's even sample channels in the hobbit distribution.

I think it'd take a couple of days of programming (and learning how MOM is different than Microsoft's WMI is different than WBEM). Unfortunately I'm the only one in the group who can code. And with everything else that's going on, the chances of me taking a couple of days is exceptionally slim.

Oh well, maybe somebody else will read this and think it's a cool, easy idea.

--Joe

Need to build a secure (public) download site

2006-06-02T10:06:00.000-04:00

I have a fairly simple task in front of me: Provide a place for random internet users to download (via anon. ftp, http and/or https) one of a set of several 300MB files. (Oh yeah, and they have no budget for hardware)

From this, I add the "usual" Enterprise Systems requirements: It has to be

manageable
secure
reliable

Seems straightforward: We have a Solaris 10 system in the DMZ in the central datacenter, it has enough mirrored disk space (over 20GB free) and it's running an application that's "more important" than this little download site, so reliability isn't a problem. If I create a zone on this server, it will be no less manageable than any of the rest (ok, the other) of the DMZ-based virtualization servers we have deployed.

That just leaves the "secure" requirement. There's lots of "interesting" opportunities there, though...

I think ideally the zone would be a mininmally installed zone (with just enough software to make apache and ftpd work) with everything mounted read-only from the global zone, and with a helper zone (only accessible to the LAN-side) having read-write access to the space (accessed via scp), with firewall rules allowing only (anyone->dlserver:80,443, and ftp) and (lan->helper:22) Oh yeah, and with traffic shaping to prevent this from eating too much of our outbound internet feed.

The firewall rules are easy... that's someone else's problem. "They" don't do traffic shaping, however, so I get to figure out the Solaris IPQOS functionality, if I get that far.

So how do you create a minimalist zone? Answers as I find them...

--Joe

Two new annoyances in one day

2006-05-26T12:03:00.000-04:00

Microsoft OWA that we use (2003) apparently uses Microsoft Word (or at least some component of MS Office) as an ActiveX control to create email messages. I noticed this when trying to send a quick email to the team from my in-laws (who don't have Office installed), and it kept popping up the "preparing to install Microsoft Office" dialog box. I cancelled the installation (since they don't have media or a license), and it fell back to a plain textbox for the message body, but the email I was responding to wasn't quoted.
Outlook apparently requires IE in order to respond to meeting requests...

Open Internet Explorer
Go to the File menu, and choose "Work Offline"
Exit IE
Outlook will still be online, will still get email, etc.
But if you try to respond to a meeting invitation, it will say that you're "Working Offline". Even though Outlook is online.

Sheesh.

Thought for the day

2006-05-03T11:16:00.000-04:00

Give a man a fish, he'll eat for a day.
Teach a man to fish, he'll eat for a lifetime.
Convince a man to focus on his core competancies while paying you to fish for him, and you have a perpetual revenue stream.

Bad day (not Enterprise)

2006-04-30T20:00:00.000-04:00

I'm having a bad computer day.

On Friday, SWMBO's work laptop (Dell D600, just like mine) started reporting "Primary Hard disk not found". Naturally, this is the system where we've been keeping a lot of our personal data while she's been on leave, and it's the one she's been using to surf the web all day... So I'm going to try to (hopefully) recover some of the data off it.

So I went out to Microcenter, bought a USB-IDE adapter (cable, really cool) and am trying to get it working. Nope. Just a little clicking sound, and the drive doesn't spin up. Damn it.

The other problem is with the home PC upstairs... Since I don't have a network cable run there, I figured it'd work to use the wireless network. I've had a D-Link DWL-122 USB wireless (802.1g) adapter (bought it a while ago for this purpose) but when I leave it plugged in, the system locks up solid. No mouse movement, no keyboard, nothing. If I don't have the wireless adapter plugged in, the system'll be fine for days. There're some net references to USB-related lockups with AMD processors and Via chipsets, some of which may be resolved by switching out the USB controller for a PCI one.

So while I'm at Microcenter, I figured I'll get a 2-port USB card to see if this'll fix the problem (assuming that I won't be able to get SWMBO's HDD back to life) so that she can be online during the day. But just as I walked in, there's a sale on PCI 802.1g wireless cards, so I pick one of those up too. I figure if the USB thing doesn't work out, the PCI card should work.

Nope. When I installed the USB card, it was recognized fine, but the DWL-122 "Cannot start. code 10". And that's the case any time it's plugged into the new card, but it works fine with the onboard ports. After a couple of retry, refresh, reinstall, reboot, reboot, reboot, reboot, I gave up. Of course the PCI card will work...

Or maybe not. About 1 minute after the I installed the card & booted, the damn thing locked up again.

And SWMBO's HDD still won't spin up.

--Joe

Oh how I hate DLTs

2006-04-28T14:20:00.000-04:00

So we have a long history of hating DLT drives here, beginning back before I had anything to do with backups when $COSA[0] would get paged almost every night because of a jukebox failure (which boiled down to the op not closing the jukebox door) going on through 3 more generations of DLT technology (7k, 8k, and now SDLT600) It seems that DLT is keeping up with its heritage.

A previous DLT8000 library had lots of problems, which we theorized was because of dust & environmental contaminants, so we switched to Qualstar libraries. These have the best dust filters in the industry. For this particular replacement, we switched to LTO-2, and have had virtually no problems.

On the other hand, $THEY, when it came time to replace their aging DLT libraries (8k 2/20, 7k 4/48, 4k 2/28) decided that they didn't want to follow our success, but would rather venture off into the brave new world of SDLT. And since the SDLT600s were current, that's what they'd get (despite the fact that these are the only drives in the company that can read this media)

So that's the background... recently, I've had to kick this particular library at least twice per week, with other interventions required (not by me) probably more often. Sometimes the drive has the 3 blinking lights, sometimes not. Usually, the drive reports that there's no tape loaded. Sometimes the library agrees. Even more rarely, Networker agrees.

Sometimes pulling & reinstalling the drive fixes the problem. Today we had to unload the drive, reseat the drive, (watch Networker load a tape before we could stop it), try to unload the drive (it couldn't), reseat the drive, then got an error message of "logical unit communication failure". So we bounced the library, and now it's working (for now)

And of course, the web interface for this library is just barely functional, so in order to actually do anything, I have to walk to the datacenter. With $COSA[1] following.

All this so that $COSA[1] can dump 250GB of filer data off to tape before he deletes it.

Sheesh
--Joe

First Post! What's it all about anyway?

2006-04-28T12:08:00.000-04:00

I'm going to try this blog thing again... The past (couple) of times I've created a website with the intention of updating it regularly haven't worked very well, mostly because I think I don't journal well.

Also because I haven't had a particular focus for the sites, they end up as just a buncha random crap that I don't bother to update, yet for some reason I keep migrating from one site to another. Maybe this will be different.

So here's my focus for this site: "Collect my thoughts about Enterprise System Administration". What does that mean? Well, this is a space for:

Ideas about server management (configuration management)
Ideas about application infrastructure (applitecture)
The interrelation between office politics, policies, and technology
Wouldn't it be cool if...
Problem solving
And ideally, if I ever get around to actually implementing some of these great ideas, they'd be here too.

Please note that these are my opinions, which are not endorsed or sponsored by my employer. There may be information that is specific to my employer's systems and landscape, and your mileage may vary.