Fighting Rogue

I'm about >< close to going "rogue sysadmin".

What does that mean? I'm very close to just saying "screw it" when it comes to any sort of collaborative decision making about technlology, and I'm just going to implement what I think is best. That's what makes me a "senior technical lead", right? That I know best? Or that I can at least make a decision of what's best for my team without having to get buy-in from a half-dozen other managers whose groups have wildly different and conflicting goals?

Why? I'm very angry about several projects that have been stalled waiting for other groups to buy-in on a framework that will solve everyone's problems. The missing functionality (monitoring) has very much come to the front over the past 30 days or so, since the big SAP upgrade. And especially with Saturday's DST patching.

--Joe

One helluva worklist

This is my first work-tuesday since november. (last tuesday counts as a work-monday since monday was a holiday) Last night, I realized I had forgotten how draining this work stuff is. I'm glad I have a supportive wife at home.

Met with $PHB yesterday from 3:00-4:50 for an hour-long meeting. Discussed what's happened since I was last in the office...

The company is going with ScaleFarce for CRM. This was a surprising turnaround given the (admittedly fourth-hand) account of the negative reception of this deal the first time. And in order to make it easy to integrate "their" systems with "ours", we're going to implement MS BizTalk as a inter-middleware layer.

Did I mention we have no experience (as a company) with MS BizTalk? And that my group is expected to deliver a production-quality landscape (including DEV and TEST systems) by early march?

Also, $COWORKER{"leftcoast"} is moving up the coast a couple of hours to work in a sales office (rather than in the building with the datacenter) And since he wants to be on a management track (he even has an MBA), he's going to be given lots of responsibility. $PHB wants to virtually split his org between "Systems Infrastructure" and "Application Infrastructure" (but since there's not enough people, everybody gets an SI hat and an AI hat)

He wants me to (long-term) sit as technical advisor/architect atop both teams. I'm probably going to be saddled with (short-term) technical team lead responsibilities over the SI hats. I think I agree with the long-term plan, but I'm not so sure about being a "team lead". I definitely don't want the administrative headache of being a real manager (with a box on the org chart), like salary, reviews, budgeting, HR issues, etc. not to mention the endless "IT Manager Meeting"s

After discussing history, we went on to list the team's major projects (as we see them) for 2006. The list filled the whiteboard. It's going to be an interesting year.

Catching up on sudokus:
1/2:
132548769
478296153
695173842
913425678
784619235
256837491
567981324
321754986
849362517
1/3:
354629718
621857394
879314526
186732945
297485163
435961872
713546289
948273651
562198437

--Joe

TOIL

Time Off In Lieu (although in this case, it's more appropriately In Labor)

I'm back to work today for what feels like the first time since before thanksgiving. That makes today one hell of a monday.

I was supposed to be back last tuesday, but my son went to the hospital that night, so I was out the rest of the week (he's fine, it was a virus that caused a high fever, but no other impact)

But that doesn't mean nothing's been happening around here since then. There was a flurry of activity the second weekend in December when I migrated the production SAP database to a new symmetrix, with a new LUN layout.

Then even more fun when I was trying to reformat parts of that new symmetrix to support our BCVs, which caused the production database to lose I/Os. That was real fun, let me tell you. (maybe in another post)

We had an unusually quiet (for me) end-of-year, $PHB sent out the holiday on-call schedule, and I was not on it. (I still got called once during my vacation, but only once)

Last week, I managed to copy the production SAP database over to the sandbox server without missing too much time at home (it was the afternoon we got back from the hospital, and I was able to get it done while he was sleeping), so the SAP team can run through another "trial" upgrade in prep for our upgrade in Feb.

Otherwise, it's been very quiet.

Oh yeah, and the networking team has moved out of my row of cubicles, over to the other side of the basement. So no more shouting over the cube wall "teh Intarweb's broke".

I'm already a week behind in my new year's resolutions:
1. 1400x1050
2. blog the "sudoku of the day"
3. Clean up my cubicle
4. Get my home computer working right (get it to stop locking up when I have the USB wireless adapter connected)
5. Install the copy of Adobe Premiere Elements SWMBO got me for christmas and learn to use it (by finishing the video of my cousin's wedding, and the kids videos)

Also on my todo list... need to set up the virtualization lab at the end of the hall. I've got 2 shiny new Dell 490s with dual dual-core (and I think HT and VT) "workstations" that I need to get working :)

So the sudoku for 1/1 reads:
162758934
784936215
539421867
295164378
613875429
847392651
378619542
956247183
421583796

--Joe

Legacy and transition

I hate trying to transition work to other people. I'm at the hopital right now helping SWMBO have our 2nd baby. So I'll be off for a while.

So I'm leaving unfinished several projects... the SAP upgrade sandbox systems, the BEA monitoring project, the Oracle installation & monitoring project, the whole EMC upgrade, the cluster implementation, as well as supporting the treasury project, the hyperion upgrade, the webfocus upgrade... not to mention the usual stuff. Much of it is in the critical path for our big SAP upgrade (4.5 to 6.0) in February.

And I guess I'm just not comfortable that I can successfully hand these projects over to the rest of my team.

Previously, I have interpreted this as a lack of communication on my part-- I haven't taken the time over the past 2 months (not like this wasn't a planned leave) to make sure that the rest of the team has the knowledge to keep these projects moving. Now, I'm not so sure that I could have done anything differently.

The members of the team that are skilled to take up any of these projects are vastly overcommitted (not all of these projects are just mine -- I just advise and consult on some of them) and I don't think I can help the remaining team learn what they would need to learn in order to make meaningful contributions to these projects (for example, they're windows administrators, and this is a solaris problem... it doesn't help if I basically use them as a speech-to-commandline interpreter)

Trouble is, I'm the most technically-skilled unix guy on the team, so I get in the critical path of so many projects. But am I realistically supposed to be able to transfer knowledge about ongoing problems where I'm also new to them?

Oh well, this post took a long time to come out, and lots of stuff has happened since then. The question still remains, though: How am I supposed to get everything done, including training a backup, when the whole team (me and all potential backups) are overcommitted?

--Joe

Fun with Filesystems

I think there's a race condition in Solaris... we had a filesystem get full with Oracle archivelogs, so I removed them, then checked to see what effect that had:

# rm D*_60[012345]?_*.dbf
# df -h .
Filesystem             size   used  avail capacity  Mounted on
/oracle/D01/saparch    5.9G 16384E   6.4G 301049486643838% /oracle/D01/saparch

A moment later, it was happy:

# df -h .
Filesystem             size   used  avail capacity  Mounted on
/oracle/D01/saparch    5.9G   257M   5.6G     5%    /oracle/D01/saparch

This is not the first time I've noticed some wierdness with removing data on S10. Last time, I wiped out a copy of our big oracle database, (rm -rf sapdata*/*) which only took a few seconds, but to unmount the filesystem took over 8 hours.

--Joe