I've got the Big Disk Server (an x4500) and since it's going to be an iSCSI target, I have to install it for now with OpenSolaris post-build-54. b54 is where the iscsitgt code got its putback.
Anyway, I've installed b63 on that monster, but before I can get everything working, I have a week-long "vacation" to learn EMC Control Center administration. So what should I do while I'm free? I'll work on setting up the rest of the administrative nicities that I need for the BDS.
Since I already have VMware Workstation on my new laptop, I'll install a b63 box, give it a couple of virtual disks for the ZFS layer, and see what code I can crank out while I'm gone.
It's never that easy, though.
Bug1: recent builds of OpenSolaris (specifically the mpt driver) cause VMware to crash. So it's IDE disks for me.
Bug2: the default X config makes the screen resolution bigger than my laptop's LCD, so I have to scroll through. Since I prefer a text-based install, I'd rather turn off X entirely.
Bug3: There's almost no documentation on how to get it to do that. There's the old "nowin" command line option (still in this version according to the docs) but I can't figure out how to pass that to grub. And the menu I'm given has 3 options (Install, Add drivers, or Shell) rather than the 7 the documentation shows.
Bug4: I don't want to install the whole distribution. The damn thing beeps if I haven't selected things correctly. Even though I have my laptop muted. And the VMware audio disconnected. And a headphone plug in the jack. How the fsck is it getting the beep through?
Bug3's workaround is to use the "Solaris Express" menu option in grub, rather than "Solaris Express Developer Edition". Grr.
--Joe
2007/06/16
NDMP tape restores
(Background: last week, "they" decreed that users should clear up unused disk space. Being a technology company, at least one user decided to write a script to clean up all his unused files, and ran this script on /net, or something. Anyway, there are now three important areas of the file shares that have no content any more. It's kinda interesting to note that all three of these areas had "test" as a component of the directory path.
The environment: Netapp filers, no snapshots of this space, monthly full backup controlled by EMC Networker via NDMP to TAN-attached SDLT600s
In the past, these sorts of problems would be handled by either Ops or my group, depending on the year, and where the (give us something to do) vs. (do it right) pendulum is swinging. Currently, it'a pointing at Ops. Except that their documentation is incomplete, so I have to get involved throughout, but on the bright side, they'll watch the tapes spin overnight. Assuming the restores go well.
Naturally, the restores aren't going well, otherwise I wouldn't be blogging about them.
My test restores (grab 1 file off tape) worked. The first restore worked using the nwrecover GUI. It was able to pull 200GB off tape and put it back onto the "autotest" share in about 24 hours.
$COWORKER's test restores (grab a couple of files off tape) didn't. They failed with an error of "NDMP Service Log: Only one path for each src and dst can be specified." Restore #2 (2GB of web content) broke with the same error message. Restore #3 (1MB of user scripts) failed also.
Well, ok, the error message reads like Networker's putting something wierd in the NDMP protocol.
A dig-in with Ethereal should help, and maybe I can figure out what inputs it needs to get the right outputs. Or not. Ethereal has some understanding of NDMP, but doesn't seem to be willing to splice back together the multi-packet NDMP requests and display them in a way that makes sense to me. Oh well.
But I know that NDMP is simply a Command & Control (C&C) protocol, the actual on-tape format is whatever native format the data server uses. In the case of Netapp, it's compatible with ufsdump on solaris. All I have to do is position the tape to the correct file, and pull the data over onto my Solaris backup server. Since I have shared tape drives, this'll be easy.
browse to the data, and restore it. This works well for restore #2. The data comes back to the backup server, and it's ready to be copied off.
The backup for restore #3, on the other hand, spans 2 tapes. This makes things much more complicated. As I mentioned, NDMP is purely C&C. When a tape gets full, NDMP simply pauses the writes until the NDMP tape server has the next one ready, then resumes the writes. There's no feedback to the dump process that the tape has been changed, so dump considers it to be a single-volume backup. And inbetween the "unload the tape" and the "next tape is ready" steps, Networker naturally puts "load the next tape" (makes sense) and "write Networker label on the tape" (which adds file marks to the tape, which I have to skip before passing the next block to ufsrestore)
So how do I fake out ufsrestore to use the 6th file on tape 1, then when that runs out of data (rather than abort with an i/o error) wait until I load the next tape, then seek forward 3 files, and continue reading? Something like "(dd if=/dev/rmt/thefirsttape ; dd if=/dev/rmt/thesecondtape) | ufsrestore -ivfb - 60" should work, except that I can't tie up both tape drives for that long, and I don't trust Ops not to break things. I need it to switch tapes in the drive.
But this doesn't work, and I don't know why. mt gets an i/o error on the 2nd tape.
This should work, right?
In the end, I've opened a call with EMC. This is apparently a bug between Networker <7.2.2 and OnTap version >=7.2.2, and it's fixed in the latest version of Networker. But in the mean time, a full-saveset recover will work, and I have that running now.
--Joe
The environment: Netapp filers, no snapshots of this space, monthly full backup controlled by EMC Networker via NDMP to TAN-attached SDLT600s
In the past, these sorts of problems would be handled by either Ops or my group, depending on the year, and where the (give us something to do) vs. (do it right) pendulum is swinging. Currently, it'a pointing at Ops. Except that their documentation is incomplete, so I have to get involved throughout, but on the bright side, they'll watch the tapes spin overnight. Assuming the restores go well.
Naturally, the restores aren't going well, otherwise I wouldn't be blogging about them.
My test restores (grab 1 file off tape) worked. The first restore worked using the nwrecover GUI. It was able to pull 200GB off tape and put it back onto the "autotest" share in about 24 hours.
$COWORKER's test restores (grab a couple of files off tape) didn't. They failed with an error of "NDMP Service Log: Only one path for each src and dst can be specified." Restore #2 (2GB of web content) broke with the same error message. Restore #3 (1MB of user scripts) failed also.
Well, ok, the error message reads like Networker's putting something wierd in the NDMP protocol.
A dig-in with Ethereal should help, and maybe I can figure out what inputs it needs to get the right outputs. Or not. Ethereal has some understanding of NDMP, but doesn't seem to be willing to splice back together the multi-packet NDMP requests and display them in a way that makes sense to me. Oh well.
But I know that NDMP is simply a Command & Control (C&C) protocol, the actual on-tape format is whatever native format the data server uses. In the case of Netapp, it's compatible with ufsdump on solaris. All I have to do is position the tape to the correct file, and pull the data over onto my Solaris backup server. Since I have shared tape drives, this'll be easy.
mminfo -q ssid=123456789 -r volume,mediafile,mediarec
sjimm 0.100.0 slot 16 drive 1
mt -f /dev/rmt/ciqsjb01_1 fsf 3
ufsrestore -ivfb /dev/rmt/ciqsjb01_1 60
browse to the data, and restore it. This works well for restore #2. The data comes back to the backup server, and it's ready to be copied off.
The backup for restore #3, on the other hand, spans 2 tapes. This makes things much more complicated. As I mentioned, NDMP is purely C&C. When a tape gets full, NDMP simply pauses the writes until the NDMP tape server has the next one ready, then resumes the writes. There's no feedback to the dump process that the tape has been changed, so dump considers it to be a single-volume backup. And inbetween the "unload the tape" and the "next tape is ready" steps, Networker naturally puts "load the next tape" (makes sense) and "write Networker label on the tape" (which adds file marks to the tape, which I have to skip before passing the next block to ufsrestore)
So how do I fake out ufsrestore to use the 6th file on tape 1, then when that runs out of data (rather than abort with an i/o error) wait until I load the next tape, then seek forward 3 files, and continue reading? Something like "(dd if=/dev/rmt/thefirsttape ; dd if=/dev/rmt/thesecondtape) | ufsrestore -ivfb - 60" should work, except that I can't tie up both tape drives for that long, and I don't trust Ops not to break things. I need it to switch tapes in the drive.
But this doesn't work, and I don't know why. mt gets an i/o error on the 2nd tape.
( # Have the first tape positioned correctly
dd if=/dev/rmt/ciqsjb01_1 bs=61440
mt -f /dev/rmt/ciqsjb01_1 offl
sjimm 0.100.0 drive 1 slot 22
sjimm 0.100.0 slot 23 drive 1
mt -f /dev/rmt/ciqsjb01_1 fsf 3
dd if=/dev/rmt/ciqsjb01_1 bs=61440 ) | ufsrestore -ivfb - 60
This should work, right?
In the end, I've opened a call with EMC. This is apparently a bug between Networker <7.2.2 and OnTap version >=7.2.2, and it's fixed in the latest version of Networker. But in the mean time, a full-saveset recover will work, and I have that running now.
--Joe
2007/05/29
Fun's over!!!
Looks like my fun with filesystems is over... http://sunsolve.sun.com/search/document.do?assetkey=1-26-102899-1 describes the same sort of output as http://enterprise-sa.blogspot.com/2006/11/fun-with-filesystems.html
Oh well.
--Joe
Oh well.
--Joe
2007/04/28
The Good-Enough trap
There's a danger out there... It grows in little alcoves and cubicles, where a Group needs a piece of software to fill a particular need.
Smart People create The Solution. Sometimes there is an explicit description of the Requirements, which does not go beyond a handful of users. With or without Requirements, the Smart People select and begin work on The Platform. This selection is based on many criteria: how familiar it is to them (or the learning curve), wanting to add "$Platform programming" to the resume, I just read an article on how $Platform makes $task easy, etc. Usually not on the list is "ability to scale" or "backup support" or "algorithmic efficiency" or "we have supported hardware to run it on" or "plays nicely with other applications"
But The Solution is created and used by the Group. And it works. And the Group is more productive because of it. So naturally, since other groups want to be more productive, they want to be Users of The Solution too. And as the Users grow in number and timezone-diversity, the limitations of The Platform become more apparent. At least to those of us on the back end. Faults, Inefficiencies, Downtimes, Management headaches, these are usually hidden from the Users (or at least aren't visible enough often enough to generate real complaints) Eventually, Leadership recognizes the value of The Solution (or at least, they recognize the value of the increased productivity), and The Solution becomes an integral part of the business.
At this point, we are put into a difficult position. Limitations of The Platform, unscalable User management tools, hardware choices, etc. mean that The Solution needs to be upgraded, improved, or otherwise replaced. But naturally, work on The Solution really doesn't fall into the area of expertise of the Group any more (or at best, fixing The Solution doesn't generate billable hours), and the Smart People who developed it in the first place have either left the company, or are too busy to reimplement it. So it comes down to IT having to choose between: 1) Support and Maintain the Unmaintainable, or 2) Replace it with "the IT way", and deal with the costs of development as well as retraining the Users (and the political cost of insulting the Smart People's Solution) Usually we're stuck with (1).
--Joe
Smart People create The Solution. Sometimes there is an explicit description of the Requirements, which does not go beyond a handful of users. With or without Requirements, the Smart People select and begin work on The Platform. This selection is based on many criteria: how familiar it is to them (or the learning curve), wanting to add "$Platform programming" to the resume, I just read an article on how $Platform makes $task easy, etc. Usually not on the list is "ability to scale" or "backup support" or "algorithmic efficiency" or "we have supported hardware to run it on" or "plays nicely with other applications"
But The Solution is created and used by the Group. And it works. And the Group is more productive because of it. So naturally, since other groups want to be more productive, they want to be Users of The Solution too. And as the Users grow in number and timezone-diversity, the limitations of The Platform become more apparent. At least to those of us on the back end. Faults, Inefficiencies, Downtimes, Management headaches, these are usually hidden from the Users (or at least aren't visible enough often enough to generate real complaints) Eventually, Leadership recognizes the value of The Solution (or at least, they recognize the value of the increased productivity), and The Solution becomes an integral part of the business.
At this point, we are put into a difficult position. Limitations of The Platform, unscalable User management tools, hardware choices, etc. mean that The Solution needs to be upgraded, improved, or otherwise replaced. But naturally, work on The Solution really doesn't fall into the area of expertise of the Group any more (or at best, fixing The Solution doesn't generate billable hours), and the Smart People who developed it in the first place have either left the company, or are too busy to reimplement it. So it comes down to IT having to choose between: 1) Support and Maintain the Unmaintainable, or 2) Replace it with "the IT way", and deal with the costs of development as well as retraining the Users (and the political cost of insulting the Smart People's Solution) Usually we're stuck with (1).
--Joe
2007/03/20
Ask the right question, and you have the answer
I really hate when I'm sending an email to ask a highly-technical question, and in the process of formulating the question find the answer.
We're looking for the Next Big Disk Array to replace the Previous Big Disk Array, which has lately been showing its age in the performance arena. This is the Big Disk Array that we use as a Networker adv_file device, where we write the Big Database backups.
There's lots of people who sell BDAs. I can pretty much characterise the product options as:
Max throughput, I generally assume, means "read whatever the optimal block size is out of cache, and imagine that the whole array is that fast" (cf/ peak transfer rates from consumer disk drives). Unless the unit supports expansion units, in which case it's "get as many expansion units as we can install, stripe a single disk group across all of them, and then report the aggregate throughput from that"
Neither is particularly helpful for me to figure out if we can write "database_backup.tar" onto the array fast enough. But I digress.
The question I was trying to ask is:
Where does it make sense to perform I/O reordering, redundancy, and cacheing:
(Which reminds me... the custom ASIC has virtually no ability to actually measure or tune any performance of the system. There is no concept of exposing performance or profiling data, and there's no way to determine that these seeks are really causing the slowness. On the solaris side, OTOH, there' s things like seeksize.d that can help figure out why the fscking thing is so slow)
Just framing the question has taken me from 60/40 in favor of JBOD to about 95% in favor of it.
We're looking for the Next Big Disk Array to replace the Previous Big Disk Array, which has lately been showing its age in the performance arena. This is the Big Disk Array that we use as a Networker adv_file device, where we write the Big Database backups.
There's lots of people who sell BDAs. I can pretty much characterise the product options as:
- Proprietary (usually web-based) interface that doesn't integrate with any other management tool (that's another rant to be ranted someday)
- Proprietary ASIC on a controller board (possibly redundant Active/Active, or Active/Passive)
- Some number of 1,2, or 4Gb Fibre and/or 1G iSCSI ports
- Cache memory, usually up to 2GB
- as many disks as will fit in that number of rack units
Max throughput, I generally assume, means "read whatever the optimal block size is out of cache, and imagine that the whole array is that fast" (cf/ peak transfer rates from consumer disk drives). Unless the unit supports expansion units, in which case it's "get as many expansion units as we can install, stripe a single disk group across all of them, and then report the aggregate throughput from that"
Neither is particularly helpful for me to figure out if we can write "database_backup.tar" onto the array fast enough. But I digress.
The question I was trying to ask is:
Where does it make sense to perform I/O reordering, redundancy, and cacheing:
- On the array's controller card (which is a custom ASIC with 2GB of cache) -or-
- In the Solaris I/O stack (including ZFS) on a server with 8GB of RAM and knowledge of the application's I/O pattern and years of performance optimization
(Which reminds me... the custom ASIC has virtually no ability to actually measure or tune any performance of the system. There is no concept of exposing performance or profiling data, and there's no way to determine that these seeks are really causing the slowness. On the solaris side, OTOH, there' s things like seeksize.d that can help figure out why the fscking thing is so slow)
Just framing the question has taken me from 60/40 in favor of JBOD to about 95% in favor of it.
Subscribe to:
Posts (Atom)