Our zonehosts have multiple NICs, on multiple subnets. This means that they have multiple default routes defined, so non-local packets are passed to those default routers in a round-robin fashion. In the past, this has not been a problem, because these default routers are actually just routers.
However now, I am creating a set of zonehosts that will be straddling a firewall. And like any good firewall, they will drop packets that are coming in on the "wrong" interface. So here's what I had to do to make this work:
Here's the config for this example:
ce0 (192.168.1.10/24) -> fw interface 192.168.1.1
ce1 (192.168.2.10/24) -> fw interface 192.168.2.1
On the global zone, edit /etc/ipf/ipf.conf to add the following rules for each interface
block out quick on ce0 to ce1:192.168.2.1 from 192.168.2.0/24 to any
block out quick on ce1 to ce0:192.168.1.1 from 192.168.1.0/24 to any
Now all the packets are put on their correct interface.
The only remaining question is "how does this deal with IPMP and link failures". That's something for this afternoon's research.
--Joe
2007/12/05
2007/11/26
Solution to a VMware license problem
I'd been having a problem with our VirtualCenter installation: I'd removed an ESX server from the inventory, then tried to add it back. This operation would fail with an error message of "There are not enough licenses to perform the operation", and an event would show up reading "Not enough CPU licenses".
Now, we have plenty of VC agent licenses, (especially since I'd just removed that same server from the inventory) so I opened a call with VMware. After making the mistake of calling it a license problem (which bounced me to a FlexLM-only support group that couldn't bounce me back -- but they did validate the license file I was using, and verify that yes, we are really licensed) I was able to talk to a moderately useful representative.
We walked through the log collection process, gathered a bunch of data, discovered a corrupt VM in the inventory (removed it), gathered more logs, and I went home for the day. The next morning, the ESX server added with no problems. So I closed the case.
Now, over the weekend, we had one of our ESX servers die. I got paged and was told (third-hand: the user reported to ops who reported to another SE who told me) that something was wrong with $otherserver. Oh well, I logged in and could tell what they were complaining about -- $server was unresponsive. Unfortunately, I hadn't turned on HA on that cluster, so it didn't fix itself automatically, and I wasn't able to migrate the VMs to the other host (the VMs are on shared disk) So I deleted $server, added the VMs to the inventory via $otherserver, booted the VMs, and went on with my thanksgiving.
Today, when I booted $server (power was off, and I didn't have the DRAC configured, also the KVM was unplugged -- I think this was the original problem) and tried to add it back to the inventory, *POOF* same "There are not enough licenses to perform the operation". So do I open another mostly-useless support call? No! I'll fix it myself this time.
`strings -10 /usr/lib/vmware/vpx/vpxa | grep / | more` eventually found the config file /etc/vmware/vpxa.cfg. I "service vmware-vpxa stop" and then mv'd that config file to a backup, and added the server back in.
And the fscking thing worked. Grr. The newly-created vpxa.cfg file is exactly identical to the old one too.
--Joe
Now, we have plenty of VC agent licenses, (especially since I'd just removed that same server from the inventory) so I opened a call with VMware. After making the mistake of calling it a license problem (which bounced me to a FlexLM-only support group that couldn't bounce me back -- but they did validate the license file I was using, and verify that yes, we are really licensed) I was able to talk to a moderately useful representative.
We walked through the log collection process, gathered a bunch of data, discovered a corrupt VM in the inventory (removed it), gathered more logs, and I went home for the day. The next morning, the ESX server added with no problems. So I closed the case.
Now, over the weekend, we had one of our ESX servers die. I got paged and was told (third-hand: the user reported to ops who reported to another SE who told me) that something was wrong with $otherserver. Oh well, I logged in and could tell what they were complaining about -- $server was unresponsive. Unfortunately, I hadn't turned on HA on that cluster, so it didn't fix itself automatically, and I wasn't able to migrate the VMs to the other host (the VMs are on shared disk) So I deleted $server, added the VMs to the inventory via $otherserver, booted the VMs, and went on with my thanksgiving.
Today, when I booted $server (power was off, and I didn't have the DRAC configured, also the KVM was unplugged -- I think this was the original problem) and tried to add it back to the inventory, *POOF* same "There are not enough licenses to perform the operation". So do I open another mostly-useless support call? No! I'll fix it myself this time.
`strings -10 /usr/lib/vmware/vpx/vpxa | grep / | more` eventually found the config file /etc/vmware/vpxa.cfg. I "service vmware-vpxa stop" and then mv'd that config file to a backup, and added the server back in.
And the fscking thing worked. Grr. The newly-created vpxa.cfg file is exactly identical to the old one too.
--Joe
2007/10/04
Learn something new
Every once in a while I pick up a new trick... Here's solaris's answer to "what if a file has weird whitespace in its name". GNU userland has "find -print0" and xargs0, since you can't have a \0 as part of the filename, so it's safe to use as a delimiter.
find [ ... ] -exec cmd {} +
I had to use this when I was searching and cataloging (and checksumming) files from various previous hard drives. I had transferred all the data over to a ZFS pool (with compression) from a couple of Windows installs, and needed a good way to walk through "/tank/hdc/Documents and Settings/" and "/tank/hdc/Program Files" nicely. And out (of google) pops something that I've missed for years.
Not that I would have really expected to look for this gem in the man page, since I already knew how to use find. But there it was. I guess it's a documented interface now.
--Joe
(P.S. with multibyte characters in file names, is it really safe to assume that \0 will no longer occur?)
find [ ... ] -exec cmd {} +
I had to use this when I was searching and cataloging (and checksumming) files from various previous hard drives. I had transferred all the data over to a ZFS pool (with compression) from a couple of Windows installs, and needed a good way to walk through "/tank/hdc/Documents and Settings/" and "/tank/hdc/Program Files" nicely. And out (of google) pops something that I've missed for years.
Not that I would have really expected to look for this gem in the man page, since I already knew how to use find. But there it was. I guess it's a documented interface now.
--Joe
(P.S. with multibyte characters in file names, is it really safe to assume that \0 will no longer occur?)
2007/09/18
wishful feature: zfs splitfs
Example:
I have a tank/data filesystem, with my important "stuff" in it, including /tank/data/oracle and /tank/data/webcontent. This is a production system, so I can't shut down to move the data around. I need to quota off the web content so it doesn't run Oracle out of space.
So what I'd like to do is...
zfs splitfs tank/data/webcontent
zfs set quota=5g tank/data/webcontent
Conceptually, it seems simple enough. Just create the appropriate new zfs filesystem entries in the pool with its root inode pointing at an existing directory. No data copying necessary.
Unfortunately, I think it would not work because there may be open files on the new (-ly partitioned) filesystem, so the (fsid,inode) pair on those open files would have to be changed to be (newid,inode) on all processes. Atomically. As part of the update to the zpool metadata. Or else the kernel would have to be able to realize that the same inode is referenced by two different filesystems. :(
--Joe
I have a tank/data filesystem, with my important "stuff" in it, including /tank/data/oracle and /tank/data/webcontent. This is a production system, so I can't shut down to move the data around. I need to quota off the web content so it doesn't run Oracle out of space.
So what I'd like to do is...
zfs splitfs tank/data/webcontent
zfs set quota=5g tank/data/webcontent
Conceptually, it seems simple enough. Just create the appropriate new zfs filesystem entries in the pool with its root inode pointing at an existing directory. No data copying necessary.
Unfortunately, I think it would not work because there may be open files on the new (-ly partitioned) filesystem, so the (fsid,inode) pair on those open files would have to be changed to be (newid,inode) on all processes. Atomically. As part of the update to the zpool metadata. Or else the kernel would have to be able to realize that the same inode is referenced by two different filesystems. :(
--Joe
2007/09/17
Installing SunCluster
We've bought into Sun Cluster (AKA Java Availability Suite), and it's my job now to install it. I have 3x SunFire V490s, 32G memory each, 4 physical CPUs (8 cores), each has a quad-gig card (plus its 2 onboard nics) and a dual-port SAN card.
Solaris 10 update 4 (08/07). Sun Cluster 3.2.
Notes on the installation:
So far, it's been pretty straightforward except when I was trying to create the cluster. When it rebooted the first node, it never noticed that the node had rebooted in cluster mode.
This is because the rpc/bind (portmap) service is set to only allow local connections out of the box. A quick "svccfg network/rpc/bind setprop config/local_only=false" (on all nodes) and cluster is now up and running.
Bug opened.
--Joe
Solaris 10 update 4 (08/07). Sun Cluster 3.2.
Notes on the installation:
So far, it's been pretty straightforward except when I was trying to create the cluster. When it rebooted the first node, it never noticed that the node had rebooted in cluster mode.
This is because the rpc/bind (portmap) service is set to only allow local connections out of the box. A quick "svccfg network/rpc/bind setprop config/local_only=false" (on all nodes) and cluster is now up and running.
Bug opened.
--Joe
Subscribe to:
Posts (Atom)