Racism and misinformation

Today, while eating at one of my favorite restaurants, Idle Hour in Spring Lake, MI (I've literally been eating there since I was able to eat solid food) I found some printouts on the lazy susan on the middle of the table.

Naturally, I picked them up, and one set of them was filled with a printed email containing a racist "joke" about painting a golf caddy robot black and having it fail to show up for work, apply for welfare, or become president, or something along those lines. Someone has reproduced this "joke" here

It was prefaced with the words:



I corrected this with my sharpie to then read:



I think it's important to call a spade a spade, and let other people figure it out. You can't hide behind blatant racism with wimpy claims of "political correctness". This isn't someone saying "A black man" rather than "A man of African American decent". It's a straight up racist joke. If you think it's funny, that's fine, as long as you realize it's actually racist, not just some stupid jab in the culture wars.

I then set it back in its place. Found another email printout, this time some email forward talking about how Obama is screwing old people by putting a 3.8% sales tax on all home sales. I got the correct information from factcheck.org stating that there is such a tax, but you have to have more than $200k a year in income, and it taxed the sale in excess of $250k for single people, and $500k for married filing jointly, and only to the extent that it exceeded those values would it be taxed (a sale of a home for $501k would only have $38 in additional taxes, as long as the person had $200k in income that year, otherwise it'd still be not taxed). So I made massive corrections to that email and put it back as well.

I don't think the owners were involved, or possibly even aware. But I won't allow racism or disinformation to go unchallenged.

xenserver2 issue – power supply

The power supply in xenserver2 failed. As I don't have a spare on hand at the moment, I brought it back to life by smacking on the case.

Percussive maintenance being the temporary fix it is, I'm not putting anything critical on this box. I'm going to tee up both of these machines for a power supply replacement, and look at either getting a third machine, or increasing RAM in these two drastically.

As i write this, the filesystem fell out from under xenserver2, so I strongly suspect something far more unseemly is happening, perhaps still power supply related.

So at this point it sits cold in the datacenter for a few days until I get the resources (time/money/etc) together to get this bad boy back to health. If anyone's feeling generous, it's an ATX power supply, pretty standard, actually. It needs the standard compliment of connectors modern supplies have. 450w or better.

affected vms:
and a few test vms of mine.

Maintenance – 7/11/2012

On 7/11/2012, I'll be performing maintenance that will take xenserver2 offline for a short period of time.

Unfortunately, due to the nature of the maintenance (a reboot to clear a likely kernel panic of the dom0) it is unlikely I will be able to live migrate the virtual machines off this host first.

Affected hosts will be:

Others will be affected, but I cannot any longer access the management console to find out which ones. Nor can I SSH into it (the connection is immediately reset).

I will try to live migrate the hosts from the console, but I don't expect much in the way of useful console.

(this is of course the OTHER server – xenserver1 freaked out last week. ugh!)

How to handle a (temporary or permanent) failure of a master server in Xenserver


OK. So you found you logged in, and you see one of the servers down, but don't see the virtual machines that were on that host anywhere in the resource pool. It's as if the entire system just freaked out and the server, and the VMs running on it, disappeared without a trace.

You've verified your shared storage is okay, and thus you're stable enough to begin remediation.

Now you have a problem, but it's easy to fix. Begin by connecting to the remaining server as root with ssh.

The following commands will get you back to a known good state. Make sure your master server is good and dead (unreachable and unlikely to come back without intervention of some sort – in my case it was a kernel panic, and a reboot and fsck -y / fixed the issue, this procedure will work fine in this scenario)

Last login: Fri Mar 18 16:42:03 2011 from (hostname redacted)
Type "xsconsole" for access to the management console.
[root@xenserver2 ~]# xe pool-emergency-transition-to-master
Host agent will restart and transition to master in 10 seconds...
[root@xenserver2 ~]# xe pool-recover-slaves
[root@xenserver2 ~]#

You are now at a point where the cluster is now usefully reporting a master server, and any remaining slaves are now pointing at it.

Now for the finishing touch – marking the powered down VMs as dead, so you can restart them on other servers. Note that if you're not correct on this, really awful inconsistent things will happen. You have been warned.

[root@xenserver2 ~]# xe vm-reset-powerstate --force --multiple
operation failed on ae820fb7-8416-68ee-35c8-c37REDACTEDf18: The operation could not be performed because a domain still exists for the specified VM.
vm: ae820fb7-8416-68ee-35c8-c37REDACTEDdf18 (REDACTED)
domid: 51
operation failed on aa2bc87f-9ef4-dd8e-492e-bREDACTED4: The operation could not be performed because a domain still exists for the specified VM.
vm: aa2bc87f-9ef4-dd8e-492e-b4REDACTE14 (REDACTED)
domid: 54
operation failed on e4f76313-2be1-0d4e-1e78-78e1REDACTED75: The operation could not be performed because a domain still exists for the specified VM.
vm: e4f76313-2be1-0d4e-1e78-78e1REDACTED5 (REDACTED)
domid: 2
operation failed on 557c9030-b995-fbaa-ab0b-61REDACTED4df: The operation could not be performed because a domain still exists for the specified VM.
vm: 557c9030-b995-fbaa-ab0b-61e4abb074df (REDACTED)
domid: 21
[root@xenserver2 ~]#

This marked every unreachable VM as "powered off" and thus it can now be restarted. You'll see it mercifully decided not to hurt my VMs that were confirmed to be operational. You may want to be more cautious and use host-id=your-uuid-here rather than –multiple but that's up to you.

More information available here: http://docs.vmd.citrix.com/XenServer/4.0.1/reference/ch02s06.html