Category Archives: Technical

Maintenance – 7/11/2012

On 7/11/2012, I'll be performing maintenance that will take xenserver2 offline for a short period of time.

Unfortunately, due to the nature of the maintenance (a reboot to clear a likely kernel panic of the dom0) it is unlikely I will be able to live migrate the virtual machines off this host first.

Affected hosts will be:
coffee.ofdoom.org
ignignokt.timminstechnologies.com

Others will be affected, but I cannot any longer access the management console to find out which ones. Nor can I SSH into it (the connection is immediately reset).

I will try to live migrate the hosts from the console, but I don't expect much in the way of useful console.

(this is of course the OTHER server – xenserver1 freaked out last week. ugh!)

How to handle a (temporary or permanent) failure of a master server in Xenserver

http://flickr.com/photos/kevincollins/74279815/
http://flickr.com/photos/kevincollins/74279815/

OK. So you found you logged in, and you see one of the servers down, but don't see the virtual machines that were on that host anywhere in the resource pool. It's as if the entire system just freaked out and the server, and the VMs running on it, disappeared without a trace.

You've verified your shared storage is okay, and thus you're stable enough to begin remediation.

Now you have a problem, but it's easy to fix. Begin by connecting to the remaining server as root with ssh.

The following commands will get you back to a known good state. Make sure your master server is good and dead (unreachable and unlikely to come back without intervention of some sort – in my case it was a kernel panic, and a reboot and fsck -y / fixed the issue, this procedure will work fine in this scenario)

Last login: Fri Mar 18 16:42:03 2011 from (hostname redacted)
Type "xsconsole" for access to the management console.
[root@xenserver2 ~]# xe pool-emergency-transition-to-master
Host agent will restart and transition to master in 10 seconds...
[root@xenserver2 ~]# xe pool-recover-slaves
[root@xenserver2 ~]#

You are now at a point where the cluster is now usefully reporting a master server, and any remaining slaves are now pointing at it.

Now for the finishing touch – marking the powered down VMs as dead, so you can restart them on other servers. Note that if you're not correct on this, really awful inconsistent things will happen. You have been warned.

[root@xenserver2 ~]# xe vm-reset-powerstate --force --multiple
operation failed on ae820fb7-8416-68ee-35c8-c37REDACTEDf18: The operation could not be performed because a domain still exists for the specified VM.
vm: ae820fb7-8416-68ee-35c8-c37REDACTEDdf18 (REDACTED)
domid: 51
operation failed on aa2bc87f-9ef4-dd8e-492e-bREDACTED4: The operation could not be performed because a domain still exists for the specified VM.
vm: aa2bc87f-9ef4-dd8e-492e-b4REDACTE14 (REDACTED)
domid: 54
operation failed on e4f76313-2be1-0d4e-1e78-78e1REDACTED75: The operation could not be performed because a domain still exists for the specified VM.
vm: e4f76313-2be1-0d4e-1e78-78e1REDACTED5 (REDACTED)
domid: 2
operation failed on 557c9030-b995-fbaa-ab0b-61REDACTED4df: The operation could not be performed because a domain still exists for the specified VM.
vm: 557c9030-b995-fbaa-ab0b-61e4abb074df (REDACTED)
domid: 21
[root@xenserver2 ~]#

This marked every unreachable VM as "powered off" and thus it can now be restarted. You'll see it mercifully decided not to hurt my VMs that were confirmed to be operational. You may want to be more cautious and use host-id=your-uuid-here rather than –multiple but that's up to you.

More information available here: http://docs.vmd.citrix.com/XenServer/4.0.1/reference/ch02s06.html

So let's say, for the sake of argument, you found a Linksys WIP-330 in your couch

OK, so anyway it's no secret we're madly, hopelessly disorganized around here.

But anyway, I have this "friend" who may or may not have lost a very expensive (at the time) 802.11g IP phone that ran windows mobile. Let's say, for the sake of argument, it was sometime around 2006.

And let's say that in 2012, he had a toddler that was jamming toys in the couch, and again in this hypothetical world, let's say that his wife was trying to clean the toys out of the couch and 6 YEARS LATER found this device. And it was still in good working order, only having been used for a few months before disappearing. But it had firmware from 2006 on it. And it was lost to the sands of time when Cisco bought Linksys. It's running one version up from the bone stock firmware, 1.00.06. Not 1.00.06A, mind you.

This is a story of that "friend", since clearly while being disorganized I'd never lose a $350 phone in my couch for 6 years, after all. That'd be embarassingly careless.

Aaaaanyway. Here's the story of how to upgrade it. You'll need an 802.11b or g access point. No, 802.11n won't work even though it should supposedly fall back to 802.11g. It won't until you get the new firmware on there, so forget it. Go to a coffeshop or something if you have to.

First, take it to version 1.00.06A (that A is important, it is crucial for the next steps. I found the web browser upgrade interface to be useful for nothing else than getting 1.00.06A on, anything but that tends to just make the web server error out and reset the connection.

This adds the ability to firmware upgrade from the handset directly. From there you need to get the wip330_sbe.bin file onto the handset. Once you have the upgrade from handset options, you're good to go. (You can get all these files, by the way, at http://psyduck.mudkips.net/~paul/wip330 and can use that url with your handset if you want, it's no skin off my back. Mind you, it's only 1.5m connection, but hey, at least you can get a copy. Linksys/Cisco don't host it anymore)

At this point you have a fancy windows ce .net machine, no sip phone, no nothing. But now, you can get to the new firmware release! YAY!

Once you have this, the on-handset upgrade can take you to wip330_v1_03_18S.bin, where you will have access to any network running open, WEP, or WPA1 (not WPA2!). If you are connecting to a WPA network and getting a prompt about WEP keys, make sure you have your authentication set to WPA/WPA2 (or WPA1/WPA2) PSK. It assumes WPA2 is WEP, no idea why. But it won't work. Mixed authentication mode will work okay, and leaves the more secure WPA2 mode available for other devices.