Yea. Fucking LJ or the client ate my last post about my spam problems.

Anyway, the statistics are as follows:
I'm using 15% CPU on a dual P2 450 doing DNS lookups for the mail relay.
I'm currently tracking 4400 connections on my edge filtering machine. I had to up its net.ipv4.ip_conntrack_max sysctl to 100,000 because the default of 16,384 was overflowing at points.

My mailserver's log is growing at the rate of 10k every 2 seconds.

in one minute I recieve approximately 1300 new connections to my SMTP port on the relay for timmins.net
That's about 22 new connections every second.

My mailserver is still accepting new connections and delivering valid mail.

Ntop claims connections from 100 pages of unique systems. It looks like it puts somewhere between 25 and 50 hosts per page.

I can't believe I'm still alive from all this on the network, but I am. Thank god.

If I ever see a spammer on the street, it'll make the 6 o' clock news. I swear it.

I know service has sucked today. It's not a bandwidth issue. I'm undergoing a gigantic joe job right now. Some spammer is generating hundreds of thousands of emails with the sender address being random addresses @timmins.net.

Since timmins.net and telcodata.us share much of the same architecture (but not quite) it is also affected.

My apologies for the shitty service, but it's not my fault. I'm doing what I can, but under the load I've already had one backup MX bail out on me (quite understandably and with my blessing) for the course of the attack, and had to switch my offsite backup MX because it was crushing that server too. Mainframe has over 30,000 backed up emails in its mail queue that are almost all invalid, just waiting to be bounced.

It's a fucking mess.

I'll summarize the rest of the day with little plus and minus signs:
+ Optimizing my monitoring system at work
– Fucking up the summary cache database on it in the progress, but rebuilding it was part of the above optimization and made a HUGE difference.
+ 100 Megabit Full Duplex connection in the detroit datacenter at work now
– I think (provider) is inaverdently forcing my redundant connection to 10/half. Grr.
+ Renewed my badge at (colo provider) today
– Got bitched at for not signing in guests. I've been taking guests for like what, 2-3 years now? and they decide to throw a bitch fit while my router is in several pieces on my lap? Fuck that noise. My guests come to help me do shit, like be my gopher or help me organize my crap.
+ K index of 9 today. Heard the Northern lights were cool. Look up at the sky and it's fucking like daylight because I'm a city boy. I hop in my truck and take off on the first road that went north I could find. Took it until I seen dark skies.
– Didn't see a damn thing
+ Got lost up in BFE, and hilarity ensues as one of my backup MXen calls me with far more patience than I would have had under the circumstances, and asks me to flip my MX record to the proper machine. My bad. I go to grab my data cable for my cell phone. It's not there. Nope, it's back 50 miles south at home. I plug in my wireless card, and remember that up in BFE, they don't use wireless ethernet. I actually manage to find an AP to find out it's not actually connected to the internet, or anything else interesting for that matter. Ugh. End up calling my other backup MX to have him edit the DNS record. More hilarity ensues as I am apparently in a non existent town.

Welcome to my day today. Hi.