Playing with Emacs
I am playing around with Emacs, and this file seemed as good a candidate for some fun as any other file.
It is nice to see that things continue to work well.
Just Goofing Around
It has obviously been a long time since I have written anything here. I suppose that shouldn't surprise anyone that knows me, seeing as how I have a perfectly good Journal. Theoretically I could turn this into a blog, but I am not particularly interested in doing any amount of typing in some interface besides Emacs.
Denial of Service
Starting on Sunday 0catch.com was hit with a massive denial of service attack which, at times, knocked the site completely off of the net. Essentially what happened is that thousands of web clients decided to load a single URL 0catch.com hosted every few seconds. At one point 0catch.com was fielding over a million hits an hour on top of the already large amount of traffic the sites already respectable traffic levels.
This resulted in slow load times for every site associated with 0catch.com, and for that I am truly sorry. On the bright side I now believe that the situation is in hand, and 0catch.com now has a pile of new tools to help guard against problems like this in the future.
Pneumonia is not my Friend
Todd got pneumonia over the weekend and so I have been on the phones. By itself that would be enough to ruin my week. I like Todd, I hate to see him sick, and I also hate being on the phones. To make matters worse new accounts at Bluefish Hosting are not getting created correctly, so I have a pile of users with brand new sites that don't work.
I am not amused.
I Hate Hardware
One of the drives went out in the new catchusers box. Of the 8 drives in the box we are currently only using two, but as you would guess the drive that failed was one of the two drives we are actually using. I switched out the drive with one of the drives that aren't being used and the mirror is currently being rebuilt.
Now I need to figure out how to return the bad drive and get a new one. Sweet.
I Nearly Killed Bambi
This morning at about 3:00 am my pager woke me up with the urgent message that our web cluster was down. It turned out that the stat box, while it still responded to pings, was not serving up files via NFS, and I could not ssh into the box.
So my Sunday morning started early with a drive into the data center to reboot the stat box. I didn't mind that much because I still had my brother's Vespa scooter, and the chance to take a late night drive seemed like fun.
What I wasn't expecting was to come within a couple of feet of hitting a deer up on the old WordPerfect campus. Luckily I slowed down when I saw the first deer in the herd cross the street in front of me. I wasn't entirely sure what it was that I had seen. The deer was on the edge of what I could see with my headlight and it was moving fast. At first I thought it was a dog, but it seemed to be moving too fast for a dog that size. Moments after slowing down several more deer burst across the street. I mised the one closest to me by just a few feet.
Once the adrenaline wore off the whole incident was pretty funny. The last thing I expected to see in the middle of Orem on a Sunday morning was an entire herd of deer. The stat box came up without problem (although I probably should start working on a failover stat box), and the site was only down for about a half hour. Cleanup took considerably longer, but the ride home was nice.
I really need to get myself a scooter.
ELI Dropped Us Off The Map
Saturday at a little after 10:00 a.m. our link to ELI dropped. Since 0catch.com is not multi-homed we disappeared until around 4:00 p.m. when the link came back up. I talked to Steve and he said he would get me the information I needed to get 0catch.com multi-homed.
Combined with the BYU loss today was a pretty disheartening day.
Where Did Jason Go?
This week has been crazy, and it made me realize that it is high time that I start writing down what has been happening again.
I spent Monday on the phones, and I have spent most of the rest of the week either battling email problems, issues with the ftp server, or space issues on catchusers5.
The good news is that I was able to resolve the problems with the ftp server by deleting keylogger accounts that had filled up their quota. These accounts would continually try to log on and the ftp server would continually have to calculate whether or not they had space left causing the load on the ftp server to spike and the service to become unavailable for actual customers. That was fixed by creating a tool that goes through the log files and finds accounts that keep trying to upload files to a full account.
The issue of disk space on catchusers5 was easier to fix, if somewhat more labor intensive. I simply spent some time moving paid customers from catchusers5 to catchusers7.
The real problem has been (and continues to be) the email server. It would appear that we are just getting more incoming spam than the one server can handle. Unfortunately, I am not entirely sure what to do about the problem.