Unexpected server downtime

[Jeffrey Hill]

Just curious... did anyone try to visit the board between 10:30 and 11:15 last night? If so, was the board down? I've gotten a couple emails that the server that the board is hosted on had problems and so they had to swap out hardware.

In any case, slicehost seems to be very professional about notifying their customers about downtime and getting everything back up in a timely manner. Also, this is the first time in the 13 months I've been with them that the server has had any problems to my knowledge... I know a couple of months ago I looked at the uptime and the server had been running for around 300 days straight. I'm pleased

[Charlie Dees]

Yes it was down.

[Jeffrey Hill]

Don't know what happened but it was down this morning... I ended up having to force a hard reboot..... this had better stop happening...

[Jeffrey Hill]

Yeah, it was down again earlier today... I've probably got something set up wrong that's causing this to occasionally happen...

[Jeffrey Hill]

Conveniently enough, it looks like the server stopped responding again this afternoon...... and this time a table was apparently corrupt when I got it back up and had to be repaired. Let me know if you notice anything weird on the board (or the MOQBA or MACA websites).

[Jeffrey Hill]

Woohoo, it happened again... at least this time I might have a lead on what the problem might be.

[Jeffrey Hill]

I was thinking to myself this morning before getting on my computer that it would be inconvenient for the server to go down this weekend... and it looks like the server had been down since around 3:40 this morning. I'm going to have to start manually restarting the server about once every 2-3 weeks as a workaround to keep this from happening since it looks like it's happening about once a month now... I apologize for the repeated downtime.

[Jeffrey Hill]

Server crashed sometime after 2:30 this morning. Guess I'll make it every other week on the planned reboots. My hosting plan is supposed to be going through a transition to a different pricing model soon so once that happens I'll hopefully be able to upgrade to a better plan that I think will help with this recurring problem.

[Jeffrey Hill]

REALLY... less than 36 hours?! I don't know what has changed that is suddenly causing the server to crash so frequently but I'm definitely going to have to do something about it soon.

[Jeffrey Hill]

The server has been acting weird this morning so I wouldn't be totally shocked if the site becomes unresponsive at some point today...

[Jeffrey Hill]

Site went down sometime between 2 and 3:45 pm...

[Jeffrey Hill]

OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.

[L-Town Expatriate]

OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.

How the heck did we get 608 users? Sounds like a spambot barrage.

[Jeffrey Hill]

Short version: I've made some more configuration changes to hopefully completely fix the out of memory instances that have caused the server to go down intermittently. Let me know if you notice the site getting significantly slower or anything like that.

Long, semi-technical version: Today when I got home from work the site wasn't responding, but I was able to log in remotely, which usually didn't work when the site went down in the past (although it was incredibly slow). I dug around trying to figure out what was going on, and all of a sudden the server rebooted. It turns out that the server was in such a bad state that it was constantly swapping memory to and from disk that it was adversely affecting the servers of other customers running on the same physical host server, and so the host initiated a hard reboot (which I would have done immediately had I not been able to log in remotely). Needless to say, I got a warning from my host to figure out what caused this.

I think I had the server configured to accept way too many simultaneous connections and some bot tried to make tons of requests at once. (The access logs appear to indicate this was the case, and I have blocked the offending IP.) A while back I had already reduced the maximum number of clients, but it apparently wasn't enough. I think that reduction may have actually contributed to the server going into a constant swap state rather than crashing entirely. So, I have slashed the number of simultaneous clients even further. Hopefully this won't adversely affect your experience on this board, moaca.org, and moqba.org, but if you notice that the sites are significantly slower or anything else doesn't seem to be working right, let me know.

I hope this finally fixes the problem, because if this exact situation happens again it sure sounds like it's going to be a lot more urgent for me to solve it once and for all..