Page 1 of 1

Unexpected server downtime

Posted: Thu Dec 23, 2010 8:13 am
by Jeffrey Hill
Just curious... did anyone try to visit the board between 10:30 and 11:15 last night? If so, was the board down? I've gotten a couple emails that the server that the board is hosted on had problems and so they had to swap out hardware.

In any case, slicehost seems to be very professional about notifying their customers about downtime and getting everything back up in a timely manner. Also, this is the first time in the 13 months I've been with them that the server has had any problems to my knowledge... I know a couple of months ago I looked at the uptime and the server had been running for around 300 days straight. I'm pleased :)

Re: Was the board down last night?

Posted: Thu Dec 23, 2010 12:38 pm
by Charlie Dees
Yes it was down.

Re: Was the board down last night?

Posted: Wed Jan 05, 2011 7:21 am
by Jeffrey Hill
Don't know what happened but it was down this morning... I ended up having to force a hard reboot..... this had better stop happening...

Re: Was the board down last night?

Posted: Sun Mar 20, 2011 1:30 pm
by Jeffrey Hill
Yeah, it was down again earlier today... I've probably got something set up wrong that's causing this to occasionally happen... :?

Re: Was the board down last night?

Posted: Wed Nov 23, 2011 5:24 pm
by Jeffrey Hill
:evil: Conveniently enough, it looks like the server stopped responding again this afternoon...... and this time a table was apparently corrupt when I got it back up and had to be repaired. Let me know if you notice anything weird on the board (or the MOQBA or MACA websites).

Re: Was the board down last night?

Posted: Mon Dec 19, 2011 8:01 am
by Jeffrey Hill
Woohoo, it happened again... at least this time I might have a lead on what the problem might be.

Re: Was the board down last night?

Posted: Fri Jan 27, 2012 9:04 am
by Jeffrey Hill
I was thinking to myself this morning before getting on my computer that it would be inconvenient for the server to go down this weekend... and it looks like the server had been down since around 3:40 this morning. I'm going to have to start manually restarting the server about once every 2-3 weeks as a workaround to keep this from happening since it looks like it's happening about once a month now... I apologize for the repeated downtime.

Re: Unexpected server downtime

Posted: Sun Feb 19, 2012 7:57 am
by Jeffrey Hill
Server crashed sometime after 2:30 this morning. Guess I'll make it every other week on the planned reboots. My hosting plan is supposed to be going through a transition to a different pricing model soon so once that happens I'll hopefully be able to upgrade to a better plan that I think will help with this recurring problem.

Re: Unexpected server downtime

Posted: Mon Feb 20, 2012 1:28 pm
by Jeffrey Hill
REALLY... less than 36 hours?! I don't know what has changed that is suddenly causing the server to crash so frequently but I'm definitely going to have to do something about it soon. :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil:

Re: Unexpected server downtime

Posted: Wed Mar 21, 2012 7:51 am
by Jeffrey Hill
The server has been acting weird this morning so I wouldn't be totally shocked if the site becomes unresponsive at some point today... :(

Re: Unexpected server downtime

Posted: Sun Apr 22, 2012 6:15 pm
by Jeffrey Hill
Site went down sometime between 2 and 3:45 pm...

Re: Unexpected server downtime

Posted: Wed Apr 25, 2012 10:27 pm
by Jeffrey Hill
OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.

Re: Unexpected server downtime

Posted: Fri Apr 27, 2012 12:00 am
by L-Town Expatriate
U. Lou Sthagaim wrote:OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.
How the heck did we get 608 users? Sounds like a spambot barrage.

Re: Unexpected server downtime

Posted: Tue Oct 01, 2013 8:49 pm
by Jeffrey Hill
Short version: I've made some more configuration changes to hopefully completely fix the out of memory instances that have caused the server to go down intermittently. Let me know if you notice the site getting significantly slower or anything like that.

Long, semi-technical version: Today when I got home from work the site wasn't responding, but I was able to log in remotely, which usually didn't work when the site went down in the past (although it was incredibly slow). I dug around trying to figure out what was going on, and all of a sudden the server rebooted. It turns out that the server was in such a bad state that it was constantly swapping memory to and from disk that it was adversely affecting the servers of other customers running on the same physical host server, and so the host initiated a hard reboot (which I would have done immediately had I not been able to log in remotely). Needless to say, I got a warning from my host to figure out what caused this.

I think I had the server configured to accept way too many simultaneous connections and some bot tried to make tons of requests at once. (The access logs appear to indicate this was the case, and I have blocked the offending IP.) A while back I had already reduced the maximum number of clients, but it apparently wasn't enough. I think that reduction may have actually contributed to the server going into a constant swap state rather than crashing entirely. So, I have slashed the number of simultaneous clients even further. Hopefully this won't adversely affect your experience on this board, moaca.org, and moqba.org, but if you notice that the sites are significantly slower or anything else doesn't seem to be working right, let me know.

I hope this finally fixes the problem, because if this exact situation happens again it sure sounds like it's going to be a lot more urgent for me to solve it once and for all..