Support Question

[RV Guy]

My question is on mySQL support.  Obviously there was a problem with the mySQL DBs yesterday morning.  It was down between when I first tried to update my site (5:30 am) till I had to leave (9:30 am).  I have no way of knowing how long it was down before 5:30 or after 9:30.

I assume techs are notified if a server goes down, but what about the mySQL DBs?  I would venture to guess that it was only fixed once someone from your tech area tried getting onto the site (maybe this forum) and found it wasn't working.  I would think if someone knew about it the minute it went down it wouldn't have taken hours to fix.

So, what's the scoop?  Maybe I am wrong and it really did take hours to bring it back up.  But if not, what about the future.  The sites I'm working on now are not really live, so it wasn't devastating for me.  But I know that there were many live sites using those mySQL DBs that were not functional.  And what about once I do publish my site live?

I look forward to a reply, and hope you have a blessed day.

RV Guy

[AlphaWolf]

My question is on mySQL support.  Obviously there was a problem with the mySQL DBs yesterday morning.  It was down between when I first tried to update my site (5:30 am) till I had to leave (9:30 am).  I have no way of knowing how long it was down before 5:30 or after 9:30.

I assume techs are notified if a server goes down, but what about the mySQL DBs?  I would venture to guess that it was only fixed once someone from your tech area tried getting onto the site (maybe this forum) and found it wasn't working.  I would think if someone knew about it the minute it went down it wouldn't have taken hours to fix.

So, what's the scoop?  Maybe I am wrong and it really did take hours to bring it back up.  But if not, what about the future.  The sites I'm working on now are not really live, so it wasn't devastating for me.  But I know that there were many live sites using those mySQL DBs that were not functional.  And what about once I do publish my site live?

I look forward to a reply, and hope you have a blessed day.

RV Guy

Well the scoop is that we did not notice it for about 30 minutes.  Had it been on any of the servers other than Phoenix it would have taken longer since MySQL was reporting as up in auto monitoring services.  But once it was noticed, we restarted the service and it would overload again in a few minutes.  Someone(s) were either running bad scripts or were being inundated with either referrer spam or DDOS to the point of overloading the connections to MySQL.  We tried upping the connections and within minutes all of those would be in use.  Total connections are set for 500, which is standard for a shared server environment.  Phoenix averages less than 75 connections per second.

So we started suspending sites, including alphaone-tech.com, that were highest on the list of CPU/RAM usage figuring we would find the culprit that way.  Long story short, we never did find the culprit, but the overload just finally stopped on its own.

MySQL was up and down about 100 times between 3:30am - 8:30am CST.  Last night I actually sat at the remote console screen all night watching mysql to see, (hope), it would happen again so we could track down who or what caused it.  Unfortunately nothing happened.

Because MySQL responded as "up" to all 4 of our remote monitoring services, the only way we had of finding out there was a problem was because the tech support center stopped responding.  The level 1 chat techs noticed that and paged me as soon as it was discovered.  Like I said, had it been on any of the other servers, we would have had no way of knowing there was a problem until someone reported it.  There is no way to monitor it when the service itself reports as being up.

peace

WOlf