Phoenix Server Down from 9:12 am

First off we apologize. None of our servers has been down for more than 30 minutes at a time in over 4 years! Actually, in the past 18 months none of our servers has been down more than 10 minutes a month MAXIMUM for rebooting.

Today Phoenix server web services only was down from 9:12 am to 12:32 pm CST.

We owe you an explanation and more.

This morning a user requested that we add two extensions to apache for them to use with joomla. A sysadmin complied and set them to become active at the next reboot. She then scheduled a reboot for 2:30 am CST tomorrow morning so this would then have had the least impact on all customers. It was anticipated that the recompile would take 30-45 minutes.

However, 15 minutes after the sysadmin set this all up. A user implemented a process in one of the OpenSource applciations that basically ran away. it began draining more and more server resources until the server, which normally runs at LESS than 50% load
even in peak periods, was at 85% load and issued an automatic alarm. I responded to this alarm personally and saw which user it was. I rebooted the server. Not, of course, thinking of the compile that would run as soon as the server rebooted. However I would have had to reboot anyway since this users application would have locked Phoenix with its process anyway.

The reboot began the compile of the web services. however, the application that had run away auto restarted itself after reboot! Which stopped the compile and began the process again.

I went in and suspended the user processes and rebooted, but the damage to apache services had already been done. This damage made it impossible to restart apache. Three of us and the data center worked steadily on this problem before being able to
bring apache back up. It turns out one of the extensions is NOT compatible with Cpanel so some downtime would have resulted even in the best of circumstances.

This downtime puts Phoenix at, baring any other issues, a 99.5% uptime for the month. While this is decent, it is not good enough & we will issue a 10% refund credit for the month to any user who puts in a ticket to Billing in the tech center at www.alphaone-tech.com/tech-support. If there were a way for us to have the billing system do an across the board credit for all accounts on Phoenix we would do so, but unfortunately there is no program to do so. When you put in your ticket, please
include your domain name.

Again, my personal apologies for this comedy of errors.

Peace

Wolf