HACKING and YOUR website!

[AlphaWolf]

Myself and 2 SysAdmins have spent most of the past two days trying to track down a sudden occurence of almost a dozen sites that stopped responding on January 11th.  Ordinarily we could have safely said immediately this was the result of a malicious script that went looking for unsecured Mambo, Joomla and Xoops sites and hacked them.  However, because all reported problems were on one of our 3 physical hard drives and because NO problems had been reported on any other server, we decided to be proactive and make sure something odd was not happening to that hard drive.

This resulted in 20 hours of work plus 90 total minutes of downtime for the entire Phoenix server.  Extensive file integrity and drive checking showed that there was no issue with any hardware or operating system software on the server.  Then 2 more sites on our John Gault server reported the SAME issues.  This confirmed our suspicion that this was hacking related.

I then took one of the sites and delved into it in depth and found that ALMOST every directory under the sites main Joomla directory had brand new .htaccess files.  Compared that to a fresh install of Joomla on a test site showed that these were NOT present in a normal install.

I deleted ALL .htaccess files that had the same date and time stamp, plus some newly added .PHP files.  These were added through injection from within the Mambo, Joomla or Xoops application and had the same date/time stamp.  This site IMMEDIATELY became operational.

The point is this.  Those of you who's sites were hacked, (and anyone else who does NOT keep up with security patches for their PHP applications), caused every user on the Phoenix server to suffer downtime.  Because we wanted to thoroughly test, Phoenix was down for MORE time yesterday than in 5 months previously combined!  Because our tech support center is also housed on Phoenix, it too was affected.

If you are running ANY PHP/MySQL based application you need to subscribe to alert services that will warn you when there are security issues.  You need to make sure you patch your site and/or stay current with versions.  Please take this time to protect yourself AND everyone else who shares the server with you.

Peace

Wolf

[AlphaWolf]

We have just been informed by a user that deleting .htaccess files does NOT fix the issue of this hack!

The only way to remove this hack appears to be to wipe your account completely and restore a recent backup done via CPanel.

If you are affected and wish us to assist you by recreating your account, please submit a Customer Service ticket in the Customer Support Center at www.alphaone-tech.com/tech-support

[jariggs]

I have tried to get rid of the soho files.  I went down through list and deleted the files in soholist.  Fantastico now allows me to try and install sohoadmin again, however, I get the following error after trying to install:

You cannot install more than one script in the root directory of a domain.

Any idea what else I might need to delete.  I assume the root directory actually means public_html....correct?

This is what my public_html directory looks like:


Click on a name to view its properties.
 

 / public_html / (Current Folder)   
 Create New Folder   
 Upload file(s)   
 Up one level   
 NewUserTemplates  755
 WysiwygPro  755
 _private  755
 _vti_bin  755
 _vti_cnf  755
 _vti_log  755
 _vti_pvt  750
 _vti_txt  755
 cgi-bin  755
 imagesbu  775
 importbu  775
 mediabu  755
 phpbb  755
 shoppingbu  775
 srpoker  755
 subscriptionbu  775
 tCustombu  775
 templatebu  775
 Create New File   
 .htaccess 0 k 0644
 .wysywigPro_edit_postinfo_html.php 5 k 0644
 0.gif 0 k 0644
 1.gif 0 k 0644
 2.gif 0 k 0644
 3.gif 0 k 0644
 4.gif 0 k 0644
 5.gif 0 k 0644
 6.gif 0 k 0644
 7.gif 0 k 0644
 8.gif 0 k 0644
 9.gif 0 k 0644
 About_Us.html 4 k 0644
 Contact_Us.html 5 k 0644
 FAQs.html 3 k 0644
 Home_Page.html 7 k 0644
 Products.html 3 k 0644
 Products.html.LCK 0 k 0644
 Shop_Now.html 4 k 0644
 Support.html 2 k 0644
 Test.html 2 k 0644
 _vti_inf.html 1 k 0644
 about_us.html 4 k 0644
 compare.html 1 k 0644
 contact_us.html 5 k 0644
 currentuserbu.log 0 k 0644
 fantastico_fileslistbu.txt 0 k 0644
 faqs.html 3 k 0644
 hitcounter.txt 0 k 0644
 home_page.html 7 k 0644
 old_index.html 1 k 0644
 postinfo.html 2 k 0644
 products.html 3 k 0644
 shop_now.html 4 k 0644
 support.html 2 k 0644
 test.html 2 k 0644


NOTE:  the files with BU added are files I renamed because the FTP program wasn't changing the permissions and I could not delete complete directory...Is that the problem?

Thanks,

Jeff R

[AlphaWolf]

Hey Jeff

Put in a support ticket.  Make sure to give them your domain name and user name.  I have a feeling its fantastico files causing the issue.  Is there anything else on the account you need to keep?  It might be easier to delete the entire account and recreate it.

peace

Wolf

[jariggs]

It occurred to me after my last post that one might just delete and recreate the same website.  Sounds from your reply that it should work.  For the particular website, I can probably get away with that since it was not too substantial and still in development.

NOW, based on your earlier assertion that databases are probably ok, can I delete site, then recreate it, reinstall the Fantastico scripts I had and then upload my old databases?  Should this work?

I have another site with the same hack attack, and that site has a little more work in it and will be even more work to replace, which is why I tackled the easier one first.

If my wife's retail site went down...there'd be real trouble, not to mention the extensive and time consuming work I've done on that project.

SO, the important question is what do we do to protect ourselves and our clients from attacks, or at least how do we back up such that we can reinstall quickly?  It doesn't really matter the source of site destruction, I suspect most hosting accts clients will not stick around to see if it happens again, UNLESS it is very easy to reinstall.

SO, I think we need a step by step backup and reinstall tutorial/instruction/script to resolve this problem.  I know backups are good, but it is not real clear how, when and what to back up.  I know we can back up from the CPanel, but as you know there are various options and it is not real clear which is best, preferred, or will allow total reinstallation.  I've run the current tutorial and tried to figure this out, but am still unclear.

I hope to be here for the long haul.  I like the general feel of Alphaone and the people.  In order for us all to enjoy the service and succeed in our personal endeavors, it is absolutely critical that are websites be secure in the first instance, and easily backed up and reinstalled when security is breached.  I suspect you will tell us that security lies significantly in our hands, but as Reseller, I don't want my accounts to have to worry about security any more than is absolutely necessary.  And to the extent it is necessary, I want them to have a detailed instruction sheet on what they are to do, how they are to create and maintain backups, and how they can reinstall when and if the need arises.  In the end, I can't help them if I don't know myself.

As a further note Wolf, I continue to be very interested in the Reselling/Hosting aspect.  I want Alphaone to assume all of the infrastructure and as much of the programming and support as possible.  In WebHosting 101, we started by looking where we wanted to end, but I suspect it is the mechanics hosting/reselling that needs to be addressed first.  We think we want to do this, but what is really involved? How do we serve those few customers that we already have.  (Marketing 101 - take care of your current customers...) That is where I think WH101 comes in, with maybe the business plan/marketing aspect saved for a little later.  This security/hacking issue underscores our need, as Resellers, to have a better grasp of the mechanics of the hosting business and our role as Reseller, starting with how we protect our clients accounts from hackers, as well as themselves...

Luckily, I only have two "resold" accounts, and both of those are for free to my family.  So they can't complain too much.  Even more fortunate, their sites survived the attack...I, alone, was the lucky one...And I actually was lucky in a way, because had their sites gone down, I would have had no idea what to tell them to do....Some Reseller I'd be...

This is our head's up, and getting the sites back up will only fix the symptom.  Without ironclad security and simple but detailed backup instructions and procedures on Alphone's part, and timely backups on the user's part...we will continue to be at risk not only of losing our sites, but our customers.

Like I said, I like where I am, and I encourage everyone to act as a community to promptly address these issues.  Wolf and Alphaone have got to lead the way (but that's only fair if you call yourself "alpha one" and you make a little money while you're doing this), but in the end are collective knowledge and wisdom is what can make this hosting site great (and send Wolf's kids to college...).  That's all good, as long as my sites and my customers keep rollin' on...

My apologies for the diatribe...but it is with the best of intent...

BTW: HAPPY MARDI GRAS TO Y'ALL FROM LAFAYETTE, LOUISIANA!!!!!!!!

Jeff R

[AlphaWolf]

Hi Jeff

All very valid concerns.  I will table the ones on the web hosting class for now - I sent you two emails on this to the email address that shows in the forum user record.  You can email me off forum to discuss this further.

Restoring Database Backups
I deally that would work.  It SHOULD work.  However, that said, it did NOT work for one of our customers.  I am suspecting that when he reinstalled the application with fantastico he either a) named a different admin name b)a different admin password c)a different path name or d)forgot to delete the OLD database before running the Fantastico install - this caused fantastico to create a totally different database name.

When this did not work, we restored a main site backup that was pre-hack.  But he had no MySQL backups from pre-hack.  So the restore worked fine, just could not access the database.  He had to start from scratch.

Security
Our servers are just about as secure as you can get them without restricting ALL access to signed in user accounts.  Something you can't do with a public server or no one would ever see your websites.

However, the biggest, by far, security holes are in PHP applications - both home grown and commercial or open source.  And there is nothing we can do about these besides having register globals off, which we do on all our servers.  PHP 5 will change some of that, but not all.  The dilemna is that if we upgrade to PHP 5 99% of ALL PHP applications will immediately fail - either because they do not support PHP 5 OR because there are special configurations that need to be done in the application to support PHP 5.

The two ways to significantly decrease your hack-ability are:
1 - watch all official support boards for ALL software you have installed on your sites and upgrade ASAP as needed
2 - Go through and painstakingly, through trial and error, turn OFF the write and executable bit for world for as many files as you can.  (right now there are over 700,000 files on one server that have write or executable turned ON for world.  In most case - MOST - that is not needed

Recovering quickly
Backup early and often and FTP those backups to your local computer.  Backup full home directory AND MySQL.  Keep backups at LEAST a month.  In this last hack the actual files were inserted two WEEKS before the hacker activated them.

If you do this, we can delete your account and recreate it from backups.  It isn't necessarily fast, (depending on when it happens and what else is going on, it can take between 6-24 hours)

One thing we will be offering for sale to all accounts soon is DiskSync backups.  You tell us what directories to backup and how often.  They are then backed up to a DiskSync server.  You will STILL have to backup your databases since all backups of MySQL databases can result in file errors since the database may be changing during a backup.  But with DiskSync, restoring the files you have designated as needing to be backed up, is as simple as putting in a ticket.  Your site will be restored within 1-8 hours, depending on workload and when it occurs.

The downside to DiskSync is that, say you want directory A backed up every day.  Every day the backup re-writes OVER the previous backup. (Same as with our nightly backups - weekly backups increment)  A lot of users went days and WEEKS without even noticing they had been hacked.  By that time, unless you have backups on your local machine to upload for us from pre-hack, we have no way to assist you.  Much as it pains me to say that.

I can tell you the routine we follow for AOT website backups if that will be helpful.

In addition to the serverwide nightly backups, every 3 nights we DiskSync the forums, the ordering system and the tech support center.
Once a week, CPanel full site and all MySQL databases are backed up and FTP'd to Brad's home server.

Even with all this, in a crisis it can take 1-8 hours to restore these vital components and up to 24 hours to restore our Webmaster Resources since that is only backed up incrementally.

Backup Instructions
If you are using our customized XPEvolution Cpanel skin, clicking Backup in Cpanel will have a link to the tutorial right on the backup page.  If you are not, you can see the same tutorial at www.alphaone-tech.com/clients/tutorials.htm.  However as a reseller, you have to purchase these tutorials for use by your customers.  Legally these can only be used by direct customers of AlphaOne and its divisions.

LOL...as for sending my kids to college.  Its a darn good thing I don't have any - they would be standing on the corner with signs saying "Will work for tuition money" 

I tallied labor costs yesterday for what this hack cost us.  And bear in mind the hacker did NOT use holes in our server, operating system or OUR software to gain access.  Labor to deal with tech support issues and help restore what sites we could restore, cost over $5,000.  That doesn't include about 80 hours of my own time.  I think we lost about 10 accounts over this where webmasters just decided to give up running a site completely, and one who could not seem to understand that it was HIS not updating his software that caused the hack and decided to switch hosting companies.

Given those costs, time and the goodwill lost, if it were remotely in our power to prevent this, we'd do so in a heartbeat.  Given the size of our company and our low hosting margins, it will take us 5 months to recover what it cost us - and those are just the tangible, trackable costs.  The only way we can add more protection to this is to ONLY host sites we manage on a server.  Managing sites to the level needed to make sure that all software for the site was up to date...  Hosting costs would be in excess of $100 a month...which is certainly not an option.

I do appreciate your concerns here, and any web host who knows what they are doing shares these concerns.  We are just limited by what is and is not viable for shared server, public website environments.

peace

Wolf

[AlphaWolf]

I spent a few hours yesterday hunting around.  Thinking that since WE can't sell our customers a service that monitors their sites for vulnerabilities, (it would be too cost prohibitive to really be of value), there had to be someone out there who does.

We are now in negotiations for reduced pricing for our customers through Control Scan.

Control Scan has 3 plans ranging from $49.95 - $99.95 a month to scan your domain daily for over 7,000 known vulnerabilities.  Their vulnerability database is updated DAILY.
In addition, anyone who subscribes annually also gets BitDefender Firewall, Anti-Virus, Anti-Spam & Anti-Spyware for their personal PCs.  We are working with them to try and get pricings on this down 40% or more.  And it looks likely.

Once we have details, we will be able to offer all our customers a free 30 day trial so you can see if is advantageous to you.  We will be setting up a pay-as-you go tech support center that will be staffed by vulnerability consultants so that if there are vulnerabilities that are beyond your expertise to correct, you can gain assistance from an expert.

AlphaOne signed up this morning and as soon as we get a copy of our vulnerability report, I will be posting.  I have no doubt that even OUR site has vulnerabilities - some can't be addressed without breaking applications, but others are probably there due to installation applications and having unnecessary write/execute permissions.

Peace

Wolf

[AlphaWolf]

One of our resellers who had a couple of his sites hacked mentioned something to me in email.  Since less than 20% of all sites that have out of date versions of software on them were actually hacked, I questioned why it was THESE sites and not more of them.

In theory it should have been.  Its really quite simple once a hacker has a hack for a known vulnerability in a piece of software, there is nothing to prevent them from hacking every installation that has that software vulnerability in it.  I could go right now and find a vulnerable Mambo site, hack IT, then find its shared IP address, do reverse lookup and dump all other domains that have that shared IP address into a script.  Then run the script against all domains in the list and attempt to hack into the automatically injecting the hack where it could be injected.  It takes NO more time or skill to do it this way.

But that isn't what happened.  Almost 80% of sites with out of date, unsecure, versions of Mambo or Joomla were NOT hacked.

Eric, one of our resellers, mentioned that the sites of his that WERE hacked had one thing in common - configuration files set to world RWE so they were open to anyone who knew the name of the configuration file.  Not hard since all open source apps can be downloaded by anyone to get config file names.  I pulled up 10 of the sites we restored from backups, (most of which have NOT yet been updated by their webmasters...sigh), and discovered that each of THEM had their configuration files set to world RWE.  I also checked one account where not only were the files hacked, but the hacker got in to the MySQL database as well.  In that one I discovered that not only was the configuration file world RWE, BUT the darn admin user name and password was inside it, readable, and NOT even encrypted!  So yes, of course it would be easy for any kid on the streets to hack.

I changed those 10 accounts to have their config files NOT RWE by world, but advise everyone running PHP applications to check your OWN configuration files and make sure they do not allow RWE by world.

We can protect the servers from being hacked, and we have - all of them hold up against all known server hacks and DDOS attacks during vulnerability penetration testing.  But if you, or your applications insist on letting anyone wre your configuration files, or if you dont keep up to date with your software's security patches, you will go through this more than once.

Or maybe not - there are still over 150 out of date applications and even MORE world RWE config files on just one of our servers.  One site has 3 year old software on it, having been with us a bit over 3 years.  They have never been hacked.  They are lucky.  Luck does run out folks.

PLEASE see our 3rd party application directory and register with the official support sites for the software you run! http://www.alphaone-tech.com/client/app_support.htm

PLEASE go through your files and directories and make sure your configuration files are not accessible.

[jariggs]

I pulled up my site, and now get my file list.  I pulled up the separate html pages created in sohoadmin and they have these entries at the top of the page.  What was the hacker trying to do?

WWW.COMPUTRENEUR.COMbag ski snowboard
WWW.COMPUTRENEUR.COMdating site
COMPUTRENEUR.COMdj slon bumer mp
WWW.COMPUTRENEUR.COMsound forge :crack
WWW.COMPUTRENEUR.COMtemplatemonster password
WWW.COMPUTRENEUR.COMhentaikey diablo
COMPUTRENEUR.COMnew jersey antitrust law
Welcome to computreneur.comsexparty. tv .com
Welcome to computreneur.comsexparty. tv .com

The text I input is still there, so I can copy and paste the text and I'll be able to save alot of time.

One thing I do not understand is how it was that I failed to properly undate.  On one site, all I installed was sohoadmin through fantastico.  You have discussed Mambo and Jumla, and I have never intentionally used either.  So how did they get involved in my sites?  Does sohoadmin make use of these?  If so, how would I know?

Additionally, none of my sites are old, as they are Sept 2005 sites.  Installations were even after that.

I think I remember seeing some suggested updating, but I am not sure it warned of security issues.  Are you saying that when the fantastico says new version available that we need to make sure we update it asap?  Will site work the same, or are we taking risks that it will not work?

While I have read the warnings about Alphaone not supporting the 3P apps (does the warning at least give the proper link to the securtiy board - not just the general site). I think there should be some monitoring of these apps by alphaone since the ease of installing the aps is part of what you're selling.  Why doesn't AT monitor these sites and post when new version is available or when security issues posted?  It would seem that would be workable if you have enough users to spread that overhead cost.  You must realize that many of us have computer knowledge, some webmaster knowledge, but we're learning.  That's what attracted me to your site to begin with.

To those other Webmasters on AT.  The level of discourse has fallen noticeably, and if we want this to be a good website/host, we all have to work together to come up with workable solutions to the issues that affect us all.

Jeff R

[jariggs]

Wolf:

I have no explanation for why, but none of your emails are making it to my address, could it be because a hyphen is in the name?

At any rate, I've changed it to one of my AOT accounts.  I am getting billing info from AOT there ok.

Sorry, but this may explain the communication problems we've been having.

Please resen or FWD emails re WH101 to me.

Thanks,

Jeff R

[AlphaWolf]

I pulled up my site, and now get my file list.  I pulled up the separate html pages created in sohoadmin and they have these entries at the top of the page.  What was the hacker trying to do?

WWW.COMPUTRENEUR.COMbag ski snowboard
WWW.COMPUTRENEUR.COMdating site
COMPUTRENEUR.COMdj slon bumer mp
WWW.COMPUTRENEUR.COMsound forge :crack
WWW.COMPUTRENEUR.COMtemplatemonster password
WWW.COMPUTRENEUR.COMhentaikey diablo
COMPUTRENEUR.COMnew jersey antitrust law
Welcome to computreneur.comsexparty. tv .com
Welcome to computreneur.comsexparty. tv .com

It looks like they may be trying to auto-force people to another site.

One thing I do not understand is how it was that I failed to properly undate.  On one site, all I installed was sohoadmin through fantastico.  You have discussed Mambo and Jumla, and I have never intentionally used either.  So how did they get involved in my sites?  Does sohoadmin make use of these?  If so, how would I know?

Now here is an issue that we can't get answers to.  Soho claims they have no known vulnerabilities.  But 4 Soho sites were hit as well.  And it wasn't just easy to guess passwords, which is what Soho suggested it might be.  Because these have been injeted directly into template files - not the database!  And its not even easy for a soho webmaster to change the templates.  They are set, permission wise, so that they can ONLY be updated by a php app that is run under the soho app or the soho app itself.  I wish I had an answer for you.  I spent hours trying to get one myself.

Additionally, none of my sites are old, as they are Sept 2005 sites.  Installations were even after that.

 

That doesn't matter - one month PHPNuke had FIVE security updates during the month.  SMF, after having none for 9 months, had two in one month.

I think I remember seeing some suggested updating, but I am not sure it warned of security issues.  Are you saying that when the fantastico says new version available that we need to make sure we update it asap?  Will site work the same, or are we taking risks that it will not work?

Unfortunately with the way these darn applications are written, there are constantly security holes being found in all of them.  Some more than others.  But even fantastico can NOT keep up with it.  Fantastico usually is at least 1 month behind on updated software.  Though that is better than 5-12 months behind.  Oh yeah there are risks in upgrading.  Many work just fine auto-upgraded, but we have seen several in the last two months where even the fantastico backup done before upgrading did not return the site to normal.  That is why we tell everyone to use Cpanel to backup both the database and the full site BEFORE ever upgrading.  That way if it does not work, you can upload those and we can restore to previous state.

While I have read the warnings about Alphaone not supporting the 3P apps (does the warning at least give the proper link to the securtiy board - not just the general site). I think there should be some monitoring of these apps by alphaone since the ease of installing the aps is part of what you're selling.  Why doesn't AT monitor these sites and post when new version is available or when security issues posted?  It would seem that would be workable if you have enough users to spread that overhead cost.  You must realize that many of us have computer knowledge, some webmaster knowledge, but we're learning.  That's what attracted me to your site to begin with.

Unfortunately most software doesnt have a defined security support area so we cant point to it.  We recommend that everyone subscribe to www.frsirt.com alerts.  For over a year we posted applicable ones here in the forums.  Our team spent over 10 hours a week making sure the info was available here.  Bottom line, 1-2 people subscribed to the threads and read them.  And with not just the 50 or so apps fantastico can install but literally thousands OTHER apps, many of which we find people installing and using on our servers, we can't keep up with that.

Ease of install via fantastico is what thousands of web hosts are selling, and it is a good selling point.  But not one of those hosts that I have ever seen even tries to do what we did by providing security alerts in forums or announcements.  Because its not manageable AND we may miss one or miss an application that someone is using.  Its not only a matter of cost and finding someone with time to do it, its being a tad reluctant to have people rely on us for this when we dont know who is running what application and we are not experts on the applications.

If one of the resellers wants to take this on, we will offer them a discount on monthly fees to do so, but I still would NOT recommend that anyone just rely on our internal support board to keep on top of things with their software.

To those other Webmasters on AT.  The level of discourse has fallen noticeably, and if we want this to be a good website/host, we all have to work together to come up with workable solutions to the issues that affect us all.

We have security as tight as we can make it without breaking every single site on each server, and have been looking in to outside services who will offer our customers a deal on daily security scans... but none of them scan for out of date software and report it.

And truthfully, I believe the best protection is to do weekly Cpanel backups and save them for at least a month.  That way if you are hacked and dont catch it before the next nightly backup of the entire server, we can still do a restore for you without having to call the data center and have them send the tapes back from off site storage.
We are also working on a plan to keep 3 days worth of backups on the server itself for easier restore - but restoring STILL takes 4 physical hours of server time due to the compression and storage method of files.

Bottom line - if you are running a PHP application you can be 90% sure that it could be hacked.  Its the nature of the beast.  But that said, if you consider that in at least the past two years that I can remember, only 2 sites have ever been hacked prior to this major infestation.  And tons of sites have huge security holes in them.  This hacker was just a tad more determined than most I figure.

Security is a balance between effort and results.  And while I keep saying update your software, protect yourself, realistically if I were a webmaster of anything except an e-commerce application I would take the route of backing up early and often and checking my site to make sure it is functioning twice a day over trying to stay on top of and implementing every single security patch and upgrade that comes out.  I KNOW that goes against what I preach, but the truth is the time involved can be daunting and so many times an upgrade breaks something that then takes time to fix.

Jeff thanks much for your constructive input into this.

peace

Wolf