Site Status - Important - Please Read

[AthenaOfDelphi]

Hi all,

As I'm sure you are all aware, the performance of the site has been pretty bad the last couple of days. Investigating the cause, I've found there are currently ALOT of hits for three scripts on the site... members, register and search. To try and ascertain whats going on, I have temporarily renamed these so they are not accessible.

Unfortunately, the net result is that the 'Whats New' button doesn't work, nor do signups and obviously member details won't work either.

The problems appear to be stemming from the fact that the server is being throttled because we have exceeded some operational limits regarding resource usage. I am investigating, but it does appear that PGD is putting the server under a somewhat large load.

I'll keep you all updated as and when I (a) make progress or (b) get more information.

[AthenaOfDelphi]

I have just banned three IP addresses that appear to have been making a concerted effort to request the same files over and over and over again, clearly not a user so they are banned. Having just looked up the IP's I've banned, 3 out of the 4 IP's I've added tonight are from Ukraine or Moscow. They have all been making concerted efforts to register lots of accounts and access member data.

The issue appears to have started on the 12th when we exceeded our maximum allowed process count for quite a while. As a consequence the hosting account has been throttled. I'm hoping that the throttle will be removed when the server notices that we're now behaving ourselves.

Now I've banned a bunch of bad clients, I'm going to re-enable the search form, but the member profiles and registration scripts will remain offline for at least tonight.

[code_glitch]

Ah... Indeed things have just sped up BIG time.

[Murmandamus]

You may have a vulnerability, or they may think you still have it. If you haven't patched vBulletin for this, please do so at once, as you could be at risk for a data breach:

http://www.securityfocus.com/bid/47281
http://www.vbulletin.com/forum/showthread.php/376995-vBulletin-4.X-Security-Patch

It is only about a month old.

[Ñuño Martínez]

Ah... Indeed things have just sped up BIG time. I'm afraid not. :(

[code_glitch]

On my end they have sped up noticeably since athena posted that... I'm not getting an 404s from time outs and pages' load times now are like usual after a long delay rather than really slow after a long delay...

[Traveler]

It's still slow on this end too. Pages take from 30 secs up to a minute to load. :(

[chronozphere]

Still too slow for me to use. :( Browsing the forum is just frustrating, so I don't bother until someone fixes this.

[code_glitch]

Oh its frustrating I agree... But look on the bright side for us admins: no SPAM! YAY!

[Murmandamus]

Yeah, but you could have much bigger problems brewing if vB is not patched to fix the vulnerability. Given the type of attacks you are experiencing, it looks like they are trying to exploit this vulnerability or one like it.

Usually, the way these things go, a vulnerability is discovered by an automated scanner targeting a specific piece of site software, then, once found, it gets put on a list that gets shared on various hacker underground sites (potentially even sold), then the criminals take over and start attacking the site, looking for private information to exploit. Even if the vulnerability gets patched, the attacks may take a few days to a week to subside. The only thing that can be done is make 100% sure the webserver and associated app software is fully patched, clean/scan the server for rogue processes and rootkits, disable attacked facilities, and block DoSing IP addresses.

[Murmandamus]

Yeah, but you could have much bigger problems brewing if vB is not patched to fix the vulnerability. Given the type of attacks you are experiencing, it looks like they are trying to exploit this vulnerability or one like it.

Usually, the way these things go, a vulnerability is discovered by an automated scanner targeting a specific piece of site software, then, once found, it gets put on a list that gets shared on various hacker underground sites (potentially even sold), then the criminals take over and start attacking the site, looking for private information to exploit. Even if the vulnerability gets patched, the attacks may take a few days to a week to subside. The only thing that can be done is make 100% sure the webserver and associated app software is fully patched, clean/scan the server for rogue processes and rootkits, disable attacked facilities, and block DoSing IP addresses.

[Murmandamus]

Just figure it should be asked, was there any chance of a data breach during the "event"? If so, we all probably should change our passwords, and be on the lookout for any phishing attempts sent to our emails.

[WILL]

Hi Murmandamus!

I don't think you should worry about your passwords. I couldn't retrieve them if I wanted to so I'd assume you are still safe there. I know with the current issues plaguing the PSN where your mind is thinking, but it's not that WE were haked, I think it was more about A Small Orange, the company that we host under being attacked and forced to go down. It could be a load of spam bots tried to come after us, but then again it's still the same amount of spam we've been receiving since before the slowdowns.

I'd like to tell everyone what exactly was going on, but truth be told I don't even know. Dom (savage) has asked the hosting company what is going on and they seem to want to blame some kind of plug-in we didn't even have. I'm not too happy with the hosting company and they way they've been handling this incident and others in the past to tell you the honest truth and I'm considering a move once I financially take it over myself. I've had nothing but good experiences from DoDaddy and I host Pascal Gamer and Red Ant Games both with them so I'd be willing to try them on for PGD in the future providing I can get the domain redirected properly without a bucket-load of headaches. :)

GoDaddy also has unlimited bandwidth as a part of it's core hosting features so this would take care of that issue as well.

[Murmandamus]

I don't think you should worry about your passwords. I couldn't retrieve them if I wanted to so I'd assume you are still safe there.

Actually, you can, and so can anyone else who gets ahold of your database. While it is true that the passwords cannot be reversed, they can be brute-forced, the only difficulty being how slow the hashing process is. Plus, having emails harvested for spam is not much better. Fortunately, I have a solution for that already, but not everyone does.

The main reason I brought it up is that it is generally a good idea to disclose the possibility/probability of a data breach occurring, because it lets your users be proactive in protecting themselves. If you tell me that, after your (hopefully expert) forensic analysis, the probability is nil or very low, then I won't worry about it. But if you think you found material evidence that it did occur, it is always the best policy to let your users know ASAP. It isn't bad on you to do so, even if it really didn't happen. But, if it did, and you don't tell us, that isn't too cool.


I know with the current issues plaguing the PSN where your mind is thinking, but it's not that WE were haked

No, it's just the way I think, since one of the hats I wear is an IT Security Analyst.


I'd like to tell everyone what exactly was going on, but truth be told I don't even know. Dom (savage) has asked the hosting company what is going on and they seem to want to blame some kind of plug-in we didn't even have. I'm not too happy with the hosting company and they way they've been handling this incident and others in the past to tell you the honest truth and I'm considering a move once I financially take it over myself. I've had nothing but good experiences from DoDaddy and I host Pascal Gamer and Red Ant Games both with them so I'd be willing to try them on for PGD in the future providing I can get the domain redirected properly without a bucket-load of headaches. :)

I'm sorry to hear that. Usually, small hosting providers provide much better service than the larger ones (of course, this coming from a small hosting provider :) ). Unfortunately, I've had the opposite experience with customers on GoDaddy, and am currently in the process of getting another site off of them onto a smaller, more capable provider. GoDaddy's support is decent, but their hosting solutions are subpar. I would expect that the performance of your site to suffer significantly if you do switch to them (which is the problem I am dealing with right now).


GoDaddy also has unlimited bandwidth as a part of it's core hosting features so this would take care of that issue as well.

What issue are you referring to? The slowdown? I don't think that will make any difference. "Unlimited bandwidth" just means that they won't bill you if you go into many terabytes of usage, not that said usage won't take your site down handily. Most providers won't charge you for bandwidth from DDoS attacks and such, as long as you didn't cause it, so I don't think that would be an issue.

Anyway, I am not intending to be critical of you here; I'm just being a concerned (and informed) user and, ultimately, just trying to help.

[WILL]

Not at all Murmandamus. I openly welcome your input. As far as GoDaddy is concerned, I've not had any problems, but that doesn't meant I wouldn't really. I've really only tried them out with a set of simplistic showcase sites that I made from scratch or using iWeb. (Bring it on all you want haters, it's a quick and easy way of making simple sites. ;)) PGD might not do well on GoDaddy, but then there are packages they offer to improve site stability/performance for those busier sites from what I know.

From what I can tell no data was taken from PGD, we just had our servers slammed. If there was data taken from the server, we were not told and I fully hold and will hold the server hosts (of which we have a contract with I'm sure) responsible. I'm not in full control of the site hosting as of yet, Dom is still taking care of this aspect, but is passing the responsibilities on to me for the next year. I'll be covering the fees and managing the hosting package once this year's term ends.

[dazappa]

I have never heard of anyone being satisfied with GoDaddy's web hosting, so I would really advise against it. If you want to go with a big hosting company, I would recommend Host Gator, which offers pretty much the same unlimited deals as GoDaddy (if you're going to be drawn to that offering) Also in the case of hostgator they offer cheaper prices for the longer you prepay. With any webhost, I would recommend against prepaying for more than 6-12 months in case the service has begun to disappoint you.