Expose:
MediaTemple’s Continued Inadequacy Issues
November 26, 2009 by Kyle BradyTags: Attack, Exploit, Failure, Hack, Hosting, MediaTemple, Security
As required by the FTC, a Full Disclosure is available - this piece adheres to the Code of Ethics
Almost a month ago, I made MediaTemple, and the world, aware of an attack that seemed to be a large security issue, and they eventually admitted it was their problem to deal with, rather than blaming it on software like hosting companies like to do. But, weeks later, the problem is not yet resolved, and the public is largely still in the dark.
In the last week, I’ve been notified twice to change my FTP/SSH passwords, and the request yesterday came with an odd statement: the passwords had been previously stored as plaintext, rather than being encrypted or hashed, and that the attackers somehow had access to this - this was MediaTemple’s sole explanation of the massive security issue.
Entirely unacceptable.
After initially making this issue public, both here and at the Inquisitr, I received a phone call from Andrew Won (VP of Customer Service) and Chris, whose position I can’t remember, on 11/16/2009 saying that they discovered the issue, had patched the necessary software, and had submitted patches to the software’s vendors – but asked me to not say anything because of the “security process”. They didn’t give me enough details as to what was actually happening for me to matter, but I kept quiet.
The traffic on my blog, and the comments, continued to mount in the days that followed and it became clear that the issue had not been resolved – people were still being hit with this hack/attack. On 11/19/2009 I asked for an update from Andrew and received a reply stating:
Unfortunately, we still don't have anything public yet. We have already resolved all of the issues and this issue will not recur.
Well, as the attacks continued for other people’s accounts, even through today (11/26/2009), it’s obvious that they had not resolved the issue. When I irately called late last night (11/26/2009 early morning PST), the tech had no answers and neither did his supervisor – in fact, I knew more about the situation than they did, and I was given the partyline: “our engineers are aware of the issue and working to address it.” Further conversation with Andrew, via email, resulted in nothing but doublespeak and sidestepping my questions.
It’s obvious, at this point, that they are either incompetent or lazy – I’m not sure which. They were slow to respond to this in the first place, and have made one misstep after another, which isn’t giving the affected customers much faith in their hosting company, let alone those unaffected that hear the horror stories. The fact that passwords were stored in such an insecure way might be part of the issue, but there are larger problems: discovery, point of entry, depth of access, and execution – none of which (mt) is, in any way, addressing.
When I mentioned this to Andrew, he responded by effectively saying they still have no idea what the problem is or how to fix it:
We are still in the process of investigating this. Unfortunately, while we have a lot of theories and assumptions, we still do not have anything definitive. So please bear with us while we investigate this. We are taking all precautionary measures and locking down many external and internal systems. We will continue to closely monitor our systems and take appropriate actions.
And they even want to dispute the fact that it’s been almost a month, while downplaying the large number of customers affected:
It was not a matter of resolving over the period of 3 weeks. It was a matter of continuing to take steps, monitor and then take further steps. The number of sites actually affected is very small, but due to recent events, we decided that we needed to take a more blanket security approach and change all (gs) Grid Service Server Admin passwords as a precautionary measure.
The “security protocol”, mentioned above, is essentially a “don’t talk about it until it’s fixed” process, but it assumes that those involved are actually trying to fix it, and (mt) is using this as both a crutch and deflector shield – in addition to assuming unaware customers are happier than aware ones:
Chris and I advised you of security protocol, which is what we were following. And security protocol states that you do not publish public info until you are absolutely certain that the issue is resolved and that you are reasonably certain that the attacks or hacks have stopped.
We didn't have much choice in this matter. As we explained to you before, security is a very sensitive issue and by making information public, you are also feeding information to your attackers. We also alerted all affected sites and accounts of the issue and informed them of the steps that we have taken at the moment and time. This issue was still evolving when we last spoke.
Finally, when asked about compensation to customers for their utter failure as a semi-secure hosting company, which they haven’t actually fixed yet, Andrew once again sidesteps the issue by choosing to blame the users/customers instead of themselves:
We do encrypt passwords, but there was a separate file that was kept for the purpose of allowing customers to view their FTP and mySQL passwords through their Account Center. This was a feature many customers asked for in the past. However, we have decided that this feature comes at a price and we are no longer willing to take that risk. Yes, we have learned our lesson. We definitely do understand that this was a headache for ours customers, it was a huge one for us, so we can only imagine it was a much bigger for our customers. We will make sure to discuss a concession of some sort for those customers that were actually affected by this issue.
In summary:
- these attacks are the result of MediaTemple’s failure as a hosting company
- they chose to wait three weeks to even address the issue publicly
- they claimed to have solved the issue long ago, when they hadn’t
- they still haven’t solved the security issue, three-or-more weeks on
- they continue to not reveal any details to users, while sidestepping most questions
- they seem to have no idea of what is truly occurring
They’re going to lose alot of customers over this, especially since they are known for having large-scale problems on a regular basis.
--- --- ---
Update (11/26/2009 5:30pm PST): I had a lengthy phone conversation with Andrew, and while I can't comment on the details, I feel more confident in MediaTemple's abilities and in what they're doing to solve this large security issue. More concrete details as they come, but I would suggest that we have more patience with (mt) on this.
Update (11/30/2009 4:35pm PST): MediaTemple is slowly opening up about this, although the full story doesn't seem to be public yet. Details as/if they come.
Kyle can be found on Twitter and MySpace, or reached via email.
