This is the second interview in the “Hosting+Security” series.
WordPress – is the most frequently used tag on this blog. There are two major reasons for this:
Thousands (if not millions) of WordPress bloggers have already experienced various security related problems: malicious scripts and spammy links injected into their web pages, rogue users, corrupted databases — they know how much time and efforts it may take to recover their sites, and how it can be frustrating to find their blogs hacked again just a few weeks later.
In this interview, I will talk with Jason Cohen about WordPress security and his new WPEngine hosting company that aims to protect your WP blogs better than any traditional shared hosting provider.
Q: Hello Jason. Could you briefly introduce yourself and your company – WPEngine?
WPEngine is the best WordPress hosting provider on Earth. We make WP fast, scalable, and secure, and we’ve added new features not available anywhere else, like blog staging.
Q: I noticed that your team consists of developers and entrepreneurs. None of you seem to have web hosting background. Who is engaged in system administration? Who configures and maintains hardware? Well, many IT guys are capable of installing and configuring a web server (I did it myself on my VPS), but the devil is in details and without relevant expertise it is easy to miss some important details. Why people should trust you?
A: On the contrary, Aaron Brazell worked at B5 Media — one of the most prestigious and largest WordPress hosting company, and Ben Metcalfe has run 100s of WordPress blogs and ran server configuration for the BBC.
No one configures hardware — nowadays almost everyone outsources that. It’s about application-level and server-level configuration, and we’re really good at that.
Most people are coming from self-hosting, and of course it’s easy to “trust us” more than you trust yourself!
Q: It looks like WPEngine is something between WordPress.com where bloggers can only use pre-installed plugins and themes and a generic shared hosting where webmasters can install whatever they want (not only WordPress).
A: Correct, we do only WordPress. It’s how we’re able to deliver on our promises of speed, scalability, and security. Obviously any host that allows “anything” is a master of nothing.
Q: As far as I understand, users can only upload their own files to a blog’s wp-content directory. Right? What will happen if someone writes a PHP script that tries to modify files outside of that directory?
A: Not true again! You have SFTP access to the entire installation. However you can’t write outside your own blog area — that would be an obvious security problem. We “chroot” your access, so you can’t even try to write files outside your area — there’s no way to even address the path.
Q: What do you mean by the “blog area”? Directory with the wp-config.php file and everything below it? Does it mean that people can remove all files there (including wp-admin and wp-includes directories) and install, say Joomla?
A: Correct, the root directory including wp-config.php.
No you cannot remove and install Joomla. Well, you can SFTP anything you want anywhere you want, and essentially it will just break.
Q: What is the level of isolation between different user accounts on the same server? It’s quite a prevalent attack vector when hackers use one compromised account to break into other sites on the same server.
A: Right! Again we use “chroot” for complete isolation at the filesystem level. Then everyone is in a different database with different login credentials so you can’t hop between databases.
Also we have hardware appliance security devices to identify attacks before it even hits our load-balancer, so most attack-attempts don’t even make it to Apache.
Q: I believe that quite a few recent massive WordPress hacks used weak permissions of wp-config.php files. Hackers could easily guess absolute paths to wp-config.php files of neighbor WP sites and read MySql credentials. Having stolen the database credentials they worked directly with WordPress tables to either create rouge admin accounts or inject malicious code into blog posts. How do you protect wp-config.php files?
A: True, but once again “chroot” prevents that kind of cross-contamination.
Q: And what about MySql database? Does every user have their own database or do you just change table prefixes? Do you provide direct access tools like phpMyAdmin?
A: No, everyone has their own database with their own credentials. We don’t have a separate phpMyAdmin, but you do have access to that from inside your normal WordPress login.
Q: Every one knows that the single most important thing you can do to keep your WordPress blog secure is to upgrade it to the most recent version as soon as possible. I noticed that some of the WPEngine-hosted blogs used different versions of WordPress (3.0.1 and 2.9.2) What is your WP upgrade policy? Do you force upgrades?
A: Actually that’s not true. When a new major version comes out the last thing you want to do is upgrade immediately. Better to wait until some of the initial bugs and attacks have happened and have been fixed.
That is true for minor upgrades that patch for security. In that case, yes, we do upgrades automatically.
Q: Do you facilitate WordPress upgrades? I personally keep my WordPress under version control (SVN) and upgrades are as easy as one “svn sw” command.
A: Yes you can upgrade any time you want. Nowadays upgrading just means clicking the “upgrade” button in WordPress — we’ve hard-coded the normally-pain-in-the-butt procedure that requires putting in FTP credentials.
Q: Your website says that users can install whatever plugins (and themes?) they want and “the only time [you] would say no is if [you] have discovered a plugin to have detrimental effects” How do you evaluate whether a plugin has detrimental effects? Do webmasters have to ask for your permission before installing a plugin (so that you can check it)? Or users can install whatever plugins they want, but once you detect some problems you may require webmasters to uninstall troublesome plugins?
A: Generally we allow people to do whatever, and we discover problems after the fact and discuss them. With 1000s of plugins registered, just as many custom installations, and new versions coming out all the time, it’s impossible for us to do anything else.
Q: What if some user installs a malicious plugins and themes? I guess you’ve heard that recent story about BlogPress SEO plugin with a backdoor code and some spying/spamming functionality. Or themes that contain encrypted code that injects hidden spammy links into blogs.
A: If we find spammy outbound activity of course we’ll stop it. If you just mean making blogs that are really spam sites, that stinks but it’s not a security problem.
Q: Your dedication to secure protocols is commendable. SFTP instead of plain text FTP? I’ve heard about malware that monitors FTP traffic and steals user credentials that can be used to upload malicious content to legitimate websites. So ditching FTP is a great move. Moreover, all modern FTP client support SFTP so the move from FTP to SFTP is completely transparent to users. Theoretically, there may be problems with corporate proxies and firewalls that block connections to high ports though. Did you hear any complaint about not supporting plain FTP?
A: Of course FTP sends username/passwords in plaintext (and everything else) so obviously that’s not secure, ever. Correct, modern FTP clients all support SFTP, so we haven’t yet encountered anyone begging for FTP access. In any case, we would have to refuse.
Q: SFTP alone is not enough to prevent passwords leaks. Some webmasters save their passwords in FTP clients (who doesn’t like one-click login!). Unfortuantely, most FTP clients can’t properly protect passwords (e.g. FileZilla saves them in plain text in .xml files). So if a webmaster’s computer is infected then malware can easily steal SFTP credentials (username/password/host/port) and feed them to botnets that hack websites. In my experience, this was the #1 attack vector in 2009. Do you warn users about dangers of saving passwords in FTP clients? How do you address attacks that use stolen SFTP credentials?
A: We really can’t control what people do outside our servers. You can store your WordPress username/password in plain text too, or put it on a post-it note, or put it in an online organizer that turns out to be insecure. Or you can email it to a consultant who ends up ripping you off.
It’s endless; we really can’t do much about that.
Q: What other secure protocols do you offer? SFTP implies SSH?
A: Better yet — we just don’t offer other protocols. You can’t SSH. In fact we SSH into the boxes using a VPN, so you’d have to compromise multiple devices and multiple username/passwords to log in.
We don’t offer email — another typical attack vector. We recommend Google Apps for wonderful free email, and for outbound email we use CloudSMTP — a third-party email delivery system where we never have to buffer email ourselves.
Q: What about SSL/HTTPS? It’s a good practice to provide access to WP admin interface via HTTPS so that it is not possible to steal you WordPress credentials or hijack your admin session. You might have heard all the latest buzz around the Firefox extension called Firesheep that sniffs packets on open wireless networks, intercepts unencrypted cookies from popular sites and allows you to hijack other people’s sessions. As far as I can see, wp-login.php pages of WPEngine hosted sites are served using plain HTTP by default, which means that if your customers decide to log into their blogs using a public wireless network (say, in an airport) they will be susceptible to this kind of attack. There has been 788,468 downloads of the Firesheep plugin to the moment so if you are using WiFi in a public place, the chances are that someone monitors your traffic. (Luckily, this plugin doesn’t target your clients’ sites, but it is open sourced so this functionality can be easily added)
A: Yes we know about all that.
Supporting wp-admin over HTTPS is one of the few things we should be doing but today we are not doing, and there’s a reason.
WordPress requires that the admin interface be located on http://std-domain.com/wp-admin where “std-domain.com” is the user’s own domain. Even though we have our own backup domain names for everyone’s blog (e.g. http://account.wpengine.com), the admin interface always redirects you to the canonical one.
That means that we’d have to have SSL certificates for everyone’s domains individually, not e.g. a *.wpengine.com certificate. And it means having a separate IP address for everyone, which most people don’t want to pay for. And of course much more IT admin and server resources.
At the $99/mo level (instead of the base $49/mo level) we do support SSL in this manner. We just can’t do it for everyone.
We’re actually looking into a way to support this with our own subdomains, rewriting the URLs on the back end. So hopefully we’ll make this a reality and we’ll close that loop too.
In the meantime you’re absolutely right that this is a vulnerability.
Q: Can you tell us about the advertised “hardware-based intrusion detection that protects your site from Distributed Denial of Service (DDOS) attacks and rogue scanning?“
A: We have both packet-scanning and DoS appliances in front of our load balancer. The scanner gets daily updates. There’s also a hardware firewall and separate hardware VPN applicances so none of that is a load on our worker servers.
Q: Do you use any malware scanners?
A: Yes, but we don’t find them to be very effective.
Q: Let’s imagine one of your clients’ blog is compromised (for whatever reason). Hackers uploaded backdoor files, injected malicious and spammy content into WP theme, messed with a database. What are your (WPEngine’s) steps to mitigate such an issue? What kind of support can your customers expect?
A: First we can shut down access to the blog immediately. Second we have backups going back for weeks, so we can check the differences there looking for a version we can safely revert to. Then we revert, and put the blog back online. In the meantime there’s a (relatively) friendly error message.
Other than that, of course you have to deal with it case-by-case.
Q: Now imagine that one day hackers discover a yet-unknown WordPress vulnerability (or vulnerability in one of the popular plugins) and start to exploit it. They hack thousands of WordPress blogs around the web every day. There is no patch available. What can you do for your clients in such a situation?
A: If there’s no patch available, and there’s no way of detecting anything about the URLs involved (i.e. to filter those out, even if it means filtering out some OK ones, until a real fix is available), your last resort is to make everything read-only. That means the entire files system is 444 or less, and the database account is allowed to SELECT only.
Sure you can’t post, sure you can’t add comments (unless you use 3rd-party commenting software), but at least the blog is available.
Of course, WPEngine can’t magically make your blog 100% secure — it can still be compromised if you don’t follow basic security practices (e.g. don’t protect your passwords, install untrusted plugins, etc.) — but it looks like their narrow WP specialization ensures that new installations are properly configured and that any issues can be resolved in quite a short time.
Anyway, I encourage you to take the interview with a grain of salt. If you want to ask Jason more questions or share your concerns with us, you can do it in the comments.