Author: @roybot

What to do when your website is hacked/exploited

So your website has been “hacked”? You load your website in your browser, and are redirected to some spammers website, or maybe you google’d yourself (naughty), and found a few thousand indexed pages for knock off prada gear? Ok, so how do you fix this, and more importantly, how do you learn how they did it so you can defend against it later.

site-exploited-001

Secure the scene

The first thing I do is take a snapshot of the hacked web site. I want the entire webroot, and the access and error logs so that I can review them.

cd /var/www/vhosts/mysite.com
tar -zcf 20150101-siteexploit-evidence.tar httpdocs logs/access_log* logs/error_log*

At this point I want to either shutdown the web server, or just restore the site and get back to the investigation. I would shut down if I felt the exploit was serious enough to warrant more serious action. Otherwise, just restore from backup, or checkout your repo, whatever you gotta do.

Here is where the fun begins. Try to figure out when, and how, your site was exploited. First things first, take an inventory of the scene.

Looking For Clues to how the site was hacked

Run the last command for suspicious logins. “last” will show a listing of last logged in users. Maybe someone SSH’d into your machine (hopefully not!) or logged in via FTP. If you see a suspicious IP in last then chances are someone has initiated the attack by simply uploading a file through FTP or SCP.

Check for modified files. If you are seeing files modified recently that you did not edit yourself they are suspect and probably contain malicious code. Check the modification timestamp on those files. That date will be useful when looking through the access logs. If you know that you have not made changes to your site in {n} days use that as the basis for your search:

find . -type f -mtime -{n} -print > modified_files.txt

Replace {n} with an actual number.

Scan all files in the webroot for common exploit signatures. If you find a lot of eval(gzinflate then you can almost be certain that it is malicious code. A good rule of thumb is eval is “evil”, except when it’s not (protip: it usually is).

grep -ir 'gzinflate\(base64_decode' *.php > suspect_files.txt

Here is a bash script that checks for common exploits:

#!/bin/bash
 
pattern='r0nin|m0rtix|upl0ad|r57shell|c99shell|shellbot|phpshell|void\.ru|phpremoteview|directmail|bash_history|\.ru/|brute *force|multiviews|cwings|vandal|bitchx|eggdrop|guardservices|psybnc|dalnet|undernet|vulnscan|spymeta|raslan58|Webshell|get_pass|PhpConfigSpy|SubhashDasyam|(eval.*(base64_decode|gzinflate|\$_))|\$[0O]{4,}|FilesMan|JGF1dGhfc|IIIl|die\(PHP_OS|posix_getpwuid|Array\(base64_decode|document\.write\("\\u00|sh(3(ll|11))|earnmoneydo'
searchpath=/var/www/vhosts
grep $pattern $searchpath -roE --include=*.php* | sort

That will check all .php files under /var/www/vhosts for common php shells and exploits.

Collect Evidence

Ok, so you have a date, maybe a date range, and a list of potentially suspect files. With that information it’s time to start looking at logs. The goal is to be able to determine when, and potentially how, the attacker exploited the site.

There are 5 common vulnerabilities attackers exploit.

  1. Remote code execution
  2. SQL injection
  3. Format string vulnerabilities
  4. Cross Site Scripting (XSS)
  5. Username enumeration

All web developers should be familiar with the OWASP top 10 too. Anyways, you want to look for signs of those types of attacks. I scan my access logs and typically check for the following things:

  • Check for POST requests to suspicious locations, such as a file upload location, or other areas that are not accessed directly (like an includes directory, or theme directory).
  • Check for GET or POST requests with a remote URL as a parameter, such as /page/?url=http://atackersite.com/malicious_script
  • Check for GET or POST requests to the files you found when scanning the site for malicious code
  • Check for very odd query parameters in GET and POST requests
  • You should know your site, check for requests that are just out of the ordinary

When you find suspicious entries in your logs, take note of the IP addresses. Scan your logs for all entries for that IP, you might be able to find when the exploit occurred. If you do, you then know the requests that might have been the culprit, or at least have a date range for when the exploit took place. Check the logs around that date range. It could be that the attacker initiated the exploit through a proxy, and finished the work through another one.

Take Action

I recently had a clients website get exploited. The attacker modified dozens of PHP scripts, uploaded about 30 different PHP shells, and just generally made a mess of things. Most of the modified files had the same modification timestamp, so using that I checked the logs and found that someone had been crafting POSTs to a file upload plugin for a wysiwyg editor. Looking into that plugin, I found that they were able to upload a PHP shell. I quickly removed the plugin, restored the modified files, and removed all of the uploaded PHP shells.

The exploit showed me that the security of the website and server were severely lacking. SomeĀ  fundamental security precautions were not implemented. Such as allowing script execution in a user upload directory. That should just never happen. I setup some rules to disable script execution in specific directories, disable direct access to other directories, secured file and directory permissions.

I also setup fail2ban on the clients server. Fail2ban is great for monitoring logs and looking for certain signatures, and then banning the offending IP. You can define other actions to take, such as emailing you or something, but I just want to block their access to the server. So, ban via iptables. I noticed that this particular attacker was sending a lot of POST requests at the same time, so I setup a fail2ban filter to handle a postflood. You will probably want to setup filters for dealing with failed log in attempts to the server, and to address any of the other suspicious activity you found in your logs.

You will definitely want to update your CMS and update any 3rd party dependencies. Doing that alone will go a long way towards keeping your site exploit free.

Wrap up

Security is hard. It’s a never ending battle. And you will never, ever, make your site 100% safe and secure. The best thing to do is have a recovery plan. When your site is exploited can you recover? If you can answer yes to that, you are in a good place.

Best Laid Plans

I had originally planned to play this set for Zombie Crawl, but ended up changing gears once the party go going. I liked this one though, so here we are.

[update 11/19]

This mix has been featured on the Source Tribe podcast.

Check it out: http://thesourcetribe.com/2014/11/17/source-tribe-35-with-roy-lindauer & http://tunein.com/radio/The-Source-Tribe-p443630/

Somehow this thing has gotten over 7000 views in the last 2 days! Crazy

Virtualbox Bug related to SendFile

I have been doing more web development with Vagrant and VirtualBox. It’s a nice way to keep my dev environment nearly the same as my production environments. Recently I was doing some front-end coding and was running into the most bizarre errors with my JavaScript.

It pays to read the documentation. Had I read it more thoroughly, I would have known about this before debugging it and wasting time. Oops!

Turns out there is bug with VirtualBox’s shared folder support and sendfile. This bug was preventing the VM from serving new versions of any file in the shared directory. Obviously this is not good for web development.

The solution is easy enough. You just have to disable sendfile in your web server.

In apache:

EnableSendFile Off

In nginx:

sendfile off;

Update:

The Vagrant documentation does include some information it: https://docs.vagrantup.com/v2/synced-folders/virtualbox.html

Shift – 06_2014

Tracklist:

  • Give It To Me (Original Mix) Dirty Channels
  • Diggin’ On You (Solomun Remix) Elekfantz
  • Forget (Original Mix) Patrick Topping
  • Manitox (Original Mix) German Brigante
  • Nightwalker (Original Mix) Raxon
  • Truesoul (Original Mix) Mendo, Danny Serrano
  • She Loves Me (PBR Streetgang Remix) Lula Circus
  • Ice (Original Mix) Sandeman
  • Emperor feat. Kali (Maceo Plex Last Disco Remix) Kali, Ali Love
  • Silhouette Of A Ghost (Original Mix) Ida Engberg
  • Crossfade (Maceo Plex Mix) GusGus Crossfade

Use Ubuntu 12.04 as an Airplay source

XBMC supports Airplay out of the box, but I don’t really want to run an entire media server just for one feature. Plus, I need the service to run in the background so I can do _work_.

I found a great tool called ShairPort that accomplishes what I need. It is easy enough to get started. On your Ubuntu machine you should install the following dependencies:


sudo apt-get install libssl-dev libavahi-client-dev libasound2-dev build-essential

Now install shairport. You will need git installed if you do not already have it for some reason.


cd /tmp
git clone https://github.com/abrasive/shairport.git
cd shairport
./configure
make

At this point you should be able to run the app.


./shairport -a "My ShairPort"

To see a list of available parameters, run ./shairport -h.

You should see it show up on your Mac now. Go to System Preferences > Sound > Output. I named mine “roy-ubuntu ShAIRport”

airplay with shairport

I recommend moving the shairport directory someplace, like /opt.

Create a symlink to the app:


sudo ln -s /opt/shairport/shairport /usr/local/bin/shairport

That’s it! So far it’s working out pretty well. The defaults for shairport have been fine for me.