🕒 Reading Time: 8 minutesEarly in March, while studying the ChinaZ threat, it became readily apparent that default passwords were being used for more than just a supplementary attack vector. Several bots relied heavily, if not exclusively, on systems with weak and/or default passwords to spread. We setup a system with weak and default passwords to capture any and all malware spread in this fashion. For this first test, I selected 5 sets of passwords; admin/admin, guest/guest, ubnt/ubnt, cisco/cisco and ADMIN/ADMIN (the last for picking up folks scanning for Supermicro IPMI devices). Unsurprisingly, it took just under 3 hours for the first infection to hit. What did surprise us though was what password combination was first to be hit; ubnt/ubnt.
As the study continued, the vast majority of the malware that hit was ChinaZ related; MrBlack and IptablesX were the predominant infection types. The non-China related malware was primarily Gafygt malware and perl bots (or “Pastebots” as I call them.). After several months of watching, late July brought an interesting change.
On July 22nd, a bot popped up that was unlike the majority of malware that targeted default passwords thusfar. It was custom-packed, used a peer-to-peer spreading mechanism, and rarely called home.It had four very distinct characteristics: 1) it runs on port 1337 and 9000; 2) it connects to mail.ru, twitter and reddit, trying to create accounts and/or communicate with accounts that it generates from a list names; 3) it is always stored in /tmp/.xs; and 4) the malware checks to see if hosts it infected are still infected by running a POST /check to port 9000.
The first thing that this malware does is create its file structure. The path /tmp/.xs is hardcoded in the malware, and this does not change. The file structure is as follows: list2: List of IPs to scan good2: List of known-good credentials, the host it belongs to and the host information login2: The list of user names and passwords to try (root;root, admin;admin; and ubnt;ubnt) recheck: A file that is read upon malware start, and then every 5 minutes, to ensure that the hosts it infected are still infected daemon.log: An empty log file. Probably left over from debugging daemon.i686.mod: 32-bit ELF LSB executable for x86 machines (statically linked and stripped) (SHA256: 0ffa9e646e881568c1f65055917547b04d89a8a2150af45faa66beb2733e7427) daemon.mipsel.mod: 32-bit ELF LSB Executable for MIPS-I (statically linked and stripped) (SHA256: 5c8c41253aa68adeb955e7d1c7b8e084e06537f75eff12c3f3a0f3cb30cb2152) daemon.mips.mod: 32-bit ELF MSB Executable for MIPS-I (statically linked and stripped) (SHA256: 86fbdd7df9486a17e9c408c7e50635e26402fdf297c9e97f1a5256100401dcc5) daemon.armv4l.mod: 32-bit ELF LSB Executable for ARM version 1 (statically linked and stripped) (SHA256: 9c2848962733846bf50b490fd8f6c7ce9ecade2d3f2f530f5ecbba283af87d3a) srv_cc: A file that contained obfuscated data (non-printable), which also appeared in files/srv_report files/srv_report: A file that had a link with this format: http://46.228.204.44/r0r/index.php?<data_from_srvcc>
The amount of effort that went into this malware was quite extensive. As I mentioned earlier, the malware was custom packed and I was still not able to unpack the malware completely. However, combining IDA’s remote debugging system, core dumps and packet captures, I was able to understand a lot about this malware. There are four groups of communication involved;bot to non-bot, bot to bot, controller to bot and bot to social media. Due to it’s sensitivity the last group will not be covered in this article.
r0_bot has a very specific target group; devices running with credentials root:root, admin:admin, and ubnt:ubnt. Once it has a list of hosts (unknown how these are provided, but they can be found in the list2 file), it runs through all of them once. Any host that is successful, it attempts to copy over the malware. The filenames, and target directory are hard-coded into the malware itself, and since there are parts of this malware that are heavily obfuscated (and I am not the best malware analyst in the world), I was unable to inject anything into it’s routine.
This is possibly the most interesting part of this malware. As I mentioned earlier, this bot runs on ports 1337 and 9000. In the two months that I have been studying this malware, no communication has taken place over port 1337. However, port 9000 is a different story. Once the bot has a list of compromised hosts, it does regular checks to see if those hosts are still infected. This is done via the recheck file. This is essentially a peer-to-peer system, which is why the bot has survived without it’s controller since early August. I first discovered this in a packet capture early on, and realized that we could analyze how many bots were infected just by checking port 9000 for /check. If the malware is running, it will respond with {‘status’:’1′}. Again, John Matherly of the Shodan Project came through and ran a scan for me two days after I originally discovered this malware. On July 24th, the scan completed, returning a total of 14,183 unique hosts running the malware. I compared these results against some of the hosts I knew to be compromised from a few good2 files scraped from several devices (with permission), and it appeared that only 1 out of 4 infected were actually running the malware. Upon further investigation, it appeared that some of those devices were not able to run the malware due to wrong architechture, no shell environment (bash, sh or otherwise) was available, and/or the system had no space. After reviewing the results, I requested another scan be run. On July 28th, there were 18,942 bots running the malware. That was 4000 new bots added in just 4 days. Even now, a quick search of Shodan nearly 28,900 hosts infected and running the malware.
This communication was the most puzzling and something I will probably never fully understand. Most of the communication between the bot and controller was encrypted, however, using the magic of coredumps combined with packet captures, I was able to pull out some very interesting information. For each bot the following JSON is created, and this is then sent up to the controller (if the controller is running, which it is not). It sends this on regular schedule (pulled from core dump on ). I have annotated the purpose of the field next to the field.
A lot of information is sent back to the controller, however, much of this remains a mystery as the controller was taken offline while I was attending Bsides and Defcon in early August.
Much of this bot is still shrouded in mystery. The malware was compiled with the full OpenSSL 1.0.2c library, libcurl and libssh, and was completely stripped. It operated completely within /tmp/.xs and is memory-friendly. The part that is the most puzzling about it all is how much attention was used in creating this bot, but yet the controller was single entity, with no redundancy. The malware originally called home to custom TwistedPython server running on port 80, but also made a lot of API-style requests to port 9000
Possibly the most eye-opening, and most disturbing part of this, was how successful these bots are by just targeting a very narrow pool of credentials. Let’s do some simple math; At peak of this bot, let’s assume 28,880 hosts were infected and running the malware succesfully. If the ratio of 1 infected host running malware per 4 infected holds true, you are looking at nearly 115,200 systems that were compromised; Just using three pairs of credentials. Many of these numbers are incredibly fuzzy because of the nature of residential targets and dynamic addressing. I was able to get some statistics together on the hosts that were compromised, thanks to the amount of detail provided in the good2 and recheck files. During this investigation, we did out best to notify as many organizations as possible without compromising the investigation itself. As of the writing of this article, we have collected a total of 1770 unique bots that were compromised (via good2 and recheck files). We have notified their respective parties and have been working with authorities in the respective regions.
r0_Bot
On July 22nd, a bot popped up that was unlike the majority of malware that targeted default passwords thusfar. It was custom-packed, used a peer-to-peer spreading mechanism, and rarely called home.It had four very distinct characteristics: 1) it runs on port 1337 and 9000; 2) it connects to mail.ru, twitter and reddit, trying to create accounts and/or communicate with accounts that it generates from a list names; 3) it is always stored in /tmp/.xs; and 4) the malware checks to see if hosts it infected are still infected by running a POST /check to port 9000.
The Files
The first thing that this malware does is create its file structure. The path /tmp/.xs is hardcoded in the malware, and this does not change. The file structure is as follows: list2: List of IPs to scan good2: List of known-good credentials, the host it belongs to and the host information login2: The list of user names and passwords to try (root;root, admin;admin; and ubnt;ubnt) recheck: A file that is read upon malware start, and then every 5 minutes, to ensure that the hosts it infected are still infected daemon.log: An empty log file. Probably left over from debugging daemon.i686.mod: 32-bit ELF LSB executable for x86 machines (statically linked and stripped) (SHA256: 0ffa9e646e881568c1f65055917547b04d89a8a2150af45faa66beb2733e7427) daemon.mipsel.mod: 32-bit ELF LSB Executable for MIPS-I (statically linked and stripped) (SHA256: 5c8c41253aa68adeb955e7d1c7b8e084e06537f75eff12c3f3a0f3cb30cb2152) daemon.mips.mod: 32-bit ELF MSB Executable for MIPS-I (statically linked and stripped) (SHA256: 86fbdd7df9486a17e9c408c7e50635e26402fdf297c9e97f1a5256100401dcc5) daemon.armv4l.mod: 32-bit ELF LSB Executable for ARM version 1 (statically linked and stripped) (SHA256: 9c2848962733846bf50b490fd8f6c7ce9ecade2d3f2f530f5ecbba283af87d3a) srv_cc: A file that contained obfuscated data (non-printable), which also appeared in files/srv_report files/srv_report: A file that had a link with this format: http://46.228.204.44/r0r/index.php?<data_from_srvcc>
Communication
The amount of effort that went into this malware was quite extensive. As I mentioned earlier, the malware was custom packed and I was still not able to unpack the malware completely. However, combining IDA’s remote debugging system, core dumps and packet captures, I was able to understand a lot about this malware. There are four groups of communication involved;bot to non-bot, bot to bot, controller to bot and bot to social media. Due to it’s sensitivity the last group will not be covered in this article.
Bot to Non-bot
r0_bot has a very specific target group; devices running with credentials root:root, admin:admin, and ubnt:ubnt. Once it has a list of hosts (unknown how these are provided, but they can be found in the list2 file), it runs through all of them once. Any host that is successful, it attempts to copy over the malware. The filenames, and target directory are hard-coded into the malware itself, and since there are parts of this malware that are heavily obfuscated (and I am not the best malware analyst in the world), I was unable to inject anything into it’s routine.
Bot to Bot
This is possibly the most interesting part of this malware. As I mentioned earlier, this bot runs on ports 1337 and 9000. In the two months that I have been studying this malware, no communication has taken place over port 1337. However, port 9000 is a different story. Once the bot has a list of compromised hosts, it does regular checks to see if those hosts are still infected. This is done via the recheck file. This is essentially a peer-to-peer system, which is why the bot has survived without it’s controller since early August. I first discovered this in a packet capture early on, and realized that we could analyze how many bots were infected just by checking port 9000 for /check. If the malware is running, it will respond with {‘status’:’1′}. Again, John Matherly of the Shodan Project came through and ran a scan for me two days after I originally discovered this malware. On July 24th, the scan completed, returning a total of 14,183 unique hosts running the malware. I compared these results against some of the hosts I knew to be compromised from a few good2 files scraped from several devices (with permission), and it appeared that only 1 out of 4 infected were actually running the malware. Upon further investigation, it appeared that some of those devices were not able to run the malware due to wrong architechture, no shell environment (bash, sh or otherwise) was available, and/or the system had no space. After reviewing the results, I requested another scan be run. On July 28th, there were 18,942 bots running the malware. That was 4000 new bots added in just 4 days. Even now, a quick search of Shodan nearly 28,900 hosts infected and running the malware.
Bot to Controller
This communication was the most puzzling and something I will probably never fully understand. Most of the communication between the bot and controller was encrypted, however, using the magic of coredumps combined with packet captures, I was able to pull out some very interesting information. For each bot the following JSON is created, and this is then sent up to the controller (if the controller is running, which it is not). It sends this on regular schedule (pulled from core dump on ). I have annotated the purpose of the field next to the field.
{
"uname": {
"machine": "x86_64",
"nodename": "host1",
"sysname": "Linux",
"release": "3.12-kali1-amd64",
"version": "#1SMPDebian3.12.6-2kali1(2014-01-06)"
},
"localaddr": [
"192.168.0.2"
],
"lip": "192.168.0.2",
"os_version": "Linuxversion3.12-kali1-amd64(debian-kernel@lists.debian.org)(gccversion4.7.2(Debian4.7.2-5))#1SMPDebian3.12.6-2kali1(2014-01-06)\n",
"hwaddr": "12: 34: 56: 78: 9A: BC",
"cpuinfo": (I have removed this field due to the immense nature of it. It is a json-formatted version of /proc/cpuinfo. If you have multiple cores, it will list them all)
"bid": "858e11563c4326dc629d7c2249ff89c6", (Bot ID; Each system had it's own)
"botnet": "unknown", (Probably for future functionality)
"ver": "1.0.15", (A version number; I'm not entire sure what this was used for, however, several requests had ver 1.0.14 in them)
"arch": "i686", (Used in the request home, http://46.228.204.44:9000/i686?ver=0)
"ip": "12.34.56.78", (Public IP, which we have changed for this example)
"auth": {
"good": "172.16.128.16;22;admin;admin;Linuxversion2.6.21.7(root@AONT)(gccversion3.4.3(release)(CodeSourceryARMQ3cvs2004))#1MonNov511: 19:, (This is pulled from the good2 and has been truncated for brevity )
"timestart": 1437762838, (start of bot in UNIX time)
"timescan": 400 (Scanning time seconds)
"timecheck": 0, (Time since check)
"timerecheck": 524, (Time since Recheck in seconds)
"countcheck": 0, (How many we are checking)
"countscan": 0, (How many we have scanned)
"timelive": 1437763805, (Current time in UNIX time)
"countrecheck": 0, (How many are rechecking)
"countcheckgood": 0, (How many are still good in the good2 file)
"countrecheckgood": 0, (How many are still good in the recheck2 file)
"countloop": 1 (unknown)
}
}
This information is sent back to the controller on a regular interval, usually in a call such as seen below.
A lot of information is sent back to the controller, however, much of this remains a mystery as the controller was taken offline while I was attending Bsides and Defcon in early August.
Much of this bot is still shrouded in mystery. The malware was compiled with the full OpenSSL 1.0.2c library, libcurl and libssh, and was completely stripped. It operated completely within /tmp/.xs and is memory-friendly. The part that is the most puzzling about it all is how much attention was used in creating this bot, but yet the controller was single entity, with no redundancy. The malware originally called home to custom TwistedPython server running on port 80, but also made a lot of API-style requests to port 9000
The Targets
Possibly the most eye-opening, and most disturbing part of this, was how successful these bots are by just targeting a very narrow pool of credentials. Let’s do some simple math; At peak of this bot, let’s assume 28,880 hosts were infected and running the malware succesfully. If the ratio of 1 infected host running malware per 4 infected holds true, you are looking at nearly 115,200 systems that were compromised; Just using three pairs of credentials. Many of these numbers are incredibly fuzzy because of the nature of residential targets and dynamic addressing. I was able to get some statistics together on the hosts that were compromised, thanks to the amount of detail provided in the good2 and recheck files. During this investigation, we did out best to notify as many organizations as possible without compromising the investigation itself. As of the writing of this article, we have collected a total of 1770 unique bots that were compromised (via good2 and recheck files). We have notified their respective parties and have been working with authorities in the respective regions.
- Password Counts
- ubnt: 719
- admin: 760
- root: 291
xxx.xxx.xxx.xxx;22;ubnt;ubnt;Linux version 2.6.32.54 (buildd@builder) (gcc version 4.1.2) #1 Wed Feb 5 18:25:32 EET 2014; xxx.xxx.xxx.xxx;22;admin;admin;Linux version 2.6.22.15 (root@localhost.localdomain) (gcc version 4.3.4 (GCC) ) #2 SMP Wed Dec 18 10:42:42 CST 2013; xxx.xxx.xxx.xxx;22;ubnt;ubnt;Linux version 3.10.20-UBNT (root@ubnt-builder2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_3_1_0_p2 build 34) ) #1 SMP Thu Oct 16 16:29:39 PDT 2014; xxx.xxx.xxx.xxx;22;admin;admin;Linux version 3.4.6 (root@BuildServer45-1) (gcc version 4.2.1) #1 Thu Aug 20 08:22:20 CST 2015In many cases, just visiting the compromised device’s login page told volumes about the device. Using this method we were able to extract some statistics from these 1770 devices. After visiting a few of the pages, a device could be identified by it’s subsequent line in the /tmp/.xs/good2 file.
- Device Statistics:
- Ubiquity Devices
- EdgeMax Series Devices: 116
- AirOS Series Devices: 594
- NAS Systems
- QNAP NAS: 52
- Synology Diskstation: 5
- Home Routers
- Mikrotik: 21
- Alcatel-Lucent I-240W-Q: 314
- OpenWRT: 10
- Modems: 200
- Servers
- SPARC Servers: 2
- x86/64 Servers: 12
- Ubiquity Devices
What’s next?
Default passwords are by no means a new threat. When we wrote about the plaintext password download vulnerability in Supermicro BMCs, nearly 10% still had the default ADMIN:ADMIN combination. This is quickly becoming an epidemic and will only get worse with the advent of IPv6 and the “Internet of Things”.
Many of the users that are affected by these infections will probably never read this article, which is no fault of their own. We cannot expect a user to understand the intricacies of device that was sold to them as “plug and play”.
At the same time, many of these devices aren’t even the responsibility of the end user, but rather they are owned by their ISP.
This juxtaposition of an unresolved pluralistic prerogative has lead to possibly the largest infection vector we have seen to date, with no definitive solution at hand. Unlike vulnerabilities such as Shellshock and Heartbleed, this is not a “patch-able” problem. This bot has not stopped growing; where one system is taken offline and has it’s password updated, there is two more to take it’s place.
This threat does not stop at just spreading malware. When combined with public record information such as ARIN records and historical rDNS records, this moves from spreading malware to data collection and targeted data collection. For example, many business internet packages come with a subnet that is dedicated to a customer. As per normal practice, the subnet is re-allocated via the Shared Whois Project (SWIP) and the basic customer information is registered for those given IPs. In the course of this investigation, we stumbled across several businesses, including dentist offices, a bakery and an engineering firm; each business all registered with their business address and subnet size. This is not difficult to obtain. Normally, this is not an issue and considered to be best practice. Many Ubiquiti EdgeOS devices have the ability to packet capture from the interface, which is not unusual and incredibly helpful in troubleshooting problems. A default password, mixed with publicly available information regarding an specific IP or IP range and the ability to packet capture brings a world of hurt to those involved. There is no easy solution to this issue. As I mentioned earlier, we cannot simply force users to have better passwords nor can we force companies to make their product more difficult to use. In the coming months, we will be continuing to post research regarding our findings into the malware that utilizes this vector, how it works, and what it targets. If you have any questions, comments or concerns, please contact us at sirt@cari.net. For those researchers interested in this malware, I do have an IDB which can be shared with LEA and security professionals . Regards, Zachary Wikholm Head of Security CARI.net sirt@cari.net