🕒 Reading Time: 4 minutes Hello! It has been a lot longer than I had planned but we have so much going on it’s been hard to find the time to write! I would like to deviate from my discussion on “Best Practices” to discuss recent events dealing with Plesk and Fedora, and how it gave birth to our newest addition, the CARI.net Incident Response Team. Parrallel’s Plesk, for those who are unfamiliar, is a commercial web hosting management program used to manage websites on a server. It can be installed on a variety of operating systems and has been in use for many years. In the past couple months we have seen a variety of 0-day vulnerabilities exploited on older versions of Plesk running on Suse and Fedora. Let me take you back to that fateful Tuesday… Tuesday March 19th 2013 9:00am We received multiple reboot requests, which when the server was rebooted, it never came back online. By the third one, we began to get very suspicious. Upon investigation, I noticed that these three servers were actually wiped of all functioning data; no OS, just some files (32 to be exact) spread across various directories. By 9:30am, we have 5 of them on the bench. Each of the servers had one thing in common: Plesk. By 2:00pm that same day, I had created a profile in which we could identify these servers, due to the fact that they were being used to attack a Romanian ISP and then wiped, and therefore had a huge amount of outgoing traffic (80Mbps+) . Over the next two weeks this became my life. Searching these servers down and notifying the customers. Now over the next 24 hours we were able to identify at least 18 compromised servers and we notified the clients. We also noticed that the servers were being controlled by several C&C servers, which we then blocked at our edge. The last server to be fully compromised and destroyed happened early Wednesday morning. By March 29th, we had eradicated all active and attacking hosts and we had notified clients via the Panel to watch out for these attacks. By April 9th, I felt that we had taken out the locatable servers and we then invited anybody who was curious about the state of their server to contact us and we would look in to it. And thus marked the beginning of the CARI.net Incident Response Team. These events opened our eyes to a dire need. Communication. Like most datacenter companies, we have an abuse contact that we manage and all abuse complaints are sent to that email and we forward them to our clients, without much interaction. However, through careful analysis of these complaints, I was able to discern a pattern which helped us locate dormant hosts. I began to use the same tactics on discovering the early signs of compromise or that a server had been compromised and was just starting to be used to attack others. In May I proposed the idea of us moving towards creating a dedicated team of trained individuals who could assist in matters like this. As goes with most corporate dealings, I began my first “official” day as leading up IRT in early June. Now I know you are wondering why I’m bringing this up now. Why is this so important? We have taken a new approach to our client-provider relationship. We want to work with our clients to ensure the integrity of our address space as well as to provide the most up-time possible for the client and their purchased services. We understand that many of our clients are using their equipment for their day to day business needs, and we also know that most clients are not aware of the most recent security issues or even how to detect them. At the time of me writing this, I have begun to set forth a few changes in how we handle your Abuse Cases but first let me make sure this is clear. We are not system administrators, and we do not guarantee any sort of solution or promise we can “secure” your server, as this is outside our scope as support staff. However, we want to assist you in any way possible. Security is a community responsibility, and we would like to have that kind of a relationship with our clients. We still maintain strict guidelines for responses to abuse cases, but we also work with you. If you have a question about a certain complaint, we would be happy to assist you with it. In the coming months, we will be looking more into this idea of a proactive relationship with our clients, especially in this department. If you have any questions or concerns about any of this or if you just want to show some interest, you can do so by opening a case (be sure to mention this blog and my name, Zach) or shoot me an email directly at zwikholm@cari.net. I should mention that Plesk has just recently admitted to having some very critical flaws in several of their products, close to 3 months after the 0-day impact. Cisco wrote a blog about this attack seen here http://blogs.cisco.com/security/plesk-0-day-targets-web-servers/ quite some time after we discovered it. There is also a great article on the issue here http://arstechnica.com/security/2013/06/more-than-360000-apache-websites-imperiled-by-crticial-vulnerability/ . I would also review the plesk website and see if any patches have been released for your version Plesk. (see kb.parallels.com). I know this was quite the extensive blog post, but it is a very exciting concept and I hope to hear from you soon! Regards, Zach W. Security Engineer Incident Response Team Cari.Net 858-974-5080 x200