Your Amazon EC2 Abuse Report

Question

I created amazon ec2 instance to run Tomcat application, but Amazon abuse team sent mail with the following log

<<<
AWS Account: ********
Report begin time: 14-12-2017 02:02:28 UTC
Report end time: 14-12-2017 02:03:28 UTC

Protocol: TCP
Remote IP: ...
Remote port(s): 80

Total bytes sent: 294167550
Total packets sent: 291255
Total bytes received: 0
Total packets received: 0

AWS Account: ********
Report begin time: 14-12-2017 02:03:15 UTC
Report end time: 14-12-2017 02:04:15 UTC

Protocol: TCP
Remote IP: ...
Remote port(s): 80

Total bytes sent: 1050081850
Total packets sent: 1039685
Total bytes received: 0
Total packets received: 0

Is it application issue or have I missed some security configuration?

We found one unknown executable application in /tmp as below, after killed this again came up after few seconds. Seems its generating unknown traffic, so we go ahead and powered off servers now.

[root@ip-172-19-24-90 tmp]# file Lixsyn
Lixsyn: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
[root@ip-172-19-24-90 tmp]#

raw   104832 426240 0.0.0.0:6                   0.0.0.0:*                   7           19719/Lixsyn
raw   104832 228096 0.0.0.0:6                   0.0.0.0:*                   7           19719/Lixsyn


root@ip-172-31-40-123 tmp]# ifconfig
eth0      Link encap:Ethernet  HWaddr ********
          inet addr:******** Bcast:172.31.47.255  Mask:255.255.240.0
          inet6 addr: ********/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:1136962 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2081358186 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:860855089 (820.9 MiB)  TX bytes:2130697820190 (1.9 TiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:11338 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11338 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:21689998 (20.6 MiB)  TX bytes:21689998 (20.6 MiB)

Answer 1

For those who are new to AWS that didn't have a team to read all the details, here is the reminder : all data transfer out from your EC2 to the internet will cost you money after the first free 1 GB.

Data Transfer OUT From Amazon EC2 To Internet

First 1 GB / month $0.00 per GB

Up to 10 TB / month $0.09 per GB

Next 40 TB / month $0.085 per GB

Since EC2s are not mean to be used as web hosting service, it is timely that AWS start sending abuse email to the admin when there is massive data download OUT from EC2 to internet.

You should check whether this is the intended traffics usage of your Tomcat webapps.

1. If you don't anticipate the traffics e.g. there is nobody connect to it from outside the internet, check the website for possible compromised, e.g. check your Security rules, establish a connection to only trust particular IP range than to everyone in the internet.

2. If you anticipate the traffics, e.g. there is multiple user connect to your apps and generate lots of traffics, relook your bandwidth needs and start consider CDN(content delivery network) to relieve the required load. Because EC2 internet outgoing traffics is not cheap, since it is not intends for content intensive download.
   Bare in mind that, if you don't move this traffics to a CDN, a DDoS will cause AWS to send you a massive traffic bills.

(update) As point out by above @Abhijit Jagtap , your server might have been compromised. perhaps at the moment you open up the server to the Internet. A botnet can easily scan through possible web services version and perform the attack. It is highly probably your Web services(Tomcat, jre/jdk, web server etc ) version contain some vulnerabilities. You should recreate the instance, perform all hardening measurement. Please don't bother to "clean up" , it is just wasting time. If you want to make some forensic later, you can create a snapshot for the compromised EC2 instance.

Answer 2

"Lixsyn" Looks like a linux backdoor virus. Has nothing to do with app Linux.BackDoor.Gates

Takashi Katsuki, a researcher at Antivirus firm Symantec has discovered a new cyber attack ongoing in the wild, targeting an open-source Web server application server Apache Tomcat with a cross platform Java based backdoor that can be used to attack other machines.

The malware, dubbed as "Java.Tomdep" differs from other server malware and is not written in the PHP scripting language. It is basically a Java based backdoor act as Java Servlet that gives Apache Tomcat platforms malicious capabilities.

Because Java is a cross platform language, the affected platforms include Linux, Mac OS X, Solaris, and most supported versions of Windows. The malware was detected less than a month ago and so far the number of infected machines appears to be low.

You may think that this type of attack only targets personal computers, such as desktops and laptops, but unfortunately that isn’t true. Servers can also be attacked. They are quite valuable targets, since they are usually high-performance computers and run 24x7.

Java worm seeks out for the system having Apache Tomcat installed-running and then attempts to log-in using the password brute-force attack using combinations of user names and passwords.

After installation, the malware servlet behaves like an IRC Bot and able to receive commands from an attacker. Malware is capable of sending-downloading files from the system, create new processes, update itself, can setup SOCKS proxy, UDP flooding i.e. Can perform massive DDoS Attack. They have mentioned that the command-and-control servers have been traced to Taiwan and Luxembourg. In order to avoid this threat, ensure that your server and AV products are fully patched and updated.

Please Add your suggestions....

Answer 3

AWS sends abuse reports when ec2 instances trigger unknown traffic to servers outside your account.

This can be due to some unknown applications which got inside your ec2 instances due to publicly open ssh/rdp ports.

What you can do here is

1. Configure the security group of your ec2 instances to allow ssh access (port 22) with source ip as your own homr/office ip only
2. Do a netstat command to find what all process ids are accessing the remote ip and port numbers mentioned in thw abuse reports
3. Once you get the process ids, find all processes associated to it using ps -ef command and remove all the related paths
4. Change all user credentials and disable ssh login without password.

Answer to your second question on preventing attacks tomcat servers on ec2.

1. Avoid exposing tomcat manager app on default url/path
2. Use "name" property in server.xml to avoid exposing the. container name and version through http headers
3. Better to put tomcat behind a web layer/proxy like nginx/haproxy to limit impact of bruteforce attacks
4. Use very strong passwords for tomcat users.