CODEWITHSUNDEEP
google-cloud-platformddos
dgarg
dgarg

Reputation: 318

GCP Network Egress charges are growing exponentially

Our Network egress charges are growing month on month. Going by the cost, we are egressing upwards of 800GB in a month, to the tune of 300KB/s on avg (600Kb/s daytime and 200kb/s night time)

I analyzed all possible scripts which are sending out data. But none of them is sending out data at this volume. I turned them off one by one but it didn't make much difference.

I momentarily turned on VPC logs, downloaded and analyzed the logs.. it is all distributed across IPs.. about 300 different IPs in a min with average of 10-12kb so about 33Mb/min. there were no IPs which stood out.

I noticed most of them using port 443.

When I use nethogs to identify the process which is doing most of the egress.. it only gave Apache & only showed me 50Kb/s. Where is the rest of the egress??

I mooted the possibility of a DDoS attack but that should show up in the Apache access logs. Apache access logs do not show any suspicious IP/url.

Looking for hints/direction I should take. Apologize if I am missing to give any crucial detail for you to analyze the issue. I will keep adding more details to the question.

Upvotes: 1

Views: 568

Answers (2)

Wojtek_B
Wojtek_B

Reputation: 4443

What you're experiencing is most probably a DDOS attack just as sankar wrote.

According to you nothing stands out particurarily in the logs which makes DDOS theory more probable.

Using Cloud Armor seems the easiest way to protect your server/app out of the box without too much effort since one of it's key features is Adaptive Protection;

Google Cloud Armor Adaptive Protection helps you protect your Google Cloud applications, websites, and services against L7 distributed denial-of-service (DDoS) attacks such as HTTP floods and other high-frequency layer 7 (application-level) malicious activity. Adaptive Protection builds machine-learning models that do the following:

  • Detect and alert on anomalous activity
  • Generate a signature describing the potential attack
  • Generate a custom Google Cloud Armor WAF rule to block the signature

This way you will be able to avoid most of that kind of attacks and save money. Even the fact that you pay for this feature should be beneficial to you in terms of money - not to speak that your server will be a lot more secure and you can focus more on other things.

---------- U P D A T E --------------

There may be one more reason.

A rootkit typically patches the kernel or other software libraries to alter the behavior of the operating system. Once this is happening, you cannot trust anything that the operating system tells you.

This way typical tools won't show the traffic or any suspicious processes.

Have a look at the list of tools that may be helpful to detect any rootkits.

Upvotes: 1

sankar
sankar

Reputation: 278

If you suspect the DDos attack happened already i would recommend to use the Cloud Armour but before that see if you have followed all the mitigations to avoid the DDos attack.

https://cloud.google.com/files/GCPDDoSprotection-04122016.pdf

Upvotes: 1

Related Questions