CODEWITHSUNDEEP
amazon-web-servicesamazon-emramazon-vpc
VB_
VB_

Reputation: 45692

Monitor subnet bandwidth

I have a batch job within one EMR (Spark cluster) that do bulk insert into another EMR (HBase cluster). Both EMRs are located within the same private subnet.

How can I monitor bandwith between two EMRs? AFAIK they are not using NAT Gateay, right? Can I have metrics only for EMR to EMR data transfer, if some other staff happens within the same subnet? Suppose another Spark jobs performs heavy shuffling.

P.S. Is that true AWS provides you with ~ 10 gygabits/sec bandwidth between EC2 instances within the same AZ? Is this bandwidth shared between all instances within the same subnet?

Upvotes: 0

Views: 127

Answers (1)

Madhan S
Madhan S

Reputation: 937

  • First there is no way to monitor traffic for subnet wise, network stats like number of bytes(packets as well) coming out or coming in is published in cloudwatch for each instance level.

  • If you need at the cluster level you can logically group instance based on its cluster type and aggregate the corresponding metrics and have custom stats for each cluster, not supported out of box.*

  • Yes instances with private IP in a same VPC doesn't require NAT to communicate between them.

  • Network bandwidth depends on instance type, newer types like c5 claims to have 100 Gbps speed, for IO heavy jobs like EMR those instances with high network bandwidth are recommended.

(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html)

Upvotes: 1

Related Questions