Flume to fetch logs over network

Question

I have been working in Flume to fetch logs from a server machine to HDFS. I was able to achieve this if the server and client machines are connected in same network. But how can i achieve the same if the server and client are in different networks.

Do i need to write a custom source for this? [Just checked with twitter example from cloudera in which they're using their own custom source to fetch twitter tweets.]

Any help would be greatly appreciated.

Thanks,

Kalai

Answer 1

If you have a multihomed host joining two non-talking networks you'd like to ship across, you can have a flume agent running there to bridge logs incoming from one network and deliver it to the other one. So your multihomed host will act as a sort of proxy. I don't know if this is necessarily a good idea, as your proxy is probably already busy doing other things if it's the only link between the networks. But if you can set this up, you won't need custom sinks or sources.

If you have two disjoint networks that can both see the internet, you can have one agent post to a web server over HTTP (or TCP for that matter, but it's more work), and another fetch it from the same website. You'll need to write two custom agents (source & sink) for that to work in a performant, reliable and secure fashion, not to mention the web service itself.

Finally, if you have two networks that are completely disconnected (with an air gap), then you may consider writing a custom sink that will, for example, autodetect an inserted tape and copy logs to the tape. Then you take the tape, walk over to another network, plug it in, and have another agent there autodetect it as well and ingest the data :)

Answer 2

Flume agents need to be able to connect to transport events. This means they need to be on the same network.

I'm not sure I understand your question. Why would you expect it to work at all?