Apache Flume to transfer text file contents between two Ubuntu VMs

Question

I am prototyping Apache Flume. My task is to transfer text file contents between two Ubuntu VMs I set up using VirtualBox. I have almost no knowledge of Flume, although I have been going through their documentation.

With VirtualBox I was able to create an Internal Network, and the two VMs are successfully able to ping each other.

I was also able to download and configure flume as shown: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started

My point being that Flume is installed and works on both VMs (or so I think).

This post give me a slight idea: https://stackoverflow.com/questions/19112465/flume-data-transferring-to-server a

So, do I implement exactly that flume.conf file on each VM? And do I need write Java to do this?

Also, I found this video, around the 18 min mark this guy does almost exactly what I want. However, I would like the input to be a text file.

Answer 1

I solved this problem by using the sink type as file_roll. I also had to identify the second VM's IP so the two could connect.

I found the step-by-step instructions on how to do this at this link: http://thisandthat.io/blog/flume-part3/

Also, you do not need to write any Java to do this. However, as is standard with Flume, ensure your JAVA_HOME is declared properly in the flume-env.sh file. All other parts are handled by the .conf file created (as described in the above link).