Now, create JAR ‘ MyTwitterSourceForFlume.jar’ as- jar cfm MyTwitterSourceForFlume.jar Manifest.txt flume/mytwittersource/*.class Please note that you have to hit enter key at end of this line. MyTwitterSourceForFlume.java MyTwitterSource.javaįirst, create Manifest.txt file using a text editor of your choice and add below line in it- Main-Class: Step 3) Compile source code using the command- javac -d. Step 2) Set CLASSPATH to contain /lib/* and ~/FlumeTutorial/flume/mytwittersource/* export CLASSPATH="/usr/local/apache-flume-1.4.0-bin/lib/*:~/FlumeTutorial/flume/mytwittersource/*" Step 1) Go to the directory containing source code files in it. – All flame JARs i.e., flume-ng-*-1.4.0.jar from Load data from Twitter using Flume sudo chmod +rrr /usr/local/apache-flume-1.4.0-bin/lib/twitter4j-core-4.0.1.jar I revoked it as below- sudo chmod -x twitter4j-core-4.0.1.jarĪfter this command gives ‘read’ permission on twitter4j-core-4.0.1.jar to all. In my case, twitter4j-core-4.0.1.jar was having to execute permission. So, revoke execute permission on such JAR. This may cause an issue with the compilation of code. It is possible that either or all of the copied JAR will have to execute permission. This directory will be referred to as in rest of the article.Ĭopy twitter4j-core-4.0.1.jar, flume-ng-configuration-1.4.0.jar, flume-ng-core-1.4.0.jar, flume-ng-sdk-1.4.0.jar to This command will create a new directory named apache-flume-1.4.0-bin and extract files into it. Step 3) Copy the downloaded tarball in the directory of your choice and extract contents using the following command sudo tar -xvf apache-flume-1.4. Step 2) Download ‘Apache Flume’ from a site- Īpache Flume 1.4.0 has been used in this Flume tutorial. Copy files MyTwitterSource.java and MyTwitterSourceForFlume.java in this directory.Ĭheck the file permissions of all these files and if ‘read’ permissions are missing then grant the same.Give a read, write and execute permissions.Step 1) Create a new directory with the name ‘FlumeTutorial’ sudo mkdir FlumeTutorial Example: Streaming Twitter Data using FlumeĪ Flume agent is a JVM process which has 3 components – Flume Source, Flume Channel and Flume Sink– through which events propagate after initiated at an external source.In this Apache Flume tutorial, you will learn. Apache log4j (enable Java applications to write events to files in HDFS via Flume).‘tail’ (which pipes data from a local file and write into HDFS via Flume, similar to Unix command ‘tail’).Apache Flume is used to collect log data present in log files from web servers and aggregating it into HDFS for analysis.įlume in Hadoop supports multiple sources like – It has a simple yet flexible architecture based on streaming data flows. Apache Flume is a reliable and distributed system for collecting, aggregating and moving massive quantities of log data.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |