Create Your First FLUME Program - Beginner's Tutorial

Prerequisites:
This tutorial is developed on Linux - Ubuntu operating System.
You should have Hadoop (version 2.2.0 used for this tutorial)
already installed and is running on the system.
You should have Java(version 1.8.0 used for this tutorial) already installed on the system.
You should have set JAVA_HOME accordingly.
Before we start with the actual process, change user to 'hduser' (user used for Hadoop ).
su - hduser

Steps :
Flume, library and source code setup
1. Create a new directory with name 'FlumeTutorial'
sudo mkdir FlumeTutorial
1. Give read, write and execute permissions sudo chmod -R 777 FlumeTutorial
2. Copy files [Link] and [Link] in this directory.
Download Input Files From Here
Check the file permissions of all these files and if 'read' permissions are missing then grant the
same-

2. Download 'Apache Flume' from site- [Link]

Apache Flume 1.4.0 has been used in this tutorial.

Next Click

3. Copy the downloaded tarball in the directory of your choice and extract contents using the following
command
sudo tar -xvf [Link]

This command will create a new directory named apache-flume-1.4.0-bin and extract files into it. This
directory will be referred to as <Installation Directory of Flume> in rest of the article.
4. Flume library setup
Copy [Link], [Link], [Link], [Link] to
<Installation Directory of Flume>/lib/
It is possible that either or all of the copied JAR will have execute permission. This may cause issue with the
compilation of code. So, revoke execute permission on such JAR.
In my case, [Link] was having execute permission. I revoked it as belowsudo chmod -x [Link]

After this command give 'read' permission on [Link] to all.

sudo chmod +rrr /usr/local/apache-flume-1.4.0-bin/lib/[Link]
Please note that I have downloaded- [Link] from [Link]
- Allflume JARs i.e., flume-ng-*-[Link] from [Link]
Load data from Twitter using Flume
1. Go to directory containing source code files in it.
2. Set CLASSPATH to contain <Flume Installation
Dir>/lib/* and ~/FlumeTutorial/flume/mytwittersource/*
export CLASSPATH="/usr/local/apache-flume-1.4.0bin/lib/*:~/FlumeTutorial/flume/mytwittersource/*"

3. Compile source code using commandjavac -d . [Link] [Link]

[Link] jar
First,create [Link] file using text editor of your choice and add below line in itMain-Class: [Link]
.. here [Link] is name of the main class. Please note that you
have to hit enter key at end of this line.

Now, create JAR '[Link]' asjar cfm [Link] [Link] flume/mytwittersource/*.class

5. Copy this jar to <Flume Installation Directory>/lib/

sudo cp [Link] <Flume Installation Directory>/lib/

6. Go to configuration directory of Flume, <Flume Installation Directory>/conf

If [Link] does not exist, then copy [Link] and rename it to [Link]
sudo cp [Link] [Link]

If [Link] does not exist, then copy [Link] and rename it to [Link]
sudo cp [Link] [Link]

7. Create a Twitter application by signing in to [Link]

a. Go to 'My applications' (This option gets dropped down when 'Egg'

button at top right corner is clicked)

b. Create a new application by clicking 'Create New App'

c. Fill up application details by specifying name of application, description
and website. You may refer to the notes given underneath each input box.

d. Scroll down the page and accept terms by marking 'Yes, I agree' and click on button 'Create your
Twitter application'

e. On window of newly created application, go to tab, 'API Keys' scroll down the page and click
button 'Create my access token'

f. Refresh the page.

g. Click on 'Test OAuth'. This will display 'OAuth' settings of application.

h. Modify '[Link]' (created in Step 6) using these OAuth settings. Steps to modify '[Link]' are
given in step 8 below.

We need to copy Consumer key, Consumer secret, Access token and Access token secret to update
'[Link]'.
Note: These values belongs to the user and hence are confidential, so should not be shared.
8. Open '[Link]' in write mode and set values for below parameters[A]
sudo gedit [Link]

Copy below [Link] = Twitter

[Link] = MemChannel
[Link] = HDFS
[Link] = [Link]
[Link] = MemChannel
[Link] = <Copy consumer key value from Twitter App>
[Link] = <Copy consumer secret value from Twitter App>
[Link] = <Copy access token value from Twitter App>
[Link] = <Copy access token secret value from Twitter App>
[Link] = guru99
[Link] = MemChannel
[Link] = hdfs
[Link] = hdfs://localhost:54310/user/hduser/flume/tweets/
[Link] = DataStream
[Link] = Text
[Link] = 1000
[Link] = 0
[Link] = 10000
[Link] = memory
[Link] = 10000
[Link] = 1000

[B]
Also, set [Link] as below,
[Link] = hdfs://<Host Name>:<Port Number>/<HDFS Home
Directory>/flume/tweets/

To know <Host Name>, <Port Number> and <HDFS Home Directory> , see value of
parameter '[Link]' set in $HADOOP_HOME/etc/hadoop/[Link]

[C]
In order to flush the data to HDFS, as an when it comes, delete below entry if it exists,
[Link] = 600
9. Open '[Link]' in write mode and set values for below parameters,
JAVA_HOME=<Installation directory of Java>
FLUME_CLASSPATH="<Flume Installation Directory>/lib/[Link]"

10. Start Hadoop

$HADOOP_HOME/sbin/[Link]
$HADOOP_HOME/sbin/[Link]
11. Two of the JAR files from the Flume tar ball are not compatible with Hadoop 2.2.0. So, we will need to
follow below steps to make Flume compatible with Hadoop 2.2.0.
a. Move [Link] out of '<Flume Installation Directory>/lib'.
Go to '<Flume Installation Directory>/lib'
cd <Flume Installation Directory>/lib
sudo mv [Link] ~/

b. Find for JAR file 'guava' as below

find . -name "guava*"

Move [Link] out of '<Flume Installation Directory>/lib'.

sudo mv [Link] ~/

c. Download [Link] from [Link]

Now, copy this downloaded jar file to '<Flume Installation Directory>/lib'

12. Go to '<Flume Installation Directory>/bin' and start Flume as./flume-ng agent -n MyTwitAgent -c conf -f <Flume Installation Directory>/conf/[Link]

Command prompt window where flume is fetching Tweets-

From command window message we can see that the output is written
to /user/hduser/flume/tweets/ directory.
Now, open this directory using web browser.
13. To see the result of data load, using a browser open [Link] and browse file system,
then go to the directory where data has been loaded, that is<HDFS Home Directory>/flume/tweets/

You Might Like

Cassandra Data Types & Expiration Tutorial
VBA Operators
What is VBA?
Execute JavaScript based code using Selenium Webdriver

Prev

Next

3 Comments
Recommend

1 Login

Guru99

Share

Sort by Best

Join the discussion

rohit 2 years ago

Thanks for the blog. It is really simple to understand beginner like me. So I started working as you direct while doing
./flume-ng agent -n MyTwitAgent -c conf -f <flume installation="" directory="">/conf/[Link] is giving me a
following error. I set java_home=/usr/lib/jvm/java-1.7.0-openjdk-[Link].x86_64

see more

Reply Share

Farheen Nilofer a year ago

At the step number 7 following instructions are given:

If [Link] does not exist, then copy [Link] and rename it to [Link]
If [Link] does not exist, then copy [Link] and rename it to [Link]
Problem : I don't get [Link] and [Link] rather [Link] in
my case.
Can anyone tell me what to do now?

see more

Reply Share
SIVANAGARAJU CHILAKALA 2 years ago

iam Getting this error

Failed to search tweets: 429:Returned in API v1.1 when a request cannot be served due to the application's rate
limit having been exhausted for the resource. See Rate Limiting in API v1.1.([Link]
message - Rate limit exceeded
code - 88
Relevant discussions can be found on the Internet at:
[Link] or
[Link]
TwitterException{exceptionCode=[506c3b98-13a0928b], statusCode=429, message=Rate limit exceeded,
code=88, retryAfter=-1, rateLimitStatus=RateLimitStatusJSONImpl{remaining=0, limit=180,
resetTimeInSeconds=1425626432, secondsUntilReset=126}, version=4.0.1}
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
see more

Reply Share

ALSO ON GURU99

SAP HANA Reporting

Business

1 comment 6 months ago

1 comment 8 months ago

Subhash456 Very

SAP HANA SQL -

Abel Chithila Kwagwanji

Application Testing

SAP HANA SQL -

Application Testing

1 comment 6 months ago

1 comment 5 months ago

balu aggala

Subscribe

d Add Disqus to your site

Jaivik Brahmbhatt

Privacy

1. System Administrator
2. Linux Programs
3. Current Job Openings
4. Learn Java Programming
5. HTML Templates

BigData Tutorials
1) What Is Big Data
2) Learn Hadoop
3) Installation
4) Learn HDFS
5) MAPReduce
6) First Hadoop Program
7) Counters & Joins In MapReduce
8) Flume and Sqoop
9) Sqoop vs Flume vs HDFS in Hadoop
10) HANDS-ON : FLUME
11) Hadoop PIG
12) Learn OOZIE
13)Big Data Testing: Functional & Performance

About
About us
Advertise with Us

Jobs
Privacy Policy

Contact Us
Contact us
FAQ
Write For Us
Follow Us

Certifications
ISTQB Certification
MySQL Certification
QTP Certification Testing Certification
CTAL Exam
Execute online
Execute Java Online
Execute Javascript
Execute HTML
Execute Python
Interesting!
Books to Read!
Contest
Quiz

Copyright - Guru99 2016