Hadoop Programing with Maven and Netbeans - CodeProject

    This post shows how to use netbeans IDE to write hadoop java program with maven. This post has been written using Hadoop 2.7.3 , Java 8 and  mvn 3.3.9.
     

    Demo for hadoop on netbeans with maven

    Example uses wordcount example available with Hadoop.If something is not readable, click on the image to get full view.

    1. Open  Netbeans IDE and create Maven Project as in below screenshot

     

    2 Click Next, to go to below screen. Enter project and package name.

    3. Next add dependencies for Maven. We need two jars - Hadoop common and hadoop core.  Follow the screenshots for both as below.

    Right click the Dependencies under your project. Select Add dependencies, this takes you to below pop-up. In the query, type "hadoop". You will see a list below, select the highlighted option or option suited for your Hadoop install.

    Here we add hadoop-core. Caution - First time maven will build and download repository details, so will take a while (30 mins it took me). It will be ok from next time onward.

    For the impatient: Skip adding dependencies, move to step 5. Where we create our pom.xml manually.

    4. Next we add hadoop-common

    5. Next check your POM.xml file to verify all is added ok. This file is located under "project files".

    If you have skipped step-4, your pom.xml will be empty and will not have dependencies listed. Add these as below:-
      

    <dependencies><br />        <!--<br />        <dependency><br />            <groupId>org.apache.hadoop</groupId><br />            <artifactId>hadoop-common</artifactId><br />            <version>2.7.3</version><br />        </dependency><br />        <!--<br />        <dependency><br />            <groupId>org.apache.hadoop</groupId><br />            <artifactId>hadoop-mapreduce-client-core</artifactId><br />            <version>2.7.3</version><br />        </dependency><br /><br />    </dependencies>

     This is how the file should look.

    6. Next create java class under package wordcounthadoop and name it WordCount, as in below screen shot. You can download code from here[1].
    Since the dependencies are not yet downloaded, netbeans will mark all imports are erroneous i.e not found.
    It's ok. Go to next step to build the code.


    7.  Last we build our project. Right click your Project and select "Build" as below. At this point your dependencies will be downloaded. Dependencies are saved in your local machine, so next  time these are available locally.
    You will see the status at bottom right of tour screen.

    8. Next we run the project. We can do that from Netbeans itself or from terminal. First from netbeans.
    Right click the project and go to properties, as in screen below.

    9. In the properties screen go to actions -> runproject. As in screen below.

    10. In set properties, below are the 3 properties you need to copy. Remove everything else.
    • First one is, classpath to the project jar file.
    • Next is command line arguments: <package name.Class name>  <input dir> <outpur dir>. The <output dir> should not be existing directory in HDFS.
    • Last is full path of hadoop binary, change it according to your environment.

    Env<span style="color: #333333;">.</span>HADOOP_CLASSPATH<span style="color: #333333;">=<</span>Project Location<span style="color: #333333;">>/</span>WordCountHadoop<span style="color: #333333;">/</span>target<span style="color: #333333;">/</span>WordCountHadoop<span style="color: #333333;">-</span><span style="color: #6600ee; font-weight: bold;">1.0</span><span style="color: #333333;">-</span>SNAPSHOT<span style="color: #333333;">.</span>jar<br /><span style="color: #008800; font-weight: bold;">exec</span><span style="color: #333333;">.</span>args<span style="color: #333333;">=</span>wordcounthadoop<span style="color: #333333;">.</span>WordCount  <span style="color: #333333;"><</span>hdfs <span style="color: #007020;">input</span> directory<span style="color: #333333;">></span> <span style="color: #333333;"><</span>hdfs output directory<span style="color: #333333;">></span><br /><span style="color: #008800; font-weight: bold;">exec</span><span style="color: #333333;">.</span>executable<span style="color: #333333;">=/</span>usr<span style="color: #333333;">/</span>local<span style="color: #333333;">/</span>hadoop<span style="color: #333333;">/</span><span style="color: #007020;">bin</span><span style="color: #333333;">/</span>hadoop<br />

    11. Now right click your project again and click "Run". (as you did with build). All goes well you should see success in the status window.

    12. Now time to check the output. Again no need to go to command line. Go to Windows -> IDE Tools -> Terminal.

    I used some garbage file, hence the output but it is ok.

    13. Below is how you run this from command line. Use Root logger setting to avoid the information scroll when running the program. Below output is with different file. 

    <span style="color: #008800; font-weight: bold;">kamal@kamal-Lenovo-G505:~$ export HADOOP_CLASSPATH=/home/kamal/NetBeansProjects/wordcounthadoop/target/wordcounthadoop-1.0-SNAPSHOT.jar <br />kamal@kamal-Lenovo-G505:~$ export HADOOP_ROOT_LOGGER=ERROR,console<br />kamal@kamal-Lenovo-G505:~$ hadoop wordcounthadoop.WordCount /user/kamal/input/wordcount /user/kamal/output/wordcount<br />kamal@kamal-Lenovo-G505:~$ hdfs dfs -cat /user/kamal/output/wordcount/*<br />a 2<br />b 3<br />c 2<br />cd 1<br />d 1<br />g 5<br />h 3<br />kamal@kamal-Lenovo-G505:~$ </span><br />

    So all done. Read next blog here, to write your first serious hadoop program and understand the basics of Input Splits.

     

    [2]

References

  1. ^ here (gist.github.com)
  2. ^ blog here (my-bigdata-blog.blogspot.com)

Comments

Post a Comment

Popular posts from this blog

New Article Posted: Seven Things You Should Do in Your AMS

Given how advanced and thorough the modern association management software (AMS) system has become, there are many opportunities for your organization to leverage it. Here are seven ways I believe every association should be using their AMS. Click here to read the rest of the article . Of course, if you’re already on my  announcements list , you would have received notice of this new article right in your email box. Not signed up yet?  Click here to sign up . Go to the article
Read more

10 Common Mistakes Biblical Counselors Sometimes Make, Part 8

Image
A Word from Bob: You’re reading Part 8 of a 10-part blog series on 10 Common Mistakes Biblical Counselors Sometimes Make . Mistake #1: We Elevate Data Collection Above Soul Connection . Mistake #2: We Share God’s Eternal Story Before We Listen Well and Wisely to Our Friend’s Earthly Story . Mistake #3: We Talk at Counselees Rather Than Exploring Scriptures with Counselees . Mistake #4: We Practice Half-Biblical Counseling If We Address Sin but Neglect Suffering . Mistake #5: We Fail to Follow the Trinity’s Model of Comforting Care . Mistake #6: We Tend to View People One-Dimensionally . Mistake #7: We Devalue Emotions Rather Than Seeing Emotions as God’s Idea . Mistake #8: We Minimize the Complexity of the Soul-Body Interconnection Because we are soul physicians and soul care-givers, our tendency is to focus on matters of the soul. Most biblical counselors are not trained in the medical field, so we wisely avoid offering medical guidance. Yet, that appropr...
Read more