-1

I am trying to run this script on for running map reduce on hadoop. But when I run this script, it is giving me the error attached in the screen shot. Script:

rm -rf /home/sk/Desktop/abc/wordcountc/

rm /home/sk/Desktop/abc/wordcountj.jar

mkdir /home/sk/Desktop/abc/wordcountc/

sudo chmod 777 /home/sk/Desktop/abc/wordcountc

cd /usr/local/hadoop

start-dfs.sh

start-yarn.sh

jps

bin/hdfs dfs -rm -r -skipTrash /user

bin/hdfs dfs -rm -r -skipTrash /opt

bin/hdfs dfs -rm -r -skipTrash /opt2

bin/hdfs dfs -mkdir /user

bin/hdfs dfs -put '/home/sk/Desktop/input' /user

cd /home/sk/Desktop/abc/

javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar -d /home/sk/Desktop/abc*.java

mv /home/sk/Desktop/abc/*.class /home/sk/Desktop/abc/wordcountc/

cd /home/sk/Desktop/abc

jar -cvf wordcountj.jar -C /home/sk/Desktop/abc/wordcountc .

cd /usr/local/hadoop

bin/hadoop jar /home/sk/Desktop/abc/wordcountj.jar OutDegree /user/input outputwc

Let me know where is my mistake so class is not being generated.enter image description here

  • 3
    Mistake one was sending a picture of text. Just copy and paste the text. – DopeGhoti Apr 20 '17 at 20:38
1

The -d flag to javac expects an argument that is a path to a directory.

Oracle says this about this flag:

Set the destination directory for class files. The directory must already exist; javac will not create it. If a class is part of a package, javac puts the class file in a subdirectory reflecting the package name, creating directories as needed.

However, you give the argument /home/sk/Desktop/abc*.java which doesn't look at all like a path to a directory.

Did you mean /home/sk/Desktop/? Or do you actually need the -d flag at all? It's possible that you don't and that abc*.java are the Java source files that you'd like to compile.

Alternatively, you want -d /home/sk/Desktop/ /home/sk/Desktop/abc*.java.

  • Applied your fix but it don't work either. And I have a java file named "OutDegree.java" placed in a folder "abc" which is present on desktop. – Hasan Iqbal Apr 20 '17 at 21:57
  • @HasanIqbal Aha, so abc is a folder? Then why do you say abc*.java? I thought you had file whose names started with abc and ended in .java which is what abc*.java will match. – Kusalananda Apr 20 '17 at 22:52
1

The javac command in your script is not working. This is indicated by the Usage: javac <options> <source files> message in your screen shot. This in turn caused the mv command to fail because no java class files were generated to move. Which in turn caused the class loader to fail and generate an error stack.

To fix this you need to change the javac command from:
javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar -d /home/sk/Desktop/abc*.java

To:
javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar -d /home/sk/Desktop/abc/ /home/sk/Desktop/abc/*.java

This will compile all the java files in /home/sk/Desktop/abc and create a corresponding class file for each java file.

This may be dangerous as in the future you may put more java files into this directory than just the word count file you are currently working with.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.