Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [hadoop]

Hadoop provides High Availability services by distributing processing of large data sets across clusters of machines.

0
votes
1answer
43 views

bash: pig: command not found

I am trying to find out what version of pig I am using. I thought I already installed it # yum install hadoop\* mahout\* oozie\* hbase\* hive\* hue\* pig\* zookeeper\* When I try to enter a pig ...
0
votes
0answers
50 views

How to copy files from HDFS recursive to the local file system

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder? Example export hdfs_folder=/app/lib export local_folder=/home/work_app while under /app/lib we ...
0
votes
1answer
23 views

how to run CLI from root with user hdfs

when I run the following cli from hdfs user its running well # su hdfs $ hadoop fs -du -s /home/test/* | awk '{ sum += $1 } END { print sum }' 4182692 but when I run it from root , ...
0
votes
0answers
59 views

How to copy files recursive from HDFS to a local folder?

The simple way to copy a folder from HDFS to a local folder is like this: su hdfs -c 'hadoop fs -copyToLocal /hdp /tmp' In the example above, we copy the hdp folder from HDFS to /tmp/local_folder ...
2
votes
3answers
174 views

how to know the right swap size for hadoop Linux machines

we are using many hadoop clusters for now we are using swap size - 16G free -g total used free shared buff/cache available Mem: 125 ...
2
votes
1answer
113 views

why sometime need to stop process by kill -9

we have kafka machines in hadoop cluster the script that stop the kafka process do the following kill PID but we notice that the script that stop the kafka not really kill the process therefore we ...
1
vote
1answer
131 views

how to manage cleaning of /tmp better on hadoop machines

As all know the content of /tmp should be deleted after some time. In my case we have machines ( redhat version 7.2 ) that are configured as following. As we can see the service that is triggered ...
1
vote
0answers
27 views

How to create empty dataframe? [closed]

I want too create empty dataframe in spark 2.3?? i was trying with sparkContext it goes well but with sparksession it is not happening.
0
votes
2answers
30 views

How to sum value of specified column by specific date in kornshell?

I'm working in a unique validation framework that validates data. For each validation job there is a SQL job with an accompanying KSH job (kornshell). The SQL queries something in the database, and ...
1
vote
0answers
30 views

how to move files based on birth time in hadoop [closed]

I have some 200k files in a folder, i want to organize them based on time of birth(creation time). i wrote the below script but it is way too slow. i want to improve it. how can i do it? #!/usr/bin/...
0
votes
1answer
80 views

Hadoop : find the hostname or IP address based on the process id

Is it possible to find the IP address or hostname of who submited a job based on the processid ? We have some hadoop jobs running for hundreds of hours. We need to know from which local machine it ...
1
vote
1answer
2k views

localhost: Permission denied (publickey,password,keyboard-interactive)

I was trying to run Hadoop in Mac OS and I get the following errors, $ hstart WARNING: Attempting to start all Apache Hadoop daemons as chaklader in 10 seconds. WARNING: This is not a recommended ...
1
vote
1answer
184 views

Not able to exit from interactive mode for yarn top command

I have a bash script which is on serverA. This script will ssh to serverB and runs yarn top command, pulls the metrics and put into the file(test.txt) on serverA. Below is the command which I am using:...
0
votes
1answer
42 views

Can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
0
votes
0answers
121 views

jumbo frame + fine tune

we have hadoop cluster ( machines are linux redhat 7 ) ,and we set jumbo frame according to documentation first we set MTU=9000 on all nodes in the cluster but we notice about some problems , so ...
0
votes
1answer
406 views

how to find the right value of MTU Jumbo frame [closed]

We've made the decision to set jumbo frames on all our Linux machines. We have a hadoop cluster with master machines, workers machines and Kafka machines. Our switches (Cisco) are suitable to Jumbo ...
0
votes
1answer
131 views

install hadoop_2_6_1_0_129-hdfs

tried to install Hadoop cluster App Timeline Server Install returned error: 2018-02-26 19:31:49,406 - Installing package hadoop_2_6_1_0_129-hdfs ('/usr/bin/yum -d 0 -e 0 -y install ...
0
votes
1answer
370 views

systemd - define a service without ExecStop and be able to stop it without “fail”

I am with CentOS 7, and I want to start Kafka standalone producer (File Connector) as a service. The command is: /opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties /...
1
vote
0answers
103 views

jps process issue about installing Hadoop

I'm trying to install hadoop with fully distributed mode in Centos6.4 (I use 4 Virtual boxes). server1 NameNode server2 SecondaryNameNode, Datanode server3 datanode server4 datanode Maybe I guess I'...
1
vote
0answers
218 views

Hadoop cluster not listening on port that I configured. What is wrong?

I set up a Hadoop cluster with RHEL 7.4 servers. There is no firewall between them. I am running Hadoop 3.0. On the namenode the core-site.xml file is configured to use port 54310. I run this ...
2
votes
0answers
285 views

The command “hdfs dfsadmin -report” fails because “failed to connect to server”

I am trying to configure a multi-node cluster of open source Hadoop. I have Hadoop 3.0 installed on the namenode and the data node. Both are running Linux (SUSE and Ubuntu). None are CentOS, RedHat ...
3
votes
0answers
1k views

How do you get Hadoop commands to work when you get the error “Invalid HADOOP_COMMON_HOME”?

I had a Hadoop version 1.x installed on Linux SUSE 12.3. I moved the directory somewhere else to back it up. I tried to install Hadoop 3.0. I expect Hadoop commands to work based on what I did. I ...
0
votes
1answer
55 views

Hadoop cluster with Ambari - best way to modify configurations

We have an Hadoop cluster, clients are installed on Linux Red Hat machines and central management via Apache Ambari. yum list | grep ambari-server ambari-server.x86_64 2.5.0.3-7 ...
0
votes
1answer
101 views

Ambari and Spark cant start from CLI

From the Ambari GUI we can not start the Spark service. So we want to start it by command line as the following: [spark@mas01 spark2]$ ./sbin/start-thriftserver.sh --master yarn-client --executor-...
0
votes
1answer
1k views

Passing inline arguments to shell script being executed on HDFS

I am running a shell script stored on HDFS (so that it can be recognized by my oozie workflow). to run this script I am using hadoop fs -cat script.sh |exec sh However I need to pass inline ...
1
vote
0answers
146 views

How to connect Kerberos with multiple LDAP servers?

My actual task is to make our kerberized Hadoop cluster usable by all our teams. Right now we have a very weird setup in our company: The Hadoop cluster has a dedicated KDC (openSUSE Kerberos with ...
3
votes
1answer
18k views

RPC: Port mapper failure - Unable to receive: errno 113 (No route to host)

I am trying to mount hdfs on my local machine(ubuntu) using nfs by following the below link:-- https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cdh_ig_nfsv3_gateway_configure.html#...
1
vote
1answer
1k views

mount.nfs: mount system call failed

I am trying to mount hdfs on my local machine running Ubuntu using the following command :--- sudo mount -t nfs -o vers=3,proto=tcp,nolock 192.168.170.52:/ /mnt/hdfs_mount/ But I am getting this ...
0
votes
1answer
370 views

How to upload file into File View on Ambari Web?

I am using HDP 2.6 and ambari 2.5. on a 5 node cluster. The cluster was setup with vagrant following these instructions https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+for+New+VM+Users ...
1
vote
1answer
2k views

Installing Oracle JDK 1.7 — 404 error

I'm trying to install Hadoop 2.7.3. on Elementary OS (which ~ Ubuntu, I believe) following the instructions in the BUILDING.txt that came with the Hadoop files. The file indicates that I need to ...
-1
votes
2answers
134 views

Linux Hadoop shell script giving .class error

I am trying to run this script on for running map reduce on hadoop. But when I run this script, it is giving me the error attached in the screen shot. Script: rm -rf /home/sk/Desktop/abc/wordcountc/ ...
5
votes
2answers
285 views

LD_LIBRARY_PATH lost when using mount command

TL;DR When a fuse filesystem is mounted via the mount command, the environment variables are not passed to the fuse script. Why? Context I am trying to mount hdfs (hadoop file system) via fuse. ...
0
votes
1answer
63 views

How do I install an .rpm that fails with an error about an .so file not being found from Python 2.6?

When I try to install cloudera-manager-agent 5.7 from an .rpm I get an error. The error says that a dependency has not been met because yum could not find libpython2.6.so.1.0(64bit). I would expect ...
-1
votes
2answers
1k views

CentOS7 SSH can't connect localhost?

I'm configuring Hadoop environment. I have use $ ssh-keygen -t rsa -P "" to generate id_rsa.pub and id_rsa. And use cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys to set password-free login. ...
2
votes
1answer
764 views

False error starting Hue

I have installed Hue in CentOS 7 from Cloudera CDH5 repository. Upon starting it reports an error: # systemctl status hue hue.service - SYSV: Hue web server Loaded: loaded (/etc/rc.d/init.d/hue) ...
1
vote
2answers
2k views

Delete files 10 days older from hdfs

I am writing a ksh script to clean up hdfs directories and files at least 10 days old. I am testing the deletion command in a terminal, but it kept saying it is wrong: $ hdfs dfs -find "/file/path/...
0
votes
3answers
911 views

How to split the date range into days using script

I have this input: startdate end date val1 val2 2015-10-13 07:00:02 2015-10-19 00:00:00 45 1900 in which one line specifies a date range that spans multiple ...
1
vote
1answer
108 views

Which OS should I use for Hadoop cluster?

I have a client setting up a Hadoop cluster. We have all used and are very familiar with CentOS 7. I was told Scientific Linux maybe better optimized for Hadoop. Is there any truth to that?
0
votes
1answer
51 views

How to move a directory?

I have a directory in HDFS with subdirectories that contain part-xxxxx files, created by Spark. I want to move that directory (and everything inside it) into a new directory. How to? My attempt: [...
2
votes
1answer
5k views

Change column datatypes in Hive database

Can I change the datatype in Hive database? Below is a complete information about the same. I have a database named "test". It has a table "name". Below is a query I had used while creating a column ...
0
votes
1answer
2k views

copyFromLocal Hadoop No such file or directory error

I have the following code from the Fedora command terminal : [root@localhost python_code]# pwd /root/python_code [root@localhost python_code]# cd .. [root@localhost ~]# ls anaconda-ks.cfg ...
0
votes
1answer
65 views

Development setup on linux

On a linux system should we install and use our development tools and setup like java ee and hadoop as a root user or as user.
0
votes
1answer
2k views

mv: target 'hadoop/' is not a directory

I am using Fedora 23 which is installed on Oracle VM. I am at the Downloading Hadoop step of the documentation here. When I run the following command in Fedora, I can see the hadoop user listed: [...
1
vote
1answer
805 views

Unexpected EOF while looking for matching ' " '

I'm trying to configure hadoop using cygwin.. It is my first time to use unix environment but while configuring hadoop-env.sh i get the this EOF error # Licensed to the Apache Software Foundation (...
1
vote
1answer
101 views

sqoop2 fail to start

My sqoop2 suddenly die and unable to start since 2 days ago, I have not changed any settings on sqoop2, I checked the /var/log/sqoop2 catalina log it says: Mar 14, 2016 2:06:17 PM org.apache.catalina....
0
votes
1answer
714 views

bash: /home/hadoopuser/hadoop/sbin: Is a directory

I am in Ubuntu 14.04 LTS. Based on how to set a Hadoop cluster I am executing this: # Add Hadoop bin and sbin directory to PATH export PATH=$PATH:$HADOOP_HOME/bin;$HADOOP_HOME/sbin where I know that ...
2
votes
1answer
1k views

Validate start-dfs.sh

I am trying to setup a Hadoop cluster, where master is my laptop and slave is the virtualbox, following this guide. So, I did, from master: gsamaras@gsamaras:/home/hadoopuser/hadoop/sbin$ sudo ./...
0
votes
1answer
8k views

Make virtualbox get a different IP address than my laptop

I am trying to setup a 2-node Hadoop cluster (following this guide), where the master is going to be my Ubuntu laptop and the slave a virtualbox running ubuntu. I also read about networking in ...
1
vote
1answer
90 views

Connect Symbolic Link to program or script that returns a file

I wanted to know if I can make a symlink to a file say in a Database. This is a wild guess, but is it possible to intercept the link-follow say using a C function and then let function return the ...
3
votes
3answers
2k views

bind failure, address in use: Unable to use a TCP port for both source and destination?

I'm debugging Hadoop DataNodes that won't start. We are using saltstack and also elasticsearch on the machines. The Hadoop DataNode error is pretty clear: java.net.BindException: Problem binding ...