Questions tagged [hadoop]

Hadoop provides High Availability services by distributing processing of large data sets across clusters of machines.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Find command only showing one match

I'm looking for files with .txt, but all I'm getting is a single file, Though I have multiple files with .txt extension and those are not coming in the result. Command : find . -name *.txt.
-1
votes
0answers
8 views

Hadoop services not starting (atleast not all) after rebooting Kali

I have installed Hadoop in Kali in VirtualBox. I have installed it as root user that I regularly use. After installing I was able to manually start all daemons one by one but after I restarted Kali ...
0
votes
0answers
13 views

Hadoop auto start up using init.d - not starting after boot

I have following startup script /etc/init.d/hadoop #!/bin/bash # # # Starts a Hadoop Master # # chkconfig: 2345 90 10 # description: Hadoop master source /lib/lsb/init-functions export HADOOP_HOME=/...
0
votes
0answers
89 views

Disabling Transparent Hugepages (THP) what is the diff between two options

we need advice about the following what is the difference between the two approach of Disabling Transparent Huge pages we have hadoop cluster , and hadoop recommendation to disable the t`ransparent ...
0
votes
0answers
31 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
0
votes
0answers
29 views

how to verify HDFS version on hadoop cluster

there are many types of hadoop clusters some of them are installed from hortronworks kits and some of them are from cloudera kits and so on . how to identify the HDFS version? second , how to get ...
0
votes
0answers
28 views

Using SSH for hadoop nodes connection

We are trying to set-up a two-nodes hadoop clusters on two different machines. SSH to localhost without password have been setted up on both machines. And hadoop can run well as single node on each ...
1
vote
0answers
391 views

ssh: connect to host localhost port 22: Connection refused

I have installed hadoop and ssh. hadoop was working fine, then today I am getting the error below when I run the command sbin/start-dfs.sh: Starting namenodes on [localhost] localhost: ssh: connect ...
1
vote
0answers
86 views

master: ssh: connect to host master port 22: Connection refused

i am trying to start my hadoop cluster using the command "start-dfs.sh" but am getting errors as shown below Starting namenodes on [master] master: ssh: connect to host master port 22: Connection ...
0
votes
0answers
35 views

curl is stuck when trying to get NameNodeStatus

we have two namenode in the cluster hadoop cluster this is good example of one of the name node that return good status first namenode machine IP - 92.3.44.2 curl -v http://92.3.44.2:50070/jmx?qry=...
-2
votes
1answer
30 views

Hadoop cluster + designing number of disks on data node machine and min requirements

we are using HDP version - 2.6.5 , and HDFS Block replication is 3 we are try to understand data nodes disks min requirements for production mode and according to the fact that Block replication=3 ...
0
votes
0answers
9 views

centOS 7 access denied when logging on kerberized environment

When I am trying to log in to one of my hadoop nodes running centos 7. I am receiving a 'access denied' error. Typically, in these situations, I login as the local admin for the box, go to /tmp and ...
0
votes
0answers
19 views

how HDFS balance the data on the disks

we want to know how HDFS balance the data on disks with different size lets say we have 2 data-node machines one data node have disk sdb with 100G and the second data node have disk with 10G what ...
1
vote
1answer
64 views

What is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
0
votes
0answers
26 views

How to check disks health in Hadoop clusters and Kafka machines

We have many Hadoop clusters. Each datanode holds many disks (between 10 and 20). All disks are used for HDFS. We use well-known tools like smartctl to check the disks. We want to know if there is a ...
0
votes
1answer
123 views

Ansible Gather facts failing at findmnt command for some hosts

ANSIBLE VERSION ansible 2.4.6.0 config file = /home/xxxxxx/ansible.cfg configured module search path = [u'/home/xxxxxx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python ...
1
vote
1answer
837 views

Copy files from a hdfs folder to another hdfs location by filtering with modified date using shell script

I have 1 year data in my hdfs location and i want to copy data for last 6 months into another hdfs location. Is it possible to copy data only for 6 months directly from hdfs command or do we need to ...
1
vote
0answers
166 views

Unable to reach Hortonworks HDP on port 8080:This site can't be reached

My goal is to run HDP on VM VirtualBox Image shows my port forwarding rules When I try to launch dashboard My host machine is 18.04 Ubuntu. Port mapping output 8000/tcp open http-alt 8042/tcp ...
0
votes
1answer
409 views

bash: pig: command not found

I am trying to find out what version of pig I am using. I thought I already installed it # yum install hadoop\* mahout\* oozie\* hbase\* hive\* hue\* pig\* zookeeper\* When I try to enter a pig ...
0
votes
1answer
281 views

how to run CLI from root with user hdfs

when I run the following cli from hdfs user its running well # su hdfs $ hadoop fs -du -s /home/test/* | awk '{ sum += $1 } END { print sum }' 4182692 but when I run it from root , ...
1
vote
3answers
683 views

how to know the right swap size for hadoop Linux machines

we are using many hadoop clusters for now we are using swap size - 16G free -g total used free shared buff/cache available Mem: 125 ...
1
vote
1answer
445 views

why sometime need to stop process by kill -9

we have kafka machines in hadoop cluster the script that stop the kafka process do the following kill PID but we notice that the script that stop the kafka not really kill the process therefore we ...
1
vote
1answer
752 views

how to manage cleaning of /tmp better on hadoop machines

As all know the content of /tmp should be deleted after some time. In my case we have machines ( redhat version 7.2 ) that are configured as following. As we can see the service that is triggered ...
0
votes
2answers
39 views

How to sum value of specified column by specific date in kornshell?

I'm working in a unique validation framework that validates data. For each validation job there is a SQL job with an accompanying KSH job (kornshell). The SQL queries something in the database, and ...
0
votes
1answer
341 views

Hadoop : find the hostname or IP address based on the process id

Is it possible to find the IP address or hostname of who submited a job based on the processid ? We have some hadoop jobs running for hundreds of hours. We need to know from which local machine it ...
1
vote
1answer
6k views

localhost: Permission denied (publickey,password,keyboard-interactive)

I was trying to run Hadoop in Mac OS and I get the following errors, $ hstart WARNING: Attempting to start all Apache Hadoop daemons as chaklader in 10 seconds. WARNING: This is not a recommended ...
1
vote
1answer
434 views

Not able to exit from interactive mode for yarn top command

I have a bash script which is on serverA. This script will ssh to serverB and runs yarn top command, pulls the metrics and put into the file(test.txt) on serverA. Below is the command which I am using:...
0
votes
1answer
74 views

Can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
0
votes
0answers
153 views

jumbo frame + fine tune

we have hadoop cluster ( machines are linux redhat 7 ) ,and we set jumbo frame according to documentation first we set MTU=9000 on all nodes in the cluster but we notice about some problems , so ...
0
votes
1answer
639 views

how to find the right value of MTU Jumbo frame [closed]

We've made the decision to set jumbo frames on all our Linux machines. We have a hadoop cluster with master machines, workers machines and Kafka machines. Our switches (Cisco) are suitable to Jumbo ...
0
votes
1answer
255 views

install hadoop_2_6_1_0_129-hdfs

tried to install Hadoop cluster App Timeline Server Install returned error: 2018-02-26 19:31:49,406 - Installing package hadoop_2_6_1_0_129-hdfs ('/usr/bin/yum -d 0 -e 0 -y install ...
0
votes
1answer
891 views

systemd - define a service without ExecStop and be able to stop it without “fail”

I am with CentOS 7, and I want to start Kafka standalone producer (File Connector) as a service. The command is: /opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties /...
1
vote
0answers
133 views

jps process issue about installing Hadoop

I'm trying to install hadoop with fully distributed mode in Centos6.4 (I use 4 Virtual boxes). server1 NameNode server2 SecondaryNameNode, Datanode server3 datanode server4 datanode Maybe I guess I'...
2
votes
0answers
340 views

Hadoop cluster not listening on port that I configured. What is wrong?

I set up a Hadoop cluster with RHEL 7.4 servers. There is no firewall between them. I am running Hadoop 3.0. On the namenode the core-site.xml file is configured to use port 54310. I run this ...
2
votes
0answers
529 views

The command “hdfs dfsadmin -report” fails because “failed to connect to server”

I am trying to configure a multi-node cluster of open source Hadoop. I have Hadoop 3.0 installed on the namenode and the data node. Both are running Linux (SUSE and Ubuntu). None are CentOS, RedHat ...
5
votes
0answers
2k views

How do you get Hadoop commands to work when you get the error “Invalid HADOOP_COMMON_HOME”?

I had a Hadoop version 1.x installed on Linux SUSE 12.3. I moved the directory somewhere else to back it up. I tried to install Hadoop 3.0. I expect Hadoop commands to work based on what I did. I ...
0
votes
1answer
88 views

Hadoop cluster with Ambari - best way to modify configurations

We have an Hadoop cluster, clients are installed on Linux Red Hat machines and central management via Apache Ambari. yum list | grep ambari-server ambari-server.x86_64 2.5.0.3-7 ...
0
votes
1answer
163 views

Ambari and Spark cant start from CLI

From the Ambari GUI we can not start the Spark service. So we want to start it by command line as the following: [spark@mas01 spark2]$ ./sbin/start-thriftserver.sh --master yarn-client --executor-...
0
votes
1answer
2k views

Passing inline arguments to shell script being executed on HDFS

I am running a shell script stored on HDFS (so that it can be recognized by my oozie workflow). to run this script I am using hadoop fs -cat script.sh |exec sh However I need to pass inline ...
1
vote
0answers
198 views

How to connect Kerberos with multiple LDAP servers?

My actual task is to make our kerberized Hadoop cluster usable by all our teams. Right now we have a very weird setup in our company: The Hadoop cluster has a dedicated KDC (openSUSE Kerberos with ...
5
votes
2answers
31k views

RPC: Port mapper failure - Unable to receive: errno 113 (No route to host)

I am trying to mount hdfs on my local machine(ubuntu) using nfs by following the below link:-- https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cdh_ig_nfsv3_gateway_configure.html#...
1
vote
1answer
3k views

mount.nfs: mount system call failed

I am trying to mount hdfs on my local machine running Ubuntu using the following command :--- sudo mount -t nfs -o vers=3,proto=tcp,nolock 192.168.170.52:/ /mnt/hdfs_mount/ But I am getting this ...
0
votes
1answer
561 views

How to upload file into File View on Ambari Web?

I am using HDP 2.6 and ambari 2.5. on a 5 node cluster. The cluster was setup with vagrant following these instructions https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+for+New+VM+Users ...
1
vote
1answer
3k views

Installing Oracle JDK 1.7 — 404 error

I'm trying to install Hadoop 2.7.3. on Elementary OS (which ~ Ubuntu, I believe) following the instructions in the BUILDING.txt that came with the Hadoop files. The file indicates that I need to ...
-1
votes
2answers
233 views

Linux Hadoop shell script giving .class error

I am trying to run this script on for running map reduce on hadoop. But when I run this script, it is giving me the error attached in the screen shot. Script: rm -rf /home/sk/Desktop/abc/wordcountc/ ...
4
votes
2answers
420 views

LD_LIBRARY_PATH lost when using mount command

TL;DR When a fuse filesystem is mounted via the mount command, the environment variables are not passed to the fuse script. Why? Context I am trying to mount hdfs (hadoop file system) via fuse. ...
0
votes
1answer
82 views

How do I install an .rpm that fails with an error about an .so file not being found from Python 2.6?

When I try to install cloudera-manager-agent 5.7 from an .rpm I get an error. The error says that a dependency has not been met because yum could not find libpython2.6.so.1.0(64bit). I would expect ...
-1
votes
2answers
1k views

CentOS7 SSH can't connect localhost?

I'm configuring Hadoop environment. I have use $ ssh-keygen -t rsa -P "" to generate id_rsa.pub and id_rsa. And use cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys to set password-free login. ...
2
votes
1answer
1k views

False error starting Hue

I have installed Hue in CentOS 7 from Cloudera CDH5 repository. Upon starting it reports an error: # systemctl status hue hue.service - SYSV: Hue web server Loaded: loaded (/etc/rc.d/init.d/hue) ...
1
vote
2answers
3k views

Delete files 10 days older from hdfs

I am writing a ksh script to clean up hdfs directories and files at least 10 days old. I am testing the deletion command in a terminal, but it kept saying it is wrong: $ hdfs dfs -find "/file/path/...