-3

Could you help me for making scripting and good scripting please ? I am done, i am really stuck... no errors when i set -x. just... i dunno what i call that is. I am newcomer in scripting bash shell... So, let me provide some piece of my scripts...

#!/bin/bash
export PATH=$PATH
set -x
Years=$(date +"%Y")
Months=$(date +"%m")
Days=$(date +"%d")

MAINS=/home/usr_engineer/url_prj

CKA=/data/disks1/url_log/
JT2=/data/disks2/url_log/
BKS=/data/disks3/url_log/
SLP=/data/disks4/url_log/
KBB=/data/disks5/url_log/
BOO=/data/disks6/url_log/
GBL=/data/disks7/url_log/
HDFS=/data/landing/mrs/url_log

kinit -kt /home/usr_engineer/usr_engineer.keytab usr_engineer

fCKA1() {
    hadoop fs -put $i $HDFS/cka-np1p/$Years/$Months
    sleep 1
    IFS=' '
    while read names rows sizes; do
        echo '`[CKA]`' > $MAINS/logs/cka1
        echo $names >> $MAINS/logs/cka1
        echo $rows | sed ':a;s/\B[0-9]\{3\}\>/.&/;ta' >> $MAINS/logs/cka1
        echo $(awk 'BEGIN {printf "%.2f GB\n",'$sizes'/1073741824}') >> $MAINS/logs/cka1
    done < $i.ctrl
    echo "[Hadoop Metadata]" >> $MAINS/logs/cka1
    Sizes_Tele=$(hadoop fs -ls $HDFS/cka-np1p/$Years/$Months/$i | awk '{print $5}')
    Sizes_Telec=$(awk 'BEGIN {printf "%.2f GB\n",'$Sizes_Tele'/1073741824}')
    echo "$i ($Sizes_Telec)" >> $MAINS/logs/cka1
    cat $MAINS/logs/cka1 | telegram-send --stdin --format markdown
    rm -rf $i $i.ctrl
}

fCKA2() {
    hadoop fs -put $i $HDFS/cka-np2p/$Years/$Months
    sleep 1
    IFS=' '
    while read names rows sizes; do
        echo '`[CKA]`' > $MAINS/logs/cka2
        echo $names >> $MAINS/logs/cka2
        echo $rows | sed ':a;s/\B[0-9]\{3\}\>/.&/;ta' >> $MAINS/logs/cka2
        echo $(awk 'BEGIN {printf "%.2f GB\n",'$sizes'/1073741824}') >> $MAINS/logs/cka2
    done < $i.ctrl
    echo "[Hadoop Metadata]" >> $MAINS/logs/cka2
    Sizes_Tele=$(hadoop fs -ls $HDFS/cka-np2p/$Years/$Months/$i | awk '{print $5}')
    Sizes_Telec=$(awk 'BEGIN {printf "%.2f GB\n",'$Sizes_Tele'/1073741824}')
    echo "$i ($Sizes_Telec)" >> $MAINS/logs/cka2
    cat $MAINS/logs/cka2 | telegram-send --stdin --format markdown
    rm -rf $i $i.ctrl
}

fJT21() {
    hadoop fs -put $i $HDFS/jt2-np1p/$Years/$Months
    sleep 1
    IFS=' '
    while read names rows sizes; do
        echo '`[JT2]`' > $MAINS/logs/jt21
        echo $names >> $MAINS/logs/jt21
        echo $rows | sed ':a;s/\B[0-9]\{3\}\>/.&/;ta' >> $MAINS/logs/jt21
        echo $(awk 'BEGIN {printf "%.2f GB\n",'$sizes'/1073741824}') >> $MAINS/logs/jt21
    done < $i.ctrl
    echo "[Hadoop Metadata]" >> $MAINS/logs/jt21
    Sizes_Tele=$(hadoop fs -ls $HDFS/jt2-np1p/$Years/$Months/$i | awk '{print $5}')
    Sizes_Telec=$(awk 'BEGIN {printf "%.2f GB\n",'$Sizes_Tele'/1073741824}')
    echo "$i ($Sizes_Telec)" >> $MAINS/logs/jt21
    cat $MAINS/logs/jt21 | telegram-send --stdin --format markdown
    rm -rf $i $i.ctrl
}

fJT22() {
    hadoop fs -put $i $HDFS/jt2-np2p/$Years/$Months
    sleep 1
    IFS=' '
    while read names rows sizes; do
        echo '`[JT2]`' > $MAINS/logs/jt22
        echo $names >> $MAINS/logs/jt22
        echo $rows | sed ':a;s/\B[0-9]\{3\}\>/.&/;ta' >> $MAINS/logs/jt22
        echo $(awk 'BEGIN {printf "%.2f GB\n",'$sizes'/1073741824}') >> $MAINS/logs/jt22
    done < $i.ctrl
    echo "[Hadoop Metadata]" >> $MAINS/logs/jt22
    Sizes_Tele=$(hadoop fs -ls $HDFS/jt2-np2p/$Years/$Months/$i | awk '{print $5}')
    Sizes_Telec=$(awk 'BEGIN {printf "%.2f GB\n",'$Sizes_Tele'/1073741824}')
    echo "$i ($Sizes_Telec)" >> $MAINS/logs/jt22
    cat $MAINS/logs/jt22 | telegram-send --stdin --format markdown
    rm -rf $i $i.ctrl
}

sleep 2
cd $CKA
sleep 2
for i in $(ls -lh $CKA | grep -v .ctrl | grep url | awk '{print $9}');do
    echo $i | grep cka-np1p
    if [ $? -eq 0 ]; then
        fCKA1
    else
        echo $i | grep cka-np2p
        if [ $? -eq 0 ]; then
            fCKA2
        fi
    fi
done

sleep 2
cd $JT2
sleep 2
for i in $(ls -lh $JT2 | grep -v .ctrl | grep url | awk '{print $9}');do
    echo $i | grep jt2-np1p
    if [ $? -eq 0 ]; then
        fJT21
    else
        echo $i | grep jt2-np2p
        if [ $? -eq 0 ]; then
            fJT22
        fi
    fi
done

I save and run that script using this command to save log

nohup bash name.sh > name.log 2>&1 &

and the results here https://pastebin.com/12yhttgG

I have added sleep, changed for, grep models But... if i separate that script such as cka.sh, jt2.sh and run those in the same time, the script would not be broken. If you see the pastebin link, the error started on 181st line. After that line, it should run hadoop command :'(

OMG... i spent 6 hours for this... pls pls help me... teamviewer or anything to solve this very welcome

  • Part of the question is off site. Put it all in the question. Also reduce code to include only the problem. (You may solve it this way). But as it is there is too much to look through. – ctrl-alt-delor Mar 22 '18 at 9:04
1

So it seems the problem you're seeing is that you're running a for i in $(ls ...), expecting to see one file at a time, and instead just getting one entry with the whole list of files in a single multi-line string.

This is caused by the IFS=' ' assignments you have in your functions. They set space as the only separator, which means newline will no longer be considered one (and it was needed for that ls to split into multiple lines.)

As you're setting IFS inside fCKA1 and fCKA2, you end up seeing the problem on the second loop, since it runs after those functions.

You can work around that by, for instance, saving the original IFS before you set it and restoring it at the end of the function.

For example:

fCKA1() {
    hadoop fs -put $i $HDFS/cka-np1p/$Years/$Months
    sleep 1
    save_IFS=$IFS
    IFS=' '
    while read names rows sizes; do
        ...
    rm -rf $i $i.ctrl
    IFS=$save_IFS
}

Perhaps better, you can set it only for the "read" command like so:

IFS=' ' read names rows sizes

In the context of your function, what you get is:

fCKA1() {
    hadoop fs -put $i $HDFS/cka-np1p/$Years/$Months
    sleep 1
    while IFS=' ' read names rows sizes; do
        ...
    rm -rf $i $i.ctrl
}

There's also the question of whether you need to set IFS at all... The default is to split on whitespace, any whitespace... Do you really need to split on space and space alone? Try removing the IFS=' ' settings altogether, it might just work for you just as well!

  • Since this is bash, better to use local IFS=... instead of saving and resetting, since an empty IFS is not the same as an unset IFS. – muru Mar 22 '18 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.