5

The following configuration file (example 1) isn't configured as it should be.

Each line in file should contain the /grid/sdX (a to z) as described in example 2.

I need to find a way to write a bash script for this task. How to append the missing /grid/sdX in the end of the lines?

example 1

more dfs_data_dir_mount.hist


/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

expected results (example 2)

/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg
6

sed solution:

sed -Ei 's~^(/[^/]+/[^/]+)(.*,)/$~\1\2\1~' dfs_data_dir_mount.hist
  • ~ - treated as sed subcommand separator
  • [^/]+ - match one or more character(s) except slash /
  • ^ $ - are the start and end of the line respectively
5

I was able to accomplish this with the following awk command:

awk -F'/' '{OFS="/";}{print $1,$2,$3,$4,$5,$6,"grid",$3}' input

awk

  • -F'/' - Delimit input by /
  • {OFS="/";} - Delimit output by /
  • {print $1,$2,$3,$4,$5,$6,"grid",$3}' - Print /grid/sd*/hadoop/hdfs/data,/ (fields taken from input) and grid/sd* (manually inputting grid and adding field 3 again)
  • nice but awk cant update the file itself ( as sed -i .... ) , so need to send results to other file and then copy it to the orig file – yael Dec 31 '17 at 20:01
  • 1
    @yael yea, although apparently gawk supports -i inplace from v4.1.0 and later. – Jesse_b Dec 31 '17 at 20:04
5

awk:

awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1'
  • -F/ -v OFS='/' sets the input and output field separator as /
  • !$NF {$0=$0 $2 OFS $3}; 1 if the last field is empty, we're rebuiling the record in the desired format. 1 is truthy in awk and is to print the records.

Example:

% cat file.txt 
/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

% awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1' file.txt
/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg
3

Perl oneliner:

perl -i.bak -pe 's;^/(grid/sd.)/hadoop/hdfs/data,/\K$;$1;' input

Match the string, picking up the grid/sdX from the start on the way, then forget (\K) the main part of the string, but still match the end of line $, which is then replaced by the part captured by the parenthesis and available in $1.

-i.bak makes the changes in-place, and saves the original file with the extension .bak.

$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
$ perl -i.bak  -pe 's:^/(grid/sd.)/hadoop/hdfs/data,/\K$:$1:' input
$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
1

Alternative (field-based) perl approach using the .= string concatenation operator:

perl -F/ -lpe '$_ .= "$F[1]/$F[2]" unless defined $F[7]' dfs_data_dir_mount.hist

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.