5

The following configuration file (example 1) isn't configured as it should be.

Each line in file should contain the /grid/sdX (a to z) as described in example 2.

I need to find a way to write a bash script for this task. How to append the missing /grid/sdX in the end of the lines?

example 1

more dfs_data_dir_mount.hist


/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

expected results (example 2)

/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg

5 Answers 5

5

sed solution:

sed -Ei 's~^(/[^/]+/[^/]+)(.*,)/$~\1\2\1~' dfs_data_dir_mount.hist
  • ~ - treated as sed subcommand separator
  • [^/]+ - match one or more character(s) except slash /
  • ^ $ - are the start and end of the line respectively
0
5

I was able to accomplish this with the following awk command:

awk -F'/' '{OFS="/";}{print $1,$2,$3,$4,$5,$6,"grid",$3}' input

awk

  • -F'/' - Delimit input by /
  • {OFS="/";} - Delimit output by /
  • {print $1,$2,$3,$4,$5,$6,"grid",$3}' - Print /grid/sd*/hadoop/hdfs/data,/ (fields taken from input) and grid/sd* (manually inputting grid and adding field 3 again)
2
  • nice but awk cant update the file itself ( as sed -i .... ) , so need to send results to other file and then copy it to the orig file
    – yael
    Commented Dec 31, 2017 at 20:01
  • 1
    @yael yea, although apparently gawk supports -i inplace from v4.1.0 and later.
    – jesse_b
    Commented Dec 31, 2017 at 20:04
5

awk:

awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1'
  • -F/ -v OFS='/' sets the input and output field separator as /
  • !$NF {$0=$0 $2 OFS $3}; 1 if the last field is empty, we're rebuiling the record in the desired format. 1 is truthy in awk and is to print the records.

Example:

% cat file.txt 
/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

% awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1' file.txt
/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg
3

Perl oneliner:

perl -i.bak -pe 's;^/(grid/sd.)/hadoop/hdfs/data,/\K$;$1;' input

Match the string, picking up the grid/sdX from the start on the way, then forget (\K) the main part of the string, but still match the end of line $, which is then replaced by the part captured by the parenthesis and available in $1.

-i.bak makes the changes in-place, and saves the original file with the extension .bak.

$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
$ perl -i.bak  -pe 's:^/(grid/sd.)/hadoop/hdfs/data,/\K$:$1:' input
$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
1

Alternative (field-based) perl approach using the .= string concatenation operator:

perl -F/ -lpe '$_ .= "$F[1]/$F[2]" unless defined $F[7]' dfs_data_dir_mount.hist

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .