bash + how to append path to the end of the line according to existing path

Question

The following configuration file (example 1) isn't configured as it should be.

Each line in file should contain the /grid/sdX (a to z) as described in example 2.

I need to find a way to write a bash script for this task. How to append the missing /grid/sdX in the end of the lines?

example 1

more dfs_data_dir_mount.hist


/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

expected results (example 2)

/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg

Answer 1

sed solution:

sed -Ei 's~^(/[^/]+/[^/]+)(.*,)/$~\1\2\1~' dfs_data_dir_mount.hist

• ~ - treated as sed subcommand separator
• [^/]+ - match one or more character(s) except slash /
• ^ $ - are the start and end of the line respectively

Answer 2

I was able to accomplish this with the following awk command:

awk -F'/' '{OFS="/";}{print $1,$2,$3,$4,$5,$6,"grid",$3}' input

awk

• -F'/' - Delimit input by /
• {OFS="/";} - Delimit output by /
• {print $1,$2,$3,$4,$5,$6,"grid",$3}' - Print /grid/sd*/hadoop/hdfs/data,/ (fields taken from input) and grid/sd* (manually inputting grid and adding field 3 again)

Answer 3

awk:

awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1'

• -F/ -v OFS='/' sets the input and output field separator as /
• !$NF {$0=$0 $2 OFS $3}; 1 if the last field is empty, we're rebuiling the record in the desired format. 1 is truthy in awk and is to print the records.

Example:

% cat file.txt 
/grid/sdk/hadoop/hdfs/data,/
/grid/sdi/hadoop/hdfs/data,/
/grid/sdh/hadoop/hdfs/data,/
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/

% awk -F/ -v OFS='/' '!$NF {$0=$0 $2 OFS $3}; 1' file.txt
/grid/sdk/hadoop/hdfs/data,/grid/sdk
/grid/sdi/hadoop/hdfs/data,/grid/sdi
/grid/sdh/hadoop/hdfs/data,/grid/sdh
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde
/grid/sdd/hadoop/hdfs/data,/grid/sdd
/grid/sdb/hadoop/hdfs/data,/grid/sdb
/grid/sdf/hadoop/hdfs/data,/grid/sdf
/grid/sdg/hadoop/hdfs/data,/grid/sdg

Answer 4

Perl oneliner:

perl -i.bak -pe 's;^/(grid/sd.)/hadoop/hdfs/data,/\K$;$1;' input

Match the string, picking up the grid/sdX from the start on the way, then forget (\K) the main part of the string, but still match the end of line $, which is then replaced by the part captured by the parenthesis and available in $1.

-i.bak makes the changes in-place, and saves the original file with the extension .bak.

$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/
/grid/sde/hadoop/hdfs/data,/grid/sde
$ perl -i.bak  -pe 's:^/(grid/sd.)/hadoop/hdfs/data,/\K$:$1:' input
$ cat input
/grid/sdc/hadoop/hdfs/data,/grid/sdc
/grid/sdj/hadoop/hdfs/data,/grid/sdj
/grid/sde/hadoop/hdfs/data,/grid/sde

Answer 5

Alternative (field-based) perl approach using the .= string concatenation operator:

perl -F/ -lpe '$_ .= "$F[1]/$F[2]" unless defined $F[7]' dfs_data_dir_mount.hist