5

I am meeting some difficulties when using bash to insert multiple empty lines into a file (called file1) according to an index file(called file2). (these files can be treated as passed variables) the index file (file2) looks like:

-
-
-
M
H
A
-
N
X
X
M
-
-
-
F
G
A
...

the file1 looks like this:

M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

the output should be looks like this:

-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

if file2 deletes the '-', the content as well as the order will always be the same as the first column in file1.

I tried dataframe in Python to deal with it, but it's too slow. So I was wondering how to use bash to figure this out. Thanks!

8
  • 1
    would the letters in the index file / file2 always be in the same order as in the data file / file1? I.e. that there's no need think about reordering any lines? I guess so, since the letters in the data file aren't unique, so the rules for reordering would need to be complicated
    – ilkkachu
    Apr 19 at 13:14
  • 1
    Do you really mean bash or do you just mean use the command line? This is a very bad problem for shell programming, are you OK with solutions using awk or perl or any other tool?
    – terdon
    Apr 19 at 13:30
  • 1
    yes, It's always in the same order @ilkkachu sorry, I thought bash contains awk, sed, grep, etc.@terdon
    – Jiao
    Apr 19 at 13:39
  • 1
    @Jiao, Bash is really just the shell, the tool that implements the command-line where you start other commands. Awk and sed etc. are distinct tools but standard. Don't do text processing with just the shell, that's a bad idea :)
    – ilkkachu
    Apr 19 at 13:58
  • 1
    I see. maybe I can just call them bash or shell tools or specify which tool
    – Jiao
    Apr 19 at 14:10

5 Answers 5

6

Assuming the letters in the index file are always the right ones in the right order (so we can ignore which letter we see), and that the empty lines actually contain the dashes and aren't totally empty, maybe this should work:

$ awk -v datafile=data.txt '$1 == "-" { print "-"; next} { getline < datafile; print }' < index.txt 
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

It reads the index file one line at a time; if the first field there is exactly -, prints that; and otherwise reads and prints a line from the other file. (Which means that if a totally empty line comes up in the index file, it will also go to the next line from the data file.)

9
  • 1
    a stupid question: if I use datafile="$data" it will not work. So, how to pass that variable?
    – Jiao
    Apr 19 at 14:13
  • 1
    @Jiao, hmm, I wonder why not? How exactly are you doing it? awk -v var=something sets the awk variable var to something. So if you have e.g. foo=123; awk -v bar="$foo" 'BEGIN {print bar}', you have the shell variable foo, that gets assigned to the awk variable bar that the awk script then prints. (Note the -v.)
    – ilkkachu
    Apr 19 at 14:15
  • 1
    data=`cat data.txt ` index=`cat index.txt` awk -v datafile="$data" '$1 == "-" { print "-"; next} { getline < datafile; print }' <(echo "$index")| less the output was the content of $index
    – Jiao
    Apr 19 at 14:18
  • 1
    @Jiao, ah, you're loading the whole contents of the files into the $data and $index variables in the shell. Hmm, I guess you can do that, but one usually doesn't. That makes it harder for the awk script to access the other file. You'd need to do essentially the same as the shell does with the process substitution (<(...)). But... do you have to use the variables in between?
    – ilkkachu
    Apr 19 at 14:37
  • 1
    the reality is that I have thousands of files that need to deal with. maybe I can generate them into a tmp folder. Thanks
    – Jiao
    Apr 19 at 14:44
3

Using GNU sed, also assuming that the order of the initial letters is strictly the same in the two file ignoring the - lines, read and insert one line from file1 for each "letter" line of file2:

$ sed -e '/^[A-Z]/{R file1' -e 'd;}' file2
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
2

With file2 being the index file and file1 as the data file, using POSIX sed , using the index file we first generate a sed code that will then be applied to the data file to get the final desired output.

sed '
  1{
    x;s:.*:H;s/.*/-/;x:;x
    i\
x;s/.*/-/;x
    :loop
      /^-/!d;g;n
    b loop
  }
  /^-/s/.*/G/;t
  c\
n
' file2 |sed -f - file1

Output:-

-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0

2

[1] Perl Supply data file on the stdin of Perl program, and take the input from stdin unless the current line begins with a dash.

perl -pe '
  $_ = <STDIN> if !/^-/;
' file2 < file1

[2] GNU sed

sed '
  /^-/!Rfile1
  $!N;//P;D
' file2

[3] Python

python3 -c 'import sys
data_file,index_file = sys.argv[1:]

with open(data_file) as f, open(index_file) as g:
  for l in g:
    var = l if l.startswith("-") else next(f)
    print(var,end="")

' file1 file2

1
for j in $(awk '/-/{print NR}' file2); do sed -i ''$j'i -' file1; done

output

cat file1
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
1
  • 1
    The problem here is that you're running one instance of sed for each empty line added. That's not as bad as reading input line by line and running five instances of cut for every line, but it's still uselessly slow with all the processes being launched (and probably slower than whatever they did with Python dataframe). You could mitigate that by stacking multiple insert commands to the sed script, though I'm not sure how well that would scale. (Also, the quoting in ''$j'i -' seems off, why not just double-quote the whole thing, "${j}i -"?)
    – ilkkachu
    Apr 20 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.