2

I have a file called path_resume.txt which contains some paths to other files.

Inside path_resume.txt:

./a2111oi/sky130_fd_sc_hd__a2111oi_0.txt
./a2111oi/sky130_fd_sc_hd__a2111oi_2.txt
./a2111oi/sky130_fd_sc_hd__a2111oi_4.txt 

Each path points to another text file that has the same structure as follows:

HEAD
INFO
BEGIN sky130_fd_sc_hd__a2111oi_0
...
...
END BEGIN
END INFO
END HEAD

I am trying to read each of these .txt files from the path_resume.txt, copy all lines between BEGIN and END BEGIN and save incrementally to another file called output.txt:

BEGIN sky130_fd_sc_hd__a2111oi_0
...
...
END BEGIN
BEGIN sky130_fd_sc_hd__a2111oi_2
...
...
END BEGIN
BEGIN sky130_fd_sc_hd__a2111oi_4
...
...
END BEGIN

When I run:

awk '{while((getline a < $0)> 0) print a}' path_resume.txt

I can read each file from the path_resume.txt correctly, but I can't remove the unwanted lines.

When I run:

awk '/BEGIN/{flag=1}/END BEGIN/{flag=0}flag' ./a2111oi/sky130_fd_sc_hd__a2111oi_0.txt
 >> output.txt

I can remove the unwanted lines, however I had to pass the path to the file manually. I do not know how to merge the two comands to achieve my goal. I appreciate any help.

1

5 Answers 5

6

You could build the same logic with the print/no-print flag inside the while(getline) loop. Something like this:

awk '{ while((getline a < $0) > 0) { 
    if (a ~ /BEGIN/) p=1;
    if (p) print a;
    if (a ~ /END BEGIN/) p=0;
} }' path_resume.txt > output.txt

The only thing here is that you need to explicitly use if statements, instead of implicit conditionals like you can on the top level of the AWK script.

Or in Bash, slurp the list of filenames to an array and pass them all to AWK in one go:

readarray -t filenames < path_resume.txt
awk '/BEGIN/ {p=1}; p; /END BEGIN/ {p=0}' "${filenames[@]}" > output.txt

(I put the print action before the END BEGIN check to also print that end delimiter.)

4

Something like the following, untested, will do what you want using any awk:

awk '
    NR == FNR { ARGV[ARGC++]=$0; next }
    $1 == "BEGIN" { f=1 }
    f
    $0 == "END BEGIN" { f=0 }
' path_resume.txt > output.txt

or this might be very slightly faster but see http://awk.freeshell.org/AllAboutGetline so you don't fall into the trap of using getline unnecessarily or incorrectly:

awk '
    BEGIN {
        file = ARGV[--ARGC]
        while ( (getline line < file) > 0 ) {
            ARGV[ARGC++] = line
        }
    }
    $1 == "BEGIN" { f=1 }
    f
    $0 == "END BEGIN" { f=0 }
' path_resume.txt > output.txt
1

Using xargs (and without using double awk to read each line from the file) you can pass all the paths you have in your text file:

xargs awk  '/BEGIN/{flag=1}/END BEGIN/{print;flag=0}flag'  < path_resume.txt > out.txt

I'm assuming the your file path_resume.txt does not contain spaces in any path

In awk I used {print;flag=0} to also print the line END BEGIN.


If you had paths with spaces in their filenames then you can use this:

tr '\n' '\0' < path_resume.txt | xargs -0  awk  '/BEGIN/{flag=1}/END BEGIN/{print;print "";flag=0}flag' > out.txt
10
  • 2
    GNU xargs has xargs -d '\n' to read intact lines, but using tr to allow -0 looks smart!
    – ilkkachu
    Commented Dec 23, 2022 at 20:47
  • 1
    translating newlines to NULs so xargs can process NUL-terminated lines is pointless vs just having xargs process newline-terminated lines. If a file name contained a newline in the middle of it then tr '\n' '\0' would convert that to a NUL and so break the file name so the only way translating newlines to NULs would work would be if none of the file names contained newlines but if that's the case they can just be processed as newline-terminated with xargs -d '\n'. Both -0 and -d are GNU xargs extensions so using -0 instead of -d '\n' wouldn't even be making the script more portable
    – Ed Morton
    Commented Dec 24, 2022 at 1:21
  • 1
    @EdMorton, except that telling xargs to read newline-terminated lines isn't that simple. -d '\n' is only available in GNU xargs, and e.g. -L 1 would tell it to read one line, but it also enables trailing spaces to allow continuing a line to the next, so it doesn't process the lines intact. -0 is more commonly supported (e.g. the BSDs, and Busybox), and the tr solution works with that even if -d isn't available. The fact that the filenames here can't have newlines comes from the problem statement already, so it's hardly an issue with how xargs is used here.
    – ilkkachu
    Commented Dec 24, 2022 at 11:29
  • 1
    @EdgarMagallon how any given command works in it's POSIX mode is different from the POSIX definition of a command, as we'd previously discussed, since even in "POSIX mode" any command can do whatever it likes with functionality that's undefined by POSIX (e.g. the value of $0 in the END section of an awk script) and still be POSIX-compliant.
    – Ed Morton
    Commented Dec 24, 2022 at 12:40
  • 1
    @EdMorton thanks, I think I only read posix manual of some commands to know what options are supported. Because if I read the xargs posix manual the -0 option is not documented so I think that's useful to know if using some option is portable or no. Commented Dec 24, 2022 at 19:03
1

Assuming all pathnames in the input file are properly quoted, concatenate all the listed files together and use sed to extract the parts between lines starting with BEGIN and end with END BEGIN:

xargs cat <path_resume.txt |
sed -n '/^BEGIN/,/^END BEGIN/p' >output.txt

This uses xargs to call cat with the sisted files in path_resume.txt as operands. This creates a single stream of all data concatenated from all the files in the order listed in the input file.

The stream is then filtered by sed which prints all lines in the blocks we're interested in.

One could reformulate the sed expression so that it instead deletes all lines that occur outside of the blocks we're interested in (allowing us to drop the -n option):

xargs cat <path_resume.txt |
sed '/^BEGIN/,/^END BEGIN/!d' >output.txt

You could obviously also call sed directly from xargs:

xargs sed '/^BEGIN/,/^END BEGIN/!d' <path_resume.txt >output.txt

Or, using awk,

xargs awk '/^BEGIN/,/^END BEGIN/' <path_resume.txt >output.txt
1
  • 1
    If the input is properly quoted for xargs, that is. It follows slightly different quoting rules than the shells do, which probably only comes up in really weird cases, but it's annoyingly one more syntax to learn.
    – ilkkachu
    Commented Dec 24, 2022 at 11:35
-1

How about this approach? It's kinda lazy, but it will do the job. You read the lines of path_resume.txt and use every line as a file for awk to work with

for i in `cat path_resume.txt`; do awk '/BEGIN/{flag=1}/END BEGIN/{flag=0}flag' $i >> output; done
3
  • that'll work for "nice" filenames, but it's worth noting that it doesn't take every line as a filename, but every whitespace separated string. E.g. if the file has the single line foo bar, that'll get treated as two filenames. Plus it'll try to expand any glob characters in the file, so better not do that if any filenames can contain any of *?[] at least.
    – ilkkachu
    Commented Dec 23, 2022 at 20:34
  • you might even get away with awk '/BEGIN/{flag=1}/END BEGIN/{flag=0}flag' $(cat path_resume.txt) since awk can take multiple filenames to process. Though if one file is missing the END BEGIN part, the flag would be set from the start of the next file...
    – ilkkachu
    Commented Dec 23, 2022 at 20:34
  • 5
    Emphasizing ilkkachu's comment: don't read lines with "for" Commented Dec 23, 2022 at 20:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .