0

I have a lot of .gz files in a folder

/a/b/c1.gz
/a/b/c2.gz
/a/b/c3.gz

and so on.

Some of the files have a single pipe delimiter, some have two, three and four and so on, in such a way:

xyz|abc
xyz|abc|wty
xyz|abc|wty|asd

and so on.

How do I find all the files that have two pipe delimiters overall, three delimiters overall etc ?

1
  • thank you @steeldriver
    – Sicario007
    May 14 '20 at 2:05
1

Assuming that in any given file, the number of |-delimited columns is constant, then it's enough to inspect the first line of a file to determine the number of columns in it.

The following will do that for a file called name.gz:

gzip -dc name.gz | awk -F '|' -v name="name.gz" '{ print NF, name } { exit }'

So, with a simple loop, you would be able to output the number of columns and the filenames of, e.g., all files matching the pattern /a/b/c*.gz:

for name in /a/b/c*.gz; do
    gzip -dc "$name" |
    awk -F '|' -v name="$name" '{ print NF, name } { exit }'
done

If you want to only output the names of the files with a certain number of columns (n=3, for example), then use

n=3
for name in /a/b/c*.gz; do
    gzip -dc "$name" |
    awk -F '|' -v n="$n" -v name="$name" 'NF == n { print name } { exit }'
done
0

Let's create three test files:

echo 'xyz|abc' > c1
echo 'xyz|abc|wty' > c2
echo 'xyz|abc|wty|asd' > c3
gzip c*

Files containing one pipe in a line:

$ zgrep '^[^|]*|[^|]*$' *.gz
c1.gz:xyz|abc

For any other numbers (including one pipe in a line), you can use the following pattern:

Two pipes in a line:

$ zgrep -E '^([^|]*\|){2}[^|]*$' *.gz
c2.gz:xyz|abc|wty

Three pipes in a line:

$ zgrep -E '^([^|]*\|){3}[^|]*$' *.gz
c3.gz:xyz|abc|wty|asd

Two or three pipes in a line:

$ zgrep -E '^([^|]*\|){2,3}[^|]*$' *.gz
c2.gz:xyz|abc|wty
c3.gz:xyz|abc|wty|asd

Max. three pipes in a line:

$ zgrep -E '^([^|]*\|){,3}[^|]*$' *.gz
c1.gz:xyz|abc
c2.gz:xyz|abc|wty
c3.gz:xyz|abc|wty|asd

If you only need the filename, add option -l, i.e. zgrep -lE ...


My zgrep version doesn't support the recursive -r option.

You could use find for a recusive search and run zgrep on the result:

$ find . -type f -name '*.gz' -exec zgrep -lE '^([^|]*\|){3}[^|]*$' {} \;
./c3.gz
0

You can pipe file names to awk and find number of |-s in each. For example: echo 'A|B|C' |awk -F\| '{print NF-1}'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.