2

I would like to filter many text files. Each file contain a very long list of numbers. Filtering the files must occur by the number of the consecutive digits within each number.

Example list form one of the files:

Input file data.log:

12365
91738
349874
128152639
1234
7654
08767
1234567

I would like to:

1- filter this list by numbers containing two consecutive digits, the expected output should be:

12365
349874
128152639
1234
7654
08767
1234567

2- filter this list by numbers containing three consecutive digits, the expected output should be:

12365
349874
1234
7654
08767
1234567

3- filter this list by numbers containing four consecutive digits, the expected output should be:

1234
7654
1234567

4- filter this list by numbers containing five consecutive digits, the expected output should be:

1234567

If the order of the consecutive digits in the numbers is small to large (e.g. 1234 ...etc) or large to small (e.g. 54321) should be included in the output.

  • 1
    What is the purpose of this exercise and what have you tried so far? – Kusalananda Sep 22 '18 at 22:01
  • 1
    Honestly, this is homework and I don't know bash :(( – Zahi Sep 22 '18 at 22:02
  • The three and four consecutive digit filter output lists should include 1234567. – agc Sep 24 '18 at 0:39
1

Using grep, tee, and rev, make a tricky little function full of bash-isms:

dqs() { a=${2:-123456789} ; [ "$1" -ge 2 ] &&  
        grep -iF "$(eval eval printf '%s\\\\n' \\$\\{a:\{0..$((${#a}-$1))\}:$1\\} |
                    tee >(rev) )"
       }

Test it:

dqs 5 < data.log 
1234567
dqs 4 < data.log 
1234
7654
1234567
dqs 3 < data.log 
12365
349874
1234
7654
08767
1234567

How it works:

printf prints a list of sequences of the desired length, (like 123, 234, etc.), tee appends a mirror-image (i.e. right-to-left, or backwards) copy using rev, then grep -f <(...) searches standard input for anything in that list.

To make that list of sequences usually would require a loop, or seq, or even both, but here we cheat by using a bash sequence expression, combined with a substring expansion, and some arithmetic. But it's impossible, because the bash interpreter cannot execute these in the desired order. Therefore eval eval and several strategic \\\s are used to force bash to do things in the right order.

The [ "$@" -gt 0 ] && is not functionally necessary here, but it's safer to have it. It makes sure that dqs has one and only one numeric parameter, or grep won't run. This prevents eval eval from doing anything evil.

Bonus: Adding a 2nd argument can change the 123456789 to any other sequence and the code should still work. For example dqs 4 123456789ABCDEF would search for four digit hexadecimal sequences, (and reverse sequences), and dqs 3 $(printf %s {a..z}) would search for three letter alphabetic sequences.

# search `man bash` for the three most popular words 
# that have 3 three char alphabetic runs
man bash | tr ' ' '\n' | sort | uniq -c | sort -gr  | 
dqs 3 $(printf '%s' {a..z}) | head -3

Output:

     92 first
     76 default
     38 environment
  • Very very nifty ! And thanks for the wiki.wooledge.org "eval" link. – Cbhihe Sep 24 '18 at 10:29
  • That's some crazy bash code golf - love it! – Ian McGowan Sep 24 '18 at 22:53
  • Maybe grep -F "$(eval eval printf "'\"%s\n\"'" "'\${a:'{0..$((${#a}-$1))}':$1}'")" data.log. No <(…), no "fence" `\\\`, and "fixed strings" (no pipe -> faster) used. – Isaac Sep 26 '18 at 3:22
  • @Isaac, Thanks! The -F is better, since the substring list is never going to be too large. See revised answer. The advantage of changing the quoting seems less obvious. Using a hard coded file name... my earlier versions of this, (before it was posted), included the file name, but leaving it out saved space, and was more versatile. Since it's a function, we could just make the 2nd argument the filename, but STDIN seems simpler. – agc Sep 26 '18 at 3:58
0

If you have a lot of very large files, regex matching in awk will be slow. One approach is to leverage grep to do the hard lifting, and awk to build the list of strings to search for (since you don't want to hard code those). I.e.

$grep -E '12|98|23|87|34|76|45|65|56|54|67|43|78|32|89|21' data.log

Does the trick for two characters, but we want to be able to do this for up to 9 characters. You need -E for extended grep to support searching multiple patterns (12|98 are two patterns) - plain ol' grep doesn't let you do that.

Awk can loop over the string 123456789 pulling out successive pieces, but we want to go forward and backward, so:

$awk 'BEGIN {f=123456789 ; b=987654321 ; for(i=1;i<9;i++) print substr(f,i,2),substr(b,i,2)}'
12 98
23 87
34 76
45 65
56 54
67 43
78 32
89 21

Let's add something so the length isn't hard coded to two (-vn=3 sets the variable n=3 inside the awk script):

$awk -vn=3 'BEGIN {f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}'
123 987
234 876
345 765
456 654
567 543
678 432
789 321

And (almost there!) get the pipe symbol grep -E wants by changing output record separator (ORS) and output field separator (OFS) to |

$awk -vn=3 'BEGIN {ORS="|" ; OFS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}'
123|987|234|876|345|765|456|654|567|543|678|432|789|321|

We have to get rid of the final pipe after 321, or grep will match everything, so add sed '.$//' to replace the last character before the end of the string ($) with nothing:

$awk -vn=3 'BEGIN {ORS="|" ; OFS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}' | sed 's/.$//'

And now we can put it together in a shell script that lets us do the search in general:

$cat t.sh
#!/bin/bash
grep -E `awk --assign n=$1 'BEGIN {OFS="|" ; ORS="|" ; f=123456789 ; b=987654321 ; for(i=1;i<11-n;i++) print substr(f,i,n),substr(b,i,n)}' | sed 's/.$//'` $2

$chmod 775 t.sh
$./t.sh 4 data.log
1234
7654
1234567
-1

Many big files indicate that this needs to be done quickly. That means a while read loop is out of the question. One thing to realize here is that each of these exercises can be simplified into matching (at least) one of a small set of patterns, and this can be done really fast using grep or similar tools like rg or ack. For example, for five digit sequences:

grep -e 12345 -e 23456 […] -e 65432 -e 54321

See man grep for more information, and use Greg's Wiki to learn Bash quickly.

  • Why the downvote? OP said they were learning Bash, so it's probably counter-productive to just give the final solution ready to run. – l0b0 Sep 26 '18 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.