0

I have >100 lz4 files named like this:

-rw-r--r-- 1 root root 210M Apr 11 10:11 compressedfile.1-0.lz4
-rw-r--r-- 1 root root 208M Apr 11 11:35 compressedfile.1-1.lz4
-rw-r--r-- 1 root root 185M Apr 11 12:49 compressedfile.2-0.lz4
-rw-r--r-- 1 root root 193M Apr 11 13:06 compressedfile.2-1.lz4
-rw-r--r-- 1 root root 201M Apr 11 14:28 compressedfile.3-0.lz4
-rw-r--r-- 1 root root 236M Apr 11 15:02 compressedfile.3-1.lz4
....

The files are huge csv files that look like this:

10.27.221.233,11,TCP,SSL,66,8578,0,,(null),510-12
10.133.205.134,10,UDP,ICMP,26,3470,1,,(null),515-10
10.92.160.173,10,TCP,SSL,66,8578,0,,(null),510-15
10.132.81.71,11,TCP,SSL,0,2,0,,(null),511-10

I need to filter out IP addresses that use SSL. My method is like this:

lz4 -dc compressedfile.1-0.lz4 | awk -F, '{if ($4=="SSL") print $1}'

that's only for one file. I tried to process multiple files with wildcards, like this one:

lz4 -dc compressedfile.*.lz4 | awk -F, '{if ($4=="SSL") print $1}'

Warning : compressedfile.1-1.lz4 won't be used ! Do you want multiple input files (-m) ? 
Warning : compressedfile.2-0.lz4 won't be used ! Do you want multiple input files (-m) ?
Warning : compressedfile.2-1.lz4 won't be used ! Do you want multiple input files (-m) ?
....
10.27.221.233
10.92.160.173
10.132.81.71
10.140.81.238
10.92.5.90
....
<it ends with the IP (with SSL) on compressedfile.1-0.lz4>

then I tried adding -m option to lz4:

lz4 -mdc compressedfile.*.lz4 | awk -F, '{if ($4=="SSL") print $1}'

there are compressedfile.* uncompressed files created :(

I need your advice on using wildcards on lz4. I'm trying to avoid using a for loop if possible.

  • lz4 -mdc *.lz4 | whatever... works fine here with lz4 v 1.7.4 - it decompresses multiple files to stdout and does not create any file. – don_crissti Apr 11 '17 at 13:11
  • @don_crissti that’s interesting; are you using Fedora or RHEL by any chance? The Fedora version works as you describe, whereas the Debian version which I tested before answering exhibits the behaviour described by the OP. – Stephen Kitt Apr 12 '17 at 8:30
  • yes, i use RHEL, it's freezed ( ̄︿ ̄) – dragon21century Apr 12 '17 at 10:40
1

lz4 files can be concatenated and processed as a unit, so this will work:

cat compressedfile.*.lz4 | lz4 -dc | awk -F, '{if ($4=="SSL") print $1}'
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.