0

I have 3 files with millions of records each. I'd like to filter those files in order to find some data more easily. Unfortunately, there is no common format from one record to other. For example:

record1 will contain: A B C D E
record2 will contain A B D E
record3 will contain B C D

The only common part that I have are the first 23 characters which contain some identifier (from which only last 15 are unique between records).

I'd like to filter those files to contain the identifier and the D part. How can I do that? I know that the delimiter between fields is tab and inside each part, there is comma.

Example of the D part:

ROAMENT,INT,15
ROAMENT - constant part which identifies the field
INT - some value which is a must if ROAMENT exists
15 - some other value which is optional
2

perl -ane 'if (/\t(ROAMENT[^\t\n]*)/) { print $F[0], $1, "\n"; }' inputfile

Explanation:

  • -a invokes autosplit, splitting input fields into the @F array. Any whitespace will split fields, if you only want tabs you can add -F"\t"
  • -n invokes an automatic loop over your input data, not printing the input line at the end
  • -e defines an expression to run in each loop
  • The expression checks for < tab >ROAMENT and then any characters up to a tab or newline; if found it prints the first field from the autosplit array and the found field.
| improve this answer | |
  • Thank you! Is there something similar in bash script? – user1977050 Sep 26 '14 at 8:34
  • Did you mean if (/(\tROAMENT[^\t\n]*)/) or something like that? – ilkkachu Oct 26 '16 at 11:51
  • Hmm yes... I blame it on formatting :) I've edited my answer to fix this – wurtel Oct 28 '16 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.