0

Lets say I have the output of this command saved to a file.

cat /dev/urandom | tr -dc '[:graph:]' | fold -w 1000 | perl -pe 's/(.)(?=.*?\1)//g' | head -n 50

I would like to compare only the first n characters on each line in a file and return only the first line containing the first instance of those characters.

So, for example, without having to sort the file, I'd like to look at the first four characters on every line in the file. I want to find the first instance of any four character string on each line and print only the lines containing the first instances.

I would appreciate it if the command could be modified to look at four, five, or six characters etc on each line.

Thank you very much for your time and assistance. I have really been struggling to figure this out.

  • MelBurslan, the content of the string should be irrelevant but the output of the above - now corrected - command is every character I can type on an English language keyboard. Below are two sample lines.

    k!>d#&)"EtXN`;*9TaD7BcL84z5[y{$Q?_Y%fCw6F0Vgn\|]ImqR.:1l<^}u'+Ms/hjS@e~2vxWO(3,bJiprP-=UAZGoHK
    3'O$#Eg5&,`l>vn491M"cVZR\7J.H[XTw*:q}Kz8hf;W_P|i<6@CAytF^Dmkb]GBsU+{Y?xje%oIQ-~r!2Sap=/)N0du(L
    
  • glenn, yes the first instance of a four character string/key/token. I also needed the matching to be variable so I can manipulate the character matching as needed.

  • wildcard, this worked wonderfully, thank you.

  • thrig, this worked wonderfully, thank you.

  • 1
    an example snippet from the file you mentioned and a match pattern would be very helpful, as the command doesn't give enough clue into what you are trying to accomplish – MelBurslan Apr 5 '16 at 19:21
  • 1
    Let me attempt to paraphrase: "The key is the first 4 characters of the line. Print a line only if it is the first appearance of its key" -- is that right? – glenn jackman Apr 5 '16 at 19:40
3

Assuming glenn jackman's paraphrase of your question is correct, here is a solution using awk and substr():

awk '{key = substr($0,1,4)}; !(key in printed); {printed[key]}' file

This sets "key" to the first four chars of a line, then prints the line unless it has seen that key before, then keeps track of the fact that that key has been printed.

1

Use a hash?

% (echo foo; echo bar; echo foobar) \
  | perl -ne '/(.{3})/; print unless $seen{$1}++'
foo
bar
% 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.