-4

I have millions of lines with these 2 formats all in one merged file.

hash:salt:password
hash:password
hash:hash

What I want is just keep the password, then remove duplicates and after remove duplicates remove all lines that contain a string.

Sample file:

695ecf9185cb3f1a415821ddd69cc6be:4e85e5:671014
695f1cc6765cac14c3478505c300ec9a::0d7b9438b5ea5d9b6883b43a656357d9775688
695f50345da6811b712ad3d08ac107cf::54dbf42fdafabe13e64c1eb7c7813e52b16188
695fe18dfdf073769841193ca98aaea3::405d98ab893b610144bf4645e6c132cb5eaf12
6960138fc7d1e2f9ec1c3df49ea72172::d7698e6198301e4a6df04d8b087a197a3d72f5
696073574b563bcfbf44a13a2848484a::9f3d22ab9249f9a05f9a7854f0ce2c6017cad2
6960a12e5357adbef3cb9df2d0ecd9c7::ca5fad391e11eb7e860bb1102528e77eb88474
6960c557ff9cc514771d3e6d855f3f8d::f3431013320c3864519b4c9269b524025bb6cf
69613e3ad5e001cdb1321012a554785e:c20954:214828
cc81a83d6bcd13b89519c45c7f5a1cfd:Refineryhotel2
696fb1bf9f806c772ac44882a28c2888:ca28a5:8561535
696fbe253ff887d5725d2438387b3bc1::60eb49f1f213fcc2f959edfec4feed89031747
696ffadc81d15371b88815c8394329a9:e5ad83:123hacker
6970031dffaac00607ce7663fa916686:4c4ba0:8109201
69706a943cc96169919476026a7de70c:ebd0af:0611314
69709651482c9e5c44c222aed4ece9aa::7a626fc07ccceb9b11c35c55941b2b5eb082c5

Expected output:

671014
214828
Refineryhotel2
8561535
123hacker
8109201
0611314

closed as unclear what you're asking by Kusalananda, Jeff Schaller, msp9011, G-Man, dr01 Sep 3 '18 at 12:49

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    I count three formats, not two (and in a comment you also mention hash:salt). How would you distinguish the second form from the third? – Kusalananda Sep 2 '18 at 20:58
  • 1
    I don’t understand what you’re asking for.  Please (1) try harder to clearly explain what you want, and (2) give some examples of input and output.  … … … … … … … … … … … … … … … … …  You say “I have … one merged file.  …  What I want is … remove all lines that contain a string.”  It almost sounds like cat /dev/null will give you the result you describe.  … … … … … … …  Please do not respond in comments; edit your question to make it clearer and more complete. – G-Man Sep 2 '18 at 21:33
0

Remove all before last “:” in Terminal

Try this,

awk -F ':' '$2 != ""{print $NF}'  file

671014
214828
Refineryhotel2
8561535
123hacker
8109201
0611314
  • will print only the text after the last : if the second field is not null.
0

With your new sample data (that don't contain duplicates), try

awk -F: '$2 != "" {print $3}' file
671014
214828

8561535
123hacker
8109201
0611314

What do you mean by "contain a string"? Can't be inferred from your sample.

If that Refineryhotel2 line is NOT an artefact ( it's the only one with just two fields), try

awk -F: '$3 == "" {print $2; next}; $2 != "" {print $3} ' file
671014
214828
Refineryhotel2
8561535
123hacker
8109201
0611314

Not the answer you're looking for? Browse other questions tagged or ask your own question.