0

I have a very large text list and need a way to extract lines beginning with the same 2 characters, then save those lines to separate files named after those 2 characters.

Example List:

abWEye7kgw7  
abff34ZSrZf  
abke8mzMyma  
b2R5mPZGbCb  
b2zhhCeLZzZ  
b2q2T5rkACp  
k9ekzbc8nUh  
k9QzXBUrNT7  
k92RtdXntZ3  
vrTtR9GmbWG  
vraVM9QXWzY  
vrME9QnksBf  

Desired Output:

ab* > ab.txt  
b2* > b2.txt  
k9* > k9.txt  
vr* > vr.txt

The list is rather large and there are lots of first 2 character combinations.

4
$ awk '{ f = substr($0,1,2) ".txt"; print >f }' file.in

$ ls
ab.txt  b2.txt  file.in k9.txt  vr.txt

$ cat ab.txt
abWEye7kgw7
abff34ZSrZf
abke8mzMyma

This can obviously be solved in the shell too, but awk is better suited for parsing text files. The substr() picks out the first two characters of each line in the input file, and this is assigned to the variable f with .txt added to the end. The print will output the current line to the file whose name is in f.

I believe you can do away with the f variable and use the substr() expression directly after >, but not in the awk implementation that I'm using on OpenBSD (this is possibly a bug).


If the number of different combinations of two first characters are too many, you may have issues with too many open files.

The following variation will take care of that:

awk '{ f = substr($0,1,2) ".txt"; print >>f; close(f) }' file.in
  • @StéphaneChazelas Yes, if the number of different two-character combinations are too many. I will add an alternative solution soon. – Kusalananda Jan 10 '18 at 14:37
1

awk solution:

awk -v FS='' '{ print > $1$2".txt" }' file

One of the resulting files:

$ cat k9.txt 
k9ekzbc8nUh
k9QzXBUrNT7
k92RtdXntZ3
  • Since you're not explicitly closing the output files after each print, they will not be truncated if the input file isn't sorted, so it'll will work on unsorted data too. (if it wouldn't work, it wouldn't work on sorted data either) – Kusalananda Jan 10 '18 at 11:01
  • @Kusalananda, let's say: It should work (in our case) – RomanPerekhrest Jan 10 '18 at 11:08
  • 1
    It should be noted that the behaviour for an empty FS is unspecified per POSIX. – Stéphane Chazelas Jan 10 '18 at 14:41
0

Try this:

cat list.txt | while IFS= read -r st; do echo $st >> ${st:0:2}.txt; done

  • 1
    Why the cat? Use while ...; do ...; done <list.txt instead. – Kusalananda Jan 10 '18 at 10:59
  • The old bad habit. Yes, you're right, it would be better. – Wild Zyzop Jan 10 '18 at 11:13
0

I have used below method to achieve the result. Tested its works fine

 for j in `awk '{print substr($1,1,2)}' k.txt  | uniq -c | awk '$1 >=2 {print $2}'`; do sed -n "/^$j/p" k.txt > $j.txt; done

It extracts the first two character of each line. If First 2 character is same in more than 1 line. it will prints the related line save the lines in filename of first 2 charater

Output

 cat ab.txt
    abWEye7kgw7
    abff34ZSrZf
    abke8mzMyma

    cat b2.txt
    b2R5mPZGbCb
    b2zhhCeLZzZ
    b2q2T5rkACp

     cat k9.txt
    k9ekzbc8nUh
    k9QzXBUrNT7
    k92RtdXntZ3

 cat vr.txt
vrTtR9GmbWG
vraVM9QXWzY
vrME9QnksBf

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.