1

I have a huge file, ~4000000 lines that look like that:

file:

1 
1 
2 
3 
2 
4 
...

basically, each line contains a number. Considering this example I would like to generate a file that will look as follows:

newFile:

1 
1 
2 
2 
3 
3 
3 
2 
2 
4 
4 
4 
4 
....

I would like that each line to be repeated n times when n is the number stored in the original line.

As I mentioned, the file is very large so Im looking for the fastest way to do so. I think that currently, with the code I implemented the bottleneck is the fact that I need to reopen the new file for writing.

This is how Im doing it now but its very very slow

while read i; do seq 1 ${i} | xargs -i -- echo $i >> newFile ;done < file

Thank you very much.

  • Isn't this just plain sorting? – Braiam Aug 26 '16 at 2:10
  • @Braiam no, the output is much bigger than the input; and the output isn't sorted, in the example there are two 2 lines appearing after 3... – Stephen Kitt Aug 26 '16 at 11:19
7

In awk:

awk '{ for (i = 1; i <= $1; i++) { print $0 } }' file > newFile

This will read the input file once and write the output file once, never needing to re-open anything.

7

In perl, using the x string repetition operator:

$ perl -ne 'print $_ x $_' file
1
1
2
2
3
3
3
2
2
4
4
4
4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.