6

I have a large text file (~500K lines) with short sentences (couple of words long). Additionally, there is some XML markup in most of the lines. Finally, the text file has been sorted before the markup has been added! Adding the XML markup changes the alphabetic sort but this is desired.

My question is: How can I print random lines respecting the order of the source file?

I know I could just use the shuf command and sort the result. The problem is that the markup will mess up the sort.

I could also write a python script which loads the text file in a list, generates some random numbers, sorts them and uses them as indices to pull out the lines. If possible, I would prefer standard *nix command-line tools.

Sample data:

<CITY>anaconda</CITY> city is in <STATE>montana</STATE>
let's go to <CITY>rome</CITY>
please find <CITY>berlin</CITY>
where is <CITY>cairo</CITY> in <COUNTRY>egypt</COUNTRY>

For example, it would be great if I could pull out the line 2 and 3. Lines 1,3 and 4 are also good. If I get the line 3, 1 and 4, this is not good.

16

Use this:

nl file | shuf -n2 | sort -n | cut -f2-
  • nl to number the lines,
  • shuf to shuffle and limit the output to 2 lines (-n),
  • sort to rebuild the original order,
  • and cut to remove the numeration of nl.

It will print 2 lines of your file in the original order of the file. Use shuf -n X, where X can be any number.

7

The selection of a random line from a file without sorting (or even knowing how many lines there are!) is covered in "The Art of Computer Programming", Volume 2, Section 3.4.2 by Donald E. Knuth. This is trivial to implement, e.g.:

(echo foo; echo bar; echo zot) \
| perl -nle 'rand $. < 1 && ( $line = $_ ); END { print $line }'

Or try shuf, which allows the selection of a certain number of lines, though may thus require more memory than the select-one Knuth algo.

5

You could do it with a while loop and $RANDOM like:

while read line; do
    if ((RANDOM%2)); then
        echo $line;
    fi;
done < _path_

which will print about half the lines, which you could control with the condition in the if

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.