How can I delete multiple random lines from a text file using sed?

Question

I want to delete 10 random lines from a text file that has 90 lines and then output this to a new file. I've been trying to do this using sed, but I have two problems. I'm using:

sed -i $((1 + RANDOM & 90))d input.txt > output.txt

and then running the command 10 times (I assume there is a better way to do this!)

The first problem I have is that I get the error:

sed: -e expression #1, char 2: invalid usage of line address 0

I assume this has something to do with the fact that it might have already deleted line 1 and it is trying again.

The second problem is that sometimes nothing is written to the output file, even though it worked before using the same command.

Answer 1

You probably wanted to use RANDOM % 90 rather then &. That's where the zeroes come from (deleting line 1 is OK, on the next run, the lines will be numbered 1 .. 89).

There is a problem, though: The formula could generate the same number several times. To prevent that, use a different approach: shuffle the numbers and pick the first ten:

shuf -i1-90 -n10 | sed 's/$/d/' | sed -f- input > output

If you don't like sed generating a sed script, you can use printf, too:

sed -f <( printf %dd\;  $(shuf -i1-90 -n10) ) input > output

Answer 2

If you don't have GNU shuf, portably, you could do:

awk -v n=90 -v p=10 '
  BEGIN {srand()}
  rand() * n-- < p {p--; next}
  {print}' < file

It's also going to be more efficient than the shuf+sed approach with high values of p since it's in o(n), while shuf+sed is in o(n*p). With n=1000000, the breaking point on my system is around p=35 with GNU sed vs GNU awk and with p=1 with GNU sed vs mawk (as in mawk is always faster).

Answer 3

I think the challenge here is delete one of the 90 lines, then one of the remaining 89 lines, etc. -- we can't delete the 90'th line when only 89 remain.

eval $(for i in {90..81}; do CMD="$CMD | sed $(( (RANDOM % $i)+1 ))d"; done; echo cat infile $CMD) > outfile

The for loop accumulated a series of strings forming a pipeline in the form | sed NNd where NN is a random number from the shrinking range starting at 1 to 90 and ending at 1 to 81 resulting in | sed 88d | sed 12d | sed 36d...

After the command CMD is formed, we prepend cat infile to the pipeline CMD (noting that CMD starts with a | from the for loop). CMD now looks like cat infile | sed 88d | sed 12d...

Finally, we eval the CMD string to execute the command and put the result in outfile

Answer 4

if there is no matter with performance, use this:

cat PATH_OF_SOURCE_FILE | \
grep -n ^ | \
grep -E "^($(seq 1 90 | shuf | head -n 80 | paste -s -d '|')):" | \
sed 's/[0-9]*:\(.*\)$/\1/' > PATH_TO_TARGET_FILE

first grep index lines; second grep select 80 random lines, and sed remove line number added by first grep.

Note: pipe the last output to shuf if you do not need the output order