3

I want to delete 10 random lines from a text file that has 90 lines and then output this to a new file. I've been trying to do this using sed, but I have two problems. I'm using:

sed -i $((1 + RANDOM & 90))d input.txt > output.txt

and then running the command 10 times (I assume there is a better way to do this!)

The first problem I have is that I get the error:

sed: -e expression #1, char 2: invalid usage of line address 0

I assume this has something to do with the fact that it might have already deleted line 1 and it is trying again.

The second problem is that sometimes nothing is written to the output file, even though it worked before using the same command.

6

You probably wanted to use RANDOM % 90 rather then &. That's where the zeroes come from (deleting line 1 is OK, on the next run, the lines will be numbered 1 .. 89).

There is a problem, though: The formula could generate the same number several times. To prevent that, use a different approach: shuffle the numbers and pick the first ten:

shuf -i1-90 -n10 | sed 's/$/d/' | sed -f- input > output

If you don't like sed generating a sed script, you can use printf, too:

sed -f <( printf %dd\;  $(shuf -i1-90 -n10) ) input > output
  • 2
    @don_crissti, I don't think the OP wants to shuffle the whole file and remove n lines, just remove n random lines from the file. – glenn jackman Nov 20 '15 at 15:58
  • 2
    @choroba, improvement, replace 90 with $(wc -l <input) – glenn jackman Nov 20 '15 at 15:59
  • Also shuf -n10 -e {1..90}d – Stéphane Chazelas Nov 20 '15 at 16:55
5

If you don't have GNU shuf, portably, you could do:

awk -v n=90 -v p=10 '
  BEGIN {srand()}
  rand() * n-- < p {p--; next}
  {print}' < file

It's also going to be more efficient than the shuf+sed approach with high values of p since it's in o(n), while shuf+sed is in o(n*p). With n=1000000, the breaking point on my system is around p=35 with GNU sed vs GNU awk and with p=1 with GNU sed vs mawk (as in mawk is always faster).

0

if there is no matter with performance, use this:

cat PATH_OF_SOURCE_FILE | \
grep -n ^ | \
grep -E "^($(seq 1 90 | shuf | head -n 80 | paste -s -d '|')):" | \
sed 's/[0-9]*:\(.*\)$/\1/' > PATH_TO_TARGET_FILE

first grep index lines; second grep select 80 random lines, and sed remove line number added by first grep.

Note: pipe the last output to shuf if you do not need the output order

-1

I think the challenge here is delete one of the 90 lines, then one of the remaining 89 lines, etc. -- we can't delete the 90'th line when only 89 remain.

eval $(for i in {90..81}; do CMD="$CMD | sed $(( (RANDOM % $i)+1 ))d"; done; echo cat infile $CMD) > outfile

The for loop accumulated a series of strings forming a pipeline in the form | sed NNd where NN is a random number from the shrinking range starting at 1 to 90 and ending at 1 to 81 resulting in | sed 88d | sed 12d | sed 36d...

After the command CMD is formed, we prepend cat infile to the pipeline CMD (noting that CMD starts with a | from the for loop). CMD now looks like cat infile | sed 88d | sed 12d...

Finally, we eval the CMD string to execute the command and put the result in outfile

  • Perhaps I can learn from the down vote about how to improve this. I think that this address the original question well, preserves order in the file (that shuf infile does not), and addresses the subtle question about a decreasing range of lines from which to delete in each successive iteration. I'm open to constructive input. – MikeV Nov 20 '15 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.