6

How can I read a series of lines from a file and then remove each line?

Obviously I can read like this:

while read -r line; do
echo $line
done < myfile.txt

But if I do this:

while read -r line; do
echo $line
sed '1d' myfile.txt
done < myfile.txt

I only get the very first line

I'm looking for an elegant way to read a line, do something with it and then remove it before moving on to the next line in the file.

  • 1
    Are you trying to make the program reentrant? If so, you probably want to instead write the last processed line number to a state file for every line. Much faster, and just as safe. – l0b0 Jan 23 '15 at 11:08
2

I'd prefer to invoke sed just one time then on each line iteration. If you like to change original file you can add -i(--in-place) option to sed.

unset n
while read -r line; do
  echo $line
  : $((n++))
done < myfile.txt
sed "1,$n d" myfile.txt
0

You can do:

sed -n '$d;w file2' >file1 <<IN
$(cat file1; echo .)
IN

Which will safely write every line from file1 into file2 while simultaneously deleting every line in file1. The shell handles the intermediate temp file securely - it creates one and then deletes it from its filesystem before ever writing a single byte to it.

As soon as sed releases the file-descriptor (at EOF or when it terminates - whichever comes first) the file ceases to exist at all. Unlike the common sed -i option, this does not depend on sed cleaning up after itself (which may not happen if its processing is unexpectedly interrupted) and nor does it carry with it the same permissions inconsistencies the -i option implies.

And the application need not be global as demonstrated - the lines can be either deleted from the original and/or written to the separate output selectively...

sed -n '$d;/pattern1/w file2
           /pattern2/p' >file1 <<IN
$(cat file1; echo .)
IN

...demonstrates how only lines from file1 which match /pattern2/ will be printed back into file1 and only lines from same which match /pattern1/ are written to file2. Any other sed command is of course applicable as well.

It is probably worth noting, though, that for most shells this will not carry over NUL bytes from file1 (which isn't often a concern for sed work) - though such a thing is possible with zsh.

0

You can replace every line of a file once it was processed correctly with this snippet (do not remove if processing fails, so you can reprocess them if you need it):

while read; do
  if some_command_worked_correctly; then
    sed -i '1,1 d' file-to-read
  fi
done < file-to-read
  • 1
    This does not work as intended; it only deletes the first line of the file, regardless of the current line of the read loop. – ijoseph May 27 '18 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.