8

I am running the following command on a Raspberry Pi 3 Debian latest release:

cat /dev/ttyUSB0 | tee -a /media/pi/KINGSTON/klima.out | grep -F $ | tee -a /media/pi/KINGSTON/log

The command works fine and does what it should; however, when I delete (manually or by CRON) the klima.out file, it is not re-created. The command keeps running, the log file continues to be appended, but the klima.out file doesn't come back. (also no buffering). I want to delete it once a week for not letting it grow over all boundaries. Any suggestions?

4
  • 1
    @Kusalananda, only in the rc/es or fish shells. Bourne-like shells and csh-like shells should be ok. May 29, 2017 at 14:51
  • 1
    Kusalananda's answer is right; note that if your source was a growing log file instead of a device, you might want to "tail -f" so you don't re-cat what's already been processed.
    – TOOGAM
    May 29, 2017 at 19:30
  • 2
    Bear in mind that deleting a file (like with rm) is essentially an edit operation on a directory. It deletes the entry pointing to the file from the directory. The file itself will only be deleted when nothing is using it; but tee is using it. May 29, 2017 at 20:58
  • This looks like a useless use of cat. Instead, tee < /dev/ttyUSB0 -a ... | ... would work the same, but tee would read directly from the char dev instead of from a pipe from cat. May 31, 2017 at 5:05

3 Answers 3

20

I'm assuming that the pipeline of yours is running for a long time and that you are trying to remove the log file while it's running.

When you delete the file, the tee process still has it open for writing, which means that the disk space is not handed back to the system. That won't happen until all open file descriptors that references the file are closed.

It's perfectly OK to write to a file that's been deleted, as long as the file descriptor was allocated before the deletion.

You will have to restart the pipeline for the file to be recreated, and to allow the space taken by the old (now nameless) log file to be reclaimed.

To avoid having to restart the pipeline, you may choose to truncate the file, i.e. trim its size down to zero without removing it. This would allow tee to continue appending to the file without having to re-open it.

Truncating a file may be done like jlliagre showed in his answer, or by using truncate (a non-standard utility which is part of GNU coreutils):

truncate -s 0 /media/pi/KINGSTON/klima.out

See the manual for truncate for further info about that utility.

8
  • as long as the file descriptor was allocated before the deletion. Actually, you can get new file descriptors for already-deleted-but-still-open files through /proc/pid/fd. It's also perfectly fine to write through those, too. So really, as long as you have an open file descriptor, the number of directory entries for that inode is pretty much irrelevant. May 31, 2017 at 5:07
  • After truncating, the next write to the still-open fd will result in a sparse file, so yes this does recover the space but you have that many bytes of zeros before the new text, because it doesn't reset the file position of already-open fds. (And some filesystems don't support sparse files, e.g. FAT or HFS+, so the next write will actually write zeros). Anyway, I don't think there's any way around that (other than a ptrace / debugger hack that resets the file position of tee's fd as well as truncating). May 31, 2017 at 5:15
  • Anyway, you could also just punch a hole out to the current len with fallocate -p, but then you need the right length and all it gains you is saving one inode update (never needs to set the length to zero). May 31, 2017 at 5:17
  • @PeterCordes I tested this and it does not seem to create a sparse file. A 100M file opened in append mode, then truncated as above. The next write to it will occur at the start of the now empty file, not after some sparse gap, as far as I can see.
    – Kusalananda
    May 31, 2017 at 9:12
  • 1
    @PeterCordes Ah, yes, if you use >, but not if you use >> or, as the OP was doing, tee -a. Writing in append mode always adds to the logical end of the file whereas not using append mode will write at the physical location after the last write.
    – Kusalananda
    May 31, 2017 at 13:01
13

If you want to recover the file blocks, you need to blank the file, not unlink it:

This portable way should work with most shells :

: > /media/pi/KINGSTON/klima.out

Unlinking the file (i.e. rm) is removing the directory entry but doesn't affect the file contents (inode) as long as the file is kept open by readers or writers.

11
  • 1
    Thanks, Yes, I just want to shorten the file. Sorry to bother, but I don't completely understand your suggested code. What does the : stand for?
    – hans
    May 29, 2017 at 14:47
  • 3
    : is the null command, i.e. it does nothing specific. The only feature used here is provided by the shell: the redirection is resetting the file to an empty content.
    – jlliagre
    May 29, 2017 at 15:04
  • 3
    @jlliagre You're perfectly correct on both points.
    – Kusalananda
    May 29, 2017 at 20:23
  • 1
    @Björn Indeed, although not strictly equivalent, echo "" would write one byte to the file. Note that echo alone does the same as echo "".
    – jlliagre
    May 30, 2017 at 8:40
  • 1
    @Björn Well, echo -n behavior is implementation defined. Depending on the shell, this command might replace the file content with the string -n instead of fully truncating it. A portable way to write "nothing" is printf "".
    – jlliagre
    May 30, 2017 at 10:15
5

You don't understand how the system is handling files.

You delete the file entry, but the file still exists as long as the program keeps an handle on it. So tee is never notified the entry was deleted and it still writes to the file !

A unique file can have many entries thanks to the hard links (created by the ln command).

You could write your own version of tee which close and open the file on every line it writes to the file, but it would be very under-performing as it would generate so many system calls.

Here is a shell-function which will split its input to several files :

splitSizeInKio=100

splitInput(){
    local PS4='+splitInput+ '
    set -x
    local i=0
    local fname="$1"
    local ii

    while true
    do if [ $i -lt 10 ]
       then ii=0$i
       else ii=$i
       fi
       local outfile="$fname".$ii
       dd of="$outfile" bs=1024 count=$splitSizeInKio
       i=$((i+1))
    done
}

(You could use "head" instead of "dd" if you spilt on a number of lines instead of a size.)

With bash, you can use "process substitution" like this :

prog1 | tee >( splitInput somefilename ) | prog2
2
  • 1
    Your splitInput might work ok using dd, since it "knows" exactly how many bytes to read from it's stdin, but using head is highly likely to result in corrupted data, due to buffering. May 29, 2017 at 16:52
  • Yeah, head will read more data from stdin than it writes, since it doesn't read 1 char at a time. And it doesn't try to "put back" any over-read beyond the number of lines it was looking for (if that's even possible on pipes). May 31, 2017 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.