Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I know this is sort of a duplicate of another question (Why this sort command gives me an empty file?) but I wanted to expand on the question in response to the answers given.

The command

shuf example.txt > example.txt

Returns a blank file, because the shell truncates the file before shuffling it, leaving only a blank file to shuffle. However,

cat example.txt | shuf > example.txt

will produce a shuffled file as expected.

Why does the pipeline method work when the simple redirection doesn't? If the file is truncated before the commands are run, shouldn't the second method also leave an empty file?

share|improve this question
    
    
@Gilles The question is more specifically about why the command works in a different format, and has this point has not been answered yet –  toryan Jan 23 at 19:43

4 Answers 4

The problem is that > example.txt starts writing to that file, before shuf example.txt starts reading it. So as there was no output yet, example.txt is empty, shuf reads an empty file, and as shuf makes no output in this case, the final result stays empty.

Your other command may suffer from the same issue. > example.txt may kill the file before cat example.txt starts reading it; it depends on the order the shell executes those things, and how long it takes cat to actually open the file.

To avoid such issues entirely, you could use shuf example.txt > example.txt.shuf && mv example.txt.shuf example.txt.

Or you could go with shuf example.txt --output=example.txt instead.

share|improve this answer
    
> example.txt doesn't start writing to the file, it opens the file for writing with truncation. Compare with 1<> example.txt which opens it read-write without truncation. –  Stéphane Chazelas Jan 24 at 16:02
    
@StephaneChazelas, opening a file for writing with truncation is usually how you start writing to a file. It's quite hard to do without opening it. ;) –  frostschutz Jan 24 at 16:05
    
Your wording is confusing. > file doesn't write anything and the real problem is that it truncates. If it only opened for writing (as 1<> file does), there wouldn't be any problem (at least not with shuf). –  Stéphane Chazelas Jan 24 at 16:10
    
If > file didn't write anything, it would work on a read-only filesystem. It only doesn't write anything when you take it literally (as in the syscall write()). You're riding on technicalities there that IMHO are way more confusing than my simplified explanation. –  frostschutz Jan 24 at 16:28

The package moreutils has a command sponge:

   sponge  reads  standard input and writes it out to the specified file.
   Unlike a shell redirect, sponge soaks up all its input before  opening
   the output file. This allows constricting pipelines that read from and
   write to the same file.

with that way you can do:

shuf example.txt | sponge example.txt

(unfortunately the moretuils package also has a util named parallel that is far less useful than gnu parallel. I removed the parallel installed by moreutils)

share|improve this answer

You are just quite lucky running

cat example.txt | shuf > example.txt

doesn't empty example.txt like this command is doing.

shuf example.txt > example.txt

Redirections are performed by the shell before the commands are executed and pipeline components are executed concurrently.

Using the -o / --output option would be the best solution with shuf but if you like taking (very slight) risks, here is a non traditional way to avoid the processed file to be truncated before being read:

shuf example.txt | (sleep 1;rm example.txt;cat > example.txt)
share|improve this answer
2  
This, of course, assumes that 1 second is long enough. It would better to instead use sponge or a temporary file. –  Chris Down Jan 23 at 7:22
    
@ChrisDown sponge or a temporary file would be definitely more secure, especially with a remote or non inode based file system but in usual cases, 1 second should be long enough for shuf to open the file unless this is a very loaded system. –  jlliagre Jan 23 at 7:39

From the GNU bash manual (see also, for the details, section 3.7 Executing Commands):

3.1.1 Shell Operation

The following is a brief description of the shell’s operation when it reads and executes a command. Basically, the shell does the following:

  1. Reads its input from a file (see Shell Scripts), from a string supplied as an argument to the -c invocation option (see Invoking Bash), or from the user’s terminal.
  2. Breaks the input into words and operators, obeying the quoting rules described in Quoting. These tokens are separated by metacharacters. Alias expansion is performed by this step (see Aliases).
  3. Parses the tokens into simple and compound commands (see Shell Commands).
  4. Performs the various shell expansions (see Shell Expansions), breaking the expanded tokens into lists of filenames (see Filename Expansion) and commands and arguments
  5. Performs any necessary redirections (see Redirections) and removes the redirection operators and their operands from the argument list.
  6. Executes the command (see Executing Commands).
  7. Optionally waits for the command to complete and collects its exit status (see Exit Status).

Consider the situation where your file does not exist. Yet your second example will create the file most of the times. If the file had not been so created and it didn't exist, and redirection hadn't occurred, cat would complain there is no such file... Most of the time it won't. To reproduce the shuffle you got with that second command, I needed a great many tries. So indeed the second expression should leave an empty file most of the times.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.