3

Why does this simple command fails using emacs shell (eshell)?

cat file.txt | wc 

I have a file with 10241 lines. Each line has less than 50 characters. Around 90% of the times I launch this command, it gives the wrong result, namely line count. Nonetheless, no error messages are given.

Looks like broken pipe is a very common topic, but I haven't found any reasonable explanation. Also, no one proposes any workarounds. How can I get this simple command working reliably?

Of course, I could've just run wc file.txt. But I'm looking for a more general solution in which any tool would work fine piped cat: cat file.txt | any_tool_here.

Details

I'm using CentOS 5. This issue appears when using eshell (emacs shell). I'm using GNU Emacs 24.5.2.

Experiments

Samples of results using cat file.txt | wc (expected: first column to be always 10241).

  1. 8568 25706 110571
  2. 9837 29513 126947
  3. 5395 16187 69615
  4. 9202 27608 118757
  5. 7299 21899 94199
  6. 9837 29513 126947

Sample of results using wc file.txt:

  1. 10241 30723 132156
  2. 10241 30723 132156
  3. 10241 30723 132156
  4. 10241 30723 132156
  5. 10241 30723 132156
  6. 10241 30723 132156

The cat command itself (when executed alone) is working properly. I validated it with the following command (a few times): cat file.txt > file2.txt. Then, I diff'd both files and they are identical.

  • 1
    What you are describing is not normal. Any tool does work fine with piped cat, I've never seen what you describe (and I just tested it with a file with 10240 lines of 50 characters each; it ran 1000 times with no error). Are you running this while another process is writing to file.txt? What operating system are you using? – terdon Oct 17 '16 at 9:58
  • No other process has access to file.txt. I'm using CentOS 5. – rph Oct 17 '16 at 9:59
  • OK. Please edit your question and i) mention the OS; ii) give us an example of the file you are using that reproduces the error; iii) give us the exact error message. Are you sure the problem is a broken pipe? It sounds like you have something strange in your file. Does running wc file.txt always work as expected? – terdon Oct 17 '16 at 10:01
  • 3
    Sorry guys. I'm using eshell (emacs shell). This issue does not happen using the linux terminal. – rph Oct 17 '16 at 10:16
  • 4
    Next time, please make sure to mention all these details in your question. it saves everyone a lot of time. – terdon Oct 17 '16 at 10:48
5

Gathering from the information about the shell that was used (eshell), it appears that the streaming aspect of this shell is the culprit. Normally, piping means opening two ends of a pipe + fork/exec, so you get two processes that share a file descriptor to a pipe, and communication goes directly through the kernel. This way, nothing can get lost - it's guaranteed to be safe (although if it the pipe or any involved stream are buffered, you may have to wait for the first process to exit normally to flush out the last chunk of the stream).

Judging from the excerpt from eshell manual:

Eshell is not a replacement for system shells such as bash or zsh. Use Eshell when you want to move text between Emacs and external processes; if you only want to pipe output from one external process to another (and then another, and so on), use a system shell, because Emacs’s IO system is buffer oriented, not stream oriented, and is very inefficient at such tasks. If you want to write shell scripts in Eshell, don’t; either write an elisp library or use a system shell.

eshell is not doing it the normal way, but fakes the pipe using its "buffers" (emacs' representation of open files) as intermediate deposit for data, and (without further research) I'd guess that at some point, wc performs a read, and emacs responds with an empty chunk (and returning 0 from read is a signal that the stream has ended) instead of waiting for more input from the first program to fill the buffer. If that's the case, it means that eshell is not only inefficient but buggy when it comes to pipes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.