I have a file name called temp.csv in my script one of the intermediate step is

echo "some info" > final.csv | cat temp.csv >> final.csv

at times the file final.csv is created without data which is in temp.csv (when it is running through scheduler). And then, when I rerun the job, then the final.csv is created as I expected.

Why it is happening like this (what exactly happening in the command echo "some info" > final.csv | cat temp.csv >> final.csv)?
If I replace the command in the following way:

echo "some info" > final.csv ; cat temp.csv >> final.csv 

will this modification be helpful?


You are piping the output from echo "some info" > final.csv into cat temp.csv >> final.csv, so these two commands (echo and cat) run in parallel.

Because of that, what ends up in the final.csv depends on what program gets scheduled. What you want to do is replace that | with a ;. And then the echo command will run until finished, and only then cat is started.

  • 1
    Why would the commands run in parallel if the latter one needs the former one to finish in order to receive the former's output piped to its input?
    – confused00
    Oct 30 '14 at 10:55
  • 2
    @confused00 Why would they not, the latter one only needs the former to finish, to be able to finish itself. It can process what it gets in parallel.
    – Anthon
    Oct 30 '14 at 10:58
  • @confused00 - read up on how pipes work. unix.stackexchange.com/questions/79501/…
    – slm
    Oct 30 '14 at 12:13

| mean pipe == output of the previous command piped to next command input. If you use > final.csv that redirect output of echo into file and remains nothing to pipe. So to correct you free use 2 command separatelly as you have offered above or use pipe if your want

echo "some info" | cat - temp.csv > final.csv

Other do the same

cat - temp.csv <<<"some info" > final.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.