3

I'm trying to append file by watching data that constantly adding every second and pipe data to new files that will process it later. using

tail -f 'data file' >> 'destination file'

But some how some line is duplicate so I do this.

tail -f 'data file' | awk '!seen[$0]++' | >> 'destination file'

This seem fine , But the line would be split some time so I guess I need to buffer the whole line to achieve this. But is this a good way to do this or should I do it another way ?

tail -f 'data file' | grep --line-buffered -E '^.*$' | awk '!seen[$0++]'

Full data example

[9/10/17 14:21:09] Data=[xxx|yyy|zzz] 

Split example

[9/1
[9/10/17 14:21:09] Data=[xx 

something like this. This happen after I pipe data from tail to awk. So I move grep to end of command and it's work. But data will hold in buffer instead if it was too small.

tail -f 'data file' | awk '!seen[$0++]' | grep --line-buffered -E '^.*$'

Thanks for advice and answer !

  • Full msg [9/10/17 14:21:09] Data=[xxx|yyy|zzz] Split [9/1 , [9/10/17 14:21:09] Data=[xx something like this. This happen after I pipe data from tail to awk. tail -f 'data file' | awk '!seen[$0++]' | grep --line-buffered -E '^.*$' I move grep to end of command and it's work , thanks. – Nrto Oct 10 '17 at 7:24
  • Are you actually seeing split lines in your destination file? tail -f is not line buffered, but the output from awk should be. And neither will be adding extra newlines to the data. – roaima Oct 10 '17 at 8:52
  • It isn't new line. The line isn't complete when the other process read from destination file. Hence broke the other process. – Nrto Oct 10 '17 at 9:07
  • 1
    Oh, so the program that reads from destination file is broken, and you need to workaround that? I see. – roaima Oct 10 '17 at 14:34
  • 1
    @John1024 That process is nodejs ( javascript ) that will output data to websocket and nats. I think I need to fix on that process then. Thanks. – Nrto Oct 11 '17 at 2:32
1

But some how some line is duplicate

If the duplicates are adjacent lines you can try uniq:

$ tail -f <file> | uniq

In a quick test this worked as expected, waiting for a complete line before writing the output.

If the duplicate lines are scattered over the file, adding uniq to the pipe between tail and awk might still help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.