I'm trying to append file by watching data that constantly adding every second and pipe data to new files that will process it later. using
tail -f 'data file' >> 'destination file'
But some how some line is duplicate so I do this.
tail -f 'data file' | awk '!seen[$0]++' | >> 'destination file'
This seem fine , But the line would be split some time so I guess I need to buffer the whole line to achieve this. But is this a good way to do this or should I do it another way ?
tail -f 'data file' | grep --line-buffered -E '^.*$' | awk '!seen[$0++]'
Full data example
[9/10/17 14:21:09] Data=[xxx|yyy|zzz]
[9/1 [9/10/17 14:21:09] Data=[xx
something like this. This happen after I pipe data from
So I move
grep to end of command and it's work. But data will hold in buffer instead if it was too small.
tail -f 'data file' | awk '!seen[$0++]' | grep --line-buffered -E '^.*$'
Thanks for advice and answer !