I have files containing data. This data sometimes has artifacts in them which I need to get rid of. An example line looks like this:
Or like this:
\2017.9.12_0:0:0.152507,0.02,0.82,0.10,11.76,181.8 \2017.9.12_0:0:0.554122,0.18,0.93,0.04,11.76,191.1 \2017.9.12_0:0:0.654682,0.06,0.89,0.10,11.74,184.0 \2017.9.12_0:0:0.755092,0.00,0.89,0.06,11.77,180.5 \2017.9.12_0:0:0.855754,0.02,0.87,0.09,11.76,181.4 \2017.9.12_0:0:0.955123,0.13,0.80,0.23,11.77,189.8 \2017.9.12_0:0:1.055499,0.10,0.82,0.35,11.76,187.6 \2017.9.12_0:0:1.155970,0.18,0.81,0.40,11.74,192.9 \2017.9.12_0:0:1.256581,0.15,0.91,0.44,11.74,189.3 \2017.9.12_0:0:1.356065,0.26,0.78,0.46,11.72,198.7 \2017.9.12_0:0:1.456712,0.37,0.69,0.33,11.74,208.1
In both cases some unwanted characters are in front of the date string. I need to remove those and keep everything else. Sometimes these artifacts are not in the date column, but in one of the other columns.
What I tried is using sed like this:
sed 's/[^0-9:_.,]*//g' dat.log > test.log
The aim was to delete everything but numbers, colons, underscores, dots and commas. And that works just fine. The problem is that sed doesn't write back the newline. I know that it takes it off during the process, but how can I get it back when the line is written to a file?
edit: Added more lines to the example input and added output from my sed command:
edit2: It turned out the problem was that the Raspberry Pi where the data is coming from, saved the files with the macintosh dataset. No idea why, but changing that using
tr '\r' '\n' < macfile.txt > unixfile.txt first solved the problem.
sedworks line by line -- it reads line, and it writes lines, complete with terminating newline. It cannot not write a newline after every line it outputs.