sed - preserve newline when writing to new file

Question

I have files containing data. This data sometimes has artifacts in them which I need to get rid of. An example line looks like this:

@@@@@@@@@@DK2018.4.24_0:0:0.200985,0.88,0.35,0.49,13.52,248.3

Or like this:

\2017.9.12_0:0:0.152507,0.02,0.82,0.10,11.76,181.8
\2017.9.12_0:0:0.554122,0.18,0.93,0.04,11.76,191.1
\2017.9.12_0:0:0.654682,0.06,0.89,0.10,11.74,184.0
\2017.9.12_0:0:0.755092,0.00,0.89,0.06,11.77,180.5
\2017.9.12_0:0:0.855754,0.02,0.87,0.09,11.76,181.4
\2017.9.12_0:0:0.955123,0.13,0.80,0.23,11.77,189.8
\2017.9.12_0:0:1.055499,0.10,0.82,0.35,11.76,187.6
\2017.9.12_0:0:1.155970,0.18,0.81,0.40,11.74,192.9
\2017.9.12_0:0:1.256581,0.15,0.91,0.44,11.74,189.3
\2017.9.12_0:0:1.356065,0.26,0.78,0.46,11.72,198.7
\2017.9.12_0:0:1.456712,0.37,0.69,0.33,11.74,208.1

In both cases some unwanted characters are in front of the date string. I need to remove those and keep everything else. Sometimes these artifacts are not in the date column, but in one of the other columns.

What I tried is using sed like this:

sed 's/[^0-9:_.,]*//g' dat.log > test.log

The aim was to delete everything but numbers, colons, underscores, dots and commas. And that works just fine. The problem is that sed doesn't write back the newline. I know that it takes it off during the process, but how can I get it back when the line is written to a file?

edit: Added more lines to the example input and added output from my sed command:

2017.9.12_0:0:0.051920,0.03,0.74,0.09,11.72,182.72017.9.12_0:0:0.152507,0.02,0.82,0.10,11.76,181.82017.9.12_0:0:0.253551,0.00,0.89,0.04,11.77,180.52017.9.12_0:0:0.353267,0.04,0.96,0.02,11.77,182.72017.9.12_0:0:0.453707,0.15,0.95,0.02,11.71,189.32017.9.12_0:0:0.554122,0.18,0.93,0.04,11.76,191.12017.9.12_0:0:0.654682,0.06,0.89,0.10,11.74,184.02017.9.12_0:0:0.755092,0.00,0.89,0.06,11.77,180.52017.9.12_0:0:0.855754,0.02,0.87,0.09,11.76,181.42017.9.12_0:0:0.955123,0.13,0.80,0.23,11.77,189.82017.9.12_0:0:1.055499,0.10,0.82,0.35,11.76,187.62017.9.12_0:0:1.155970,0.18,0.81,0.40,11.74,192.92017.9.12_0:0:1.256581,0.15,0.91,0.44,11.74,189.32017.9.12_0:0:1.356065,0.26,0.78,0.46,11.72,198.72017.9.12_0:0:1.456712,0.37,0.69,0.33,11.74,208.1

edit2: It turned out the problem was that the Raspberry Pi where the data is coming from, saved the files with the macintosh dataset. No idea why, but changing that using tr '\r' '\n' < macfile.txt > unixfile.txt first solved the problem.

Answer 1

Here it seems like you can do:

tr -cd '0-9:_.,\r\n' < file.in > file.out

To remove all but those characters you want to keep including the line delimiters in whatever format they were initially.

Answer 2

Why don't you get rid of the tr and include that in the sed command:

sed 's/\(^\|^M\)[^0-9:_.,]*/\n/g; s/^\n//' file

May be a bit tricky to include the control characters (^M, \r, 0x0D) and (\n, 0x0A) in the sed script, depending on your OS and sed version.