0

I have files containing data. This data sometimes has artifacts in them which I need to get rid of. An example line looks like this:

@@@@@@@@@@DK2018.4.24_0:0:0.200985,0.88,0.35,0.49,13.52,248.3

Or like this:

\2017.9.12_0:0:0.152507,0.02,0.82,0.10,11.76,181.8
\2017.9.12_0:0:0.554122,0.18,0.93,0.04,11.76,191.1
\2017.9.12_0:0:0.654682,0.06,0.89,0.10,11.74,184.0
\2017.9.12_0:0:0.755092,0.00,0.89,0.06,11.77,180.5
\2017.9.12_0:0:0.855754,0.02,0.87,0.09,11.76,181.4
\2017.9.12_0:0:0.955123,0.13,0.80,0.23,11.77,189.8
\2017.9.12_0:0:1.055499,0.10,0.82,0.35,11.76,187.6
\2017.9.12_0:0:1.155970,0.18,0.81,0.40,11.74,192.9
\2017.9.12_0:0:1.256581,0.15,0.91,0.44,11.74,189.3
\2017.9.12_0:0:1.356065,0.26,0.78,0.46,11.72,198.7
\2017.9.12_0:0:1.456712,0.37,0.69,0.33,11.74,208.1

In both cases some unwanted characters are in front of the date string. I need to remove those and keep everything else. Sometimes these artifacts are not in the date column, but in one of the other columns.

What I tried is using sed like this:

sed 's/[^0-9:_.,]*//g' dat.log > test.log

The aim was to delete everything but numbers, colons, underscores, dots and commas. And that works just fine. The problem is that sed doesn't write back the newline. I know that it takes it off during the process, but how can I get it back when the line is written to a file?

edit: Added more lines to the example input and added output from my sed command:

2017.9.12_0:0:0.051920,0.03,0.74,0.09,11.72,182.72017.9.12_0:0:0.152507,0.02,0.82,0.10,11.76,181.82017.9.12_0:0:0.253551,0.00,0.89,0.04,11.77,180.52017.9.12_0:0:0.353267,0.04,0.96,0.02,11.77,182.72017.9.12_0:0:0.453707,0.15,0.95,0.02,11.71,189.32017.9.12_0:0:0.554122,0.18,0.93,0.04,11.76,191.12017.9.12_0:0:0.654682,0.06,0.89,0.10,11.74,184.02017.9.12_0:0:0.755092,0.00,0.89,0.06,11.77,180.52017.9.12_0:0:0.855754,0.02,0.87,0.09,11.76,181.42017.9.12_0:0:0.955123,0.13,0.80,0.23,11.77,189.82017.9.12_0:0:1.055499,0.10,0.82,0.35,11.76,187.62017.9.12_0:0:1.155970,0.18,0.81,0.40,11.74,192.92017.9.12_0:0:1.256581,0.15,0.91,0.44,11.74,189.32017.9.12_0:0:1.356065,0.26,0.78,0.46,11.72,198.72017.9.12_0:0:1.456712,0.37,0.69,0.33,11.74,208.1

edit2: It turned out the problem was that the Raspberry Pi where the data is coming from, saved the files with the macintosh dataset. No idea why, but changing that using tr '\r' '\n' < macfile.txt > unixfile.txt first solved the problem.

  • sed would write back the newline, a Unix newline. If your original file is a DOS text file, then it would get converted to a Unix text file by your sed command since you would delete the carriage return at the end of each line. Do edit your question and add examples of what you're seeing. – Kusalananda Aug 9 '18 at 8:24
  • You must give an actual example, because poor sed works line by line -- it reads line, and it writes lines, complete with terminating newline. It cannot not write a newline after every line it outputs. – AlexP Aug 9 '18 at 8:25
  • added more input lines and the output from my command @AlexP. Files are coming from a Raspberry Pi and never saw DOS. – BallerNacken Aug 9 '18 at 8:30
  • Cannot reproduce the behavior. On what operating system are you seeing this apparently impossible behavior? – AlexP Aug 9 '18 at 8:36
  • 1
    Post a hexdump of your file. – RudiC Aug 9 '18 at 8:55
0

Here it seems like you can do:

tr -cd '0-9:_.,\r\n' < file.in > file.out

To remove all but those characters you want to keep including the line delimiters in whatever format they were initially.

  • That works great! And is a lot faster compared to sed! – BallerNacken Aug 9 '18 at 9:24
0

Why don't you get rid of the tr and include that in the sed command:

sed 's/\(^\|^M\)[^0-9:_.,]*/\n/g; s/^\n//' file

May be a bit tricky to include the control characters (^M, \r, 0x0D) and (\n, 0x0A) in the sed script, depending on your OS and sed version.

  • Simply because I have no clue how that works. :) Your command doesn't do anything. The characters are still there and it is still in the macintosh char set. – BallerNacken Aug 9 '18 at 9:22
  • You can enter control characters on the shell (bash) command line like Ctrl-V Ctrl-M for ^M and Ctrl-V Ctrl-J for \n. – RudiC Aug 9 '18 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.