1

I'm not sure how to phrase the question as most of the answers are about removing \r\n from a file.

I have a unique problem where compressed files are numbered randomly and in order to associate them properly with a database record, I need to list the file contents and check them.

I'm using this solutions "In bash script, how to capture stdout line by line"

Which was a great start.

Some of the contents have a name with whitespace and I found this solution: How to print third column to last column?

I discovered when attempting to update the database record, that ^M is being inserted in the results from of the awk pipe, but only for the NF column.

Not sure how to resolve this particular glitch. I don't see where ^M is being inserted, or how to remove it from the last column.

My code

This line works fine if I strip ^M

filename="$(echo "$line" | awk '{if ($3 ~ /^M$/) {sub(/^M$/,"", $3)} printf $3; printf ""}')"

This line fails:

text="$(echo "$line" | awk '{for(i=6;i<NF+1;i++) {if ($i ~ /^M$/) {sub(/^M$/,"", $i)} } printf "%s ", $i; printf ""}')"

And the simplified version fails:

text="$(echo "$line" | awk '{for(i=6;i<NF+1;i++) sub(/^M$/,"", $i) printf "%s ", $i; printf ""}')"

In vim/vi ^M is created with ctrl-V + <return key> Using \r\n has no effect.

I'm using cygwin, and have been for a long time, and I have other *nix scripts that I have written which run fine. I discovered that for some reason, this particular run of awk is adding ^M to the output.

I found this question with a similar problem, but I created my script with vim from the start so there was no windows based editor involved.

If I mount that windows folder as a samba share and run the script from linux it produces the output without a ^M, so at this point I'm wondering if this is a bug or something else. It's really strange.

UPDATE My use of the REGEX in sub() was causing the string to return empty, so I did not properly understand how to clear out the CRLF.

NF+1 was a leftover from attempting to find out what was introducing the CRLF I was using i<=NF before that.

  • 1
    do you have dos2unix installed? – αғsнιη Jan 24 at 7:01
  • Stripping carriage-returns from the end of lines is exactly one of the things that dos2unix does. – Kusalananda Jan 24 at 7:22
1

awk does not identify the ^M literally, it identifies it as the CRLF pattern \r\n, so your sub() can use the CR character representation directly as below. Also you don't have to check if the field contains a character and do a replacement. The replacement functions simply do nothing if the mentioned pattern is not found. So all you need is the following to replace the CR at the last column only.

awk '{ sub("\r", "", $NF); print $NF }' 

If there are multiple columns that need to be replaced, switch $NF with the appropriate column needed.

If you are doing this in a loop for all the columns up to the end of the file, just do

awk '{ for(i=6; i<=NF ; i++) { sub("\r", "", $i); printf "%s ", $i; } }'

Also file can only have utmost NF columns and $NF is the last column value. Change your loop to run until NF to access the last column value.

| improve this answer | |
  • I don't understand how that fits into my code. It looks like that's the sum total of the awk string. AAlso don't understand the function of the 1. – Ken Ingram Jan 24 at 6:53
  • @KenIngram: I added an explicit print statement now. You can modify it how you want to use it for your other column values – Inian Jan 24 at 6:54
  • I tried this: text="$(echo "$line" | awk '{for(i=6;i<NF+1;i++) {sub("\r", "", $NF) printf "%s ", $i; printf ""}}')" – Ken Ingram Jan 24 at 6:55
  • The key point is that the particular text is sometimes separated with whitespace. It's a name so the whitespace needs to be included. Thus the loop. Integrating that last column is proving to be a major obstacle. – Ken Ingram Jan 24 at 6:57
  • 1
    sub("\r", "", $NF) should be sub(/\r$/, "") since sub() takes a regexp as the first arg, doing the sub on $NF would force awk to rebuild each record unnecessarily, and if you have white space at the end of the line then the \r is after the white space rather than at the end of $NF. – Ed Morton Jan 24 at 16:06
3

With a few implementations of awk including GNU awk, mawk and busybox awk (the 3 implementations commonly found on Linux-based systems, Cygwin's being GNU awk by default I believe), RS the input record separator can be a regular expression (as opposed to a single character in POSIX).

In those, you can do:

awk -v RS='\r\n' '{print $NF}' < your-file.msdos

to process those files, or:

awk -v RS='\r?\n' '{print $NF}' < your-file.msdos-or-unix

to be able to process both files with \n delimiters or \r\n separators.

some MS-DOS files also tend to have the last line non-delimited, but awk will also fix that on output as it appends the output record separator (ORS which remains \n here) to all records when printing.

As far as default field splitting goes in awk, you'll also find that there is variation between implementations. POSIX says it should be split on sequences of blanks, leading and trailing ones removed. The notion of blank is locale dependant, and includes at least SPC and TAB. You'll find many awk implementations restrict it to SPC and TAB only regardless of the locale, many also add NL (only relevant when the record separator is not newline).

busybox awk includes all ASCII whitespace, so including CR, FF, VT. So in busybox awk, fields by default never contain CR. You can achieve the same behaviour with GNU awk by doing gawk -v 'FPAT=[^[:space:]]' where fields are then defined as sequences of non-whitespace.

A few more notes:

  • avoid shell loops to process text, especially here since you're already using awk which is one of the right tools to process text.
  • don't use echo on arbitrary data
  • the first argument to printf is the format, you don't want to use arbitrary data there. Use printf "%s", $3 if you want to print $3 without appending ORS, not printf $3.
  • printf "" is a no-op. It doesn't do anything. If you want to print a newline, use printf "\n" or print "" (the latter prints ORS, newline by default).
| improve this answer | |
  • Ok. Thanks for the insight. – Ken Ingram Jan 24 at 7:35
  • This was really good info. I see that I need to stop using "echo" the way I am doing it and use printf instead. This will help me level up my bash skills. Perhaps after all these years I will truly be intermediate level. – Ken Ingram Jan 24 at 8:22
  • 1
    Just be careful with RS='\r?\n' since some files, e.g. a CSV exported from MS-Excel, will use \r\n as the end of a record but use \n as linebreaks within fields so if you set RS='\r?\n' then you get parts of the record read as if it was a whole record since the \n within a field will also be interpreted as an end of a record. – Ed Morton Jan 24 at 16:03
  • 1
    Most of my ETL work is generated in a unix environment. However, this is a good caveat to tape to the toolbox. – Ken Ingram Jan 24 at 20:39
  • @EdMorton, a CSV can have \r\n or \n within a field, you can't process those CSVs with awk in any case in that way (regardless of whether RS is \n, \r\n or \r?\n). – Stéphane Chazelas Jan 24 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.