Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Currently, I have a file (Results.txt) which has 27k rows. The file content is as below.

45008657        |       A.      |               |       Long    |       0
49210987        |       A.      |               |       Steven  |       3262
49600694        |       A.      |               |       Steven    |       4772
467814  |       A.      |       Rafeal    |       Nadal|       4133
53111047        |       A.      |       Mike    |       Tyson  |       13484

I need to remove the extra white spaces, which am doing like below.

cat Results.txt | tr -d " \t\r" > Results1.txt

The file (Results1.txt) is neatly formatted with no extra white spaces.


Now, I need to convert all the alphabets to lowercase and append columns 2 to 4.

awk 'BEGIN { FS = "|" } ; { print $1"|"tolower($2) tolower($3) tolower($4)"|"$5 }' Results1.txt > Results2.txt

My Results2.txt file will look like below.


Is there any way I could accomplish all the above mentioned steps in my initial file (Results.txt) itself? I want to reduce over usage of files so that it would be easy for debugging in later stages.

EDIT: I simply do not want to remove the files once after creating a new file which doesn't seem an elegant solution to me.

share|improve this question
In the tr command, you do not need to cat the file. You can do it as follows: tr -d " \t\r" < Results.txt > tmp.txt && mv tmp.txt Results.txt – Ketan Mar 31 '14 at 18:40
For what you want to do, you can write vim macros. Then, things will get edited within vim and you won't need to save intermediate files. – Ketan Mar 31 '14 at 18:42
Thanks. I will check vim macros. For now, I think I will stick to devnull's answer with your mv approach. :) – Ramesh Mar 31 '14 at 18:43
you could first lower case with dd if=input.txt of=output.txt conv=lcase then ... let the unicorn decide – Kiwy Apr 1 '14 at 11:35
up vote 10 down vote accepted

With GNU sed:

sed -i 's/\s//g;s/\(|[^|]*\)|\([^|]*\)|/\1\2/;s/.*/\L&/' Results.txt
share|improve this answer
This command is the one I am looking for. Thanks a lot :) – Ramesh Mar 31 '14 at 19:42
Unless I'm missing something, this would skip the third field. (That is, the resulting output would be different from the expected output in the question.) – devnull Apr 1 '14 at 16:09
@devnull, indeed. I had misread the requirements. Should be OK now. – Stéphane Chazelas Apr 1 '14 at 19:59
Couldn't you do this with any sed if you just y/A-Z/a-z/? – mikeserv Apr 1 '14 at 20:01
Apologies for the nitpick. However, it seems that the question wasn't clear enough (given that the previous version of your answer was accepted)! – devnull Apr 1 '14 at 20:02

You don't need a temporary file, indeed.

awk -F'|' '{gsub("[ \t\r]", "", $0)}{print $1,tolower($2$3$4),$5}' OFS='|' inputfile

The first part of the expression does what you do using tr. The second part conbines fields 2-4 and lowercases those and prints it with the other two fields.

For your input, it'd produce:


GNU awk 4.1.0 and higher support in-place editing. You can say:

awk -i inplace -F'|' '{gsub("[ \t\r]", "", $0)}{print $1,tolower($2$3$4),$5}' OFS='|' inputfile
share|improve this answer
Yeah, but the inputfile needs to be saved as different file after getting the output right? I need only the inputfile alone. – Ramesh Mar 31 '14 at 18:40
For now, I will stick to your answer with ketan's mv approach. Thanks :) – Ramesh Mar 31 '14 at 18:43
@Ramesh GNU awk 4.1.0 and higher support in-place editing. Updating the answer. – devnull Mar 31 '14 at 19:07
unfortunately my server has awk 3.1.5 version only. But anyways thanks for the information :) – Ramesh Mar 31 '14 at 19:12
stephane's and terdon's commands were much closer to what I originally intended to achieve. :) – Ramesh Mar 31 '14 at 19:44

With your original attempt, you can put all the commands in the same pipeline if you use the sponge utility:

cat Results.txt |
  tr -d " \t\r" |
  awk 'BEGIN { FS = "|" } ;
    { print $1"|"tolower($2) tolower($3) tolower($4)"|"$5 }' |
    sponge Results.txt

sponge will store its stdin in memory until it reaches the end of input, then write to file. This should be acceptable for a file of the size mentioned.

Note you can also use a shell redirection to input a file to td rather than cat, eg:

tr -d " \t\r" <Results.txt
share|improve this answer
thanks for letting me know about sponge command. I will check it out :) – Ramesh Mar 31 '14 at 19:59

Just use a tool that allows in place editing. Perl for example:

perl -i -F'\|' -lane 's/[ \t]+//g for @F; print lc("$F[0]|$F[1]$F[2]$F[3]|$F[4]")' a
  • The -i turns on in-plcae editing, the changes are applied directly to the original file (in other words, perl deals with creating and deleting the temp file).

  • The -a turns on automatic splitting on the character given by -F, the fields are saved in the array @F.

  • The s/[ \t]+//g removes all spaces and tabs and the lc() makes everything lowercase.

share|improve this answer
it is perfect. But however, I get the output as, 45008657|a.long0| instead of 45008657|a.long|0. The command that you had mentioned seems to have the | at correct place only. Is there any other reason for the misplaced |? – Ramesh Mar 31 '14 at 19:27
@Ramesh I can't reproduce. I tried this on the example you gave and it works as expected. – terdon Mar 31 '14 at 19:31
stephane's solution was something which I originally intended to achieve :) I am still puzzled on why the | gets misplaced after the last column using the perl solution. – Ramesh Mar 31 '14 at 19:45
@Ramesh that should only happen if you are missing a | in the input file. I can't think of any other reason and as I said, I can't reproduce it with your example. Stephane's is better though (as usual). – terdon Mar 31 '14 at 19:48

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.