file :

hi hello,new york,      ,  ,brazil site   ,brazil
january,month is feb   ,      , ,indiana jones


  • delete all horizontal space (spaces and tab) between any two comas (like , ,) if there is no character other than space. so it would look like :

    hi hello,new york,,,brazil site   ,brazil
    january,month is feb   ,,,indiana jones
  • also, delete any trailing or leading horizontal space (space and tab) for each parameter so that it looks like this.

    final output:

    hi hello,new york,,,brazil site,brazil
    january,month is feb,,,indiana jones

NOTE: The parameters itself has spaces like hi hello and indiana jones etc. It should not be affected and become like hihello or indianajones. If possible a solution with awk, otherwise sed will also do.

I am unable to achive the first task with sed like this:

sed -n 's/,[[:blank:]],/,,/gp' file

but i get this as output:

january,month is feb   ,      ,,indiana jones
  • 1
    Probably I misunderstand you, but why not just sed 's/ *, */,/g' file? – manatwork Feb 27 '13 at 8:11
  • thanks @manatwork it works, u got this this was easy and i have been thinking a lot. – munish Feb 27 '13 at 8:14
  • @manatwork, better sed 's/[[:space:]]*,[[:space:]]*/,/g to catch tabs too. – vonbrand Feb 27 '13 at 12:27
  • Right. Or sed 's/\s*,\s*/,/g' file to type less. – manatwork Feb 27 '13 at 13:14

There are three possibilities with sed:

  1. Simplest one (matching a space character zero or more times before a comma, and a space character zero or more times after the comma, replacing it by just a comma—and this many times, thus using the option g for global):

    sed 's/ *, */,/g' file

    Time for this command: 3.056s, with perl -p -e instead of sed 5.932s

  2. [:space:] means all whitespace characters, thus being the same as [ \t\r\n\v\f]. This is POSIX standard:

    sed 's/[[:space:]]*,[[:space:]]*/,/g' file

    Time for this command: 10.365s, with perl -p -e instead of sed 9.060s

  3. And finally the abbreviated version with \s, which stands for the same as [:space:] but is Perl syntax:

    sed 's/\s*,\s*/,/g' file

    Time for this command: 10.507s, with perl -p -e instead of sed 6.126s

All of them result in

hi hello,new york,,,brazil site,brazil
january,month is feb,,,indiana jones

The time was taken for a file with 2M lines (containing the questions example text 1M times). The file was 97MBytes big.

The first command seems to be the fastest with sed. The last command is the fastest when using perl and taking into account, that it replaces not only spaces but also tabstops, linefeed and so on.

  • All 3 regular expressions work in perl's replacement mode too =) perl -pi -e 's/\s*,\s*/,/g' file – Matt Mar 1 '13 at 9:49
  • To have the same as sed you should have written perl -p -e to be non in place editing. Your example perl -pi -e is the same as sed -i. – erik Mar 1 '13 at 11:20
  • ah yes. was thinking file replacement from the question. – Matt Mar 1 '13 at 12:53

So, you are asking awk:

awk '{$1=$1};gsub(" ,",",")'

Will this do? Note that this have the side-effect of removing duplicated space in your data inside commas, which might not be a problem for English.

  • May you please explain the command? What is this $1=$1 for? – erik Mar 1 '13 at 6:42
  • 2
    '$1=$1' is a refreshing, that tells awk to refresh the data it reads from input, after that $0 is changed, and output is re-structured with OFS(the out put seperater) – MeaCulpa Mar 1 '13 at 6:55
  • Time for this command: 5.038s and for the easier to understand and less to write version awk 'gsub(" +,",",")' only 3.061s. Same time as for the simple sed command in my answer. – erik Mar 1 '13 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.