I have a text document that has a load of text which has an extra space added after every letter!


T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t…



Note that there is an extra space after every letter, so there are two spaces between consecutive words.

Is there a way that I can get awk or sed to delete the extra spaces?  (Unfortunately this text document is massive and would take a very long time to go through manually.)  I appreciate that this is probably a much more complex problem to solve with just a simple bash script as there needs to be some sort of text recognition also.

How can I approach this problem?

  • 2
    it is trivial to replace all spaces with nothing.. but I think you'd want to separate the words?
    – Sundeep
    Sep 10, 2016 at 12:19
  • 1
    That doesn't limit the change to spaces between letters. (Digits and punctuation aren't letters, for instance). You can do this in sed with a loop. This also is probably a duplicate. Sep 10, 2016 at 12:44
  • 1
    to restrict only between letters: echo 'T h i s ; i s .a n 9 8 e x a m p l e' | perl -pe 's/[a-z]\K (?=[a-z])//ig'
    – Sundeep
    Sep 10, 2016 at 12:55
  • 4
    @JuliePelletier: The source of the original revision shows that the spaces between words were doubled. Why did you un-double them in your edit? Sep 10, 2016 at 17:13
  • 1
    Are these really spaces, or is this UTF-16 from a Windows machine? Sep 12, 2016 at 20:39

10 Answers 10


Use wordsegment, a pure-Python word segmentation NLP package:

$ pip install wordsegment
$ python2.7 -m wordsegment <<<"T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t"
the book also has an analytical purpose which is more important
  • 1
    Using NLP is probably the most effective solution if there is nothing else to tell the words apart. NLP performs better than a look-ahead dictionary in most cases.
    – grochmal
    Sep 11, 2016 at 2:26

The following regex will remove the first space in any string of spaces. That should do the job.

s/ ( *)/\1/g

So something like:

perl -i -pe 's/ ( *)/\1/g' infile.txt

...will replace infile.txt with a "fixed" version.

  • @terdon I've noticed in recent times that people have stopped writing perl pie scripts as perl -pie - as your edit shows. What's the rationale for this? The -pie has always worked well for me, and is a great mnemonic. Has -i's behavior changed to treat anything following as an extension, rather than only those things which begin with a dot? It would seem strange for them to break something so idiomatic. Sep 14, 2016 at 22:13
  • 1
    Huh, well it's not an idiom I'm familiar with. Perl has been this way for as long as I've been using -i. On the other hand, I've only ever used it on Linux machines and I haven't known about it for more than a few years, so I can't speak as to its older behavior. On my machine though, this: perl -pie 's/a/b/' f, produces an error: Can't open perl script "s/o/A/": No such file or directory. While perl -i -pe 's/o/A/' f works as expected. So yes, the e is taken as the backup extension.
    – terdon
    Sep 16, 2016 at 12:42
  • Sadface. Ah, well, time moves on, and it just means I need to relearn a parameter order. Keeps my brain squishy, I guess. Thanks for letting me know, and for fixing my code! Sep 16, 2016 at 14:27

Based on the fact that the input includes double spaces between words, there is a much simpler solution. You simply change the double spaces to an unused character, remove the spaces and change the unused character back to a space:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | sed 's/  /\-/g;s/ //g;s/\-/ /g'


The book also has an analytical purpose which is more important

  • 5
    A sed command with a meaning "replace every occurence of a non-space character, followed by a space with just the corresponding non-space character" does the same:sed -e "s/\([^ ]\) /\1/g"
    – woodengod
    Sep 11, 2016 at 2:55
  • 3
    That is indeed a good alternative. You should post it as an answer to get credit for it. Sep 11, 2016 at 3:23

Perl to the rescue!

You need a dictionary, i.e. a file listing one word per line. On my system, it exists as /var/lib/dict/words, I've also seen similar files as /usr/share/dict/british etc.

First, you remember all the words from the dictionary. Then, you read the input line by line, and try to add characters to a word. If it's possible, you remember the word and try to analyze the rest of the line. If you reach the end of the line, you output the line.

use warnings;
use strict;
use feature qw{ say };

my $words = '/var/lib/dict/words';
my %word;

sub analyze {
    my ($chars, $words, $pos) = @_;
    if ($pos == @$chars) {
        $_[3] = 1;  # Found.
        say "@$words";
    for my $to ($pos .. $#$chars) {
        my $try = join q(), @$chars[ $pos .. $to ];
        if (exists $word{$try}) {
            analyze($chars, [ @$words, $try ], $to + 1, $_[3]);

open my $WORDS, '<', $words or die $!;
undef @word{ map { chomp; lc $_ } <$WORDS> };

while (<>) {
    my @chars = map lc, /\S/g;
    analyze(\@chars, [], 0, my $found = 0);
    warn "Unknown: $_" unless $found;

For your input, it generates 4092 possible readings on my system.

  • fails test with spaced out version of a cat a log ie a c a t a l o g Sep 10, 2016 at 13:43
  • @richard: OBOE, fixed. But it now generates too many possibilites, try to remove one letter words.
    – choroba
    Sep 10, 2016 at 13:49
  • @richard You might fight this problem with help of a non-deterministic algorithm (e.g. all possible readings are stored) and apply a parser on it. Then you could filter all 4000 possible readings to the single one with the least error count.
    – bash0r
    Sep 10, 2016 at 19:29

Note: this answer (like a few others here) is based on an earlier version of the question where words were not delimited. The newer version can be trivially answered.

On an input like:

T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t

You could try:

 $ tr -d ' ' < file | grep -oiFf /usr/share/dict/words | paste -sd ' '
 The book also has ana na l y tic al purpose which ism ore important

It processes left to right and find one longest word after the next.

Obviously, here, it's not the best selection of words as that sentence doesn't make any sense, but to come up with the right one, you'd need tools able to understand the grammar or meaning of the text or at least some statistical information on what words are likely to be found together to come up with the most probable set of words. Looks like the solution is a specialised library as found by Lynn

  • @terdon, see edit. The problem is that that question was changed from a complex and interesting one into a trivial one. Is there a way you could split it into the two questions that it was before and after the edit? Sep 11, 2016 at 17:53
  • I'm afraid not, no. Still a clever trick though, even if not perfect.
    – terdon
    Sep 11, 2016 at 17:59
  • 1
    Strictly speaking, the question was trivial from the beginning — see the first version and its source.  Unfortunately, the OP didn't understand how Stack Exchange renders text, so the correct input text wasn't visible until trichoplax fixed the formatting — and, even more unfortunately, it wasn't visible then, because the person who approved that edit immediately went and broke it. Sep 11, 2016 at 22:12

Similar to Dewi Morgan's version, but with sed:

$ echo "f o o  t h e  b a r" | sed -r "s/[ ]{1}([^ ]{1})/\1/g"
foo the bar
  • That's GNU sed only and that's not equivalent to Dewi's. The standard sed equivalent of Dewi's would be sed 's/ \( *\)/\1/g' Sep 12, 2016 at 10:01
  • note the "similar" ;-)
    – Jaleks
    Sep 18, 2016 at 9:45

Although it could (and should) be done with a Perl one-liner, a small C parser would be very fast, too, and is also very small (and hopefully very correct):

#include <stdio.h>
#include <stdlib.h>

int main()
  char c1 = '\0', c2 = '\0', tmp_c;

  c1 = fgetc(stdin);
  for (;;) {
    if (c1 == EOF) {
    c2 = fgetc(stdin);
    if (c2 == EOF) {
      if (c1 != ' ') {
        fputc(c1, stdout);
    if (c1 == c2 && c1 == ' ') {
      tmp_c = fgetc(stdin);
      if (tmp_c != EOF) {
        if (tmp_c != '\n') {
          ungetc(tmp_c, stdin);
          fputc(' ', stdout);
        } else {
          ungetc(tmp_c, stdin);
      } else {
    } else if (c1 != ' ') {
      fputc(c1, stdout);
    c1 = c2;

Compiled with

gcc-4.9 -O3 -g3  -W -Wall -Wextra -std=c11 lilcparser.c -o lilcparser

(programm is a bit less than 9kb)

Use in a pipe like e.g.:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | ./lilcparser

I tried this and it seems to work :

echo "<text here>" | sed -r 's/(\w)(\s)/\1/g'

The sed command captures two groups and returns only the first.


In c++, I would do this:

#include <fstream>
using namespace std;

int main()
    fstream is("test.txt", std::ios::in);

    char buff;

    while (!is.eof()){is.get(buff);str.push_back(buff);} //read file to string

    for (int a=0;a<str.size();++a)if (str[a] == ' ' && str[a + 1] != ' ')str.erase(str.begin()+a);

    ofstream os("test.txt", std::ios::out | std::ios::trunc); //clear file for rewrite

    os.write(str.data(), str.size() * sizeof(char)); //write chars

    return 0;

Will change the contents of the test text file, into the same string, but with spaces between letters removed. (It requires a space between every letter to be accurate).

$ echo 'F o u r  s c o r e  a n d' | \
txr -t '(mapcar* (opip (split-str @1 "  ")
                       (mapcar (op regsub #/ / ""))
                       (cat-str @1 " "))
Four score and

$ txr -e '(awk (:begin (set fs "  "))
               ((mf (regsub #/ / ""))))'  # mf: modify fields
F o u r  s c o r e  a n d
Four score and

$ awk -F'  ' '{for(i=1;i<=NF;i++)gsub(/ /,"",$i);print}'
F o u r  s c o r e  a n d
Four score and

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .