A script that deletes extra spaces between letters in text

Question

I have a text document that has a load of text which has an extra space added after every letter!

Example:

T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t…

Visually:

T␣h␣e␣␣b␣o␣o␣k␣␣a␣l␣s␣o␣␣h␣a␣s␣␣a␣n␣␣a␣n␣a␣l␣y␣t␣i␣c␣a␣l␣␣p␣u␣r␣p␣o␣s␣e␣␣w␣h␣i␣c␣h␣␣i␣s␣␣m␣o␣r␣e␣␣i␣m␣p␣o␣r␣t␣a␣n␣t…

Note that there is an extra space after every letter, so there are two spaces between consecutive words.

Is there a way that I can get awk or sed to delete the extra spaces?  (Unfortunately this text document is massive and would take a very long time to go through manually.)  I appreciate that this is probably a much more complex problem to solve with just a simple bash script as there needs to be some sort of text recognition also.

How can I approach this problem?

Answer 1

Use wordsegment, a pure-Python word segmentation NLP package:

$ pip install wordsegment
$ python2.7 -m wordsegment <<<"T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t"
the book also has an analytical purpose which is more important

Answer 2

The following regex will remove the first space in any string of spaces. That should do the job.

s/ ( *)/\1/g

So something like:

perl -i -pe 's/ ( *)/\1/g' infile.txt

...will replace infile.txt with a "fixed" version.

Answer 3

Based on the fact that the input includes double spaces between words, there is a much simpler solution. You simply change the double spaces to an unused character, remove the spaces and change the unused character back to a space:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | sed 's/  /\-/g;s/ //g;s/\-/ /g'

...outputs:

The book also has an analytical purpose which is more important

Answer 4

Perl to the rescue!

You need a dictionary, i.e. a file listing one word per line. On my system, it exists as /var/lib/dict/words, I've also seen similar files as /usr/share/dict/british etc.

First, you remember all the words from the dictionary. Then, you read the input line by line, and try to add characters to a word. If it's possible, you remember the word and try to analyze the rest of the line. If you reach the end of the line, you output the line.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $words = '/var/lib/dict/words';
my %word;

sub analyze {
    my ($chars, $words, $pos) = @_;
    if ($pos == @$chars) {
        $_[3] = 1;  # Found.
        say "@$words";
        return
    }
    for my $to ($pos .. $#$chars) {
        my $try = join q(), @$chars[ $pos .. $to ];
        if (exists $word{$try}) {
            analyze($chars, [ @$words, $try ], $to + 1, $_[3]);
        }
    }
}


open my $WORDS, '<', $words or die $!;
undef @word{ map { chomp; lc $_ } <$WORDS> };

while (<>) {
    my @chars = map lc, /\S/g;
    analyze(\@chars, [], 0, my $found = 0);
    warn "Unknown: $_" unless $found;
}

For your input, it generates 4092 possible readings on my system.

Answer 5

Note: this answer (like a few others here) is based on an earlier version of the question where words were not delimited. The newer version can be trivially answered.

On an input like:

T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t

You could try:

 $ tr -d ' ' < file | grep -oiFf /usr/share/dict/words | paste -sd ' '
 The book also has ana na l y tic al purpose which ism ore important

It processes left to right and find one longest word after the next.

Obviously, here, it's not the best selection of words as that sentence doesn't make any sense, but to come up with the right one, you'd need tools able to understand the grammar or meaning of the text or at least some statistical information on what words are likely to be found together to come up with the most probable set of words. Looks like the solution is a specialised library as found by Lynn

Answer 6

Similar to Dewi Morgan's version, but with sed:

$ echo "f o o  t h e  b a r" | sed -r "s/[ ]{1}([^ ]{1})/\1/g"
foo the bar

Answer 7

Although it could (and should) be done with a Perl one-liner, a small C parser would be very fast, too, and is also very small (and hopefully very correct):

#include <stdio.h>
#include <stdlib.h>

int main()
{
  char c1 = '\0', c2 = '\0', tmp_c;

  c1 = fgetc(stdin);
  for (;;) {
    if (c1 == EOF) {
      break;
    }
    c2 = fgetc(stdin);
    if (c2 == EOF) {
      if (c1 != ' ') {
        fputc(c1, stdout);
      }
      break;
    }
    if (c1 == c2 && c1 == ' ') {
      tmp_c = fgetc(stdin);
      if (tmp_c != EOF) {
        if (tmp_c != '\n') {
          ungetc(tmp_c, stdin);
          fputc(' ', stdout);
        } else {
          ungetc(tmp_c, stdin);
        }
      } else {
        break;
      }
    } else if (c1 != ' ') {
      fputc(c1, stdout);
    }
    c1 = c2;
  }
  exit(EXIT_SUCCESS);
}

Compiled with

gcc-4.9 -O3 -g3  -W -Wall -Wextra -std=c11 lilcparser.c -o lilcparser

(programm is a bit less than 9kb)

Use in a pipe like e.g.:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | ./lilcparser

Answer 8

I tried this and it seems to work :

echo "<text here>" | sed -r 's/(\w)(\s)/\1/g'

The sed command captures two groups and returns only the first.

Answer 9

In c++, I would do this:

#include <fstream>
using namespace std;

int main()
{   
    fstream is("test.txt", std::ios::in);

    char buff;
    vector<char>str;

    while (!is.eof()){is.get(buff);str.push_back(buff);} //read file to string

    for (int a=0;a<str.size();++a)if (str[a] == ' ' && str[a + 1] != ' ')str.erase(str.begin()+a);
    is.close();

    ofstream os("test.txt", std::ios::out | std::ios::trunc); //clear file for rewrite

    os.write(str.data(), str.size() * sizeof(char)); //write chars
    os.close();

    return 0;
    }

Will change the contents of the test text file, into the same string, but with spaces between letters removed. (It requires a space between every letter to be accurate).

Answer 10

$ echo 'F o u r  s c o r e  a n d' | \
txr -t '(mapcar* (opip (split-str @1 "  ")
                       (mapcar (op regsub #/ / ""))
                       (cat-str @1 " "))
                 (get-lines))'
Four score and


$ txr -e '(awk (:begin (set fs "  "))
               ((mf (regsub #/ / ""))))'  # mf: modify fields
F o u r  s c o r e  a n d
Four score and


$ awk -F'  ' '{for(i=1;i<=NF;i++)gsub(/ /,"",$i);print}'
F o u r  s c o r e  a n d
Four score and

Answer 11

Using Raku (formerly known as Perl_6)

Similar to the Perl answer already posted by @DewiMorgan:

~$ raku -pe 's:g/ \s (\s*) /$0/;'    file

Raku is a programming language in the Perl family. Since Raku has simplified the use of codeblocks within Regex matchers, the following answers become very readable:

~$ raku -pe 's:g/ \s+ /{$/.chop}/;'  file

#OR:

~$ raku -pe 's:g/ (\s+) /{$0.chop}/;'  file

Note differences between Perl and Raku:

• In Raku regex modifier such as :g for global go at the head of the construct.
• In Raku matches are stored in the match-variable $/ (or $<>).
• Raku starts numbering captures from $0.

Sample Input:

T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t…

Sample Output:

The book also has an analytical purpose which is more important…

https://docs.raku.org/language/regexes
https://raku.org