12

I have a text document that has a load of text which has an extra space added after every letter!

Example:

T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t…

Visually:

T␣h␣e␣␣b␣o␣o␣k␣␣a␣l␣s␣o␣␣h␣a␣s␣␣a␣n␣␣a␣n␣a␣l␣y␣t␣i␣c␣a␣l␣␣p␣u␣r␣p␣o␣s␣e␣␣w␣h␣i␣c␣h␣␣i␣s␣␣m␣o␣r␣e␣␣i␣m␣p␣o␣r␣t␣a␣n␣t…

Note that there is an extra space after every letter, so there are two spaces between consecutive words.

Is there a way that I can get awk or sed to delete the extra spaces?  (Unfortunately this text document is massive and would take a very long time to go through manually.)  I appreciate that this is probably a much more complex problem to solve with just a simple bash script as there needs to be some sort of text recognition also.

How can I approach this problem?

  • 2
    it is trivial to replace all spaces with nothing.. but I think you'd want to separate the words? – Sundeep Sep 10 '16 at 12:19
  • for ex: echo 't h i s i s a n e x a m p l e' | sed 's/ //g' – Sundeep Sep 10 '16 at 12:20
  • 1
    That doesn't limit the change to spaces between letters. (Digits and punctuation aren't letters, for instance). You can do this in sed with a loop. This also is probably a duplicate. – Thomas Dickey Sep 10 '16 at 12:44
  • 1
    to restrict only between letters: echo 'T h i s ; i s .a n 9 8 e x a m p l e' | perl -pe 's/[a-z]\K (?=[a-z])//ig' – Sundeep Sep 10 '16 at 12:55
  • 4
    @JuliePelletier: The source of the original revision shows that the spaces between words were doubled. Why did you un-double them in your edit? – El'endia Starman Sep 10 '16 at 17:13

10 Answers 10

16

The following regex will remove the first space in any string of spaces. That should do the job.

s/ ( *)/\1/g

So something like:

perl -i -pe 's/ ( *)/\1/g' infile.txt

...will replace infile.txt with a "fixed" version.

  • @terdon I've noticed in recent times that people have stopped writing perl pie scripts as perl -pie - as your edit shows. What's the rationale for this? The -pie has always worked well for me, and is a great mnemonic. Has -i's behavior changed to treat anything following as an extension, rather than only those things which begin with a dot? It would seem strange for them to break something so idiomatic. – Dewi Morgan Sep 14 '16 at 22:13
  • 1
    Huh, well it's not an idiom I'm familiar with. Perl has been this way for as long as I've been using -i. On the other hand, I've only ever used it on Linux machines and I haven't known about it for more than a few years, so I can't speak as to its older behavior. On my machine though, this: perl -pie 's/a/b/' f, produces an error: Can't open perl script "s/o/A/": No such file or directory. While perl -i -pe 's/o/A/' f works as expected. So yes, the e is taken as the backup extension. – terdon Sep 16 '16 at 12:42
  • Sadface. Ah, well, time moves on, and it just means I need to relearn a parameter order. Keeps my brain squishy, I guess. Thanks for letting me know, and for fixing my code! – Dewi Morgan Sep 16 '16 at 14:27
17

Use wordsegment, a pure-Python word segmentation NLP package:

$ pip install wordsegment
$ python2.7 -m wordsegment <<<"T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t"
the book also has an analytical purpose which is more important
  • 1
    Using NLP is probably the most effective solution if there is nothing else to tell the words apart. NLP performs better than a look-ahead dictionary in most cases. – grochmal Sep 11 '16 at 2:26
13

Based on the fact that the input includes double spaces between words, there is a much simpler solution. You simply change the double spaces to an unused character, remove the spaces and change the unused character back to a space:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | sed 's/  /\-/g;s/ //g;s/\-/ /g'

...outputs:

The book also has an analytical purpose which is more important

  • 5
    A sed command with a meaning "replace every occurence of a non-space character, followed by a space with just the corresponding non-space character" does the same:sed -e "s/\([^ ]\) /\1/g" – woodengod Sep 11 '16 at 2:55
  • 3
    That is indeed a good alternative. You should post it as an answer to get credit for it. – Julie Pelletier Sep 11 '16 at 3:23
10

Perl to the rescue!

You need a dictionary, i.e. a file listing one word per line. On my system, it exists as /var/lib/dict/words, I've also seen similar files as /usr/share/dict/british etc.

First, you remember all the words from the dictionary. Then, you read the input line by line, and try to add characters to a word. If it's possible, you remember the word and try to analyze the rest of the line. If you reach the end of the line, you output the line.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $words = '/var/lib/dict/words';
my %word;

sub analyze {
    my ($chars, $words, $pos) = @_;
    if ($pos == @$chars) {
        $_[3] = 1;  # Found.
        say "@$words";
        return
    }
    for my $to ($pos .. $#$chars) {
        my $try = join q(), @$chars[ $pos .. $to ];
        if (exists $word{$try}) {
            analyze($chars, [ @$words, $try ], $to + 1, $_[3]);
        }
    }
}


open my $WORDS, '<', $words or die $!;
undef @word{ map { chomp; lc $_ } <$WORDS> };

while (<>) {
    my @chars = map lc, /\S/g;
    analyze(\@chars, [], 0, my $found = 0);
    warn "Unknown: $_" unless $found;
}

For your input, it generates 4092 possible readings on my system.

  • fails test with spaced out version of a cat a log ie a c a t a l o g – ctrl-alt-delor Sep 10 '16 at 13:43
  • @richard: OBOE, fixed. But it now generates too many possibilites, try to remove one letter words. – choroba Sep 10 '16 at 13:49
  • @richard You might fight this problem with help of a non-deterministic algorithm (e.g. all possible readings are stored) and apply a parser on it. Then you could filter all 4000 possible readings to the single one with the least error count. – bash0r Sep 10 '16 at 19:29
6

Note: this answer (like a few others here) is based on an earlier version of the question where words were not delimited. The newer version can be trivially answered.

On an input like:

T h e b o o k a l s o h a s a n a n a l y t i c a l p u r p o s e w h i c h i s m o r e i m p o r t a n t

You could try:

 $ tr -d ' ' < file | grep -oiFf /usr/share/dict/words | paste -sd ' '
 The book also has ana na l y tic al purpose which ism ore important

It processes left to right and find one longest word after the next.

Obviously, here, it's not the best selection of words as that sentence doesn't make any sense, but to come up with the right one, you'd need tools able to understand the grammar or meaning of the text or at least some statistical information on what words are likely to be found together to come up with the most probable set of words. Looks like the solution is a specialised library as found by Lynn

  • @terdon, see edit. The problem is that that question was changed from a complex and interesting one into a trivial one. Is there a way you could split it into the two questions that it was before and after the edit? – Stéphane Chazelas Sep 11 '16 at 17:53
  • I'm afraid not, no. Still a clever trick though, even if not perfect. – terdon Sep 11 '16 at 17:59
  • 1
    Strictly speaking, the question was trivial from the beginning — see the first version and its source.  Unfortunately, the OP didn't understand how Stack Exchange renders text, so the correct input text wasn't visible until trichoplax fixed the formatting — and, even more unfortunately, it wasn't visible then, because the person who approved that edit immediately went and broke it. – Scott Sep 11 '16 at 22:12
2

Similar to Dewi Morgan's version, but with sed:

$ echo "f o o  t h e  b a r" | sed -r "s/[ ]{1}([^ ]{1})/\1/g"
foo the bar
  • That's GNU sed only and that's not equivalent to Dewi's. The standard sed equivalent of Dewi's would be sed 's/ \( *\)/\1/g' – Stéphane Chazelas Sep 12 '16 at 10:01
  • note the "similar" ;-) – Jaleks Sep 18 '16 at 9:45
1

Although it could (and should) be done with a Perl one-liner, a small C parser would be very fast, too, and is also very small (and hopefully very correct):

#include <stdio.h>
#include <stdlib.h>

int main()
{
  char c1 = '\0', c2 = '\0', tmp_c;

  c1 = fgetc(stdin);
  for (;;) {
    if (c1 == EOF) {
      break;
    }
    c2 = fgetc(stdin);
    if (c2 == EOF) {
      if (c1 != ' ') {
        fputc(c1, stdout);
      }
      break;
    }
    if (c1 == c2 && c1 == ' ') {
      tmp_c = fgetc(stdin);
      if (tmp_c != EOF) {
        if (tmp_c != '\n') {
          ungetc(tmp_c, stdin);
          fputc(' ', stdout);
        } else {
          ungetc(tmp_c, stdin);
        }
      } else {
        break;
      }
    } else if (c1 != ' ') {
      fputc(c1, stdout);
    }
    c1 = c2;
  }
  exit(EXIT_SUCCESS);
}

Compiled with

gcc-4.9 -O3 -g3  -W -Wall -Wextra -std=c11 lilcparser.c -o lilcparser

(programm is a bit less than 9kb)

Use in a pipe like e.g.:

echo "T h e  b o o k  a l s o  h a s  a n  a n a l y t i c a l  p u r p o s e  w h i c h  i s  m o r e  i m p o r t a n t  " | ./lilcparser
1

I tried this and it seems to work :

echo "<text here>" | sed -r 's/(\w)(\s)/\1/g'

The sed command captures two groups and returns only the first.

0

In c++, I would do this:

#include <fstream>
using namespace std;

int main()
{   
    fstream is("test.txt", std::ios::in);

    char buff;
    vector<char>str;

    while (!is.eof()){is.get(buff);str.push_back(buff);} //read file to string

    for (int a=0;a<str.size();++a)if (str[a] == ' ' && str[a + 1] != ' ')str.erase(str.begin()+a);
    is.close();

    ofstream os("test.txt", std::ios::out | std::ios::trunc); //clear file for rewrite

    os.write(str.data(), str.size() * sizeof(char)); //write chars
    os.close();

    return 0;
    }

Will change the contents of the test text file, into the same string, but with spaces between letters removed. (It requires a space between every letter to be accurate).

0
$ echo 'F o u r  s c o r e  a n d' | \
txr -t '(mapcar* (opip (split-str @1 "  ")
                       (mapcar (op regsub #/ / ""))
                       (cat-str @1 " "))
                 (get-lines))'
Four score and


$ txr -e '(awk (:begin (set fs "  "))
               ((mf (regsub #/ / ""))))'  # mf: modify fields
F o u r  s c o r e  a n d
Four score and


$ awk -F'  ' '{for(i=1;i<=NF;i++)gsub(/ /,"",$i);print}'
F o u r  s c o r e  a n d
Four score and

protected by Michael Mrozek Sep 13 '16 at 5:49

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.