I am cleaning up my data that is stored in text files. Each line starts with a category label followed by the actual data that I want to clean up. There are many text files in different subfolders, so I use egrep to pass the filenames on to sed.

CON: the Unix and Linux question
SEM: eins, the zwei, drei
AUTH: , the
AFF: The holy seat
TTITLE: As we go, the Kuckuck comes too

Now in every line starting with (SEM|AFF|CON) I want to replace (T|t)he[ ]* when it follows (:|\,). That is, the data should later look like

CON: Unix and Linux question
SEM: eins, zwei, drei
AUTH: , the
AFF: holy seat
TTITLE: As we go, the Kuckuck comes too

So far I tried to achieve this in two steps, one for the :-part and the other for the ,-part. But I struggle already with the first step.

First part The command/pattern to identify the files is egrep -rl ^"(SEM|CON|AFF)\: (t|T)he". This works as intended.

Now when I do

egrep -rl ^"(SEM|CON|AFF)\: (t|T)he" | xargs sed -i 's/\((SEM|CON|AFF)\: \)(t|T)he[ ]*/\1/g'

nothing happens. Is my sed part wrong? Can't I backrefer to ((SEM|CON|AFF)\: with \1?

Second part The command/pattern to identify the files is egrep -rl ^"(SEM|CON|AFF)\:.*\,[ ]*(t|T)he". This also works as intended. But every combination on sed that I tried so far deletes the content.

  • Unless you escape the ( in a Basic RE, it will be seen as a literal (. Try \(\(SEM|CON|AFF\): \)\(t|T\). Or use extended REs (sed -r), and replace all \( with just (. Since you used egrep, you're getting extended REs for free. – muru Dec 10 '14 at 11:39
  • You mean, I don't need -e when I use egrep? – MERose Dec 10 '14 at 11:47
  • egrep is like grep -E. -e is used to denote an expression. – muru Dec 10 '14 at 12:13

I would use the following:

sed -r '/(SEM|AFF|CON)/ s/([:,] *)[Tt]he */\1/g' file

Add -i option to change file in place.


Just use one sed expression (needs GNU sed):

sed -r -i -e '/(SEM|AFF|CON)/s/([:,]\s*)the\s+/\1/ig' *

The search pattern at the begin of the sed command restricts the replacement to the lines which begins with the selected categories. The i flag for the replace command (s//) makes the pattern case-insensitive, the g flag allows more than on replacement in a line. \s stands for any whitespace (space, tabs).

The sed uses extended regular expressions with the -r flag, this is for example supported by GNU sed. The -i flag enables inplace edit. The shell expands * to all files in the current directory.

The pattern ensures at least one space or tab after the so that words starting with the like theater are not replace.


Try this way:

egrep -rl "^(SEM|CON|AFF)\: (t|T)he" * | xargs sed -r -i 's/(^(SEM|CON|AFF):\s)((t|T)he[ ]*)/\1/g'
  • This works well for the first part, but how can I proceed with the second part, i.e. where the[ ]* follows a comma? Also, why the * in front of the pipe? – MERose Dec 10 '14 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.