I'm stuck - piping a text file like this:

cat file1.txt | sed '/^[0-9].*[0-9]$/d' > file2.txt

This regex catches the lines in a text editor, and it works when I use it to delete all blank lines in the same file, so no problem with (Linux/Windows) newline format I guess.

I wonder why this does not delete those lines, or how this can be done otherwise?

  • So what is your problem?
    – cuonglm
    Nov 18 '15 at 12:17
  • @cuonglm: sorry for not being clear enough; edited now.
    – Sadi
    Nov 18 '15 at 12:24
  • I don't understand the question, the sed command works. But I don't understand your second sentence. "This regex catches the lines in a text editor" what text editor? "it works when I use it to delete all blank lines" blank lines? I thought its about begin and end with a digit? You may need to give example input and desired output.
    – chaos
    Nov 18 '15 at 12:43
  • Has the source data file come from a Windows system? If so, [0-9] will not match the last character on a line.
    – roaima
    Nov 18 '15 at 13:14
  • @chaos ; no the sed command above doesn't work although I can search and replace / delete lines using the same regex, i.e. ^[0-9].*[0-9]$ in a text editor like gedit or kkedit.
    – Sadi
    Nov 18 '15 at 13:51

Some extra notes:

sed '/^[0-9].*[0-9]$/d'

Would fail to delete the lines that contain a single digit (even though they start and end in a digit).

Also note that in many locales, [0-9] matches more than the 0123456789 digits. If you want to match on those only, you'd need [0123456789] or [[:digit:]].

The more obvious command to filter lines based on a pattern is grep.

cat is the command to concatenate. It makes little sense to concatenate a single file.

Here, to address all those and ignore any leading or trailing spacing characters (like the MS-DOS trailing carriage returns in your input), you could do:

<file.txt.in grep -e '^[[:space:]]*[^[:digit:][:space:]]' \
                  -e '[^[:digit:][:space:]][[:space:]]*$' \
                  -e '^[[:space:]]*$' > file.txt.out

(that is return the lines that start or end in a non-digit (past the leading whitespace, or before the trailing whitespace), or are all whitespace.


<file.txt.in grep -vx '[[:space:]]*[[:digit:]]\(.*[[:digit:]]\)\{0,1\}[[:space:]]*' > file.txt.out

Or the same with EREs:

<file.txt.in grep -vxE '[[:space:]]*[[:digit:]](.*[[:digit:]])?[[:space:]]*' > file.txt.out

That is, filter out (with -v) the lines that start with a digit optionally followed by the rest a line ending in a digit, allowing whitespace at the start and ned.

Those ones could however fail to remove lines like 8xx<non-character>yy8 where <non-character> is a sequence of bytes that don't form a valid character in the current locale. But in any case, you can't expect much portably for this kind of line that is not valid text.



cat file1.txt | sed '/^\s*$/d' > file2.txt

works, but

cat file1.txt | sed '/^[0-9].*[0-9]$/d' > file2.txt

doesn't (although one would expect to work like the former);

considering that input file might include Windows carriage returns, then it might be safer to use this command instead:

cat file1.txt | sed '/^[0-9].*[0-9]\(\r\|$\)/d' > file2.txt

This should work in all cases, including lines ending with Linux or Windows carriage return.

  • 1
    [$] is a literal dollar sign, so the line 1$x would be deleted by this command. Jan 22 '19 at 14:29
  • @BenjaminW. thanks for the warning! Any suggestions for adding \r without using two commands?
    – Sadi
    Jan 23 '19 at 7:06
  • 1
    Something like \(\r\|$\) (or (\r|$) for sed -E) maybe? Jan 23 '19 at 14:45
  • 1
    Note that \s, \r and \| are non-standard (originally GNU) extensions. Jan 24 '19 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.