8

How can I delete all lines in a text file which have fewer than 'x' letters OR numbers OR symbols? I can't use awk 'length($0)>' as it will include spaces.

  • 1
    For clarity: is it "delete if n[alpha] < x OR n[digit] < x"? (It seems so). Or "delete if n[alnum] < x" (alnum including both number and letters)? – fra-san Apr 13 '20 at 8:10
  • 6
    ...and: what do you mean by "symbols"? It could help if you provided a small sample of text and the expected result of processing. – fra-san Apr 13 '20 at 8:22
13

Assuming you want to delete lines that contain less than n graphical symbols:

awk -v n=5 '{ line = $0; gsub("[^[:graph:]]", "") } length >= n { print line }'

This deletes all characters that does not match [[:graph:]]. If the length of the string that remains is greater than or equal to n, the (unmodified) line is printed.

The value of n is given on the command line.

[[:graph:]] is equivalent to [[:alnum:][:punct:]], which in turn is the same as [[:alpha:][:digit:][:punct:]]. It is roughly the same as [[:print:]] but does not match spaces.

Instead of [^[:graph:]], you could possibly use [[:blank:]] to delete all tabs or spaces.

With sed, following the above awk code almost literally,

sed -e 'h; s/[^[:graph:]]//g' \
    -e '/.\{5\}/!d; g'

or, simplified (only counting non-blank characters),

sed -e 'h; s/[[:blank:]]//g' \
    -e '/...../!d; g'

This first saves the current line into the hold space with h. It then deletes all non-graph characters (or blank characters in the second variation) on the line with s///g. If the line then contains less than 5 characters (change this to whatever number you want, or change the number of dots in the second variation), the line is deleted. Else, the stored line is fetched from the hold space with g and (implicitly) printed.

7
sed -e 's/[^[:space:][:cntrl:]]/&/20' -e t -e d < file

would print the lines of file that contain at least 20 non-whitespace non-control characters (see also [[:graph:]] or [[:alnum:][:punct:]], it's not clear what characters you want to include/exclude from your description; beware that on some systems, the non-breaking-space character is included in graph and not in space).

The idea is that it tries to substitute the 20th occurrence of a non-whitespace/control character with itself (&) and we branch off if that substitution succeeds (t), thus skipping the d action (to delete).

With awk, you could do:

awk 'gsub(/[^[:space:][:cntrl:]]/, "&") >= 20' < file

Relying on the fact that gsub() returns the number of substitutions it has made.

With grep:

grep -E '^([[:space:][:cntrl:]]*[^[:space:][:cntrl:]]){20}' <  file

((.*[^[:space:][:cntrl:]]){20} would also work but would be more expensive).

  • This [:stuff:] is locale-depenent. – Kaz Apr 14 '20 at 0:46
2

This will delete the lines which has 10 and less than 10 characters in it:

sed -E '/^.{1,11}$/d' filename

OR

sed -r '/^.{1,11}$/d' filename  

If you want to trim spaces you can use:

sed -E 's/^[[:space:]]*//g;s/[[:space:]]*$//g;/^.{1,11}$/d' filename

If you want to do in-place(directly inside the file) delete use -i option with it.

  • 2
    This seems to also count spaces. – Kusalananda Apr 13 '20 at 9:17
  • Yes , as per question, if within the minimum characters , if space is there , it will be deleted . – Stalin Vignesh Kumar Apr 13 '20 at 9:58
  • 1
    What if a line starts with 30 spaces and then more than 10 non-spaces? Also note that such a line should be printed, which means printing the spaces intact (the question only asks about deleting certain lines, not about modifying others). – Kusalananda Apr 13 '20 at 11:25
  • 1
    All 30 spaces will be deleted and check if non-space less than 10 , it will delete the line...i checked and working...so you the one de-voted it ah :).. :( – Stalin Vignesh Kumar Apr 13 '20 at 11:53
1
 perl -n -e 'print if tr/!-\176// > 38'

where 38 is your number of non-space characters.

If you're looking for a particular list of non-space characters, feel free to stick them inbetween the first two /'s of tar, either individually, or with "-" specifying a range: e.g.

 tr/A-Za-z0-9//

If you need - to be one of them, put - first.

If you need / to be one of them, encode it as \057.

tr/A-Za-z0-9\057+//          # match characters in Base64

Gory details

perl -e means the perl program is in the command line.
perl -n -e means iterate that program on each line of input, i.e. wrap it inside perl while (<>) { your code here }

While iterating, perl sucks the input into a metavariable called $_.

That is also the default target of most commands (including print and tr).

command if condition is a perl idiom for "if (condition) { command }"

tr/// is the transliteration operator, e.g. tr/ABC/abc/ means change every A to a, B to b, and C to c. The - operator means a range, so tr/A-Z/a-z/ means the whole alphabet. Null in the second field means don't change the string. The return value of the tr/// operator is the count of characters matched.

Blown out into fat code, this code looks like

 $textstart = "!";    # first character after space
 $textend = "\176";   # last printable character in octal; dec 126 or 0x7E 
 $textrange = $textstart . "-" . $textend; 

 while (defined ($_ = <STDIN>) ) {    # while able to suck a line from STDIN into $_

     if ( {$_ =~ tr/$textrange//}     # count desired characters within textrange 
                                  > 38  )   # compare to 38
     {                                # start if block
           print $_;                  # If valid, print raw input
     }                                # end if block   

 }  # end while
1

Using Raku (née Perl6)

raku -ne '.put if chars( S:g/\s// ) >= 10;'  filename

or

raku -ne '.put unless chars( S:g/\s// ) < 10;'  filename

or

raku -ne '.put unless chars( S:g/\W// ) < 10;'  filename

or

raku -ne '.put unless chars( S:g/<ws>// ) < 10;'  filename

Briefly, in Raku/Perl6 the "capital-S" S/// operator is used to generate a resultant string devoid of unwanted characters (e.g. blank spaces), the characters of the resultant string are counted with the chars function, compared with "n" (using 10 as an example), and finally--if the boolean is satisfied-- .put will return the original line intact.

Note 1: "S/// uses the same semantics as the s/// operator, except it leaves the original string intact and returns the resultant string instead of $/ ($/ still being set to the same values as with s///)."

https://docs.raku.org/language/regexes#S///_non-destructive_substitution

Note 2: In Raku/Perl6 regex modifiers like :g (for 'global') are known as adverbs and are (usually) placed at the head of the S/// or s/// operator, right after the S or s.

Note 3: The . dot in Raku/Perl6 is used to call a method on the topic variable $_ , therefore the first 'word' of code .put is essentially shorthand for $_.put.

https://docs.raku.org/language/5to6-nutshell#-%3E_Method_calls

Note 4: Raku/Perl6 has a much more restricted set of command line flags. The -e flag ("execute") runs Raku/Perl6 code at the command line. The -n flag runs the Raku/Perl6 code linewise, i.e. against each line of an input file one line at a time, returning a result. The two flags may be combined into an -ne flag, but regardless, the -e flag must come last.

https://docs.raku.org/language/5to6-nutshell#Command-line_flags
https://github.com/rakudo/rakudo/wiki/Running-rakudo-from-the-command-line

-1
perl -pi.bak -e '$_ = "" unless length > 20' file1 file2

This allows for lines (including the "\n") that have x+1 or more characters.

  • it seems white space should be ignored? – Sebastian Apr 14 '20 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.