Tag Info

Hot answers tagged

19

So I went to the source, and it looks like the slowness is in handling double byte characters. Essentially, for every character read in, it needs to call mbrtowc() to try to convert it to a wide character, then that wide character is tested to see if it's a word separator, line separator, etc. Indeed, if I change my locale LANG variable from the default ...


17

The command wc aka. word count can do it: $ wc -w <file> example $ cat sample.txt today is a good day $ wc -w sample.txt 5 sample.txt # just the number (thanks to Stephane Chazelas' comment) $ wc -w < sample.txt 5


17

The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...


13

At a guess, Your locale uses UTF-8 encoding, and About 10% of your file consists of characters which require more than one octet to encode into UTF-8. By the way, from man wc: -c, --bytes print the byte counts -m, --chars print the character counts


11

When ls is executed it parses various options. It also detect if output is a tty or not by isatty(). ls.c: code case LS_LS: /* This is for the `ls' program. */ if (isatty (STDOUT_FILENO)) { format = many_per_line; /* See description of qmark_funny_chars, above. */ qmark_funny_chars = true; } else { format = ...


8

Both bash and zsh have a >(pipeline) feature: psql ... | tee >(wc -l) | bzip2 Note that the > here is not a normal redirection, but a necessary part of the syntax. You would need a second > if you wanted to combine it with an actual redirection (with a space in between so it wouldn't be read as >> redirect-for-append).


7

how about feeding the pattern file back in as a data file so that each pattern finds at least one match, and then subtracting one from the final reported count for each match grep -f patterns.in logfile.txt patterns.in | cut -f2 -d':' | sort | uniq -c | awk '{print($1 - 1" "$2)}'


7

There is no end-of-file character in Unix or Linux filesystems. The read() system call returns 0 on end-of-file condition, if the file descriptor in use refers to a regular file. read() works differently on sockets and pipes. You don't get a special character to mark end of file. wc gave you 30 as a character or byte count because the first line has 12 ...


7

If you're running Linux, your wc probably comes from GNU Coreutils and has a --files0-from option to read a file (or stdin) containing an arbitrarily long list of NUL-terminated names of file to count. The GNU Coreutils wc documentation says "This is useful when the list of file names is so long that it may exceed a command line length limitation. In such ...


7

Your system should have GNU grep, that has an option -P to use Perl expressions and you can use that, combined with -c (so no need for wc -l): grep -Pvc '\S' somefile The '\S' hands the pattern \S to grep and matches all line containing anything that is not space, -v selects all the other lines (those only with space), and -c counts them. From the man ...


7

The reason why you get 3 here has already been explained, but to add a bit more about the question in the subject: By the strict (POSIX) definition of the term, a text line is always terminated by a newline character, so counting the number of newline characters is the same as counting the number of lines. Sometimes though, you find files that have data ...


6

One approach would be to make use of ls to give us a list of the files, but we want this list to be guaranteed to show only 1 file or directory per line. The -1 switch will do this for us. $ ls -1 dir1 dir2 dir3 fileA fileB fileC Example Create the above sample data in an empty directory. $ mkdir dir{1..3} $ touch file{A..C} Check it: $ ls dir1 dir2 ...


5

ls ends each filename with a newline (\n) and not a NUL (\0) (if its standard output is not a terminal). A way to list the files in the current directory, using NUL as a separator, is: find . -maxdepth 1 -print0. This will match the files starting with a period too. To ignore them, use: find . -maxdepth 1 \! -name '.*' -print0 Others ways could be: ls ...


5

If your grep is the GNU grep, here is a quick and dirty solution: grep -A1 "Prepare to remove role" | grep "Delete Successful" | wc -l The grep option -A1 tells grep to print the matching line AND one line following the matching line. The second grep then only prints the lines where the delete is successfull. Note that this will only work reliably when ...


5

Your assumption is correct. wc -l counts the number of newlines (\n). As XTian said, echo adds a newline to the end of everything it prints unless you tell it not to with -n. You can check that this is what's going on by piping through od -c: $ echo -e '\n\n' | od -c 0000000 \n \n \n 0000003 So wc -l correctly counts three lines.


4

I suspect that wc will use less CPU time, it is written in C and has been carefully optimized over time. However, there is no reason not to test this yourself. time will let you compare the performance of your perl script against wc. Using the time command, you should get output like: > time wc /usr/share/dict/words 119095 119095 1145922 ...


4

If your goal is only to detect whether there's always the same number of tabs per line (no bash, no awk): sed 's/[^\t]//g' file | sort -u | wc -l If it outputs 1, then it's good! Or, replacing sed with tr: tr -cd \\t\\n < file | sort -u | wc -l or if you like useless uses of cats and don't like concatenating options: cat file | tr -c -d \\t\\n | ...


4

Try this, and apologies for being obvious: cat *.cs | wc -l or, with git: git ls-files -z ${1} | xargs -0 cat | wc -l If you actually want the output to look like wc output, with both individual counts and a sum, you could use awk to add up the individual lines: git ls-files -z ${1} | xargs -0 wc -l | awk ...


3

find itself can not execute in parallel (not that I know of). xargs can do it, and the simplest way to do it with xargs is to wrap it in a shell script. But before that you should optimize your condition itself. cat is useless unless actually concatenating files. And you don't need to count ALL lines just to determine that a file has 10 or more. So I ...


3

Must it be done with wc? Because here I've ran into a very nice attempt to use regex as a csplit pattern. I don't have a system to test it right now but the regex itself seem to do the job. The expression looks like that: csplit input-file.txt '/([\w.,;]+\s+){500}/'


3

Use find in conjunction with xargs. The only reason I am recommending find is to take advantage of the -print0 option, which separates file names by NULs; this helps avoid issues with file names containing spaces. find . -maxdepth 1 -type f -print0 | xargs -0 wc


3

Here's a perl script that opens files (given as command line arguments) in UTF-16 (endianness detected via BOM), and counts the lines. #! /usr/bin/env perl use strict; use warnings; while (my $file = shift @ARGV) { my $fh; if (!open($fh, '<:encoding(UTF-16)', $file)) { print STDERR "Failed to open [$file]: $!\n"; next; } ...


3

I would convert the file to UTF-8 with LF line endings, so I can directly use the native tools: $ iconv -f UTF-16LE -t UTF-8 myfile.txt | dos2unix | wc -l The dos2unix part is the trickiest bit. There are many variants of this tool floating around, not all of which know how to be used in a pipeline. Sometimes it's called something else, like d2u.


3

This can be done in a very simple shell script: for file in *; do echo -n "$file" | wc -m; done Just loop through each file echoing the name to wc. The -n on the echo is so that it doesn't append a newline, which would erroneously increase the count by 1.


3

You don't need to call out to wc, bash is perfectly capable: ${#var} is the length of the value of $var. for f in *; do echo ${#f}; done reference


3

The problem is xargs which is splitting the command into multiple runs, so wc is reporting the total for each time. You have a few options, you could keep things the way they are and parse the wc output: git ls-files -z ${1} | xargs -0 wc -l | awk '/total/{k+=$1}END{print k,"total"}'; You could cat the files: git ls-files -z ${1} | xargs -0 cat | wc -l ...


2

The wc implementation from GNU coreutils tries to optimize the width of the columns. If you pass it only regular files (whether on standard input or by name), it reads each the directory entry for each file to know the file size, and it knows that all the numbers it's going to print are smaller or equal to the sum of the sizes of the files. With regular ...


2

The issue you're experiencing is that wc -l counts new lines. Since you haven't in fact typed the \n there is in fact zero new lines. excerpt from wc man page Print newline, word, and byte counts for each FILE, and a total line if more than one FILE is specified. With no FILE, or when FILE is -, read standard input. A word is a non-zero-length ...


2

It's generally the case that when trying to solve a problem that can be completely encapsulated by a single command line tool such as wc, that they're more performant than anything else. It's only when you have to start chaining multiple "commands" together does it make sense to transition the solution from chaining them together, to solving the problem in ...


2

I would do it like so: #!/bin/bash FILENAME=$1 [ -f "$FILENAME" ] || exit FILEBYTES=$(stat -c%s "$FILENAME") FILEWORDS=$(wc -w "$FILENAME" | awk '{print $1}') FILELINES=$(wc -l "$FILENAME" | awk '{print $1}') printf "Size of %s -- %s bytes, %s words, %s lines.\n" \ "$FILENAME" "$FILEBYTES" "$FILEWORDS" "$FILELINES" Example Say I have this ...



Only top voted, non community-wiki answers of a minimum length are eligible