Tag Info

New answers tagged

0

The coreutils pr command is another candidate: the only wrinkle seems to be that it is necessary to force the page width to be large enough to accommodate the output width. Using a file created using @Glenn_Jackman's 10,000,000 word generator, $ time tr '[[:blank:]]' '\n' < one.line | pr -s' ' -W 1000000 -JaT -1000 > many.line real 0m2.113s user ...


0

wonder how many msec this takes in go, just made a prototype didnt compile //wordsplit.go // go build wordsplit.go && ./wordsplit bigtext.txt package main import ( "fmt" "io/ioutil" "log" "os" "strings" ) func main() { myfile, err := os.Open(os.Args[0]) if err != nil { log.Fatal(err) } defer ...


1

Not really suitable when Number of words is a big number but if it's a small number (and ideally, no leading/trailing spaces in your one-line file) this should be quite fast (e.g. 5 words per line): tr -s '[[:blank:]]' '\n' <input.txt | paste -d' ' - - - - - >output.txt


4

Perl seems quite astonishingly good at this: Create a file with 10,000,000 space separated words for ((i=1; i<=10000000; i++)); do printf "%s " $RANDOM ; done > one.line Now, perl to add a newline after each 1,000 words time perl -pe ' s{ (?:\S+\s+){999} \S+ # 1000 words \K # then reset start of match ...


1

The same sed command can be simplified by specifying how many word-space patterns you want to match. I didn't have any big string files to test it out on, but without the loops in your original script this should run as fast as your processor can stream the data. Added benefit, it'll work equally well on multi-line files. n=500; sed -r ...


7

Use xargs (17 seconds): xargs -n1000 <file >output It uses the -n flag of xargs which defines the max number of arguments. Just change 1000 to 500 or whatever limit you want. I made a test file with 10^7 words: $ wc -w file 10000000 file Here are the time stats: $ time xargs -n1000 <file >output real 0m16.677s user 0m1.084s sys ...


3

Assuming your definition of word is a sequence of non-blank characters separated by blanks, here's an awk solution for your single-line file awk '{for (i=1; i<=NF; ++i)printf "%s%s", $i, i % 500? " ": "\n"}i % 500{print ""}' file


2

The venerable fmt(1) command, while not strictly operating on "a particular number of words" can fairly quickly wrap long lines to a particular goal (or maximum) width: perl -e 'for (1..100) { print "a"x int 3+rand(7), " " }' | fmt Or with modern perl, for a specific number of words, say, 10, and assuming a single space as the word boundary: ... | perl ...


3

The reason this command fails (even the one with the fixed typo) is that it is syntactically wrong. split -n10 -a4 -d5 somefile The -d flag does not take a numeric argument. (Contrast this with the long version --numeric-suffixes which can.) See the split invocation for the details that are unfortunately omitted from the derived man page. Use this ...


3

If you mean just calculation not line counting: awk 'NR==1 || n+10000<$2{n=$2; portion++}{print > FILENAME "." portion}' file


8

Hack one-liner version. Perhaps more suitable for Code Golf than this forum though. This generates split1, split2, split3 and so on, as filenames. awk '{if($2>b+9999){a++;b=$2}print >"split" a}' file.txt To have output files named split001, split002, split003, involves this extra sprintf: awk '{if($2>b+9999){a++;b=$2}print ...


14

awk 'NR==1 {n=$2} { file = sprintf("file.%.4d", ($2-n)/10000) if (file != last_file) { close(last_file) last_file = file } print > file }' Would write to file.0000, file.0001... (the number being int(($2-n)/10000) where n is $2 for the first line). Note that we close files once we've stopped ...


4

#!/bin/bash first=$( head -n1 file | awk -F" +" '{print $2}' ) last=$( tail -n1 file | awk -F" +" '{print $2}' ) for (( i=$first ; i<=$last ; i=i+10000 )) ; do awk -v start=$i -v end=$(($i+10000)) 'BEGIN { FS == " +" } { if ( $2 >= start && $2 < end ) print $0 }' file \ >> interval_"$i"_to_"$(( $i+10000 ))" done Test with ...



Top 50 recent answers are included