Hot answers tagged

37

You can use split and cat. For example something like $ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo. (where the input filename is foo and the last argument is the output prefix) The same command with short options: $ split -b 100k -d -a 3 foo foo. The split commands generate pieces named: foo.000, foo.001 ... For re-assembling ...


31

cut sounds like a suitable tool for this: bash-4.2$ s='id;some text here with possible ; inside' bash-4.2$ id="$( cut -d ';' -f 1 <<< "$s" )"; echo "$id" id bash-4.2$ string="$( cut -d ';' -f 2- <<< "$s" )"; echo "$string" some text here with possible ; inside But read is even more suitable: bash-4.2$ IFS=';' read -r id string ...


29

That's just what cat was made for. Since it is one of the oldest GNU tools, I think it's very unlikely that any other tool does that faster/better. And it's not piping - it's only redirecting output.


25

I would use grep: $ grep -o . <<<"StackOver" S t a c k O v e r or sed: $ sed 's/./&\n/g' <<<"StackOver" S t a c k O v e r And if empty space at the end is an issue: sed 's/\B/&\n/g' <<<"StackOver" All of that assuming GNU/Linux.


17

You may want to break on graphem clusters instead of characters if the intent is to print text vertically. For instance with a e with an acute accent: With graphem clusters (e with its acute accent would be one graphem cluster): $ perl -CLAS -le 'for (@ARGV) {print for /\X/g}' $'Ste\u301phane' S t é p h a n e (or grep -Po '\X' with GNU grep built with ...


16

This wasn't available back then but with more recent versions (≥ 8.16) of gnu split one can use the --additional-suffix switch to have control over the resulting extension. From man split: --additional-suffix=SUFFIX append an additional SUFFIX to file names. so when using that option: split -dl 10000 --additional-suffix=.txt words wrd the ...


15

Under the hood There is no more efficient way than copying the first file, then copying the second file after it, and so on. Both DOS copy and cat do that. Each file is stored independently of other files on the disk. Almost every filesystem designed to store data on a disk-like device operates by blocks. Here's a highly simplified presentation of what ...


14

awk 'NR==1 {n=$2} { file = sprintf("file.%.4d", ($2-n)/10000) if (file != last_file) { close(last_file) last_file = file } print > file }' Would write to file.0000, file.0001... (the number being int(($2-n)/10000) where n is $2 for the first line). Note that we close files once we've stopped ...


12

Here is a solution: printf 'n file-%02d.tar\n' {2..100} | tar -ML 716800 -cf file-01.tar Documents/ 2>/dev/null where 100 is a number greater or equal to the number of volumes. Edit Setting a big number should not be a problem, though I tend to not take a ridiculous one. An alternative could be a "next volume" script, that you can set with the ...


11

pdftk is able to cut out a fixed set of pages efficiently. With a bit of scripting glue, this does what I want: number=$(pdfinfo -- "$file" 2> /dev/null | awk '$1 == "Pages:" {print $2}') count=$((number / pagesper)) filename=${file%.pdf} counter=0 while [ "$count" -gt "$counter" ]; do start=$((counter*pagesper + 1)); end=$((start + pagesper - 1)); ...


10

Such tasks are best managed with the shell. Use split and then write a simple loop to rename the files. E.g. for file in wrd.* do mv "$file" "$file.txt" done would rename your wrd.01, wrd.02, etc. files so they all have a .txt extension.


10

You can use GNU parallel to do that as it can limit the number of elements to a job as well as provide a job number (for a unique zip archive name): $ touch $(seq 20) $ find . ! -name "*.zip" -type f -print0 | parallel -0 -N 5 zip arch{#} {} adding: 1 (stored 0%) adding: 10 (stored 0%) adding: 11 (stored 0%) adding: 12 (stored 0%) adding: 13 ...


10

The commands basename and dirname can be used for that, for example: $ basename /home/user/a/directory/myapp.app myapp.app $ dirname /home/user/a/directory/myapp.app /home/user/a/directory For more information, do not hesitate to do man basename and man dirname.


9

With awk you can do: awk '{print >out}; /XYZ/{out="file2"}' out=file1 largefile Explanation: The first awk argument (out=file1) defines a variable with the filename that will be used for output while the subsequent argument (largefile) is processed. The awk program will print all lines to the file specified by the variable out ({print >out}). If the ...


9

Just set IFS according to you needs and let the shell perform word splitting: IFS=':' for dir in $PATH; do [ -x "$dir"/"$1" ] && echo $dir done This works in bash, dash and ksh, but tested only with the latest versions.


9

The obvious solution would be to use the shell word splitting, but beware of a few gotchas: IFS=: set -f for dir in $PATH; do dir=${dir:-.} [ -x "${dir%/}/$1" ] && printf "%s\n" "$dir" done You need set -f because when a variable is left unquoted, both word splitting and filename generation (globbing) are performed on it and here you only ...


9

Unless I'm missing something, split does split by line if you use -l switch: -l, --lines=NUMBER put NUMBER lines per output file so split -l 1 inputfile should do what you want.


8

With any standard sh (including bash): sep=';' case $s in (*"$sep"*) before=${s%%"$sep"*} after=${s#*"$sep"} ;; (*) before=$s after= ;; esac read based solutions would work for single character (and with some shells, single-byte) values of $sep other than space, tab or newline and only if $s doesn't contain newline characters. ...


8

Using ImageMagick: $ convert -crop 800x1000 image.png cropped_%d.png Will create a sequence of files named cropped_1.png, cropped_2.png, and so on. References Tile Cropping, sub-dividing one image into multiple images ImageMagick v6 Examples -- Cutting and Bordering


8

Hack one-liner version. Perhaps more suitable for Code Golf than this forum though. This generates split1, split2, split3 and so on, as filenames. awk '{if($2>b+9999){a++;b=$2}print >"split" a}' file.txt To have output files named split001, split002, split003, involves this extra sprintf: awk '{if($2>b+9999){a++;b=$2}print ...


8

Use xargs (17 seconds): xargs -n1000 <file >output It uses the -n flag of xargs which defines the max number of arguments. Just change 1000 to 500 or whatever limit you want. I made a test file with 10^7 words: $ wc -w file 10000000 file Here are the time stats: $ time xargs -n1000 <file >output real 0m16.677s user 0m1.084s sys ...


7

Perl seems quite astonishingly good at this: Create a file with 10,000,000 space separated words for ((i=1; i<=10000000; i++)); do printf "%s " $RANDOM ; done > one.line Now, perl to add a newline after each 1,000 words time perl -pe ' s{ (?:\S+\s+){999} \S+ # 1000 words \K # then reset start of match ...


7

Not with split, but you can easily rename them afterwards, or you can do it in awk: awk '{filename = "wrd." int((NR-1)/10000) ".txt"; print >> filename}' inputfile


7

Seems like there should be a more efficient way than piping all of the contents through the system's stdin / stdout Except that's not really what's happening. The shell is connecting the stdout of cat directly to the open file, which means that "going through stdout" is the same as writing to disk.


6

You can use awk for the job: $ curl -o example.vcf http://qt.gitorious.org/qt-mobility/contacts/blobs/raw/\ d7f10927176b8c3603efaaceb721b00af5e8605b/demos/qmlcontacts/contents/example.vcf $ gawk ' /BEGIN:VCARD/ { ++a; fn=sprintf("card_%02d.vcf", a); print "Writing: ", fn } { print $0 >> fn; } ' example.vcf Writing: card_01.vcf Writing: ...


6

csplit -f vcard input.txt -z '/END:VCARD/+1' '{*}'


6

Use --pipe: cat 2011.psv | parallel --pipe -l 50000000 ./carga_postgres.sh It requires ./carga_postgres.sh to read from stdin and not from a file, and is slow for GNU Parallel version < 20130222. If you do not need exactly 50000000 lines the --block is faster: cat 2011.psv | parallel --pipe --block 500M ./carga_postgres.sh This will pass chunks of ...


6

See also pdfseparate and pdfunite from poppler-utils. pdfseparate breaks the file into one file per page which makes it relatively easy to reassemble at will later on with pdfunite, manually or (semi-)automatically. Like with zsh: autoload zargs setopt extendedglob reunite() pdfunite "$@" file-$1-$argv[-1].pdf pdfseparate file.pdf p%d zargs -n 5 ...


6

With a modern ksh here's a shell variant (i.e. without sed) of one of the sed based answers above: { read in <##XYZ ; print "$in" ; cat >file2 ;} <largefile >file1 And another variant in ksh alone (i.e. also omitting the cat): { read in <##XYZ ; print "$in" ; { read <##"" ;} >file2 ;} <largefile >file1 (The pure ksh ...


6

This is a job for csplit: csplit -sf file -n 1 large_file /XYZ/ would silently split the file, creating pieces with prefix file and numbered using a single digit, e.g. file0 etc. Note that using /regex/ would split up to, but not including the line that matches regex. To split up to and including the line matching regex add a +1 offset: csplit -sf file ...



Only top voted, non community-wiki answers of a minimum length are eligible