Tag Info

Hot answers tagged

22

You can use split and cat. For example something like $ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo. (where the input filename is foo and the last argument is the output prefix) The same command with short options: $ split -b 100k -d -a 3 foo foo. The split commands generate pieces named: foo.000, foo.001 ... For re-assembling ...


12

That's just what cat was made for. Since it is one of the oldest GNU tools, I think it's very unlikely that any other tool does that faster/better. And it's not piping - it's only redirecting output.


12

cut sounds like a suitable tool for this: bash-4.2$ s='id;some text here with possible ; inside' bash-4.2$ id="$( cut -d ';' -f 1 <<< "$s" )"; echo "$id" id bash-4.2$ string="$( cut -d ';' -f 2- <<< "$s" )"; echo "$string" some text here with possible ; inside But read is even more suitable: bash-4.2$ IFS=';' read -r id string ...


10

Here is a solution: printf 'n file-%02d.tar\n' {2..100} | tar -ML 716800 -cf file-01.tar Documents/ 2>/dev/null where 100 is a number greater or equal to the number of volumes. Edit Setting a big number should not be a problem, though I tend to not take a ridiculous one. An alternative could be a "next volume" script, that you can set with the ...


10

pdftk is able to cut out a fixed set of pages efficiently. With a bit of scripting glue, this does what I want: number=$(pdfinfo -- "$file" 2> /dev/null | awk '$1 == "Pages:" {print $2}') count=$((number / pagesper)) filename=${file%.pdf} counter=0 while [ "$count" -gt "$counter" ]; do start=$((counter*pagesper + 1)); end=$((start + pagesper - 1)); ...


9

Under the hood There is no more efficient way than copying the first file, then copying the second file after it, and so on. Both DOS copy and cat do that. Each file is stored independently of other files on the disk. Almost every filesystem designed to store data on a disk-like device operates by blocks. Here's a highly simplified presentation of what ...


9

Just set IFS according to you needs and let the shell perform word splitting: IFS=':' for dir in $PATH; do [ -x "$dir"/"$1" ] && echo $dir done This works in bash, dash and ksh, but tested only with the latest versions.


9

The obvious solution would be to use the shell word splitting, but beware of a few gotchas: IFS=: set -f for dir in $PATH; do dir=${dir:-.} [ -x "${dir%/}/$1" ] && printf "%s\n" "$dir" done You need set -f because when a variable is left unquoted, both word splitting and filename generation (globbing) are performed on it and here you only ...


9

Unless I'm missing something, split does split by line if you use -l switch: -l, --lines=NUMBER put NUMBER lines per output file so split -l 1 inputfile should do what you want.


9

You can use GNU parallel to do that as it can limit the number of elements to a job as well as provide a job number (for a unique zip archive name): $ touch $(seq 20) $ find . ! -name "*.zip" -type f -print0 | parallel -0 -N 5 zip arch{#} {} adding: 1 (stored 0%) adding: 10 (stored 0%) adding: 11 (stored 0%) adding: 12 (stored 0%) adding: 13 ...


7

Seems like there should be a more efficient way than piping all of the contents through the system's stdin / stdout Except that's not really what's happening. The shell is connecting the stdout of cat directly to the open file, which means that "going through stdout" is the same as writing to disk.


6

Such tasks are best managed with the shell. Use split and then write a simple loop to rename the files. E.g. for file in wrd.* do mv "$file" "$file.txt" done would rename your wrd.01, wrd.02, etc. files so they all have a .txt extension.


5

split is a standard utility, included in the coreutils package. This package has the priority “required” (and is marked “essential”), so a normal Debian installation would have it. I guess your server is running BusyBox utilities. BusyBox is a suite of utilities designed for systems with little disk space or little memory. Many of its features are optional, ...


5

With any standard sh (including bash): sep=';' case $s in (*"$sep"*) before=${s%%"$sep"*} after=${s#*"$sep"} ;; (*) before=$s after= ;; esac read based solutions would work for single character (and with some shells, single-byte) values of $sep other than space, tab or newline and only if $s doesn't contain newline characters. ...


5

Use --pipe: cat 2011.psv | parallel --pipe -l 50000000 ./carga_postgres.sh It requires ./carga_postgres.sh to read from stdin and not from a file, and is slow for GNU Parallel version < 20130222. If you do not need exactly 50000000 lines the --block is faster: cat 2011.psv | parallel --pipe --block 500M ./carga_postgres.sh This will pass chunks of ...


5

You can use awk for the job: $ curl -o example.vcf http://qt.gitorious.org/qt-mobility/contacts/blobs/raw/\ d7f10927176b8c3603efaaceb721b00af5e8605b/demos/qmlcontacts/contents/example.vcf $ gawk ' /BEGIN:VCARD/ { ++a; fn=sprintf("card_%02d.vcf", a); print "Writing: ", fn } { print $0 >> fn; } ' example.vcf Writing: card_01.vcf Writing: ...


5

csplit -f vcard input.txt -z '/END:VCARD/+1' '{*}'


5

One way to do this is with awk. You can set a threshold to get the first line the threshold is hit and last line it falls below. Something like this might work: awk -F, -vthreshold_up=20 -vthreshold_down=10 'BEGIN { cur = "gt"; } { if (cur == "gt" && ...


5

split is a traditional UNIX tool, that does one job only—splitting files. If you had a bunch of files to archive to individual disks, you might do it like this: ____________________ | FILESYSTEM | _________ ____________ | dir1/ dir2/ | tar | | gzip | | | file1 file3 | ...


5

You can also use awk: $ w | head -1 | awk '{print $10,$11,$12}' 0.80, 0.84, 0.93 Or, if the number of fields is variable, use: $ w | head -1 | awk '{print $(NF-2),$(NF-1),$NF}' 0.81, 0.82, 0.91 Or, the much more elegant (thanks @Letitzia): $ w | head -1 | awk -F "load average: " '{print $2}' Sed: $ w | head -1 | sed 's/.*load average: *//' Perl: ...


5

The easiest way is probably to use head and tail: $ head -n 1000 input-file > output1 $ tail -n +1001 input-file > output2 That will put the first 1000 lines from input-file into output1, and all lines from 1001 till the end in output2


4

You can also try fpart, a tool I've written (BSD-licensed) : https://sourceforge.net/projects/fpart/


4

This wasn't available back then but with more recent versions of split one can use: --additional-suffix=SUFFIX append an additional SUFFIX to file names. so adding --additional-suffix=.txt to the command means the resulting files will automatically have .txt extension. split -dl 10000 --additional-suffix=.txt words wrd


4

If the recipient has split, then you can do: tar -cvf - ~/batch/ | gzip | ssh recipient 'cd /destination && split --bytes=1024m - batch.tar.gz.seg'


4

Must it be done with wc? Because here I've ran into a very nice attempt to use regex as a csplit pattern. I don't have a system to test it right now but the regex itself seem to do the job. The expression looks like that: csplit input-file.txt '/([\w.,;]+\s+){500}/'


4

See also pdfseparate and pdfunite from poppler-utils. pdfseparate breaks the file into one file per page which makes it relatively easy to reassemble at will later on with pdfunite, manually or (semi-)automatically. Like with zsh: autoload zargs setopt extendedglob reunite() pdfunite "$@" file-$1-$argv[-1].pdf pdfseparate file.pdf p%d zargs -n 5 ...


4

You can use this script to do the job. It's called split-vcf-file. Example usage $ split_vcf.pl Error! Input VCF filename missing, -i Usage: perl split_vcf.pl -i input_file -o output_dir [OPTION] -v, Verbosity levels, 1-3 To run the script: mkdir vcf_files split_vcf.pl -i current.vcf -o vcf_files


4

Using GNU Parallel you can do: cat foo.vcf | parallel --pipe -N1 --recstart BEGIN:VCARD 'cat >{#}' See more examples: http://www.gnu.org/software/parallel/man.html Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 10 seconds installation: wget -O - pi.dk/3|sh


4

I don't think you can do this in a single regex, but since you're using gawk, you can use some gawk extensions: save the separators using the split() function use the match() function awk '{ n = split($0, a, /\[\[|\]\]|@[[:alnum:]]+/, s) for (i=1; i<=n; i++) { printf "(%s)", a[i] if (match(s[i], /^@(.+)/, m)) ...


4

You can use the split tool: split -l 1000 words.txt words- will split your words.txt file into files with no more than 1000 lines each named words-aa words-ab words-ac ... words-ba words-bb ... If you omit the prefix (words- in the above example), split uses x as the default prefix. For using the generated files with parallel you can make use of a ...



Only top voted, non community-wiki answers of a minimum length are eligible