Tag Info

Hot answers tagged

20

You can use split and cat. E.g. something like split --bytes 500M --numeric-suffixes --suffix-length=3 foo. Assuming that the shell sorts the results of shell globbing you could do something like this: cat foo.* > foo Else, you could use a combination of find/sort/xargs.


11

That's just what cat was made for. Since it is one of the oldest GNU tools, I think it's very unlikely that any other tool does that faster/better. And it's not piping - it's only redirecting output.


10

cut sounds like a suitable tool for this: bash-4.2$ s='id;some text here with possible ; inside' bash-4.2$ id="$( cut -d ';' -f 1 <<< "$s" )"; echo "$id" id bash-4.2$ string="$( cut -d ';' -f 2- <<< "$s" )"; echo "$string" some text here with possible ; inside But read is even more suitable: bash-4.2$ IFS=';' read -r id string ...


9

Just set IFS according to you needs and let the shell perform word splitting: IFS=':' for dir in $PATH; do [ -x "$dir"/"$1" ] && echo $dir done This works in bash, dash and ksh, but tested only with the latest versions.


9

The obvious solution would be to use the shell word splitting, but beware of a few gotchas: IFS=: set -f for dir in $PATH; do dir=${dir:-.} [ -x "${dir%/}/$1" ] && printf "%s\n" "$dir" done You need set -f because when a variable is left unquoted, both word splitting and filename generation (globbing) are performed on it and here you only ...


9

pdftk is able to cut out a fixed set of pages efficiently. With a bit of scripting glue, this does what I want: number=$(pdfinfo -- "$file" 2> /dev/null | awk '$1 == "Pages:" {print $2}') count=$((number / pagesper)) filename=${file%.pdf} counter=0 while [ "$count" -gt "$counter" ]; do start=$((counter*pagesper + 1)); end=$((start + pagesper - 1)); ...


8

Here is a solution: printf 'n file-%02d.tar\n' {2..100} | tar -ML 716800 -cf file-01.tar Documents/ 2>/dev/null where 100 is a number greater or equal to the number of volumes. Edit Setting a big number should not be a problem, though I tend to not take a ridiculous one. An alternative could be a "next volume" script, that you can set with the ...


8

Under the hood There is no more efficient way than copying the first file, then copying the second file after it, and so on. Both DOS copy and cat do that. Each file is stored independently of other files on the disk. Almost every filesystem designed to store data on a disk-like device operates by blocks. Here's a highly simplified presentation of what ...


8

Unless I'm missing something, split does split by line if you use -l switch: -l, --lines=NUMBER put NUMBER lines per output file so split -l 1 inputfile should do what you want.


7

Seems like there should be a more efficient way than piping all of the contents through the system's stdin / stdout Except that's not really what's happening. The shell is connecting the stdout of cat directly to the open file, which means that "going through stdout" is the same as writing to disk.


6

Such tasks are best managed with the shell. Use split and then write a simple loop to rename the files. E.g. for file in wrd.* do mv "$file" "$file.txt" done would rename your wrd.01, wrd.02, etc. files so they all have a .txt extension.


5

split is a standard utility, included in the coreutils package. This package has the priority “required” (and is marked “essential”), so a normal Debian installation would have it. I guess your server is running BusyBox utilities. BusyBox is a suite of utilities designed for systems with little disk space or little memory. Many of its features are optional, ...


5

One way to do this is with awk. You can set a threshold to get the first line the threshold is hit and last line it falls below. Something like this might work: awk -F, -vthreshold_up=20 -vthreshold_down=10 'BEGIN { cur = "gt"; } { if (cur == "gt" && ...


5

split is a traditional UNIX tool, that does one job only—splitting files. If you had a bunch of files to archive to individual disks, you might do it like this: ____________________ | FILESYSTEM | _________ ____________ | dir1/ dir2/ | tar | | gzip | | | file1 file3 | ...


4

You can also try fpart, a tool I've written (BSD-licensed) : https://sourceforge.net/projects/fpart/


4

This wasn't available back then but with more recent versions of split one can use: --additional-suffix=SUFFIX append an additional SUFFIX to file names. so adding --additional-suffix=.txt to the command means the resulting files will automatically have .txt extension. split -dl 10000 --additional-suffix=.txt words wrd


4

With any standard sh (including bash): sep=';' case $s in (*"$sep"*) before=${s%%"$sep"*} after=${s#*"$sep"} ;; (*) before=$s after= ;; esac read based solutions would work for single character (and with some shells, single-byte) values of $sep other than space, tab or newline and only if $s doesn't contain newline characters. ...


4

If the recipient has split, then you can do: tar -cvf - ~/batch/ | gzip | ssh recipient 'cd /destination && split --bytes=1024m - batch.tar.gz.seg'


4

Use --pipe: cat 2011.psv | parallel --pipe -l 50000000 ./carga_postgres.sh It requires ./carga_postgres.sh to read from stdin and not from a file, and is slow for GNU Parallel version < 20130222. If you do not need exactly 50000000 lines the --block is faster: cat 2011.psv | parallel --pipe --block 500M ./carga_postgres.sh This will pass chunks of ...


4

Must it be done with wc? Because here I've ran into a very nice attempt to use regex as a csplit pattern. I don't have a system to test it right now but the regex itself seem to do the job. The expression looks like that: csplit input-file.txt '/([\w.,;]+\s+){500}/'


4

See also pdfseparate and pdfunite from poppler-utils. pdfseparate breaks the file into one file per page which makes it relatively easy to reassemble at will later on with pdfunite, manually or (semi-)automatically. Like with zsh: autoload zargs setopt extendedglob reunite() pdfunite "$@" file-$1-$argv[-1].pdf pdfseparate file.pdf p%d zargs -n 5 ...


4

You can use this script to do the job. It's called split-vcf-file. Example usage $ split_vcf.pl Error! Input VCF filename missing, -i Usage: perl split_vcf.pl -i input_file -o output_dir [OPTION] -v, Verbosity levels, 1-3 To run the script: mkdir vcf_files split_vcf.pl -i current.vcf -o vcf_files


4

Using GNU Parallel you can do: cat foo.vcf | parallel --pipe -N1 --recstart BEGIN:VCARD 'cat >{#}' See more examples: http://www.gnu.org/software/parallel/man.html Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 10 seconds installation: wget -O - pi.dk/3|sh


4

You can use awk for the job: $ curl -o example.vcf http://qt.gitorious.org/qt-mobility/contacts/blobs/raw/\ d7f10927176b8c3603efaaceb721b00af5e8605b/demos/qmlcontacts/contents/example.vcf $ gawk ' /BEGIN:VCARD/ { ++a; fn=sprintf("card_%02d.vcf", a); print "Writing: ", fn } { print $0 >> fn; } ' example.vcf Writing: card_01.vcf Writing: ...


4

csplit -f vcard input.txt -z '/END:VCARD/+1' '{*}'


4

I don't think you can do this in a single regex, but since you're using gawk, you can use some gawk extensions: save the separators using the split() function use the match() function awk '{ n = split($0, a, /\[\[|\]\]|@[[:alnum:]]+/, s) for (i=1; i<=n; i++) { printf "(%s)", a[i] if (match(s[i], /^@(.+)/, m)) ...


4

You can use the split tool: split -l 1000 words.txt words- will split your words.txt file into files with no more than 1000 lines each named words-aa words-ab words-ac ... words-ba words-bb ... If you omit the prefix (words- in the above example), split uses x as the default prefix. For using the generated files with parallel you can make use of a ...


4

Additional explanation based on NTFS filesystem performance After writing the lower section ot this answer, the OP pointed out that the script is running it on a NTFS disk, and suspects that may be part of the problem. This would not be too surprising: There are performance problems with NTFS speciffically related to handling many small files. And we ...


3

Each character is three bytes wide, as shown in this xxd output: $ xxd chinese-bytes 0000000: e6b4 9ee5 baad e6b9 96 ......... split -b3 works for me. $ split -b3 chinese-bytes $ echo xa? xaa xab xac $ cat xaa; echo 洞 $ cat xab; echo 庭 $ cat xac; echo 湖


3

Not with split, but you can easily rename them afterwards, or you can do it in awk: awk '{filename = "wrd." int((NR-1)/10000) ".txt"; print >> filename}' inputfile



Only top voted, non community-wiki answers of a minimum length are eligible