Tag Info

Hot answers tagged

27

You can use split and cat. For example something like $ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo. (where the input filename is foo and the last argument is the output prefix) The same command with short options: $ split -b 100k -d -a 3 foo foo. The split commands generate pieces named: foo.000, foo.001 ... For re-assembling ...


18

That's just what cat was made for. Since it is one of the oldest GNU tools, I think it's very unlikely that any other tool does that faster/better. And it's not piping - it's only redirecting output.


16

cut sounds like a suitable tool for this: bash-4.2$ s='id;some text here with possible ; inside' bash-4.2$ id="$( cut -d ';' -f 1 <<< "$s" )"; echo "$id" id bash-4.2$ string="$( cut -d ';' -f 2- <<< "$s" )"; echo "$string" some text here with possible ; inside But read is even more suitable: bash-4.2$ IFS=';' read -r id string ...


12

Under the hood There is no more efficient way than copying the first file, then copying the second file after it, and so on. Both DOS copy and cat do that. Each file is stored independently of other files on the disk. Almost every filesystem designed to store data on a disk-like device operates by blocks. Here's a highly simplified presentation of what ...


10

Here is a solution: printf 'n file-%02d.tar\n' {2..100} | tar -ML 716800 -cf file-01.tar Documents/ 2>/dev/null where 100 is a number greater or equal to the number of volumes. Edit Setting a big number should not be a problem, though I tend to not take a ridiculous one. An alternative could be a "next volume" script, that you can set with the ...


10

pdftk is able to cut out a fixed set of pages efficiently. With a bit of scripting glue, this does what I want: number=$(pdfinfo -- "$file" 2> /dev/null | awk '$1 == "Pages:" {print $2}') count=$((number / pagesper)) filename=${file%.pdf} counter=0 while [ "$count" -gt "$counter" ]; do start=$((counter*pagesper + 1)); end=$((start + pagesper - 1)); ...


9

This wasn't available back then but with more recent versions (≥ 8.16) of gnu split one can use the --additional-suffix switch to have control over the resulting extension. From man split: --additional-suffix=SUFFIX append an additional SUFFIX to file names. so when using that option: split -dl 10000 --additional-suffix=.txt words wrd the ...


9

Just set IFS according to you needs and let the shell perform word splitting: IFS=':' for dir in $PATH; do [ -x "$dir"/"$1" ] && echo $dir done This works in bash, dash and ksh, but tested only with the latest versions.


9

The obvious solution would be to use the shell word splitting, but beware of a few gotchas: IFS=: set -f for dir in $PATH; do dir=${dir:-.} [ -x "${dir%/}/$1" ] && printf "%s\n" "$dir" done You need set -f because when a variable is left unquoted, both word splitting and filename generation (globbing) are performed on it and here you only ...


9

Unless I'm missing something, split does split by line if you use -l switch: -l, --lines=NUMBER put NUMBER lines per output file so split -l 1 inputfile should do what you want.


9

You can use GNU parallel to do that as it can limit the number of elements to a job as well as provide a job number (for a unique zip archive name): $ touch $(seq 20) $ find . ! -name "*.zip" -type f -print0 | parallel -0 -N 5 zip arch{#} {} adding: 1 (stored 0%) adding: 10 (stored 0%) adding: 11 (stored 0%) adding: 12 (stored 0%) adding: 13 ...


7

Seems like there should be a more efficient way than piping all of the contents through the system's stdin / stdout Except that's not really what's happening. The shell is connecting the stdout of cat directly to the open file, which means that "going through stdout" is the same as writing to disk.


7

With awk you can do: awk '{print >out}; /XYZ/{out="file2"}' out=file1 largefile Explanation: The first awk argument (out=file1) defines a variable with the filename that will be used for output while the subsequent argument (largefile) is processed. The awk program will print all lines to the file specified by the variable out ({print >out}). If the ...


6

Such tasks are best managed with the shell. Use split and then write a simple loop to rename the files. E.g. for file in wrd.* do mv "$file" "$file.txt" done would rename your wrd.01, wrd.02, etc. files so they all have a .txt extension.


6

With any standard sh (including bash): sep=';' case $s in (*"$sep"*) before=${s%%"$sep"*} after=${s#*"$sep"} ;; (*) before=$s after= ;; esac read based solutions would work for single character (and with some shells, single-byte) values of $sep other than space, tab or newline and only if $s doesn't contain newline characters. ...


6

You can use awk for the job: $ curl -o example.vcf http://qt.gitorious.org/qt-mobility/contacts/blobs/raw/\ d7f10927176b8c3603efaaceb721b00af5e8605b/demos/qmlcontacts/contents/example.vcf $ gawk ' /BEGIN:VCARD/ { ++a; fn=sprintf("card_%02d.vcf", a); print "Writing: ", fn } { print $0 >> fn; } ' example.vcf Writing: card_01.vcf Writing: ...


6

With a modern ksh here's a shell variant (i.e. without sed) of one of the sed based answers above: { read in <##XYZ ; print "$in" ; cat >file2 ;} <largefile >file1 And another variant in ksh alone (i.e. also omitting the cat): { read in <##XYZ ; print "$in" ; { read <##"" ;} >file2 ;} <largefile >file1 (The pure ksh ...


5

split is a standard utility, included in the coreutils package. This package has the priority “required” (and is marked “essential”), so a normal Debian installation would have it. I guess your server is running BusyBox utilities. BusyBox is a suite of utilities designed for systems with little disk space or little memory. Many of its features are optional, ...


5

Use --pipe: cat 2011.psv | parallel --pipe -l 50000000 ./carga_postgres.sh It requires ./carga_postgres.sh to read from stdin and not from a file, and is slow for GNU Parallel version < 20130222. If you do not need exactly 50000000 lines the --block is faster: cat 2011.psv | parallel --pipe --block 500M ./carga_postgres.sh This will pass chunks of ...


5

csplit -f vcard input.txt -z '/END:VCARD/+1' '{*}'


5

One way to do this is with awk. You can set a threshold to get the first line the threshold is hit and last line it falls below. Something like this might work: awk -F, -vthreshold_up=20 -vthreshold_down=10 'BEGIN { cur = "gt"; } { if (cur == "gt" && ...


5

split is a traditional UNIX tool, that does one job only—splitting files. If you had a bunch of files to archive to individual disks, you might do it like this: ____________________ | FILESYSTEM | _________ ____________ | dir1/ dir2/ | tar | | gzip | | | file1 file3 | ...


5

This is a perfect recipe for xargs: cat list_of_files | xargs -n 40 command Quoting from man xargs: -n number Set the maximum number of arguments taken from standard input for each invocation of the utility. An invocation of utility will use less than number standard input arguments if the number of bytes ...


5

You can also use awk: $ w | head -1 | awk '{print $10,$11,$12}' 0.80, 0.84, 0.93 Or, if the number of fields is variable, use: $ w | head -1 | awk '{print $(NF-2),$(NF-1),$NF}' 0.81, 0.82, 0.91 Or, the much more elegant (thanks @Letitzia): $ w | head -1 | awk -F "load average: " '{print $2}' Sed: $ w | head -1 | sed 's/.*load average: *//' Perl: ...


5

The easiest way is probably to use head and tail: $ head -n 1000 input-file > output1 $ tail -n +1001 input-file > output2 That will put the first 1000 lines from input-file into output1, and all lines from 1001 till the end in output2


5

Using ImageMagick: $ convert -crop 800x1000 image.png cropped_%d.png Will create a sequence of files named cropped_1.png, cropped_2.png, and so on. References Tile Cropping, sub-dividing one image into multiple images ImageMagick v6 Examples -- Cutting and Bordering


5

Here is shell code (bash, ksh, or zsh) that may do what you want: image=clock.jpg size=$( identify -ping -format "%wx%h" "${image}" ) x_upb=${size%x*} y_upb=${size#*x} x_inc=10 y_inc=10 x_tile=100 y_tile=100 for ((x=0; x<x_upb; x+=x_inc)) do for ((y=0; y<y_upb; y+=y_inc)) do convert "${image}" -crop "${x_tile}x${y_tile}+${x}+${y}" ...


5

{ sed '/XYZ/q' >file1; cat >file2; } <infile With GNU sed you should use the -unbuffered switch. Most other seds should just work though. To leave XYZ out... { sed -n '/XYZ/q;p'; cat >file2; } <infile >file1


5

This is a job for csplit: csplit -sf file -n 1 large_file /XYZ/ would silently split the file, creating pieces with prefix file and numbered using a single digit, e.g. file0 etc. Note that using /regex/ would split up to, but not including the line that matches regex. To split up to and including the line matching regex add a +1 offset: csplit -sf file ...


4

You can also try fpart, a tool I've written (BSD-licensed) : https://sourceforge.net/projects/fpart/



Only top voted, non community-wiki answers of a minimum length are eligible