The tag has no usage guidance.

learn more… | top users | synonyms

37
votes
4answers
51k views

What's the best way to join files again after splitting them?

If I have a large file and need to split it into 100 megabyte chunks I will do split -b 100m myImage.iso That usually give me something like xaa xab xac xad And to get them back together I have ...
31
votes
2answers
32k views

Break a large file into smaller pieces

How do I break a large, +4GB file into smaller files of about 500MB each. And how do I re-assemble them again to get the original file?
23
votes
4answers
79k views

Splitting string by the first occurrence of a delimiter

I have a string in the next format id;some text here with possible ; inside and want to split it to 2 strings by first occurrence of the ;. So, it should be: id and some text here with possible ; ...
15
votes
12answers
1k views

Bash script: split word on each letter

How can I split a word's letters, with each letter in a separate line? For example, given "StackOver" I would like to see S t a c k O v e r I'm new to bash so I have no clue where to start.
13
votes
3answers
27k views

Split a file by line and have control over resulting files extension

There is a standard command for file splitting - split. For example, if I want to split a words file in several chunks of 10000 lines, I can use: split -dl 10000 words wrd and it would generate ...
13
votes
8answers
1k views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
8
votes
8answers
1k views

Split text file into lines with fixed number of words

Related, but no satisfactory answers: How can I split a large text file into chunks of 500 words or so? I'm trying to take a text file (http://mattmahoney.net/dc/text8.zip) with > 10^7 words all in ...
8
votes
4answers
849 views

Splitting file for every 10000 numbers ( not lines )

I have a file that looks like the following: chr19 61336212 + 0 0 CG CGT chr19 61336213 - 0 0 CG CGG chr19 61336218 + ...
8
votes
5answers
3k views

How to split a file by using keyword boundaries

I have a vcf file that contains numerous vcards. When importing the vcf file to outlook it seems to only import the first vcard. Hence I want to split them up. Given that a vcard starts with BEGIN:...
8
votes
7answers
1k views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
8
votes
7answers
6k views

Splitting large directory tree into specified-size chunks?

I have a directory tree that I would like to back up to optical disks. Unfortunately, it exceeds the size of any one disk (it's about 60GB). I am looking for a script that would split this tree into ...
8
votes
3answers
3k views

Split: how to split into different percentages?

How can I split a text file into 70% and 30% using the split command ?
8
votes
4answers
2k views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
8
votes
1answer
509 views

K3b to split volume across media

How do I get K3b to split a volumes across several media? For example if I give it a folder that has 6 GB (not one file of 6 GB), it should create several discs of 4.4 GB automatically, instead of me ...
7
votes
3answers
6k views

Split file into multiple small files, separate by the newline symbol

Is there a utility that split file by newline symbol? e.g if a file contains the following lines, aa bbb cccc If I want to split it to 3 files, the desired output would be: aa, bbb And cccc (in 3 ...
7
votes
2answers
7k views

Can I automate tar's multi-volume-feature?

Ok, so I've just had a read through this page after a way to improve my current backup solution on my Debian server. Tar seems to be offering a quite nice multi-volume feature, although when I try it ...
7
votes
2answers
704 views

Zip Archive With Limited Number of Files

What command can I use to create zips with a file number limit? I have a folder (no subfolders) of, say, 5000 files, so I would want a command that could divide that number and create 10 individual ...
7
votes
6answers
2k views

Split large file into chunks without splitting entry

I have a rather large .msg file formatted in the UIEE format. $ wc -l big_db.msg 8726593 big_db.msg Essentially, the file is made up of entries of various length that look something like this: UR|...
6
votes
6answers
3k views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
6
votes
4answers
4k views

Split PDF into documents with several pages each

There are several resources on the web explaining how one can split a PDF into many files with on page per file. But how can you split them into chunks of, say, five pages each? I have looked into ...
6
votes
4answers
1k views

How to split text between separator into multiple files?

I have a text file contaning the following: "random textA" "random random textB" The separator is " How can I split the containt into multiple file as follow using a bash command? File 1 : ...
6
votes
4answers
4k views

Split single string into character array using ONLY bash

I want to split 'hello' into h e l l o in an array using only bash, I could do it in sed with sed 's/./& /g' but I want to know how to split a string into an array in Bash when I do not know what ...
6
votes
4answers
2k views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
6
votes
5answers
2k views

How do I break up a file like split to stdout for piping to a command?

I have a large .sql file full of SELECT statements that contain data I want to insert into my SQL Server database. I'm looking for how I could basically take the file's contents, 100 lines at a time, ...
6
votes
3answers
1k views

Tar piped to split piped to scp

So I'm trying to transfer a bunch of files via SCP. Some of these are too large to be stored on the recipient (Android phone, 4GB file size limit). The sender is almost out of space, so I can't ...
5
votes
2answers
273 views

split a string into path and application

If I have a string such as /home/user/a/directory/myapp.app or just /home/user/myapp.app how can I split this so that I just have two variables (the path and the application) e.g. path="/...
5
votes
2answers
3k views

How to split an image vertically using the comand line?

So, say I have a large 800x5000 image, how would I split that into 5 separate images with dimensions 800x1000 using the command line?
5
votes
6answers
21k views

How to split a string into an array in bash

I have a problem with the output of a program. I need to launch a command in bash and take its output (a string) and split it to add new lines in certain places. The string looks like this: battery....
5
votes
3answers
4k views

Split the lines of a text file into separate files

I have a text file that looks something like this: foo bar zip rar tar I need to use a bash script on OSX to make a new text file after every new line like this: cat text1.txt foo cat text2.txt ...
5
votes
3answers
190 views

Robustly segmenting a csv file containing noisy data

Say I have a csv file with these columns: timestamp,cpu,memory An example of the shape of the actual data would be this chart: Having the data visualized in this way, it's easy for the human eye ...
5
votes
2answers
123 views

How to wait for execution of parallelized processes and stitch together the outputs?

Quite new to doing things on Unix, looking to make a script that does the following things in order: Take main .tsv file, split into X number of files with Y lines each Run each split file through a ...
5
votes
3answers
671 views

The ERE regex to split() string between a delimiter and end-of-word

I'm using a long gawk 3.1.6 script to do a complex conversion of Zim markdown text into GtkDialog code and am stuck on the following problem... Sample ASCII input... [[link|label label]] [[link]] @...
4
votes
2answers
3k views

What's a safe and portable way to split a string in shell programming?

When writing a shell script, I often want to split a string. Here's a very simple example: for dir in $(echo $PATH | tr : " "); do [[ -x "$dir"/"$1" ]] && echo $dir done This will ...
4
votes
3answers
94 views

Split file by number of lines including header in each one

I need to split a .txt file into smaller ones containing 100 lines each, including the header. I don't know if this is relevant, but the original file is delimited like this: COLUMN1 | COLUMN2 | ...
4
votes
3answers
3k views

Splitting a large binary file into sections determined by context patterns

I have a large (2GB) file that looks like this: ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ <binary data> ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ <binary data> ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ <...
4
votes
2answers
281 views

split a large file to new file with unique file names

I need to split a file into unique file names. I can do it with sed command eg, sed -n '/scaffold135_/w 135-scaf.txt' input file.txt but it's time consuming so I need a smart way to do it faster. ...
4
votes
1answer
382 views

What does `{\} {}` mean in `find` command?

Please explain this command in detail. I am using this command to find large files above 6 MB and split them into 5 MB chunks in the same folder. Then it removes the original files (larger than 6MB) ...
4
votes
3answers
729 views

Is it possible to use split to make character chunks out of Chinese unicode bytes?

For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me. But there's one ...
4
votes
1answer
43 views

Tar directory into independent archive files no larger than a certain size

I have a directory containing many files. Together, these files take up several gigabytes of space. I'd like to compress this directory. But compressing the directory into a single file will make ...
4
votes
6answers
2k views

How can I split a large text file into chunks of 500 words or so?

I know there's the command split which can split a file into several chunks based on file size, and then there's the command wc which can count words. I just don't know how to use the two together.
4
votes
1answer
56 views

Split file and know how many files were generated

I'm using the following lines to split a file into smaller parts: split --line-bytes=100M -d $input $output/FILENAME echo "$input was split into ??? 100MB files." >> demo.log After that, I ...
4
votes
0answers
38 views

Programmatically breaking all files in directory along size boundaries?

Okay, so I have a directory that has lots of files in it. These files are highly-compressed archives of documents, and I want to store these to a set of DVDs. That is, I want to have each archive on a ...
4
votes
1answer
178 views

screen & xterm: how to select text with the mouse in one only pane when window is divided vertically

Interestingly enough, this has also been a tmux issue for some: http://daemonforums.org/showthread.php?t=7311 But currently I am sticking to screen, and I'd like to know how I can make screen "behave"...
3
votes
2answers
140 views

How to split a file by counting digit numbers within a row?

I have a file with 45000 character within each line and I I want to split the original file based on specific numbers of character within a line. as an small example my input file looks like: input....
3
votes
4answers
241 views

Split columns based on patterns in fields

Example: I'd like to split $2 in input file, putting a tab after pattern surname Input: name surname1 name surname30000 name surname456 Desired output: name surname 1 name ...
3
votes
2answers
48 views

pipe process output to command split by line and whitespace

On the system I'm using I often need to cancel jobs I run. I can check my jobs like this: $ showq|grep jquick 2001744 jquick Running 120 00:08:26 Mon Apr 11 11:06:02 2002020 ...
3
votes
2answers
614 views

Script using “split” in a loop over large files

I'm attempting to get a bash script working, and running into terrible errors along the way. The aim of the script is to break several large text files, into multiple files with a line count read from ...
3
votes
2answers
1k views

split a file, pass each piece as a param to a script, run each script in parallel

I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
3
votes
1answer
67 views

How to split images using ImageMagick?

I have an image which contains multiple images in it. I want to split the image into multiple image files, one file per image. How do I do this using ImageMagick? I have attached a sample image file.
3
votes
1answer
440 views

Split input file into X parts of size total lines / X

Currently, my bash script splits by number of lines. However, I'd like to split a file into X pieces, each of those having total lines equal to the file length divided by X. The script is run as ...