147

I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?

6 Answers 6

254

This is technically what cat ("concatenate") is supposed to do, even though most people just use it for outputting files to stdout. If you give it multiple filenames it will output them all sequentially, and then you can redirect that into a new file; in the case of all files just use ./* (or /path/to/directory/* if you're not in the directory already) and your shell will expand it to all the filenames (excluding hidden ones by default).

$ cat ./* > merged-file

Make sure you don't use the csh or tcsh shells for that which expand the glob after opening the merged-file for output, and that merged-file doesn't exist before hand, or you'll likely end up with an infinite loop that fills up the filesystem.

The list of files is sorted lexically. If using zsh, you can change the order (to numeric, or by age, size...) with glob qualifiers.

To include files in sub-directories, use:

find . ! -path ./merged-file -type f -exec cat {} + > merged-file

Though beware the list of files is not sorted and hidden files are included. -type f here restricts to regular files only as it's unlikely you'll want to include other types of files. With GNU find, you can change it to -xtype f to also include symlinks to regular files.

With the zsh shell,

cat ./**/*(-.) > merged-file

Would do the same ((-.) achieving the equivalent of -xtype f) but give you a sorted list and exclude hidden files (add the D qualifier to bring them back). zargs can be used there to work around argument list too long errors.

8
  • 25
    Beware that your quoted command will probably only do what the poster wants if they're numbered in such a way that the shell expands * in "natural" order. If you have "file1.txt...file9.txt...file14.txt" it won't work because file1?.txt will sort between file1.txt and file2.txt. You'd have to rename them to "file01.txt...file09.txt...file14.txt". Say echo * if you're not sure. Nov 4, 2010 at 21:43
  • 3
    @Warren: good point (or you can use zsh and set its numeric_glob_sort option). Nov 4, 2010 at 23:04
  • 2
    @warren-young a correct, useful warning comment. But in my actual case the order makes no difference (because files contain just simple SQL statements inserting data records which have no dependencies).
    – Ivan
    Nov 4, 2010 at 23:16
  • 2
    Beware, if the count of files exceeds a certain limit, you can run in errors like - /bin/cat: Argument list too long
    – Nupur
    Aug 5, 2015 at 13:46
  • 1
    @ARA1307 Only if the file already exists; otherwise the glob will be expanded before the shell opens the file to write to it. Good point in that situation though Sep 20, 2018 at 3:45
33

If your files aren't in the same directory, you can use the find command before the concatenation:

find /path/to/directory/ -name *.csv -print0 | xargs -0 -I file cat file > merged.file

Very useful when your files are already ordered and you want to merge them to analyze them.


More portably:

find /path/to/directory/ -name *.csv -exec cat {} + > merged.file

This may or may not preserve file order.

5
  • 2
    This is the way to go if you have a lot of files. You avoid an "argument list too long" error. May 15, 2014 at 23:17
  • 4
    You need -name "*.csv" instead of -name *.csv - without the quotes it fails.
    – Peteris
    Aug 16, 2016 at 14:15
  • The need for quotes depends on the version of the find command, specially in find and awk it's a problem when you are on a mac, the versions of both programs is a bit old. So far on ubuntu, fedora, debian and CentOS it worked smoothly without the quotes
    – 3nrique0
    Sep 15, 2016 at 12:08
  • I would expect the unquoted version to work when there are no files in the current directory matching the pattern "*.csv", since the shell would then pass the literal * to find.
    – RJHunter
    Nov 18, 2016 at 5:47
  • 2
15

The command

$ cat * > merged-file

actually has the undesired side-effect of including 'merged-file' in the concatenation, creating a run-away file. To get round this, either write the merged file to a different directory;

$ cat * > ../merged-file

or use a pattern match that will ignore the merged file;

$ cat *.txt > merged-file
2
  • 15
    cat * > merged-file works fine. Globs are processed before the file is created. If merged-file already exists, cat (mine at least) will detect that it's the output file and refuse to read it. IF the file already exists AND you have the redirect later in the pipeline, then it obviously can't do that, so then and only then do you get the runaway file.
    – Kevin
    Feb 21, 2012 at 22:48
  • cat has no way to detect if the file is the output one. The redirection happens in the shell; cat only prints on stdout.
    – bfontaine
    Sep 11, 2017 at 18:48
11

Like the other ones from here say... You can use cat

Lets say you have:

~/file01
~/file02
~/file03
~/file04
~/fileA
~/fileB
~/fileC
~/fileD

And you want only file01 to file03 and fileA to fileC:

cat ~/file01 ~/file02 ~/file03 ~/fileA ~/fileB ~/fileC > merged-file

Or, using brace expansion:

cat ~/file0{1..3} ~/file{A..C} > merged-file

Or, using fancier brace expansion:

cat ~/file{0{1..3},{A..C}} > merged-file

Or you can use for loop:

for i in file0{1..3} file{A..C}; do cat ~/"$i"; done > merged-file
1
  • 1
    Note that the string [01-03] won't work as a globbing pattern.
    – Kusalananda
    Aug 3, 2016 at 10:13
1

You can specify the pattern of a file then merge all of them as follows:

cat *pattern* >> mergedfile
0

Another option is sed:

sed r 1.txt 2.txt 3.txt > merge.txt 

Or...

sed h 1.txt 2.txt 3.txt > merge.txt 

Or...

sed -n p 1.txt 2.txt 3.txt > merge.txt # -n is mandatory here

Or without redirection ...

 sed wmerge.txt 1.txt 2.txt 3.txt

Note that last line write also merge.txt (not wmerge.txt!). You can use w"merge.txt" to avoid confusion with the file name, and -n for silent output.

Of course, you can also shorten the file list with wildcards. For instance, in case of numbered files as in the above examples, you can specify the range with braces in this way:

sed -n w"merge.txt" {1..3}.txt

Not the answer you're looking for? Browse other questions tagged or ask your own question.