Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For a project I'm doing I need to concatenate and compress (with zip) many (midi) files and compare their sizes. I've thought there must be some language that allows me to do something like this:

  1. create all pairwise concatenations (i.e. I need to merge files 01.mid and 02.mid and create file 0102.*, I don't care if it's a valid midi file or not, I just want it to have the information of both of them. Is .tar a good option?)

  2. compress all files independently (original and concatenated)

  3. check all file sizes and write them in some table which I can then import with Matlab or something

Should I use bash scripting?

share|improve this question
up vote 5 down vote accepted

You can use bash scripting, but for compression etc. it relies on other programs. You might be better of with Python, or any other scripting language which can generate CSV files for you (although CSV syntax is not necessarily difficult to generate), and that can do the concatenation and compression.

Given this directory of files (all different length, but filled with the same character):

-rw-rw-r-- 1 anthon users   24 Apr 16 15:03 01.mid
-rw-rw-r-- 1 anthon users  254 Apr 16 15:03 02.mid
-rw-rw-r-- 1 anthon users  455 Apr 16 15:03 03.mid
-rw-rw-r-- 1 anthon users   98 Apr 16 15:03 04.mid
-rw-rw-r-- 1 anthon users  444 Apr 16 15:03 05.mid
-rw-rw-r-- 1 anthon users  239 Apr 16 15:03 06.mid
-rw-rw-r-- 1 anthon users  257 Apr 16 15:03 07.mid
-rw-rw-r-- 1 anthon users  128 Apr 16 15:03 08.mid
-rw-rw-r-- 1 anthon users  228 Apr 16 15:03 09.mid
-rw-rw-r-- 1 anthon users  160 Apr 16 15:03 10.mid

and this program test.py:

#! /usr/bin/env python
# coding: utf-8

import sys
import os
import csv
import bz2

comb_name_map = {}
file_names = []
prev_name = None

# combine the names
for name in sys.argv[1:]:
    if prev_name:
        basename, ext = os.path.splitext(name)
        basename = os.path.basename(basename)
        combi_name = os.path.splitext(os.path.basename(prev_name))[0] + name
        comb_name_map[combi_name] = (prev_name, name)
        prev_name = None
        prev_name = name

for combi, org in comb_name_map.iteritems():
    file_names.append(combi + '.bz2')
    with bz2.BZ2File(file_names[-1], 'w') as outf:
        for name in org:
            with open(name, 'rb') as inf:
                buf = inf.read()
                file_names.append(name + '.bz2')
                with bz2.BZ2File(file_names[-1], 'w') as compf:

with open('filesizes.csv', 'w') as fp:
    csvf = csv.writer(fp)
    for name in file_names:
        size = os.path.getsize(name)
        csvf.writerow([name, size])

run as:

python test.py *.mid

You will get the following output files:

-rw-rw-r-- 1 anthon users 45 Apr 16 15:26 0102.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 01.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 02.mid.bz2
-rw-rw-r-- 1 anthon users 45 Apr 16 15:26 0304.mid.bz2
-rw-rw-r-- 1 anthon users 43 Apr 16 15:26 03.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 04.mid.bz2
-rw-rw-r-- 1 anthon users 44 Apr 16 15:26 0506.mid.bz2
-rw-rw-r-- 1 anthon users 43 Apr 16 15:26 05.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 06.mid.bz2
-rw-rw-r-- 1 anthon users 41 Apr 16 15:26 0708.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 07.mid.bz2
-rw-rw-r-- 1 anthon users 37 Apr 16 15:26 08.mid.bz2
-rw-rw-r-- 1 anthon users 43 Apr 16 15:26 0910.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 09.mid.bz2
-rw-rw-r-- 1 anthon users 39 Apr 16 15:26 10.mid.bz2

and a file filesizes.csv:


Make sure you have an even number of .mid files. To include the original filenames and sizes, just concatenate the name to file_names while processing.

share|improve this answer
Amazing answer, thanks! – mombius Apr 16 '13 at 16:56

Yes - bash scripting will be able to meet the requirements. But you should use whatever scripting language you are comfortable with. python, perl are other possibilities.

share|improve this answer
A further advantage of perl or python (or ruby) would be that these should all have libraries for interfacing with matlab, and much better data structures than bash for tables. – goldilocks Apr 16 '13 at 16:09

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.