5

I have about 100 files.

They are named like this.

3000_ABCD_XXXXXXX.csv
3000_ABCD_YYYYYYY.csv
3000_ABCD_XYXYZYZ.csv

3000_EFGH_XXXXXXX.csv
3000_EFGH_YYYYYYY.csv
3000_EFGH_XYXYZYZ.csv

3000_IJKL_XXXXXXX.csv
3000_IJKL_YYYYYYY.csv
3000_IJKL_XYXYZYZ.csv

Currently I'm zipping each file individually but I want to group them based on their common substring like for example ABCD.zip will store

3000_ABCD_XXXXXXX.csv
3000_ABCD_YYYYYYY.csv
3000_ABCD_XYXYZYZ.csv

EFGH.zip will store

3000_EFGH_XXXXXXX.csv
3000_EFGH_YYYYYYY.csv
3000_EFGH_XYXYZYZ.csv

etc.

I'm very new to Unix/Bash scripting. Could anyone point me in the right direction?

Edit: ABCD,EFGH,IJKL are not known in advance. Their position and width within the file name is guaranteed, though.

  • What do you mean by "pick them up using indexes"? – Joseph R. Jun 10 '14 at 17:59
  • Like I know where the 'ABCD' will be in the file name i.e it will be from index 7 to 11 for eg... but i dont knw that it is called 'ABCD' – user72070 Jun 10 '14 at 18:24
4

With zsh:

setopt extendedglob
typeset -A a
for f (./*) {
  [[ $f = (#b)*_(*)_* ]] &&
    a[$match]+=$f$'\0'
}
for z (${(k)a}) {
  echo zip ./$z.zip ${(ps:\0:)a[$z]}
}

(remove the echo to actually do it when satisfied).

Using perl (from zsh/bash or any other non-csh-like shell):

perl -e 'for (@ARGV) {push @{$a{$1}}, $_ if (/_(.*)_/s)}
  system "echo", "zip", "./$_.zip", @{$a{$_}} for (keys %a)' ./*_*_*

(again, remove the "echo", to actually do it).

3

You could do something like:

IFS='
'
set -f
for group in $(set +f; printf '%s\n' 3000_*.csv | sed 's/3000_\([^_]*\).*/\1/' | LC_ALL=C uniq)
do
  set +f
  zip "$group.zip" "3000_$group"*.csv
done

Should work in bash or a POSIX shell, provided the filenames don't contain newlines.

  • That assumes filenames don't contain space, tab newline, globbing (*, ?, [) characters. That will run the same zip commands several times. – Stéphane Chazelas Jun 10 '14 at 19:05
  • @Stéphane, fixed. Forgot the uniq at first. – Graeme Jun 10 '14 at 19:23
  • You also need to fix the locale to C (LC_ALL=C). Try for instance with a list of files like 3000_a_a.csv, 3000_a@b_a.csv, and 3000_a_c.csv in a typical en_US.UTF-8 locale. That's because in most non-C locales, _ is ignored (at first) for sorting. – Stéphane Chazelas Jun 10 '14 at 19:37
2

You can try the below script.

##The find command below finds all the csv files in the current directory. 

find ~/home/file-directory-location/*.csv -type f > filenames.txt

##We know the second substring after _ will contain the index. 
##I am sorting the file based on that second substring and getting the 
##indices into a new file for zipping.
##The uniq will specify how many zip files we are creating.  

LC_ALL=C sort -t_ -k2,2 filenames.txt | cut -d '_' -f 2 | LC_ALL=C uniq > indexes

##Now, for the created indices just zip the CSV files based on the index name. 
while read index; 
do
        tar cvzf "$index".tgz /home/file-directory-location/3000_"$index"*
done <indexes
  • If any of the cvs files in ~/home/file-directory-location are of type directory, then you will include all the regular files in there. I'm not sure what's the point of your find command there. You've got some missing quotes in there. You need LC_ALL=C for sort and uniq. – Stéphane Chazelas Jun 10 '14 at 19:08
  • I just want to get the list of files that need to be zipped before I find out the indices. So, I use the above find command to get the list of csv files. – Ramesh Jun 10 '14 at 19:09
  • @StéphaneChazelas, I added the quotes for the variable. Please let me know if this is fine. – Ramesh Jun 10 '14 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.