I have a problem in merging files for a big data sets like below. I want you help me to let me know the script that can do such task.

fine name: fluxes_year_lat_long I have 30 years of daily fluxes files for more that 100 location(lat_long) I want to merge from year 1 to year 30 of each point files.

my file looks like this

file name1: fluxes_2000_50_70

2000 1 1 5000 ....
2000 1 2 2000 ....
2000 12 31 5000
YYYY M D Fluxes

file name2: fluxes_2001_50_70

2001 1 1 5000
YYYY M D Fluxes

all the way to:

file name30: fluxes_2030_50_70

2030 1 1 5000
YYYY M D Fluxes

The number of rows for each file is the same.

I want the output to be merging the daily fluxes from year 2000 to 2030 for each point's location like

output file: fluxes_lat_long

YYYY MM DD Fluxes value.

In addition, I want the script read and merge file starting with the name fluxes_. I appreciate your response.

  • do you want to concatenate the files? are they already sorted? Seems you need cat, cut and some awk, do you want to add them in order just removing the first YYYY D M FLuxes line? – vfbsilva Aug 21 '13 at 17:42
  • paste might also be appropriate – Bernhard Aug 21 '13 at 17:46

I think you are asking that given files:

fluxes_2000_10_10   fluxes_2001_10_10    fluxes_2003_10_10


fluxes_2000_20_10   fluxes_2001_20_10    fluxes_2003_20_10

You create a "total" file for each of the above files at latitude 10_10 and then 20_10 etc.

In which case:

bash$ for lat in $(ls -1 fluxes_* | sed 's/fluxes_[0-9]*_\([0-9]*_[0-9]*\)/\1/' |sort|uniq )
    echo "processing $lat ..." 
    cat fluxes_[0-9]*_${lat} > fluxes_TOT_${lat} 

ls -1 fluxes_* | sed 's/fluxes_[0-9]*_\([0-9]*_[0-9]*\)/\1/' | sort | uniq finds all the unique latitudes (x_y) number pairs, we then step through them with a for loop and cat the contents of the files at that latitude (all the years) and outputs them to fluxes_TOT_x_y.

  • Thank you Drav!! your script worked perfectly what I wanted to do. – AiB Aug 21 '13 at 19:46
  • your script is almost working my work, but it reads all the files and folders in the folder I am working on. Can you please let the script do read and merge only fluxes_YYYY_lat_long? Thank you for helping – AiB Aug 21 '13 at 20:30
  • Ah yes it will, change it to: $(ls -1 fluxes_* | sed 's/fluxes_[0-9]*_\([0-9]*_[0-9]*\)/\1/' |sort|uniq ) - I will edit the answer accordingly – Drav Sloan Aug 21 '13 at 20:31
  • Great, but I think I may have caused confusion, you have the ability on the site to accept the answer (clicking the tick by my answer), details are on meta: meta.stackoverflow.com/help/someone-answers – Drav Sloan Aug 22 '13 at 9:19
  • Dear Drav Sloan, I encountered a problem when I am running the code. It was working earlier,now it isn't. It should process the lat not the whole file name to cat for the specific lat_long. Here is the log file. processing fluxes_2007_-65_-147... cat: fluxes_[0-9]*fluxes_2007-65_-147: No such file or directory processing fluxes_2008_-65_-147... cat: fluxes_[0-9]*fluxes_2008-65_-147: No such file or directory. there is no output result. Looking for your response. Thank you – AiB Aug 25 '13 at 8:15

I would use sort -u fluxes_*_lat_long > fluxes_lat_long`, it is by far the simplest solution.

EDIT: use in a script

for file in $(ls -1 fluxes_20*_*_*) ; do
    export ll="$(echo $file |cut -d _ -f 3,4)"  # extract lat & long
    echo "sort -u fluxes_*_$ll"  "> fluxes_$ll" # create sort instruction
done | uniq | sh
  • @ Daniel W. Crompton, Would you please let me know the whole script that could do my task? Thank you – AiB Aug 25 '13 at 7:16
  • 1
    @Abraham hope that helps. – Daniël W. Crompton Aug 25 '13 at 9:39
  • 1
    @ Daniel, Thank you very much. I is perfect. – AiB Aug 25 '13 at 11:22

If I understood your requirements, you want to merge all the flux data for one point in fluxes_lat_long with the format

01 01 Fluxes total_fluxes_for_Jan_1_over_30_years
01 02 Fluxes total_fluxes_for_Jan_2_over_30_years

If that's the case, here's a Perl script that should help:


use strict;
use warnings;

my @files = glob 'fluxes_*_*_*';
my %points;
$files[$_] =~ /_([0-9]+_[0-9]+)\Z/ and $points{$1}++ for (0..$#files);

for my $point (sort keys %points){
    my @point_files = grep { /_$point\Z/ } @files;
    my %days;
    for my $file (@point_files){
        open my $f,'<',$file or die "Failed to open file $file : $!\n";
        <$f>; #Discard the header
            my ($year,$month,$day,$number) = split;
            $days{"$month $day"}+=$number;
         close $f;

     open my $of,'>',"fluxes_$point";
     for (sort by_date keys %days){
         print $of "$_ Fluxes $days{$_}\n";

 sub by_date{
    my ($month_a,$day_a) = split /\s*/,$a;
    my ($month_b,$day_b) = split /\s*/,$b;
    my $month_sort = $month_a <=> $month_b;
    my $day_sort = $day_a <=> $day_b;
    return $month_sort ? $month_sort : $day_sort
  • Thanks Joseph. you understand my question. Actually Daniel's script did what I wanted. when I tried your scrip, I have the following error massage. – AiB Aug 25 '13 at 16:19
  • "my" variable $month_a masks earlier declaration in same scope at ./abraham.pl line 32. "my" variable $month_b masks earlier declaration in same scope at ./abraham.pl line 32. "my" variable $day_a masks earlier declaration in same scope at ./abraham.pl line 33. "my" variable $day_b masks earlier declaration in same scope at ./abraham.pl line 33. "my" variable $month_sort masks earlier declaration in same scope at ./abraham.pl line 34. "my" variable $month_sort masks earlier declaration in same scope at ./abraham.pl line 35. syntax error at ./abraham.pl line 24, near "&by_date keys" – AiB Aug 25 '13 at 16:20
  • Invalid separator character '$' in attribute list at ./abraham.pl line 35, near "$month_sort : return " Execution of ./abraham.pl aborted due to compilation errors. – AiB Aug 25 '13 at 16:23
  • @Abraham All fixed. – Joseph R. Aug 25 '13 at 16:48
  • Thank you for your help. It seems all error is fixed. But I couldn't get output when running the script. The log file is empty. Thank you – AiB Aug 25 '13 at 18:28

If you want to concatenate, you can do it easily:

cat fluxes_2001_50_70 >> fluxes_2000_50_70

But you do it when your files don't increase their volumne.

However , your question is behind of set of other question such as:

"second file should be update after merging?"


tail -qn +2 will remove the headers and start from the second line for every file, fluxes_* will glob all the files that starts with that string, and > fluxes_2000_2030 will pipe all data to a single output file:

tail -qn +2 fluxes_* > fluxes_2000_2030

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.