0

I have two arrays with the same number of elements I want to manipulate. They are read from a file into two arrays (odd numbered lines go into array one, even numbered into array 2):

arr1=("1" "1" "3" "2" "4" "7" "7" "7" "1" "2" "3" "3" "3" "3" "7" "5")
arr2=("4" "1" "3" "5" "7" "1" "2" "3" "2" "9" "2" "6" "8" "9" "4" "6")

The data refers to season numbers and episode numbers in the same position across the two arrays. So array 1 (arr1) is seasons, array 2 (arr2) is episodes, and they align by element number. So ${arr1[0]} corresponds to ${arr2[0]}, etc.

What I am trying to do is sort them so that they are numerically ordered first by season, then by episode. So the original array (with annotations for which item is which element):

       1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16
arr1=("1" "1" "3" "2" "4" "7" "7" "7" "1" "2" "3" "3" "3" "3" "7" "5")
arr2=("4" "1" "3" "5" "7" "1" "2" "3" "2" "9" "2" "6" "8" "9" "4" "6")
       1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16

Becomes:

       2   9   1   4   10  12  3   12  13  14  5   16  6   7   8   15
arr1=("1" "1" "1" "2" "2" "3" "3" "3" "3" "3" "4" "5" "7" "7" "7" "7")
arr2=("1" "2" "4" "5" "9" "2" "3" "6" "8" "9" "7" "6" "1" "2" "3" "4")
       2   9   1   4   10  12  3   12  13  14  5   16  6   7   8   15

Possible ideas:

  1. For each item i in ${arr1[@]}, write the corresponding element of ${arr2[n]} to a file. The file can then have sort run on it.
n="0"
for i in "${arr1[@]}"; do
    echo "${arr2[${n}]}" >> "${i}.txt"
    (( n++ ))
done

But I want to try avoid involving a disk write and sort if I don't have to.

  1. Sort the data into some kind of an individual array? Each season gets its own array, and then the array can be sorted with something like sort -n "${season1Arr[@]}" -- But I don't know how this would be done.

  2. Change the way the data is manipulated? I can't change the input file, but I can change the way it's processed. Perhaps instead of reading lines into two arrays based on even/odd line numbers, they could be managed some other way?

I'm trying to keep it as pure-bash as possible for compatibility, but I understand that external programs very may well need to be used. Appreciate any ideas.

4
  • 2
    bash is entirely the wrong language for what you are doing. bash is great at co-ordinating the execution of other programs that do data processing. it is absolutely terrible at doing data processing itself. try perl or awk or python. perl or awk will be most familiar if you're already used to text processing with unix command-line tools (and in some ways, you can think of perl as a language that combines all of the features of shell, grep, sed, tr, awk, sort, and more). All three also support complex data structures, including multi-dimensional arrays.
    – cas
    Mar 12 at 1:22
  • I suspect, though, that you probably don't need a multi-dimensional array. An associative array where the key is the series and episode with a separator (e.g. "1.4", "1.1", "3.1", etc) and any arbitrary value ("1" or "true" is good when you don't actually care about the value, just the presense/absence of a key). Then you can do a natural sort by key. This will be much easier in perl, awk, python, or any other language that isn't shell because shell is crap at any kind of data processing. These languages either have built-ins or library modules for sorting arrays and hashes by key or value.
    – cas
    Mar 12 at 1:28
  • 2
    Can you show the original file? It would be much easier to sort that first and then read your data sorted. Even if it is a JSON, XML or some other type of structured file format.
    – Kusalananda
    Mar 12 at 8:34
  • 3
    "But I want to try avoid involving a disk write and sort if I don't have to." -- no, stop. That's premature optimization. Without first doing it, it's hard to know how big of a performance issue that temporary file or calling out sort really is. That temporary file isn't even likely to hit disk since the OS caches writes... And, if you really have to start considering performance issues, the first thing to do is to switch from the shell to something else. (Especially since you're running Bash, which isn't fast at all.)
    – ilkkachu
    Mar 12 at 13:15

4 Answers 4

0

Here's an example of what I was talking about in my comments, using perl and an associative array (hash), but using a version sort rather than the natural sort I mentioned (a simple numeric or alpha-numeric sort would put, e.g. 1.10 before 1.2, which is clearly wrong for "series.episode"):

#!/usr/bin/perl

use strict;

use Sort::Versions;

my %data;
my ($key, $series, $episode);

while (<>) {
  chomp; # remove trailing newline from input

  if ($. % 2 == 1) {
    $series = $_;
  } else {
    $episode = $_;
    $key = "$series.$episode";
    $data{$key} = 1;
  };
}

print join(", ", sort { versioncmp($a, $b) } keys %data), "\n";

This could be shortened a bit - the only variables really needed are the %data hash and $series. I've used $key and $episode to make it obvious how the input values are being used. BTW, $_ is the default input/value/iterator. It has lots of uses in perl and many functions & syntax elements use it if another variable isn't provided - in this script, it's the value of the current line as read by the while (<>) loop. See man perlvar and search for "General Variables".

#!/usr/bin/perl

use strict;
use Sort::Versions;

my (%data, $series);

while (<>) {
  chomp; 

  if ($. % 2 == 1) {
    $series = $_;
  } else {
    $data{"$series.$_"} = 1;
  };
}

print join(", ", sort { versioncmp($a, $b) } keys %data), "\n";

Sample run (both versions produce the same output):

$ ./sort-series-episode.pl input.txt  
1.1, 1.2, 1.4, 1.15, 2.5, 2.9, 3.2, 3.3, 3.6, 3.8, 3.9, 4.7, 5.6, 7.1, 7.2, 7.3, 7.4

Note: the Sort::Versions module is not a core perl module and needs to be installed separately, either via cpan or with a distro package - e.g. in Debian, apt-get install libsort-versions-perl

2
  • input.txt is my reconstruction of the data you described. I added data series 1, epsiode 15 to show that it sorts correctly.
    – cas
    Mar 12 at 2:20
  • Also, if you don't want to install non-core modules, you can just use a custom sort function. Version sort isn't difficult to implement. See Perl sort numbers naturally and perldoc -f sort
    – cas
    Mar 12 at 2:25
0

While perl is much better for such tasks, it is possible to do it in almost pure bash (safe for external sort):

#!/bin/bash
arr1=( "1" "1" "3" "2" "4" "7" "7" "7" "1" "2" "3" "3" "3" "3" "7" "5" )
arr2=( "4" "1" "3" "5" "7" "1" "2" "3" "2" "9" "2" "6" "8" "9" "4" "6" )

# make an array with strings from both initial arrays:
arr_length=${#arr1[@]}
for ((i=0; i< $arr_length; i++)); do
   arr_combined[$i]="${arr1[$i]}!${arr2[$i]}"
done

# sort the combined strings
arr_sorted=( $(printf "%s\n" "${arr_combined[@]}" | sort) )

# split the elements of sorted array back into two arrays
for pair in ${arr_sorted[@]} ; do
    arr1n+="${pair%!*} "
    arr2n+="${pair#*!} "
done

# print the results
printf "%s\n" "${arr1n[@]}"
printf "%s\n" "${arr2n[@]}"
0

You can make life much easier if you sort the numbers in the original file first and then populate your arrays. I assume your original file contained something like this:

$ cat file1
1 1 3 2 4 7 7 7 1 2 3 3 3 3 7 5
4 1 3 5 7 1 2 3 2 9 2 6 8 9 4 6

So this command:

transpose file1 | sort | transpose

would produce what you want:

1 1 1 2 2 3 3 3 3 3 4 5 7 7 7 7
1 2 4 5 9 2 3 6 8 9 7 6 1 2 3 4

The transpose function is very simple. You can write your own. Here is one example:

transpose () 
{ 
    awk '{ for (i=1; i<=NF; i++) a[i]= (i in a?a[i] OFS :"") $i; } 
    END{ for (i=1; i<=NF; i++) print a[i] }' $1
}
0

From your description, the following may be the original input data:

1
4
1
1
3
3
2
5
4
7
7
1
7
2
7
3
1
2
2
9
3
2
3
6
3
8
3
9
7
4
5
6

To process this in the way you seem to want:

paste - - <file | sort -k 1,1n -k 2,2n

This first transforms the data into two tab-delimited columns, with the season in the first column and the episode in the second.

The lines are sorted numerically on the first column, and any lines with identical first columns are sorted numerically on the second.

With the input shown above, this yields:

1       1
1       2
1       4
2       5
2       9
3       2
3       3
3       6
3       8
3       9
4       7
5       6
7       1
7       2
7       3
7       4

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .