1

I have a project that I am trying to accomplish through shell scripts.

I have some ~30 years worth of directories of a long-running weekly radio show. Because they are from various sources, the names can be in wildly different formats. This makes it difficult to know what shows I have and what shows I am missing.

I want to create symlinks in a standard date format, and symlink the filename as date to the actual show directory, if I have it.

For instance, I want to make

'2015-09-25' -> '../Radio Show/2015-09-25 Special Guest/'
'2015-10-02' -> '../Radio Show/Very funny! 2015-10-02 Show'

There are all varieties of date formats also, but for now I am just worried about finding YY-MM-DD and YYYY-MM-DD formats.

So I created a file where each line was a date, from 1980-01-01 to 2010-12-31, using this answer.

Then, I read each line, using find to look for a directory with that string in its name. However, it took a long time to do a find on the entire directory tree for each date going back 30 years*.

So, I used find -type d . > filesystem.txt to create a file of all the directory names. Then I could just grep for each date string in that file, instead of running a find on the disk for each date string.

However, I'm having a problem loading each line from the dates file into grep.

I can get matches using $ grep -f dates.txt filesystem.txt But that gives me all results, in this format:

./complete/1996-02-18
./complete/1996-03-03
./complete/1996-03-31
...

and I can't figure out how to make the result with the string argment to get this:

'1996-03-31' -> './complete/1996-03-31'

I've tried $ grep "${dates[@]}" metadata/filesystem.txt, but that doesn't do what I thought it would:

grep: 1988-01-03: No such file or directory
grep: 1988-01-04: No such file or directory

Here's a psuedo-code version of what I want to do:

foreach ( date-string in dates.txt ) {
  grep date-string in filesystem.txt
  if (match) {
     ln -s match date-string
  }
}

How can I do this in bash?

-* I could simplify this by not using every date, but I'm not sure if the radio program fell on the same day for all its history. I'd like to be sure and not miss a date, so I want to use all dates in the 30 year span.

  • If there isn't some compelling reason to keep the original file & directory names, I think the first thing you should do is re-organise the directory structure and rename the files. For something like this, a useful structure is .../RadioShow/YYYY/ (or .../RadioShow/YYYY/MM/ but with a weekly show, you'll probably have <=5 episodes per month`. Then move all the files from a given year into their appropriate directory. You can now process a year's worth of shows at a time - 30 smallish tasks rather than one enormous one. – cas Aug 2 '16 at 1:42
  • The files in the directories should then be renamed so that each filename starts with YYYY-MM-DD. - that way, they'll sort correctly by date. Do each of your original directories have just one file (an mp3 or whatever)? or multiple files (audio file plus a .txt file with notes, transcript, etc)? – cas Aug 2 '16 at 1:49
3

To answer the question in the subject: how to use grep to find any of the elements of an array.

a=(foo bar baz)
grep "${a[@]}" files

Would be:

grep foo bar baz files

That is, search for foo in bar, baz or files which is not what you want here.

You want:

grep 'foo
bar
baz' files

instead. For that, you'd do:

IFS=$'\n'
grep -- "${a[*]}" files

The first character of $IFS is used to join the elements of the array when using the "${a[*]}" syntax. That works with all shells that support arrays (ksh, zsh, bash, yash (though the $'\n' part doesn't work in yash yet, you'd need to use a literal newline character there)).

With zsh, you can also do:

grep -e$^a files

Which is expanded to

grep -efoo -ebar -ebaz files

Which is another way to search for different strings.

(note that if the array contains fixed strings to search for as opposed to regular expressions to match against, you should use the -F option).

1

With zsh:

autoload zmv # best in ~/.zshrc
zmv -Ls -n '../Radio Show/(^*[0-9])((19|)(<80-99>~^??)|(20|)(<0-16>~^??))(-<1-12>-<1-31>~^-??-??)(^[0-9]*)' '${4:+19$4}${6:+20$6}$7'

-n is for dry-run. Remove to actually do the linking when happy with the proposed actions.

zmv takes care of avoiding conflicts or of overriding files. The zsh specific glob operators here:

  • <1-12> matches a string that resolves to a decimal integer in between 1 and 12. Note that it matches the 012 in 2012.
  • ^x: negation
  • x~y (and-not):string that matches x as long as it doesn't match y. So <1-12>~^?? matches a 2-digit number from 1 to 12 (matches 01 but not 1 nor 0001).
  • (x|y): alternation like in ERE.

It does insert the missing 19 or 20 for dates in YY-MM-DD format.

1

John1024's answer is probably best, but just for completeness here's your pseudo-code implementation:

for datestring in $(cat dates.txt)
do if match="$(grep "$datestring" filesystem.txt)"
   then echo ln -s "$match" "$datestring"
   fi
done

I've left an echo in so it does nothing until you remove it. But the above has to expand all the dates as arguments, so you should prefer this:

while read datestring
do if match="$(grep "$datestring" filesystem.txt)"
   then echo ln -s "$match" "$datestring"
   fi
done <dates.txt

I've put $datestring in double quotes though we know it doesn't have spaces, so this won't change anything.

1

According to the question's title, the problem is that you have an array containing a list of strings that you want to grep for, but grep requires either a single regular expression or multiple -e options.

(you could use -f and supply the strings from a file or pipe or process substitution, but my answer is focused on using an array. or just a list of strings).

The simple approach, of looping through each element of the array and outputting it with | as a prefix or suffix for each element ends up with a string that has a leading or trailing | to remove. Easily done, but annoying....and you shouldn't have to, there should be a built-in function that joins arbitrary strings with an arbitrary separator.

perl has a very useful join() function which can be used for this kind of thing. Unfortunately, bash doesn't....but it's easy enough to create one.

function join() {
  # input:
  # $1       - separator string
  # $2...$n  - list of items to join
  #
  # output:
  # a joined string on stdout.

  local sep result i

  sep="$1" ; shift
  result="$1" ; shift

  for i in "$@" ; do result="$result$sep$i" ; done
  printf '%s' "$result"
}

Note: this join function deliberately does not append a newline to the end of the output string. You can easily add one if you need it (e.g. with echo as in my examples below). It would also be easy to add an option to change the printf format string to %s\n...either as an optional first option before the separator arg or, to do it properly, using getopts.

Note also: the function name join conflicts with the standard utility join (which joins corresponding lines from two files). I never use /usr/bin/join (because it never quite matches what I need to do, so I end up having to write a perl or awk script instead), so i don't care. I can always override the function with command join file1 file2 or similar anyway, if i ever need to. If you do care, rename it to joinarray or joinstrings or something. I prefer join so that it's the same name as the perl function.

Anyway, with that function in your shell environment or in your script, you can do things like:

$ join '|' a b c d e
a|b|c|d|e$ _

(that's ugly, there's no newline so the next $ shell prompt appears on the same line as the output. use echo while testing this function on the command line)

$ echo $(join '|' a b c d e)
a|b|c|d|e

$ echo $(join '|' a b c 'd e')
a|b|c|d e

$ echo '^('$(join '|' a b c 'd e' 'f g h')')$'
^(a|b|c|d e|f g h)$

Remember to quote and/or escape any strings that need it.

This function is useful for more than just regexp | alternation (OR) operators.

$ csv=$(join ',' foo bar baz)
$ echo "$csv"
foo,bar,baz

$ echo $(join ':' user pass uid gid gecos home shell)
user:pass:uid:gid:gecos:home:shell

or, joining an array:

date_array=( $(cat dates.txt) )
date_ere='('$(join '|' "${date_array[@]}")')'      # extended regexp
date_re='\('$(join '\|' "${date_array[@]}")'\)'    # basic regexp

grep "$date_re" filesystem.txt

or even:

grep -E "($(join '|' $(cat dates.txt)))" filesystem.txt

awk doesn't have a join function either. here's the same function implemented for awk. Both the bash and awk functions are modelled on the perl function - separator as first argument, array/list as subsequent args.

function join(sep,array,       i) {
  result=array[1];
  for (i=2;i<=length(array);i++) result = result sep array[i];
  return result;
};
0

If I understand correctly, you have a filesystem.txt file that looks like:

$ cat filesystem.txt 
../Radio Show/Very funny! 2015-10-02 Show
../Radio Show/2015-09-25 Special Guest/

Consider this:

$ sed -E 's/.*[^[:digit:]]([[:digit:]]{2,4}-[[:digit:]]{2}-[[:digit:]]{2}).*/ln -s "&" "\1"/' filesystem.txt >script

The above creates a file called script. script looks like a series of bash commands:

$ cat script
ln -s "../Radio Show/Very funny! 2015-10-02 Show" "2015-10-02"
ln -s "../Radio Show/2015-09-25 Special Guest/" "2015-09-25"

Inspect this file and, if it looks like this does what you want, then execute it:

bash script
  • Yeeech. Why would you generate a script and then run it? This adds an unnecessary level of indirection, making the process harder to understand. It breaks on some special characters in file names. There's almost never a good excuse to generate a script like this apart from entering an obfuscated programming contest. – Gilles Jul 30 '16 at 21:44
  • 1
    @Gilles generating a script is often a good idea. sometimes it's the best way of doing something, especially something potentially dangerous or irreversible. Scripts are just text files and text files can be generated by scripts. I frequently write awk or perl scripts that process data and output a shell script. I can debug (i.e. view the output) by piping into less (or check syntax with bash -n), or I can run it by piping into bash or sh. or redirect to a file to hand-edit with vim or scp to another machine before running. – cas Jul 31 '16 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.