I want to write a bash script (MacOS 10.9.5) that inserts values associated with some strings. In the script, I will define each of the possible associated values.

For example, I can define the text associated with the link text yahoo to be www.yahoo.com as the variable


The prefix XX_ is added to avoid name collision with existing variables. My script then is to replace all occurrences of




by matching the link text within the square brackets of the \MakeLink macro with an existing variable. If a variable for some text was not provided then we use the tittle case of the link text. Thus,

\MakeLink[foo bar]{}

should become

\MakeLink[foo bar]{Foo Bar}

The script below handles the case of where

  • the link text does not have a space in it and
  • the link text variable has not been defined


As the number of possible values of the link text could range in the thousands and could have spaces in it, my questions are:

  1. Is this the best approach for this? Would it be better to use an array for the variables?
  2. How should I handle the case where the link text has a space it. For example, I would like to be able to have

    \MakeLink[the google]{}

    be replaced with

    \MakeLink[the google]{www.google.com}.


  • Ok to assume that there will be only one occurrence of \MakeLink per line.
  • The MakeTitleCase macro needs to be enhanced to have a list of words for which the case is not to be altered (as it would be in a title), but I can address that later.

Known Problems with Existing Solution:

  • There is a problem with how I am matching \MakeLink as the match still occurs even if the leading back slash is left out. See last line of first paragraph in test case.
  • If I have a ? in the file it seems that sed has an issue.
  • Don't know how to handle case where link text contains a space.



## Can't have a backslash in the values of these variables, which is ok for my purposes.

function MakeTitleCase {
    echo $(echo "$1" | awk '{for(j=1;j<=NF;j++){ $j=toupper(substr($j,1,1)) substr($j,2) }}1')

while read -d $'\n' LINE; do
    ## Extract target which is the text within the square brackets of "\MakeLink[target]{}"
    TARGET=$(echo ${LINE} | sed -e 's?\]{}.*??' -e 's?\MakeLink\[??')
    if [ -z "${!TEMP}" ]; then
        REPLACEMENT=$(MakeTitleCase "${TARGET}")

    ## Incorrect handling of leading backslash for the match.
    echo "${LINE}" | sed "s?\MakeLink\[${TARGET}\]{}?\\\MakeLink\[${TARGET}\]{${REPLACEMENT}}?";

exit 0

Sample Input File:

A very popular site on the internet was
\MakeLink[yahoo]{} but was surpassed by
\MakeLink[google]{} due to its  
MakeLink[search engine]{}.

Due to its dominance
\MakeLink[the google]{} has had to deal with
\MakeLink[antitrust issues]{}.

Current Output:

A very popular site on the internet was
\MakeLink[yahoo]{www.yahoo.com} but was surpassed by
\MakeLink[google]{www.google.com} due to its
\MakeLink[search engine]{Search Engine}.

Due to its dominance
\MakeLink[the google]{The Google} has had to deal with
\MakeLink[antitrust issues]{Antitrust Issues}.

Desired Output:

Only change from above is the associated text for the google, and that the MakeLink[search engine]{} should not be altered as it is missing a leading backslash.

A very popular site on the internet was
\MakeLink[yahoo]{www.yahoo.com} but was surpassed by
\MakeLink[google]{www.google.com} due to its
MakeLink[search engine]{}.

Due to its dominance
\MakeLink[the google]{www.google.com} has had to deal with
\MakeLink[antitrust issues]{Antitrust Issues}.

Similar answer to choroba's (I wrote this without seeing yours, I swear!), but handles the title-casing without hardcoding:

use strict;
use warnings;

my %links = (
    yahoo => "www.yahoo.com",
    google => "www.google.com",
$links{"the $_"} = $links{$_} for keys %links;

while (<>) {
        sprintf "\\MakeLink[%s]{%s}", 
            exists $links{$1} ? $links{$1}
                              : join " ", map {ucfirst lc} split " ", $1;

Running it:

$ perl link.pl input
A very popular site on the internet was
\MakeLink[yahoo]{www.yahoo.com} but was surpassed by
\MakeLink[google]{www.google.com} due to its  
MakeLink[search engine]{}.

Due to its dominance
\MakeLink[the google]{www.google.com} has had to deal with
\MakeLink[antitrust issues]{Antitrust Issues}.
  • Pretty much exactly what I need. The automation of the the ... is a great idea, but unfortunately won't be useful for me. But I might be able to adapt that idea to other needs. – Peter Grill Dec 2 '15 at 0:40

Perl to the rescue:

use warnings;
use strict;

my %replace = ( yahoo              => 'www.yahoo.com',
                google             => 'www.google.com',
                'search engine'    => 'Search Engine',
                'the google'       => 'The Google',
                'antitrust issues' => 'Antitrust Issues',

while (<>) {

You create a hash table of replacements and use it in the substitution. You can create hash tables in recent bash versions, but you can't use them directly in sed, so there's no direct bash+sed counterpart.

  • Works great, but I do need to be able to handle the case where an entry is not in the hash table. – Peter Grill Dec 2 '15 at 0:38

I haven't reviewed your script, but I see a couple of places where you're running into quoting issues (characters that have a special meaning when you don't want them to):

  • read -d $'\n' LINE (a complicated way of writing read LINE) parses backslash escapes, so it effectively eats backslashes. Make it read -r LINE. This command also drops leading and trailing spaces; to avoid this, make it IFS= read -r LINE.
  • You're substituting variables into sed scripts. The content of these variables is parsed as a sed script, not a search string or replacement text the way you intend. This is the issue with ? in the file: when it appears in $TARGET, sed sees a ?. To fix that, add backslash characters before all characters that are special in sed (and beware that in a regexp and in a replacement text, you need to escape different characters!).

Actually… don't do what I wrote above. I was just explaining what went wrong; but you should completely rewrite your script, because you're using a screwdriver to hammer in a nail.

You're using bash, which has associative arrays. Using variables with a constructed name is a hack that's convenient when nothing better is available, but it's harder to use than a proper data structure. Unless the XX_yahoo variables must actually come from the environment, use an associative array.

typeset -A targets

While parsing a file line by line in the shell with while read … is possible, it isn't really appropriate for large files (it's slow) or files with nontrivial syntax (as you've found, it's hard to parse things correctly when you're doing back-and-forth between the shell and external tools such as sed). Your task is prime material for an awk script (or perl, as shown in other answers).

And if you're going to use awk anyway, you might as well define the associative array directly in awk.

Untested code.

#!/bin/awk -f
function MakeTitleCase(text) {
    split(text, words);
    text = "";
    for (w in words) {
        text = text toupper(substr(w,1,1)) substr(w,2)
    return text;

/^ *\\MakeLink\[[^][{}]*\]{}/ {
    target_start = index($0, "[") + 1;
    target_end = index($0, "]") - 1;
    target = substr($0, target_start, target_end - target_start);
    if (target in targets) {
        replacement = targets[target];
    } else {
        replacement = MakeTitleCase(target);
    $0 = substr($0, 1, target_start-1) replacement substr($0, target_end);

  • Thanks for the detailed explanation of my bash script. With your awk solution, can the index of the targets array have spaces in it? I started using the perl solution and it works great, but it hasn't grown yet. Do you know if perl or awk would be faster once the array grows to thousands of entries? – Peter Grill Dec 3 '15 at 11:38
  • @PeterGrill Indices in an awk array can be arbitrary strings. I don't know whether awk or perl would be faster for large files; for small files I expect awk to be faster because it starts up more quickly. Thousands of entries isn't large by either awk or perl's standards. However, if you start doing more complex things such as loading the replacements from a file, perl will be easier: awk is mostly good at processing one file line by line, perl is more flexible. – Gilles Dec 3 '15 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.