1

This is an extension of the question I asked . Passing directory from command line to shell script

I have a script which writes another script using a heredoc. I need to be able to write unexpanded variables in the heredoc, so I use single quotes ('EOF'). However, I need one variable to be expanded. Given the script below, how can I write the value of $sourcedir inside the heredoc?

#!/bin/bash 

sourcedir="$1"
cd $sourcedir

find "$PWD" -maxdepth 2 -name \*_R1*.fastq.gz > list1

fastq_list=$sourcedir/list1 echo `cat $fastq_list` num_files=$(wc -l < 
 $sourcedir/list1) echo $num_files



cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
#$ -j y
#$ -cwd -S /bin/sh
#$ -l h_vmem=10G
#$ -pe smp 12

if [ -z "${SGE_TASK_ID}" ]; then   echo "Need to set SGE_TASK_ID"   exit 1 fi


 BASEDIR=$sourcedir

 echo "BASEDIR" echo $BASEDIR

 BASEFILES=$( ls *_R1.fastq.gz)
 BASEFILES_ARRAY=(${BASEFILES})
 BASEFILE=${BASEFILES_ARRAY[(${SGE_TASK_ID} - 1)]}
 echo $BASEFILE


 ...................
 ...................

 EOF

 qsub -t 1-${num_files}  run_array_job.sh

I am running this script using

bash script.sh /home/dir/data

I am able to pass /home/dir/data as $1 to sourcedir but it also needs to be passed to BASEDIR , in array script which is submitted to cluster using qsub.

  • As simple as BASEDIR=$1. You need to escape all $'s in your embedded script. If you don't get what I mean I can write an answer. – Weijun Zhou Mar 12 at 19:05
  • @WeijunZhou the actual script is much, much longer than this (I know this from chatting with the OP), so escaping all $ just to keep the value of one variable will be complicated. – terdon Mar 12 at 19:08
  • @ron it would help if you could show how you want $sourcedir in the array script. – terdon Mar 12 at 19:08
  • 1
    In this case I usually do cat > wrapperscript.sh << EOF, then put run_array_job.sh $1 in the heredoc for wrapperscript.sh. – Weijun Zhou Mar 12 at 19:10
  • 1
    You can leave the EOF quoted so that you don't need to escape the $s for run_array_job.sh, but leave out the quote when you write heredoc for wrapperscript.sh – Weijun Zhou Mar 12 at 19:12
2

You can most straightforwardly do this for your usage just by breaking your heredoc in two parts:

cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
...
EOF

printf 'BASEDIR="%s"\n" "$sourcedir" >> run_array_job.sh

cat >> run_array_job.sh<<'EOF'
echo "BASEDIR" echo $BASEDIR
...
EOF

qsub -t 1-${num_files}  run_array_job.sh

This just builds the first part of the file, appends the single line that you wanted the variable available for at the end of that first part using >>, and then joins the rest of the document on the end in the same way.

You end up with the same coherent file at the end, and only write a couple of lines extra. If you have multiple variables to pass through, you can put them all in at once as well.

  • Thanks @Michael ,@Weijun also provided the same answer in the chat room. – Ron Mar 12 at 20:53
2

That's somewhat hard to do, since unlike with quotes, you can't just stop and restart expansion in an here-doc. But we could post-process the here-doc with sed:

#!/bin/bash
sourcedir=/some/path
sed -e "s,%%sourcedir%%,$sourcedir,g" << 'EOF'
some commands here with $variables not expanded 
except for the special %%sourcedir%%, which is
EOF

Running that produces the output:

some commands here with $variables not expanded 
except for the special /some/path, which is

The sed command simply changes all instances of %%sourcedir%% with whatever the value of $sourcedir (as long as it doesn't contain commas; you'd need to change the separator for the s command to something else then.)

I changed the placeholder to another format for clarity's sake, but you could also leave it as $sourcedir and then use sed -e "s,\$sourcedir,$sourcedir,". (Though note that it would also match $sourcedirectory and other similar variables, but not ${sourcedir} even though that would be equivalent in the shell.)

Alternatively, use GNU envsubst on the document, if you have it (it's part of gettext):

#!/bin/bash
export sourcedir=/some/path
envsubst '$sourcedir' << 'EOF'
some commands here with $variables not expanded 
except for the special $sourcedir, which is
EOF
  • Is envsubst GNU only? – Weijun Zhou Mar 12 at 19:49
  • @WeijunZhou, yep, seems to be. – ilkkachu Mar 12 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.