2

I realize this might be hard to answer without you knowing how my cluster is setup, but I' am trying to submit jobs (via SGE) to a cluster, but the environment is not setup correctly and the jobs fails. Moreover, there are two different master nodes I can login into to submit jobs to the same cluster, and my scripts work on one while not on the other.

The is the machine info for the master node that my script does work on:

cat /proc/version 
Linux version 2.6.32-279.el6.x86_64 (mockbuild@x86-008.build.bos.redhat.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Wed Jun 13 18:24:36 EDT 2012

The machine it does not work on:

cat /proc/version
Linux version 3.10.0-514.6.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Thu Feb 23 03:04:39 UTC 2017

Here is a test script I' am using:

#!/bin/bash -I
#$ -wd ~
#$ -N test
#$ -o ~/test.log
#$ -j y
#$ -terse
#$ -V
#$ -notify
#$ -l h_vmem=2G -pe smp 1 -l athena=true
ls
hostname
nproc

Here is the output after running "qsub test.sh":

/bin/bash: module: line 1: syntax error: unexpected end of file
/bin/bash: error importing function definition for `BASH_FUNC_module'
/opt/sge/default/spool/execd/node156/job_scripts/1063646: line 11: ls: command not found
/opt/sge/default/spool/execd/node156/job_scripts/1063646: line 12: hostname: command not found

To add to the confusion, when I ssh directly into those job nodes (node156 in the above example) I can run the ls and hostname commands just fine!

I've been in contact with the cluster admins, and they are unable to replicate my issue (even if they login as me). We first tested that if setting ~/.bashrc and ~/.bash_profile to default settings would fix it, but it did not. Here are those files:

cat ~/.bashrc 
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

.bash_profile:

cat ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi


# User specific environment and startup programs

Any suggestions?

0

I don't have a full solution, because I don't know anything about SGE. But I can explain part of the problem.

The machine where your script works is running an old version of the operating system. This is evident not only by the kernel version number, but also by the fact that it hasn't received security updates in a while. Specifically, I think it's running a version of bash that's vulnerable to the Shellshock bug.

Bash (ab)uses the environment to pass functions. Normally the environment is only used to pass data, in the form of a series of items of the form NAME=VALUE. Older versions of bash add items of the form NAME=() {CODE}, which in some circumstances allowed injecting code by defining a variable that a script would never use — the shellshock bug. The fix for the bug changed the way functions are encoded to BASH_FUNC_NAME%%=() {CODE}.

Evidently some part of your setup dumps out the environment and parses it. This may either be a part of SGE or something specific to your setup. A plausible reason to do this is to save the environment in which a job was submitted, to execute the job in the same environment.

Something somewhere is defining a function called module in bash, and exporting it. The code would look something like

module () {
  …
}
export -f module

The fix is either to upgrade the environment parser to something that can cope with the new bash encoding, or to stop exporting functions.

  • This is the answer, thanks! "declare -F" shows that there is a function named "module" defined in the environment on the machine the code does not work on. It is defined in the "old" way as shown at the bottom of your post. Moreover, the -V option in my SGE job script copies tells the tool to copy over all environment variables. Removing -V fixes the issue. – murphycj May 30 '17 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.