Simple cp command not working (inside shell script and submitted using sbatch on a cluster)

Question

I am working on a cluster running RHEL and I submit jobs using the following command.

sbatch MyScript.sh

The content of the MyScript.sh are as below.

#!/bin/sh

# ....
# Other SBATCH related commands are here
# ....

## Script begins here

for d in lambda.*/
do

  cd ${d%?}
  echo "Changed Directory"
  cp -r ../Transfer/${d%?}/ENMIN/  ./
  echo "Transferred"
  rm -rf ../Transfer/${d%?}
  echo "Removed"
  cd ENMIN
  time mpirun -np $SLURM_NTASKS gmx_mpi mdrun -v -stepout 1000 -s enmin.tpr -deffnm enmin

  echo "Energy minimization done of $d cycle"
  echo "Copying to OutboundTransfer"
  mkdir ../../Transfer/${d%?}
  cp -r ../ENMIN ../../Transfer/${d%?}
  echo "Copied"

  cd ../../
done

The problem is the line cp -r ../Transfer/${d%?}/ENMIN/ ./, which is not working. The script is not moving beyond this line. It copies one file and that's it ..it stops doing anything productive. It prints "Changed Directory" to the log and nothing else.

Two strange things: (1) the "cp" (inside vi) is colored differently than other commands and (2) when I check squeue, the job seems to be running without anything actually happening (as if it is in a loop).

The pertinent directory structure is as below (output of tree).

|-- lambda.00
|-- lambda.01
|-- lambda.02
|-- lambda.03
|-- lambda.04
|-- lambda.05
|-- lambda.06
|-- lambda.07
|-- lambda.08
|-- lambda.09
|-- lambda.10
|-- lambda.11
|-- lambda.12
|-- lambda.13
|-- lambda.14
|-- lambda.15
|-- lambda.16
|-- lambda.17
|-- lambda.18
|-- lambda.19
|-- lambda.20
|-- lambda.21
|-- lambda.22
|-- lambda.23
|-- lambda.24
|-- lambda.25
|-- lambda.26
|-- lambda.27
|-- lambda.28
|-- lambda.29
`-- Transfer
    |-- lambda.00
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.01
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.02
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.03
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.04
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.05
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.06
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.07
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.08
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.09
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.10
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.11
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.12
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.13
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.14
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.15
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.16
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.17
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.18
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.19
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.20
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.21
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.22
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.23
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.24
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.25
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.26
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.27
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    |-- lambda.28
    |   `-- ENMIN
    |       |-- enmin.tpr
    |       `-- mdout.mdp
    `-- lambda.29
        `-- ENMIN
            |-- enmin.tpr
            `-- mdout.mdp

I have limited control over this cluster. What are my options to make this script work?

What shell are you using to launch the script? You don't have a shebang line, so how exactly are you running it? What OS is the cluster running? — terdon♦, Sep 14 '20 at 8:03
@terdon Thank you. I should have mentioned that I am using #!/bin/sh at the top of Shell script. I am editing my post. uname -a gives "el" 64 bit — Amit, Sep 14 '20 at 8:10
Thanks, the sh is the problem. But uname -a gives far more than "el" 64bit. Is this some form of Linux? — terdon♦, Sep 14 '20 at 8:13
I tried #!/bin/bash as well but same effect. el is Red Hat Enterprise Linux — Amit, Sep 14 '20 at 8:15
Then it really sounds like you should talk to the sysadmin. If the cp hangs instead of complaining you don't have permission, it isn't a permissions issue. It might simply be a huge file that takes several hours to copy or there might be some other issue that the sysadmin will know. If my solution using mv instead of cp doesn't help, you really should talk to the sysadmin. — terdon♦, Sep 14 '20 at 8:52

terdon · Accepted Answer · 2020-09-14 08:20:56Z

The ${d%?} construct doesn't work in POSIX shells like sh. Depending on your OS, your sh will either be a simple POSIX shell like dash (e.g. on Debian or Ubuntu) or perhaps bash running in POSIX mode. In either case, the ${d%?} will not be understood and that's likely what is breaking your script.

A simple solution would be to use #!/bin/bash to run it. However, your script also seems needlessly complicated. For one thing, the ${d%?} simply removes the trailing slash from the directory name, but there is no benefit in doing that at all:

$ d="bar/"
$ cd $d
$ pwd
/home/terdon/foo/bar

Having one (or more) slashes at the end of a directory name is fine. You could just as well do cd bar as cd bar/ or cd bar//////////.

That said, there is also no need to cd in the first place, that just adds a layer of complication for no good reason. Here's a simplified version of your script:

#!/bin/bash
for d in lambda.*/
do

  mv Transfer/"$d"/ENMIN/  "$d"
  echo "Transferred"
  ( ## open a subshell so the cd only hapens in the subshell
    ## and doesn't affect the script
    cd "$d"/ENMIN/ && 
      time mpirun -np $SLURM_NTASKS gmx_mpi mdrun -v -stepout 1000 -s enmin.tpr -deffnm enmin
  )
  echo "Energy minimization done of $d cycle"
  echo "Copying to OutboundTransfer"
  mkdir Transfer/"$d"
  cp -r "$d"/ENMIN Transfer/"$d"
  echo "Copied"
done

I don't know how sbatch works, but from what you describe, it sounds likely that it is running the script with sh, ignoring the shebang. You could also try sbatch bash MyScript.sh or simply try my version above which should be portable to any sh-style shell. The only bashism you had was the ${d%?} anyway, so removing that should solve all of your problems.

Thank you. I really appreciate you taking some time to explain and clean the code. It definitely went through first round of iteration (and then it "disappeared" as usual). It copied the ENMIN instead of moving it and the file name is there but the content is zero bytes. Could it be that it is a write permission issue with the parent folder in which the script is running. For the folder it says "drwxr-xr-x" and I am definitely not the root. — Amit, Sep 14 '20 at 8:44
@Amit hang on, where is this running? That's probably your main issue: you are not running the script from the directory that contains the lamda* subdirs, so lambda.*/ expands to the string lambda.*/ and nothing else happens. Try replacing lambda.*/ with /full/path/to/dir/with/lambda.*/. — terdon♦, Sep 14 '20 at 8:50
or use a subshell for the cd .. process) so that you don't have to worry about how deep the expanded directory structure is — Jeff Schaller♦, Sep 14 '20 at 16:22

Stack Exchange Network

current community

your communities

more stack exchange communities

Simple cp command not working (inside shell script and submitted using sbatch on a cluster)

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged shell-script cp cluster or ask your own question.

Hot Network Questions

Simple cp command not working (inside shell script and submitted using sbatch on a cluster)

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged shell-script cp cluster or ask your own question.

Related

Hot Network Questions