3

I've been writing a shell script which should make grep command outputs for further use. However when I pass a variable containing ~/.../multiple_dir/* to grep as input I get empty output file.

#!/bin/sh

set -u

PROGRAM="$1"
REGEXP=$(cat "$2")
INP_FILE="$3"
OUT_FILE="$4"

printf "%-30s: " $(basename ${INP_FILE})

if [ $INP_FILE = "STDIN.inp" ]
then
    cat ${INP_FILE} | ${PROGRAM} ${REGEXP} - > ${OUT_FILE} 2>&1
elif [ $INP_FILE = "MULTIPLE.inp" ]
then
    ${PROGRAM} ${REGEXP} '$(cat ${INP_FILE})' > ${OUT_FILE} 2>&1
else
    ${PROGRAM} ${REGEXP} ${INP_FILE} > ${OUT_FILE} 2>&1
fi

true

This is the script I've written, with single files or from STDIN it works like a charm, but when $(cat ${INP_FILE})=~/.../multiple_dir/* it doesn't work.

I have ./inputs, ./inputs/multiple_dir/, and ./tests directories. In ./tests directory, I have a link to ./inputs directory files, which are SINGLE (pdb file), STDIN (pdb file which I will pass as STDIN), UNREADABLE (no read bit), MULTIPLE (contains ~/.../inputs/multiple_dir/*) and EMPTY (empty).

I have a Makefile which calls shell script and passes one file at a time as a parameter. Each case works perfectly except when it passes MULTIPLE, then I get an empty output file. In all other cases output files aren't empty. The values makefile passes to script is as follows grep regExp/regExp(contains text ATOM) tests/MULTIPLE.inp(contains text ~/.../inputs/multiple_dir/*) outputs/MULTIPLE.out

To better illustrate the problem I'm experiencing i've wrote a few short scripts.

#!/bin/sh
grep ATOM ~/5as-darbas/inputs/multiple_dir/* > working.out

#!/bin/sh
input_file="$1"
#$1 is a file containing text ~/.../inputs/multiple_dir/*
echo "$(cat ${input_file})"
grep ATOM "$(cat ${input_file})" > not_working.out

Scripts results are as follows:

simas@Lenovo:~/5as-darbas/inputs$ ./working 
simas@Lenovo:~/5as-darbas/inputs$ ./not_working MULTIPLE
~/5as-darbas/inputs/multiple_dir/*
grep: ~/5as-darbas/inputs/multiple_dir/*: No such file or directory
simas@Lenovo:~/5as-darbas/inputs$ ls -l
total 2372
-rw-rw-r-- 1 simas simas       0 Dec 10 19:37 EMPTY
-rw-rw-r-- 1 simas simas      35 Dec 12 02:49 MULTIPLE
-rw-rw-r-- 1 simas simas      25 Dec 12 02:43 MULTIPLE~
drwxrwxr-x 3 simas simas    4096 Dec 12 01:52 multiple_dir
-rwxr-xr-x 1 simas simas     163 Dec 12 03:26 not_working
-rw-rw-r-- 1 simas simas     161 Dec 12 03:26 not_working~
-rw-rw-r-- 1 simas simas       0 Dec 12 03:29 not_working.out
-rw-rw-r-- 1 simas simas       0 Dec 12 01:52 s~
-rw-rw-r-- 1 simas simas  332343 Dec 10 19:38 SINGLE
-rw-rw-r-- 1 simas simas  252720 Dec 10 19:38 STDIN
--w--w---- 1 simas simas  252720 Dec 10 19:38 UNREADABLE
-rwxr-xr-x 1 simas simas      70 Dec 12 03:27 working
-rw-rw-r-- 1 simas simas     128 Dec 12 03:27 working ~
-rw-rw-r-- 1 simas simas 1554999 Dec 12 03:29 working.out

Note the size of working.out and not_working.out.

I was able to get this script to work. The thing is when you pass a path ~/path/to/somewhere/* shell for some reason can't find it, however if you pass path like this /home/name/path/to/somewhere/* it works like charm.

3
  • 1
    Can you show an example of what's wrong?
    – slm
    Dec 12, 2013 at 0:04
  • I've updated my question, so you should see now what is the problem now.
    – Simas.B
    Dec 12, 2013 at 0:32
  • 1
    'grep regex files.... 2>&1 > outfile' this will store both stdout and stderr. might help. I am wondering if that triple dot is a typo? or maybe the '..' is not correct from PWD.
    – Sean Perry
    Dec 12, 2013 at 7:29

3 Answers 3

2

Try using eval in your statement. Eval will evaluate the value of $INP_FILE before cat'ting them.

from

${PROGRAM} ${REGEXP} '$(cat ${INP_FILE})' > ${OUT_FILE} 2>&1

to

${PROGRAM} ${REGEXP} $(eval cat ${INP_FILE}) > ${OUT_FILE} 2>&1
3
  • That didn't help, still getting empty output.
    – Simas.B
    Dec 12, 2013 at 0:44
  • 1
    Is $3 a file like file.inp whose content is a list of files like ~/../multiple_dir/*? When cat didn't work I assume this is not your case. If $3 is a variable containing a list of file then there is no need to cat it. You will just need to ls it Try: ${PROGRAM} ${REGEXP} $(eval ls ${INP_FILE}) > ${OUT_FILE} 2>&1
    – alvits
    Dec 12, 2013 at 0:53
  • $3 contains full path to directory(~/../multiple_dir/*), eval didn't help.
    – Simas.B
    Dec 12, 2013 at 1:12
1

This is the error you get if you try to search in an empty directory:

$ tree
.
├── dir
│   └── file
└── empty_dir

2 directories, 1 file
$ grep foo dir/*  ## works, no error
$ grep foo empty_dir/*
grep: empty_dir/*: No such file or directory

Basically, when you run something like grep ./*, the glob (*) is interpreted by the shell which will expand it to the contents of the directory you gave. If the directory is empty, that expands to nothing and the shell returns an error. You will get the same error irrespective of which program you use:

$ ls empty_dir/*
ls: cannot access empty_dir/*: No such file or directory

So, I'm guessing that ~/5as-darbas/inputs/multiple_dir/ is empty. This is not a big deal and you can just ignore the error. If you want to deal with it more gracefully, you could give the directory name (no glob) and run a recursive grep:

$ grep -R foo empty_dir/

To do this with your current setup, change `~/5as-darbas/inputs/multiple_dir/* to ~/5as-darbas/inputs/multiple_dir/ and give grep -R as the first argument to your script:

$ myscript.sh 'grep -R' regExp/regExp tests/MULTIPLE.inp outputs/MULTIPLE.out
1

Ignore all of this, I was thinking the REGEXP was incorrectly safe guarded.

#!/bin/sh grep "$(cat $1)" $2

This simple case works with input files

tmp*

and

tmpfile

Called as

greptest re_file file_to_be_inspected
9
  • The thing is my script works when i pass a link to a file, but not when i pass $(cat ${INP_FILE}) which is ~/.../inputs/multiple_dir/*. Single quotes didn't help.
    – Simas.B
    Dec 12, 2013 at 0:18
  • 1
    The value of $(cat ${INP_FILE}) is ~/../inputs/multiple_dir/*? Is this done on the line where currently REGEXP=$(cat $2)?
    – Sean Perry
    Dec 12, 2013 at 0:34
  • 1
    I misread your code. I was looking at the first call to cat at REGEXP and not the later one. The shell is still running your input as a glob within the $(). Put just echo $(cat $1) in a script and run it with an input file of ~/../inputs/multiple_dir/* and you can see this directly. If you put double quotes (not single as I suggested earlier) around the $(cat ...) invokation you should get the correct result. I suggested the single quotes when I had missed where you were invoking cat.
    – Sean Perry
    Dec 12, 2013 at 0:53
  • 1
    Sorry for the confusion, I updated my answer to properly reflect the problem as I understand it now.
    – Sean Perry
    Dec 12, 2013 at 0:57
  • 1
    OK, clearly I lost the thread here (multi-tasking FAIL!). My answer was based on the REGEXP being busted and thus totally off base. The regexp is coming through just fine but the grep is not finding anything in any of the files specified by INP_FILE? So is there a match in any of the files in multiple_dir/? Which is what you are asking it to grep. If you are getting no output then the files do not contain ATOM.
    – Sean Perry
    Dec 12, 2013 at 1:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .