5

The command

$ find ~/foo/ -type f -iname "*.txt" -print0 | parallel -0 cat

uses GNU Parallel to print all the .txt files under ~/foo/.

I have a python script where I want to call this bash command:

import subprocess, sys

def runBashCommand(my_command):
    process = subprocess.Popen(my_command.split(), stdout=subprocess.PIPE)
    output  = process.communicate()[0]
    return None

def makeCommand(my_path):
    return "find {} -type f -iname \"*.txt\" -print0 | parallel -0 cat".format(my_path)

Issuing

>>> makeCommand('~/foo/')

returns

'find ~/foo/ -type f -iname "*.txt" -print0 | parallel -0 cat'

but issuing

>>> runBashCommand(makeCommand('~/foo/'))

yields the error

find: paths must precede expression: |
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]

what's the problem with my script?

6

You're not actually running a bash command. What you're doing is running an executable directly and passing it arguments.

Try the following script to see what is happening:

import subprocess
p = subprocess.Popen(["echo", "a", "b", "|", "rev"], stdout=subprocess.PIPE)
print p.communicate()

The output will be:

('a b | rev\n', None)

There's no redirection happening, the "|" is being passed in literally. I.e., it is as if you had typed find ... \| parallel .... Thus the error.

There's two ways to fix it.

  • The easy way: pass shell=True to subprocess.Popen. That will run it through the shell, with all that entails. If you do that, you also need to pass in a string instead of an array:

    import subprocess
    p = subprocess.Popen("echo a b | rev", stdout=subprocess.PIPE, shell=True)
    print p.communicate()
    
    # Result: ('b a\n', None)
    

    If you do this, be very careful about argument substitution into your string.

  • The robust way: open two processes using Python and pipe them together.

    import subprocess
    # First command
    p1 = subprocess.Popen(["echo", "a", "b"], stdout=subprocess.PIPE)
    # Second command's input linked to the first one's output
    p2 = subprocess.Popen(["rev"], stdin=p1.stdout, stdout=subprocess.PIPE)
    # Read from p2 to get the output
    print p2.communicate()
    
    # Result: ('b a\n', None)
    

    This is more robust and doesn't spawn an extra shell, but on the other hand it is more typing. If you do this, note that no shell substitution happens. In your case it doesn't look like you need it, but if you'd want to use, say, ~, you'd have to get it through Python (e.g. os.getenv("HOME")).

  • Can you expand on "be very careful about argument substitution into your string"? It seems that this applies to your second method but no example is given as for how it applies to the first. – Wildcard May 12 '16 at 22:36
  • 3
    @Wildcard: with shell=True, it passes the whole string into the shell. Let's say you wanted to get the output of ls for a given directory. subprocess.Popen(["ls", dir], ...) would always work. However, subprocess.Popen("ls " + dir, ..., shell=True) would fail if dir had spaces in it... let alone what would happen if dir got set to something like ";rm -rf /*". So that would be something to really pay attention to, especially if dir comes from user input. But if you don't use shell=True, this kind of thing can't happen. – marinus May 12 '16 at 22:49
0

You cannot split() your command string as it contains characters that need to be processed by the shell, eg ~ and |. Use the version:

process = subprocess.Popen(my_command, stdout=subprocess.PIPE, shell=True)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.