2

Trying to parallelize a sed operation however, while the parallel version works, it gives back a wrong output.

The sed operation that I would like to parallelize (works)

sed 's/\s.*$// ; s/\(.*\)/\L\1/' < oldfile.txt > newfile.txt

My Parallel version of the above sed operation (Does not work correctly for some reason):

parallel -a oldfile.txt -k --block $BYTES --pipe-part "sed 's/\s.*$// ; s/\(.*\)/\L\1/'" > newfile.txt
10
  • 3
    Looks like you're trying to extract the fist column/field and lowercase it; keep in mind regex is expensive (read slow) so s/\s.*$// will be much slower than e.g. cut -f1 and s/\(.*\)/\L\1/' will be much slower than e.g. tr '[:upper:]' '[:lower:]'... Optimization more often than not means using the right tools for the job. Commented May 14, 2017 at 15:07
  • @don_crissti how is tr not using regex? isn't [:upper:] a regex class?
    – HashWizard
    Commented May 14, 2017 at 15:34
  • 2
    It' a character class not a regex. Commented May 14, 2017 at 15:35
  • 1
    Come to think... to do it in a single pass you'd use awk '{print tolower($1)}' infile Commented May 14, 2017 at 17:56
  • 1
    It should be faster than sed. I don't know if it can beat the cut | tr combination (I kinda doubt that...) Commented May 14, 2017 at 22:23

2 Answers 2

5

This is usually due to double quoting.

Quoting is annoying, so try instead using a shell function:

mysed() {
    sed 's/\s.*$// ; s/\(.*\)/\L\1/'
}
export -f mysed

parallel -a oldfile.txt -k --block -1 --pipe-part mysed > newfile.txt
2

GNU parallel invokes the command that you specify through a shell. This is occasionally convenient, but often a pain, as you've discovered, because of the quoting issues and the dependency on the SHELL environment variable. (Ole and I have already had words about that.)

You have to tell parallel that what you're passing is an executable with arguments, rather than the default behavior which assumes that the arguments are a command to be executed by the program named by the SHELL environment variable, arbitrarily split into pieces that are joined together with a space in between. There's no such option; the closest thing is to tell parallel to quote the command so that the shell will end up running the correct command.

parallel -a oldfile.txt -k --block $BYTES --pipe-part -q sed 's/\s.*$// ; s/\(.*\)/\L\1/' > newfile.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .