2

Trying to parallelize a sed operation however, while the parallel version works, it gives back a wrong output.

The sed operation that I would like to parallelize (works)

sed 's/\s.*$// ; s/\(.*\)/\L\1/' < oldfile.txt > newfile.txt

My Parallel version of the above sed operation (Does not work correctly for some reason):

parallel -a oldfile.txt -k --block $BYTES --pipe-part "sed 's/\s.*$// ; s/\(.*\)/\L\1/'" > newfile.txt
  • 3
    Looks like you're trying to extract the fist column/field and lowercase it; keep in mind regex is expensive (read slow) so s/\s.*$// will be much slower than e.g. cut -f1 and s/\(.*\)/\L\1/' will be much slower than e.g. tr '[:upper:]' '[:lower:]'... Optimization more often than not means using the right tools for the job. – don_crissti May 14 '17 at 15:07
  • @don_crissti how is tr not using regex? isn't [:upper:] a regex class? – HashWizard May 14 '17 at 15:34
  • 2
    It' a character class not a regex. – don_crissti May 14 '17 at 15:35
  • 1
    Come to think... to do it in a single pass you'd use awk '{print tolower($1)}' infile – don_crissti May 14 '17 at 17:56
  • 1
    It should be faster than sed. I don't know if it can beat the cut | tr combination (I kinda doubt that...) – don_crissti May 14 '17 at 22:23
4

This is usually due to double quoting.

Quoting is annoying, so try instead using a shell function:

mysed() {
    sed 's/\s.*$// ; s/\(.*\)/\L\1/'
}
export -f mysed

parallel -a oldfile.txt -k --block -1 --pipe-part mysed > newfile.txt
2

GNU parallel invokes the command that you specify through a shell. This is occasionally convenient, but often a pain, as you've discovered, because of the quoting issues and the dependency on the SHELL environment variable. (Ole and I have already had words about that.)

You have to tell parallel that what you're passing is an executable with arguments, rather than the default behavior which assumes that the arguments are a command to be executed by the program named by the SHELL environment variable, arbitrarily split into pieces that are joined together with a space in between. There's no such option; the closest thing is to tell parallel to quote the command so that the shell will end up running the correct command.

parallel -a oldfile.txt -k --block $BYTES --pipe-part -q sed 's/\s.*$// ; s/\(.*\)/\L\1/' > newfile.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.