2

I often find that I've concocted some incantation which I'd like to reuse, but it's far too susceptible to ad-hoc tweaking to store as a function. These commands often have some critical variable component, such as a filename, buried deep within. I'd like to move that component to the end of the command.

For example, my latest witchcraft prints the table definitions of all new INSERT statements in a diff file:

for T in $(grep -oP "(?<=^\+INSERT INTO \`)[\w-]+(?=\`)" foo.diff | sort -u) ; do docker-compose exec mysql mysqldump --no-data some_database $T | grep -P "^\s*\`|^CREATE|^\) |^\s*CONSTRAINT" | sed "s/ COMMENT .*,$/,/" | sed "s/CONSTRAINT \`\w*\` //" | sed "s/^) ENGINE.*;/);/" ; done

Formatted for legibility:

for T in $(grep -oP "(?<=^\+INSERT INTO \`)[\w-]+(?=\`)" foo.diff | sort -u)
  do
  docker-compose exec mysql mysqldump --no-data some_database $T
    | grep -P "^\s*\`|^CREATE|^\) |^\s*CONSTRAINT"
    | sed "s/ COMMENT .*,$/,/"
    | sed "s/CONSTRAINT \`\w*\` //"
    | sed "s/^) ENGINE.*;/);/"
done

I'm very likely going to want to use this again in the future on a file that is not called foo.diff, so I'd like the command to end with the file name. My first inclination would be to just wrap the whole thing in a function, and have the filename be the $1 parameter.

However, it is very likely that I'm going to tweak it further as I run it. Add a bit of sed here, maybe a touch of awk there. So I'd still like the whole command to be on the CLI when I Ctrl-P to run it again. Therefore I decided to use a variable:

$ FILENAME=diff-03-04.sql

That's appropriate for this specific example as I'll be using the same file for each incantation today, but there exist other examples where I'd really like to be able to just tweak the end of the command. So this:

$ for Q in $(foo Torvalds bar) ; do baz $Q ; done

Would become something akin to (but not necessarily exactly):

$ for Q in $(foo SOME_MAGIC_HERE bar) ; do baz $Q ; done < Torvalds
11
  • That's why you'd use a script or function to encapsulate the code. Use type f for function f to see its definition. Copy and paste to suit Apr 3, 2022 at 9:28
  • 1
    It's unclear why using a variable would be inappropriate in your last example. Personally, I would put these things into scripts. I would also strongly suggest using cmd | while read loops rather than for in $(cmd) loops.
    – Kusalananda
    Apr 3, 2022 at 10:03
  • "and have the filename be the $1 parameter" – sh -c '…' sh foo.diff. This however introduces an additional level of quoting. Apr 3, 2022 at 10:18
  • @roaima Thank you. The reason that I would prefer to not wrap in a function is because there will be much tweaking from invocation to invocation, especially as I hone in on exactly the code and results I want. I want to run, and have accessible to alter, the entire command each time.
    – dotancohen
    Apr 3, 2022 at 12:40
  • 1
    For the record: (re)defining a function and calling it in the same line is possible: myfunc() { echo "$1"; }; myfunc foo.diff Apr 3, 2022 at 13:58

2 Answers 2

3

It seems to me that you have misunderstood the power of a script or function. Personally I prefer scripts because they are self contained and available regardless of the interactive shell I'm using, but a function can do exactly the same thing. For me, systemwide scripts go into /usr/local/bin and personal (junk) scripts go into ~/bin. Both are in my $PATH.

Create the directory ~/bin. Add it to your $PATH (export PATH="$PATH:$HOME/bin"). Now any executable script placed in that directory will be available to you as a new command.

Let's take your example as a case in point. You want the file name to be a use-time selection, so let's make it the first argument of a script.

Create the file ~/bin/tables:

#!/bin/bash
file=$1
for T in $(grep -oP "(?<=^\+INSERT INTO \`)[\w-]+(?=\`)" "$file" | sort -u)
  do
  docker-compose exec mysql mysqldump --no-data some_database "$T"
    | grep -P "^\s*\`|^CREATE|^\) |^\s*CONSTRAINT"
    | sed "s/ COMMENT .*,$/,/"
    | sed "s/CONSTRAINT \`\w*\` //"
    | sed "s/^) ENGINE.*;/);/"
done

Make the script executable with chmod a+x ~/bin/tables and now you have a new command.

tables foo.diff
tables another.diff
9
  • 2
    I think the whole point of the question is that it's not just the filename that will change, but also some tweaks inside the function itself depending on which filename will be processed, if I understood OP correctly. This is why every invocation will be a little different, potentially, and why OP prefers to get the whole line invoked with readline facilities.
    – ychaouche
    Apr 3, 2022 at 13:02
  • Thank you roaima. But as ychaouche notes, every invocation will be slightly different, and editing the script file each time is adding friction, not reducing it. Theoretically, I could just keep VIM open and run the file (with :!./% foo.diff or :!% foo.diff if it's in my PATH) but then doing anything else (e.g. ls) becomes a hassle.
    – dotancohen
    Apr 3, 2022 at 13:17
  • 1
    @ychaouche indeed. I thought that that too, but then the OP seemed to suggest in later comments that it was mostly about changing the filename Apr 3, 2022 at 13:18
  • @dotancohen sorry, you've lost me: where does vi/vim fit into the process? Apr 3, 2022 at 13:19
  • @roaima Changing the filename, or other arbitrary string, does happen, that is the gist of this question. "How do I take this buried string and make it specifically easier to modify, while I can still modify the whole thing" might be a verbose way of phrasing it.
    – dotancohen
    Apr 3, 2022 at 13:20
0

Everybody here has given excellent advice, in the other answer and in the comments. The best method that I've found is to wrap the command in a function and run that function in a single command:

$ _() { echo $1; } && _ beer
beer
$

Additionally, I've replaced the for T in $() syntax with cmd | while read as suggested by Kusalananda. Thus the final syntax is as follows:

_() { grep -oP "(?<=^\+INSERT INTO \`)[\w-]+(?=\`)" $1 | sort -u | while read -r T ; do docker-compose exec mysql mysqldump --no-data some_database $T | grep -P "^\s*\`|^CREATE|^\) |^\s*CONSTRAINT" | sed "s/ COMMENT .*,$/,/" | sed "s/CONSTRAINT \`\w*\` //" | sed "s/^) ENGINE.*;/);/" ; done } && _ foo.diff

Formatted for readability:

_() {
  grep -oP "(?<=^\+INSERT INTO \`)[\w-]+(?=\`)" $1
    | sort -u
    | while read -r T ;
      do
      docker-compose exec mysql mysqldump --no-data some_database $T
        | grep -P "^\s*\`|^CREATE|^\) |^\s*CONSTRAINT"
        | sed "s/ COMMENT .*,$/,/"
        | sed "s/CONSTRAINT \`\w*\` //"
        | sed "s/^) ENGINE.*;/);/"
    done
}
&& _ foo.diff

This method preserves the ability to tweak anywhere in the command that I need, yet easily change the argument (in this case, the filename), without needing to escape quotes in the commands themselves.

5
  • 2
    It's far more robust to write while IFS= read -t T and double-quote "$T" each time you use it Apr 3, 2022 at 14:55
  • 1
    The echo $1 construct can be better replaced with printf -- "%s\n" "$1" Apr 4, 2022 at 7:21
  • 1
    @roaima Thank you! I understand why the while construct is better (after googling it) but what is the advantage with printf over echo? I very much appreciate your ambition to share your knowledge, I am just as ambitions to learn!
    – dotancohen
    Apr 4, 2022 at 10:26
  • 2
    Considering the general case. Some implementations of echo action a first argument that begins with a dash. For example echo -n hello might suppress the newline after hello. Also, with an unquoted $1 you replace intentional whitepace (such as two or more spaces in a row, or leading/trailing whitespace) with a single space. The printf approach guarantees that these unexpected behaviours will not happen. Apr 4, 2022 at 11:29
  • @roaima I see, thank you.
    – dotancohen
    Apr 4, 2022 at 14:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .